nhance/app/Views/docs/background-jobs.php

<?php
/**
 * Background Jobs - content only
 * app/Views/docs/background-jobs.php
 *
 * This page documents the current queue implementation built around:
 * - app/Controllers/Jobs.php
 * - app/Controllers/JobWorker.php
 * - app/Controllers/Jobs/*.php sample handlers
 * - app/Models/JobModel.php
 */
?>

<p>
  Nhance uses a database-backed job queue for long-running or deferred work. A
  producer adds a row into the <code>jobs</code> table through
  <code>Jobs::addJob()</code>, and the CLI worker in
  <code>JobWorker</code> picks up queued rows, executes the mapped handler, and
  writes the final status and response back to the same record.
</p>

<div class="callout info">
  <span>i</span>
  <div>
    <strong>Current implementation</strong>
    This page describes the queue exactly as it exists today, including the
    handler registry in <code>JobWorker::$event_class_mapping</code> and the
    helper methods already used by controllers like <code>EmployeeController</code>
    and <code>LeadsController</code>.
  </div>
</div>

<h2 id="overview">Overview</h2>

<p>
  The queue flow is simple: enqueue, store, process, update. It is used for
  tasks such as file validation, employee processing, claim import work, API
  sync jobs, bulk mail, and batch e-card generation.
</p>

<div class="mermaid-wrapper">
  <div class="mermaid">
flowchart TD
    A[Controller or service] --> B[Jobs::addJob]
    B --> C[(jobs table)]
    C --> D[php spark cli/processjobs]
    D --> E[JobWorker::processJobs]
    E --> F[JobWorker::processJob]
    F --> G[event_class_mapping lookup]
    G --> H[Handler method or function]
    H --> I[Update status, run_time, response]
  </div>
</div>

<h2 id="queueing-jobs">Queueing jobs</h2>

<p>
  New jobs are inserted through <code>Jobs::addJob(array $payload)</code>. The
  method validates the input, generates a UUID, JSON-encodes the inner payload,
  and stores the row with status <code>queued</code> unless a custom status is
  provided.
</p>

<table>
  <thead>
    <tr><th>Key</th><th>Required</th><th>Description</th></tr>
  </thead>
  <tbody>
    <tr>
      <td><span class="param-name">job_name</span></td>
      <td><span class="badge req">required</span></td>
      <td>Name used to look up the handler in <code>JobWorker::$event_class_mapping</code>.</td>
    </tr>
    <tr>
      <td><span class="param-name">payload</span></td>
      <td><span class="badge req">required</span></td>
      <td>Array that will be JSON-encoded into the <code>jobs.payload</code> column.</td>
    </tr>
    <tr>
      <td><span class="param-name">status</span></td>
      <td><span class="badge opt">optional</span></td>
      <td>Defaults to <code>queued</code>.</td>
    </tr>
  </tbody>
</table>

<pre><code class="language-php">$job = Jobs::addJob([
    'job_name' => 'memberDataListExcelFileFormatValidation',
    'payload'  => [
        'lead_id' => $lead_id,
        'age_validation' => true,
    ],
]);
</code></pre>

<p>
  The method returns an array with <code>id</code>, <code>uuid</code>, and
  <code>job_name</code>. Real code paths already use this pattern, for example
  after lead placement data is saved and then queued for validation.
</p>

<h2 id="worker-lifecycle">Worker lifecycle</h2>

<p>
  <code>JobWorker</code> is the execution engine. It defines four statuses:
  <code>queued</code>, <code>running</code>, <code>done</code>, and
  <code>failed</code>.
</p>

<ol class="steps">
  <li>
    <strong><code>processJobs()</code> fetches all queued rows</strong>
    <p>Jobs are selected from <code>jobs</code> ordered by <code>created_dt ASC</code>.</p>
  </li>
  <li>
    <strong>Each queued row is forwarded to <code>processJob()</code></strong>
    <p>The worker can process the next queued row, or a specific <code>id</code> plus <code>uuid</code> pair.</p>
  </li>
  <li>
    <strong>Status changes to <code>running</code></strong>
    <p>Once picked, the worker updates the row before invoking the handler.</p>
  </li>
  <li>
    <strong>The handler receives the decoded payload</strong>
    <p>The worker resolves the handler from the event map and passes the job payload into it.</p>
  </li>
  <li>
    <strong>The row is finalized</strong>
    <p>After execution, the worker stores the final status, runtime, and response JSON back into the same job row.</p>
  </li>
</ol>

<div class="callout warning">
  <span>!</span>
  <div>
    <strong>Selection and locking</strong>
    The worker query uses <code>LIMIT 1 FOR UPDATE</code> when fetching a single
    job. In practice, treat the queue as database-backed and process it from CLI
    workers, not from normal web requests.
  </div>
</div>

<h2 id="handler-registry">Handler registry</h2>

<p>
  Every executable job must be registered in
  <code>JobWorker::$event_class_mapping</code>. Each entry defines a handler
  category and a target class or function.
</p>

<table>
  <thead>
    <tr><th>Type</th><th>Meaning</th><th>Example from code</th></tr>
  </thead>
  <tbody>
    <tr>
      <td><code>CC</code></td>
      <td>Controller class handler</td>
      <td><code>App\Controllers\Jobs\SubJob</code>, <code>EmployeeServiceController</code></td>
    </tr>
    <tr>
      <td><code>HC</code></td>
      <td>Helper-style class handler</td>
      <td><code>App\Helpers\HttpRequestHelper</code>, <code>App\Helpers\MailHelper</code></td>
    </tr>
    <tr>
      <td><code>HF</code></td>
      <td>Standalone function handler</td>
      <td><code>fancy_date_time_format</code></td>
    </tr>
  </tbody>
</table>

<p>
  Resolution order inside the worker is:
</p>

<ol class="steps">
  <li>
    <strong>Look up the job name in the mapping</strong>
    <p>If the name is missing, the worker throws an exception and marks the job failed.</p>
  </li>
  <li>
    <strong>Instantiate the mapped class for <code>CC</code> or <code>HC</code></strong>
    <p>The mapped class must exist.</p>
  </li>
  <li>
    <strong>Choose the callable</strong>
    <p>If a method matching the job name exists, it is used first; otherwise the worker falls back to <code>handle()</code>.</p>
  </li>
  <li>
    <strong>For <code>HF</code>, call the mapped function directly</strong>
    <p>The handler value itself is treated as the callable.</p>
  </li>
</ol>

<h2 id="sample-handlers">Sample handlers</h2>

<p>
  The <code>app/Controllers/Jobs/</code> directory currently contains two simple
  examples that show the expected pattern for small job classes:
</p>

<table>
  <thead>
    <tr><th>File</th><th>Method</th><th>Behavior</th></tr>
  </thead>
  <tbody>
    <tr>
      <td><code>app/Controllers/Jobs/AddJob.php</code></td>
      <td><code>handle($payload)</code></td>
      <td>Returns <code>$payload['a'] + $payload['b']</code>.</td>
    </tr>
    <tr>
      <td><code>app/Controllers/Jobs/SubJob.php</code></td>
      <td><code>handle($payload)</code></td>
      <td>Logs through <code>mylogger</code> and returns <code>$payload['a'] - $payload['b']</code>.</td>
    </tr>
  </tbody>
</table>

<pre><code class="language-php">namespace App\Controllers\Jobs;

use App\Controllers\PublicController;

class ExampleJob extends PublicController
{
    public function handle($payload)
    {
        return [
            'ok' => true,
            'received' => $payload,
        ];
    }
}
</code></pre>

<h2 id="status-lifecycle">Status lifecycle</h2>

<p>
  The queue storage model is <code>app/Models/JobModel.php</code>, which maps to
  the <code>jobs</code> table and allows these main fields:
</p>

<table>
  <thead>
    <tr><th>Column</th><th>Purpose</th></tr>
  </thead>
  <tbody>
    <tr><td><code>id</code></td><td>Primary key returned after enqueue.</td></tr>
    <tr><td><code>name</code></td><td>Logical job name used in the worker registry.</td></tr>
    <tr><td><code>payload</code></td><td>JSON-encoded input payload.</td></tr>
    <tr><td><code>response</code></td><td>JSON-encoded handler output or failure details.</td></tr>
    <tr><td><code>status</code></td><td><code>queued</code>, <code>running</code>, <code>done</code>, or <code>failed</code>.</td></tr>
    <tr><td><code>run_time</code></td><td>Measured execution time for the job.</td></tr>
    <tr><td><code>uuid</code></td><td>Generated at enqueue time and used when fetching a specific row.</td></tr>
  </tbody>
</table>

<div class="mermaid-wrapper">
  <div class="mermaid">
stateDiagram-v2
    [*] --> queued
    queued --> running
    running --> done
    running --> failed
  </div>
</div>

<h2 id="failure-behavior">Failure behavior</h2>

<p>
  Both worker-level failures and handler-level failures are caught and written
  back into <code>jobs.response</code> as structured JSON. The stored data
  includes the error message, file, line, trace, and whether the failure
  happened in the task or the worker wrapper.
</p>

<div class="callout danger">
  <span>x</span>
  <div>
    <strong>Special file failure branch</strong>
    When a failed job payload contains only a numeric <code>file_id</code>, the
    worker also updates the related <code>files</code> row to
    <code>status = failed</code> and writes a generic system error reason. This
    is important for file-processing jobs that surface state back to the UI.
  </div>
</div>

<h2 id="running-via-cli">Running via CLI</h2>

<p>
  The worker is exposed through CLI routes in <code>app/Config/Routes.php</code>.
</p>

<table>
  <thead>
    <tr><th>Route</th><th>Target</th><th>Purpose</th></tr>
  </thead>
  <tbody>
    <tr>
      <td><code>cli/processjob</code></td>
      <td><code>JobWorker::processJob</code></td>
      <td>Process one queued job.</td>
    </tr>
    <tr>
      <td><code>cli/processjobs</code></td>
      <td><code>JobWorker::processJobs</code></td>
      <td>Loop through all queued jobs in created order.</td>
    </tr>
  </tbody>
</table>

<pre><code class="language-bash">php spark cli/processjob
php spark cli/processjobs</code></pre>

<p>
  There is also a web route named <code>processjob</code>, but operationally this
  queue should be treated as a CLI worker flow.
</p>

<h2 id="live-runner-script">Live runner script</h2>

<p>
  For live environments where the worker should keep polling continuously, a
  simple shell loop can call the queue worker across all required applications.
  Use the root-level <code>phpqueue.sh</code> script and replace the placeholder
  app paths with your deployment-specific values.
</p>

<pre><code class="language-bash">#!/bin/bash

PHP_BIN="${PHP_BIN:-/usr/bin/php}"

# Dummy example paths for documentation.
# Replace each one with the real public/index.php path in that environment.
APP_INDEXES=(
    "/var/www/example-suite/zenith-app/public/index.php"
    "/var/www/example-suite/enrolment-app/public/index.php"
    "/var/www/example-suite/partner-api/public/index.php"
)

while true; do
    echo "Running job at $(date)"

    for index_file in "${APP_INDEXES[@]}"; do
        "${PHP_BIN}" "${index_file}" cli/processjob
    done

    sleep 1
done</code></pre>

<table>
  <thead>
    <tr><th>Dummy path</th><th>Description</th></tr>
  </thead>
  <tbody>
    <tr>
      <td><code>/var/www/example-suite/zenith-app/public/index.php</code></td>
      <td>Example public entry file for the first application queue target.</td>
    </tr>
    <tr>
      <td><code>/var/www/example-suite/enrolment-app/public/index.php</code></td>
      <td>Example public entry file for the second application queue target.</td>
    </tr>
    <tr>
      <td><code>/var/www/example-suite/partner-api/public/index.php</code></td>
      <td>Example public entry file for the third application queue target.</td>
    </tr>
  </tbody>
</table>

<div class="callout warning">
  <span>!</span>
  <div>
    <strong>Production note</strong>
    This script is an infinite loop, so it should be started under a process
    manager such as <code>systemd</code>, <code>supervisord</code>, or another
    service wrapper rather than being launched manually in a shell session.
  </div>
</div>

<h2 id="systemd-service">Systemd service</h2>

<p>
  For Ubuntu-style live deployments, keep a systemd unit file such as
  <code>nhance_php_queue_server.service</code>. The repository now includes a
  sanitized template with placeholder values.
</p>

<pre><code class="language-ini">[Unit]
Description=Nhance PHP Queue Server
After=network.target

[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/var/www/example-suite/nhance-app
ExecStart=/bin/bash /var/www/example-suite/nhance-app/phpqueue.sh
Restart=always
RestartSec=2

[Install]
WantedBy=multi-user.target</code></pre>

<table>
  <thead>
    <tr><th>Dummy value</th><th>Description</th></tr>
  </thead>
  <tbody>
    <tr>
      <td><code>/var/www/example-suite/nhance-app</code></td>
      <td>Example project root where <code>phpqueue.sh</code> is kept.</td>
    </tr>
    <tr>
      <td><code>www-data</code></td>
      <td>Example service account; replace it with the real Linux user and group used by PHP on that server.</td>
    </tr>
  </tbody>
</table>

<p>
  After replacing the placeholders, deploy and enable it with standard systemd
  commands:
</p>

<pre><code class="language-bash">sudo mv nhance_php_queue_server.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable nhance_php_queue_server.service
sudo systemctl start nhance_php_queue_server.service
sudo systemctl restart nhance_php_queue_server.service</code></pre>

<h2 id="permissions-setup">Permissions setup</h2>

<p>
  The queue runner script should live in the project root as
  <code>phpqueue.sh</code>, and its ownership plus execute permission should be
  set for the service user and group used by PHP in that environment.
</p>

<pre><code class="language-bash">sudo chown &lt;service-user&gt;:&lt;service-group&gt; phpqueue.sh
sudo chmod 750 phpqueue.sh</code></pre>

<div class="callout info">
  <span>i</span>
  <div>
    <strong>Why this matters</strong>
    In live environments, the two crucial steps are:
    using a managed system service for the queue loop, and ensuring the
    root-level <code>phpqueue.sh</code> file has the correct owner, group, and
    execute permission for that service account.
  </div>
</div>

<h2 id="checking-status">Checking status</h2>

<p>
  <code>app/Libraries/JobStatusService.php</code> now supports lookup by job
  name, by job <code>id</code>, and by job <code>uuid</code>. All methods normalize the
  decoded response and runtime before returning them.
</p>

<table>
  <thead>
    <tr><th>Method</th><th>Use when</th></tr>
  </thead>
  <tbody>
    <tr>
      <td><code>getJobStatusByName(string $jobName)</code></td>
      <td>You want the latest job row for a logical job name.</td>
    </tr>
    <tr>
      <td><code>getJobStatusById(int $jobId)</code></td>
      <td>You know the numeric queue row id and want that exact job record.</td>
    </tr>
    <tr>
      <td><code>getJobStatusByUuid(string $uuid)</code></td>
      <td>You want to track one exact job instance using the UUID returned at enqueue time.</td>
    </tr>
  </tbody>
</table>

<pre><code class="language-php">$service = new \App\Libraries\JobStatusService();
$status = $service->getJobStatusByName('bulkGenerateEcardAndStoreinS3');
</code></pre>

<pre><code class="language-php">$job = Jobs::addJob([
    'job_name' => 'exampleJob',
    'payload' => [
        'file_id' => 123,
    ],
]);

$service = new \App\Libraries\JobStatusService();
$statusById = $service->getJobStatusById((int) $job['id']);
$statusByUuid = $service->getJobStatusByUuid($job['uuid']);
</code></pre>

<p>
  The service returns <code>success</code>, <code>job_name</code>,
  <code>status</code>, <code>response</code>, <code>job_id</code>,
  <code>uuid</code>, <code>run_time</code>, and a message.
</p>

<div class="callout info">
  <span>i</span>
  <div>
    <strong>Which lookup should you use?</strong>
    Use <code>job name</code> when you only care about the latest run for a given
    handler. Use <code>job id</code> or <code>uuid</code> when the enqueue response
    is available and you need to track one specific job instance across retries
    or parallel runs.
  </div>
</div>

<h2 id="adding-a-new-handler">Adding a new handler</h2>

<ol class="steps">
  <li>
    <strong>Create the handler class or function</strong>
    <p>For small custom jobs, <code>app/Controllers/Jobs/</code> is already used as a simple home for dedicated job handlers.</p>
  </li>
  <li>
    <strong>Register the job in <code>JobWorker::$event_class_mapping</code></strong>
    <p>Pick the correct type: <code>CC</code>, <code>HC</code>, or <code>HF</code>.</p>
  </li>
  <li>
    <strong>Expose a callable method</strong>
    <p>The worker first looks for a method matching the job name, then falls back to <code>handle()</code>.</p>
  </li>
  <li>
    <strong>Queue the job through <code>Jobs::addJob()</code></strong>
    <p>Pass a stable job name and only the payload fields the handler actually needs.</p>
  </li>
  <li>
    <strong>Run the worker and inspect the job row</strong>
    <p>Validate the final status, runtime, and response before integrating the job into larger workflows.</p>
  </li>
</ol>

<pre><code class="language-php">// 1. Register in JobWorker::$event_class_mapping
'exampleJob' => [
    'type' => 'CC',
    'handler' => 'App\Controllers\Jobs\ExampleJob',
],

// 2. Queue it
Jobs::addJob([
    'job_name' => 'exampleJob',
    'payload' => [
        'file_id' => 123,
        'source' => 'manual-test',
    ],
]);
</code></pre>

<div class="callout success">
  <span>+</span>
  <div>
    <strong>Practical rule</strong>
    Keep payloads explicit and handler names stable. The job name is the contract
    between producers and the worker registry, so renaming it has queue-wide
    impact.
  </div>
</div>