Workers And Queues

The backend uses a PostgreSQL table-backed queue. Jobs are claimed with locking semantics and dead-lettered after retry exhaustion. The operator source of truth is the database plus structured worker logs and OpenTelemetry metrics.

Queues

QueueProducerConsumerPurpose
proofs.validatedProof submit routesworker-proofValidate proof, decrypt payload, create certificate, consume license, sign PDF/receipt.
certificates.to_anchorCertificate serviceworker anchorSubmit or update blockchain anchor metadata and regenerate certificate PDF.
notifications.to_sendUser, billing, webhook flowsworker notificationsSend SMTP/webhook notifications, retry failures, write delivery metadata.
billing.exportsBilling export routesworker reportsGenerate requested export artifacts.
reports.monthlyReport schedulerworker reportsGenerate monthly billing report artifacts.
proofs.retryRetry routes/schedulerworker retryRe-enqueue due failed or awaiting-license proofs.

The local PostgreSQL/PGMQ setup also seeds underscore queue aliases that resolve to the same jobs table: proofs_validated, certificates_to_anchor, notifications_to_send, billing_exports, reports_monthly, and proofs_retry. The dot-notation form above is canonical; treat the underscore form as a Compose/PGMQ compatibility alias only.

Worker Commands

WorkerCommandUseful variables
Proof/app/worker-proofWORKER_ID, WORKER_POLL_INTERVAL, WORKER_CLAIM_LIMIT, WORKER_ONCE.
Anchor/app/worker anchorSame worker variables plus blockchain/signer/storage settings.
Notifications/app/worker notificationsSame worker variables plus notification/URL policy settings.
Reports/app/worker reportsSame worker variables plus storage, billing, PAdES settings.
Retry/app/worker retrySame worker variables; enqueues due proof retries before claims.
Maintenance/app/worker maintenanceUses retention interval when WORKER_POLL_INTERVAL is unset.

Set WORKER_ONCE=true for a controlled single pass:

  WORKER_ONCE=true WORKER_CLAIM_LIMIT=10 /app/worker retry

Queue Inspection

Use SQL for baseline triage:

  SELECT queue, status, count(*)
FROM jobs
GROUP BY queue, status
ORDER BY queue, status;

DLQ summary:

  SELECT queue, reason, count(*), max(created_at) AS last_seen
FROM jobs_dlq
GROUP BY queue, reason
ORDER BY last_seen DESC;

Newest DLQ entries:

  SELECT id, original_job_id, queue, payload, reason, created_at
FROM jobs_dlq
ORDER BY created_at DESC
LIMIT 20;

Replay Rules

  1. Confirm the downstream dependency is healthy before replaying anything.
  2. Inspect representative DLQ rows and the referenced business record.
  3. Replay by inserting a new jobs row with the same queue and payload.
  4. Keep the original DLQ row for audit.
  5. Do not update a dead job back to pending. Create a new job so the replay is visible and auditable. Mutating status in place hides the recovery from queue metrics and incident review.

If a payload is malformed or references a deleted resource, leave it in DLQ and record the incident decision in the operations log. DLQ rows are retention evidence — do not delete them unless the incident process explicitly approves archival.

Proof Retry Checks 

  SELECT id, tenant_id, organization_id, status, attempt_count, next_retry_at,
       rejection_reason, received_at
FROM proofs
WHERE status IN ('FAILED', 'AWAITING_LICENSE', 'REJECTED')
ORDER BY received_at DESC
LIMIT 50;
StatusReplay guidance
FAILEDSeparate terminal validation failures from dependency failures. Replay only after fixing signer, storage, PDF, validation config, or other dependency failures. Watch attempt_count and next_retry_at; proofs at the retry limit need manual review before another replay.
AWAITING_LICENSEImport/allocate licenses or fix quota scope first, then retry.
REJECTEDTerminal validation failure. Replay only if the validation rule or code was wrong; otherwise leave it.

Confirm completion by checking that the proof has certificate_id set and status CERTIFIED or CERTIFIED_NO_ANCHOR. To enqueue a retry manually when the API retry flow is unavailable, create a proofs.retry job with payload {"proof_id":"..."}.

Worker Scaling

Proof, notifications, reports, retry, and anchor workers are stateless and can scale horizontally where the workload allows it. Use queue depth, in-flight jobs, dependency capacity, and idempotency guarantees to decide scale. The anchor and monthly report schedulers should be scaled conservatively until production chain and scheduler ownership are finalized.