Observability And Alerts link

The current implementation emits OpenTelemetry signals to the collector. The reference stack routes metrics to Prometheus, logs to Loki, traces to Tempo, and dashboards/alerts to Grafana.

The backend does not expose a Prometheus /metrics endpoint by default. Prometheus scrapes the OpenTelemetry Collector Prometheus exporter.

Local URLs link

Signal Fields link

HTTP logs include:

Use request_id for support tickets and trace_id for cross-service triage.

Alerts link

For every alert, follow this common flow before the alert-specific steps:

1. Confirm /readyz and dependency container/pod health.
2. Open Grafana Wipe Backend Overview.
3. Filter Loki by service_name and the request_id or trace_id from the failing request.
4. Follow the trace into Tempo when present.
5. Inspect the relevant queue and DLQ rows.
6. Fix dependency or configuration before replaying jobs.

WipeAPITelemetryMissing link

1. Check Process Heartbeats for component="wipe-api".
2. Confirm the API container is running and can reach the OpenTelemetry Collector.
3. Check API startup logs for observability initialization errors and verify OBSERVABILITY_OTLP_ENDPOINT.
4. If /readyz is healthy but telemetry is absent, restart only after confirming collector health and network routing.

WipeWorkerTelemetryMissing link

1. Check Process Heartbeats for wipe-proof-worker and each wipe-worker-* component.
2. Confirm the missing worker container/pod is running and has OBSERVABILITY_ENABLED=true.
3. Check the worker logs for startup/configuration errors before restarting.
4. If every worker is healthy but one heartbeat is missing, verify the OpenTelemetry Collector and Prometheus scrape target before changing worker scale.

WipeHTTP5xxRate link

1. In Grafana Wipe Backend Overview, check HTTP Request Rate by route and status.
2. Filter Loki logs by service_name="wipe-api" and the failing http_route.
3. Follow trace_id into Tempo when present.
4. Cross-check dependency readiness with /readyz, then Keycloak, PostgreSQL, MinIO, and signer configuration.

WipeQueueBacklogHigh link

1. Check Queue Depth And In-Flight Jobs by queue and status.
2. Verify the corresponding worker container is running and exporting logs.
3. Inspect wipe_queue_jobs_total by outcome for retry or dead-letter growth.
4. Drain or replay DLQ entries only after the root error in worker logs is understood.

WipeQueueJobFailures link

1. Check Queue Job Outcomes for retry, retry_error, complete_error, or dead_letter_error by queue.
2. Inspect logs for the affected worker component and queue.
3. Map the failing queue to its dependency: object storage for proof/report jobs, signer for proof jobs, SMTP/webhook targets for notification jobs, chain provider for anchor jobs.
4. Confirm the retry rate drops before replaying or creating replacement jobs.

WipeQueueDeadLetters / WipeQueueDLQNotEmpty link

1. Check Queue DLQ Entries by queue.

2. Inspect the newest DLQ rows:

     SELECT id, original_job_id, queue, reason, created_at
   FROM jobs_dlq
   ORDER BY created_at DESC
   LIMIT 20;
     

3. Apply the replay rules from Workers and queues. Do not mutate a dead job back to pending.

4. Leave retained DLQ rows in place for audit unless the incident process explicitly approves archival.

WipeProofProcessingStalled link

1. Check Pending Proofs for whichever status is accumulating.
2. Check Proof Processing Rate for retry, dead_letter, awaiting_license, and terminal outcomes.
3. For AWAITING_LICENSE, validate license grants and allocation state before retrying.
4. For VALIDATING, GENERATING_PDF, SIGNING, or ANCHORING, inspect proof-worker logs and traces.

WipeProofsFailed / WipeProofsAwaitingLicense link

1. Check Pending Proofs grouped by FAILED and AWAITING_LICENSE.

2. For FAILED, inspect the proof worker logs and rejection_reason:

     SELECT id, tenant_id, organization_id, status, attempt_count, next_retry_at, rejection_reason, received_at
   FROM proofs
   WHERE status IN ('FAILED', 'AWAITING_LICENSE')
   ORDER BY received_at DESC
   LIMIT 50;
     

3. For AWAITING_LICENSE, check license grants, allocation scope, user quotas, and revocation state before retrying.

4. For FAILED, fix the dependency or malformed configuration first, then use the retry flow from Workers and queues.

WipeSignerFailures link

1. Check Signer Operation Errors by operation: decrypt, hmac, hmac_verify, pades_sign, or receipt_sign.
2. Inspect proof-worker and API logs for signer errors around the same time window.
3. Confirm signer mode, endpoint reachability, mTLS material, bearer token, key IDs, and TSA configuration when PAdES signing fails. See Signer boundary.
4. After recovery, run a proof upload smoke test and a public verification check to exercise decrypt, HMAC, PAdES, and HMAC verification paths.

WipeAnchorFailures link

1. Check Anchor p95 Duration by chain_id and outcome.
2. Inspect anchor-worker logs for submit or confirmation errors.
3. Confirm chain configuration, authorized sender/key configuration, and provider endpoint health.
4. With the local noop chain, this alert should stay silent because no real submit/confirmation is attempted.

WipeDBConnectionsSaturated link

1. Check Database Connections by service_name.
2. Look for slow routes or workers with high in-flight jobs.
3. Check PostgreSQL health and active queries.
4. Scale the saturated component or tune pool limits only after identifying the workload causing pressure.

WipeVerificationAnomalyCritical link

1. Check Verification Anomalies by anomaly and scope.
2. Filter verification_log for the same time window and anomaly type.
3. For UNAUTHORIZED_ANCHOR_SENDER, treat the event as a potential trust-boundary incident and verify chain registry / authorized sender state before dismissing it.
4. Preserve logs and verification rows for incident response; do not purge requester metadata during active triage.

WipeHashChainIntegrityBreak link

1. Check Hash Chain Integrity for the affected chain (audit_log or consumption_receipts).
2. Inspect maintenance-worker logs around the alert time.
3. Stop any manual data repair until the incident owner confirms the expected previous/current hashes.
4. Restore from backup or apply a signed corrective migration only through the incident process.

Production Settings link

Keep logs free of secrets, raw proof payloads, canonical certificate JSON, API key secrets, JWTs, and private tenant data.