Common Pitfalls

A short list of mistakes that have either happened already or are easy to make on first contact with the platform. Read this before operating a non-local instance.

Dev Secrets Outside Local

SecretWhy it bites you
wipe_dev_agent.0123456789abcdef0123456789abcdefSeeded proof-automation API key with proofs:write and proofs:read. If accepted by a non-local instance, anyone holding the seed can submit proofs.
wipe_dev_enroll.0123456789abcdef0123456789abcdefSeeded one-time enrollment code. If enrollment is enabled in a shared environment, an attacker can enroll a rogue agent.
KEYCLOAK_MASTER_SEED_USER_PASSWORD / adminDefault passwords for seeded users. Never reuse outside DATABASE_SEED_DEV=true local stacks.
TEST_SIGNER_SEEDTest signer HMAC seed. Production must use a remote signer with real custody; the test signer is not a custody boundary.

Rule of thumb: if DATABASE_SEED_DEV=true, AGENT_DEV_ENROLLMENT_ENABLED=true, LICENSE_DEV_MODE=true, or SIGNER_MODE=test is set, the environment is not production-safe.

Master-Realm Token Scope

PLATFORM_ADMIN and BILLING_OPERATOR tokens issued by the master realm are not scoped to a tenant. Every admin request that targets tenant data must include explicit tenant_id (and organization_id when applicable):

  ✗ GET /api/v1/users                       # uses no target scope
✓ GET /api/v1/users?tenant_id=...         # platform admin targeting a tenant
✓ POST /admin/api/v1/tenants/{id}/organizations  # scope embedded in URL

A frontend must never infer tenant_id or organization_id from a master-realm token. If your tooling does, you are one mis-click away from operating on the wrong tenant.

Dev-Mode Bypass Flags

FlagEffectProduction rule
LICENSE_DEV_MODE=trueBypasses grant signature verification after instance-fingerprint check.Refused when the environment declares production; keep it false and rely on real Exaion keyring signatures.
DATABASE_AUTOMIGRATE=trueAPI/worker create missing tables via GORM AutoMigrate.Disable in production; run reviewed SQL migrations only.
AGENT_DEV_ENROLLMENT_ENABLED=trueSeeds the dev enrollment code.Disable outside local dev.
SIGNER_MODE=testIn-process deterministic signer, no custody boundary.Production requires SIGNER_MODE set to the remote signer with mTLS, key IDs, and TSA.

Idempotency-Key Mistakes

  • Reusing a key with a different payload → CONFLICT. The backend stores the original request hash and rejects replays that do not match.
  • Omitting Idempotency-Key on a protected write → BAD_REQUEST. Every POST, PUT, PATCH, and DELETE under /api/v1 and /admin/api/v1 requires a UUID v4.
  • Generating non-UUID v4 values → BAD_REQUEST. Use a proper UUID v4 generator; do not hash or derive the key from request content.

DLQ Replay Errors

  1. Do not UPDATE jobs SET status='pending' WHERE status='dead'. This hides the replay from queue metrics and audit. Insert a new jobs row with the same queue and payload, and keep the DLQ row.
  2. Do not replay before fixing the dependency. A dead-lettered proof job against a broken signer will just dead-letter again, doubling DLQ volume.
  3. Do not replay REJECTED proofs. They are terminal validation failures; replay only if the validation rule itself was wrong.

See Workers and queues for the full replay procedure.

Outbound URL Policy Surprises

IdP metadata, branding logo URLs, and webhook targets all pass the same SSRF policy. By default the backend rejects:

  • localhost, loopback, link-local, and private IP ranges.
  • Credential-bearing URLs (https://user:pass@host/...).
  • Non-http(s) schemes.

If you need an internal IdP or a private webhook target, add the host to SECURITY_OUTBOUND_ALLOWED_HOSTS explicitly. Do not set SECURITY_OUTBOUND_ALLOW_PRIVATE_NETWORKS=true outside a controlled internal deployment.

MFA Recency

When SENSITIVE_ACTION_MFA_MAX_AGE is configured (recommended 10m–15m in production), sensitive writes require a JWT with a recent auth_time and an MFA amr value such as otp or webauthn. Common mistakes:

  • Forgetting to re-authenticate before rotating an API key or revoking a certificate during an incident.
  • Treating a stale auth_time as good enough — it is not; the recency window is enforced.
  • Local stacks with MFA recency disabled lulling operators into skipping the step on production.

Public Verification Information Leaks

The public /verify endpoint never returns tenant_id, organization_id, proof_id, internal storage keys, canonical JSON, the HMAC value, or PDF object keys. If you find yourself adding “one more field” to help support debugging, you are about to break the tenant isolation contract. Route all internal debugging through authenticated admin endpoints.

Audit Log Retention Edges

When retention prunes the oldest audit rows, the first remaining row becomes the trusted start for the next chain verification. Preserve external backups before changing RETENTION_AUDIT_LOG_TTL. A shortened TTL with no backup silently destroys chain continuity evidence.