Production configuration, operations, and key rotation.

Deployment, backup, and recovery

Release order

GetRatchet runs its API and console on Vercel, stores durable work in PostgreSQL, and runs customer tool handlers in separate developer-controlled workers. Keep the PostgreSQL connection and RESULT_ENCRYPTION_KEY stable across releases. Deploy additive database migrations before API code that uses them. Production and Preview builds run prisma migrate deploy against their own database before next build. Bootstrap seeding is an explicit operator action, never automatic on deployment. Preview refuses an inherited DATABASE_URL and requires its own PREVIEW_DATABASE_POSTGRES_URL and PREVIEW_RESULT_ENCRYPTION_KEY; it disables outbound email, OAuth, AI, and OTLP exports. CI must pass before merging to main, which is connected to Vercel Production.

Workers should be deployed after the API. Keep every old name@version handler online until its queued, retry-scheduled, and leased jobs finish. New handler versions do not take over old jobs. For a breaking API or schema change, first ship a compatibility release, then migrate, then deploy new producers and workers. Never delete or edit an applied migration; record the next change in a new migration.

Required environment

Vercel Production needs DATABASE_URL, BOOTSTRAP_API_KEY, and RESULT_ENCRYPTION_KEY. Account email needs RESEND_API_KEY and EMAIL_FROM. OAuth uses the provider client IDs and secrets. Optional external AI summaries need GROQ_API_KEY; each organization must also opt in. Optional OTLP telemetry uses the documented endpoint and headers in observability. Keep secrets in the deployment secret store, never in Git, logs, support tickets, or browser code. Give producer and worker processes separate scoped API keys.

Backup and restore

Take a consistent managed PostgreSQL backup before a migration and keep the backup in access-controlled storage with a retention schedule. A logical backup for an operator-managed database can be created with pg_dump --format=custom --no-owner --no-acl --file=getratchet.dump "$DATABASE_URL". Treat the dump as sensitive: it can contain account records, API-key digests, encrypted recovery values, and incident metadata. Store RESULT_ENCRYPTION_KEY separately under secret management; without that same key, encrypted values in the restored database cannot be read. Verify your database provider's own automated backup and point-in-time recovery settings and limits.

Test the backup in a new non-production database. Create that database, then run pg_restore --no-owner --no-acl --exit-on-error --dbname="$RESTORE_DATABASE_URL" getratchet.dump. Verify _prisma_migrations, organization and run counts, a scoped API read, and a worker claim on disposable test work. Keep the restored environment isolated from production email, OAuth, and customer workers so it cannot deliver old queued side effects. The CI job runs scripts/restore-smoke.sh against local PostgreSQL: it inserts a fixture, takes a logical backup, restores it to a separate CI database, and verifies the fixture and migration history. It does not test the managed provider's point-in-time restore or encryption key recovery.

Result encryption key rotation

The first release of the keyring reader keeps writing the old unversioned format while RESULT_ENCRYPTION_KEY_ID is unset. Deploy that reader everywhere and allow previous API instances to drain before enabling versioned writes. To rotate, first back up the database and current secret. Set RESULT_ENCRYPTION_KEY_ID=v1 with the same current key and redeploy; new ciphertext then carries the v1 prefix, and old ciphertext remains readable. Once those instances are current, choose a new 32-character-or-longer RESULT_ENCRYPTION_KEY and a new ID such as v2. Set RESULT_ENCRYPTION_KEY_PREVIOUS to a JSON object mapping the old ID to the old secret, for example {"v1":"<old secret>"}. Keep that historical secret only in server-side secret management. Deploy API and workers with the same keyring; verify old queued input and idempotency lookups still decrypt. The previous-key map is bounded to eight IDs and is never sent to clients.

To retire the old secret, run CONFIRM_REENCRYPT=1 pnpm exec tsx scripts/reencrypt-results.ts --apply against the intended database with the full current and historical keyring. This resumable script scans durable input and encrypted idempotency results in batches of 100 and uses compare-and-swap updates so concurrent result writes are not overwritten. Check its counts, rerun until both counts are zero, and verify a worker claim and duplicate idempotency lookup. Keep the old key configured until the scan is complete and all API and worker deployments have switched to the new key; then remove it from RESULT_ENCRYPTION_KEY_PREVIOUS. Unfinished jobs can outlive run retention, so waiting for the retention period alone is insufficient. A wrong or missing key stops decryption rather than silently discarding data.

Rollback and incident response

If an API deployment fails, first stop new producers, keep workers at a compatible version, and use Vercel's previous Ready deployment only if it understands the current schema. Additive migrations normally remain applied; prisma migrate deploy has no automatic rollback. If a destructive data change is unavoidable, restore a verified backup into a new database, review the data loss window, switch DATABASE_URL during a controlled outage, then resume producers and workers. Never point a restored staging database at production workers.

Check /api/health for platform API, database, and queue reachability, and /api/v1/health with a scoped key for customer worker and queue conditions. An offline customer worker does not mean GetRatchet's platform is down. If queue age grows, check worker heartbeat, exact registered handler versions, key scope, service pause and circuit state, then database connectivity. If a circuit opens, inspect destination health and recent attempts before resuming. If OTLP export fails, tool execution continues; inspect collector connectivity and exporter configuration. If AI is unavailable, the deterministic investigation and recovery plan remain available. See worker operations, observability, and AI advisory.

Database isolation

Hosting isolation uses separate provider databases and secrets. It does not yet provide database-enforced isolation between customer environments. See the restricted-role migration plan for schema changes, runtime-role checks, negative tests, cutover, and rollback requirements. Never describe the current API-level customer scoping as complete database isolation.