Worker leases, retries, versions, and safe replay.

Durable worker operations

Run registerTool({ name, version, handler }) in a separate long-lived Node process, then await ratchet.worker.start({ concurrency, signal }). Deploy at least one worker for each exact name/version still present in queued or retry-scheduled jobs. Keep the old version running until its jobs finish; registering a newer version does not silently migrate them.

The producer calls enqueue with a stable idempotency key, then can request run completion. The run remains RUNNING while required steps are pending and becomes SUCCEEDED after the last one succeeds. An exhausted or cancelled step makes the run FAILED. A failed run does not automatically cancel other in-flight steps; inspect them before retrying.

The queue is PostgreSQL. A claim atomically locks a due job and its endpoint, increments the attempt and lease token, and creates an execution row. Heartbeats extend the lease. If a worker vanishes, another worker with the same handler version can reclaim the job after lease expiry. The previous attempt is marked failed with a warning that an external side effect may have completed. A stale worker report receives HTTP 409. A worker should stop normal polling on shutdown and let active calls finish; worker.stop() does that.

Failures use exponential backoff with 20% jitter, starting at 30 seconds and capped at 15 minutes. After the configured maximum attempts, the job is EXHAUSTED. An input or output schema rejection is non-retryable. A tool timeout aborts the handler's AbortSignal; handlers must propagate it to their own HTTP calls. A handler that ignores it can continue running after GetRatchet records a timeout, so destination-side idempotency remains essential.

If queue age rises, check that a worker is online, its key works, and it registers the exact queued handler versions. If a circuit is open, inspect recent attempts and the external service before resuming traffic. A paused service holds queued jobs without consuming attempts; existing in-flight calls may finish. If database access fails, producers cannot enqueue and workers cannot claim. Do not run tools outside GetRatchet to work around that failure unless the caller deliberately accepts losing the execution record.

Do not log input, output, credentials or full error bodies from the worker. GetRatchet stores encrypted recovery input and result data; the dashboard shows redacted previews. Keep RESULT_ENCRYPTION_KEY available on every API and worker deployment. Follow the rotation procedure before changing it.

Replaying a failed step

An administrator can preview GET /api/v1/steps/:id/replay and then request POST /api/v1/steps/:id/replay with {"acknowledgeDuplicateRisk":true,"maxAttempts":1}. The server locks the durable job and accepts this only after it is exhausted or cancelled. By default, replay keeps its encrypted original input and exact handler version, increments the attempt number, and writes a replay event with the actor key ID. Confirm destination-side idempotency before replay: the previous worker may have completed an external side effect even when its result was not recorded. A paused service holds the replay until resumed.

To use a different handler, first publish a new contract for the same tool name and inspect the replay preview's upgrades list. The preview validates the encrypted saved input against each target input schema without returning the raw input. An incompatible or FORBIDDEN target cannot be selected. After reviewing schema and side-effect changes, submit targetContractVersion, expectedHandlerVersion from the preview, and acknowledgeVersionChange: true along with the normal replay fields. The server rechecks the target and saved input under the job lock, updates the step's contract and handler version for the next attempt, and records both old and new versions in the replay event. Previous executions remain unchanged. The target worker must register the new exact handler version; otherwise the job stays queued. Schema compatibility only proves the saved input has the expected shape; the handler may still behave differently. Check destination-side idempotency and the target handler's effects before approving the replay.

Worker polling defaults to every 30 seconds. Shorter intervals increase database operations and Vercel function invocations; budget capacity before lowering this value. A worker needs a separate WORKER key authorized for every exact name@version it registers. A producer needs an INGEST key. The worker ID remains bound to its registering key.

Service policies are read at GET /api/v1/endpoints/:id/policy and changed by an administrator with PUT and an expectedVersion. Presets are STANDARD (five attempts, 30-second initial backoff), AGGRESSIVE (eight attempts, five-second initial backoff), and RELAXED (three attempts, two-minute initial backoff). A service also has a default timeout, maximum concurrent leased jobs, and optional attempts-per-minute limit. Durable steps store the policy version; jobs snapshot timeout, maximum attempts and retry preset. Policy updates do not change those values or previously scheduled due times. Concurrency and rate gates apply to new claims immediately. Paused services accept new jobs but do not claim them.