Trace propagation, metrics, dashboards, and alerting.

OpenTelemetry export

GetRatchet emits OpenTelemetry traces and bounded-cardinality metrics. Set OTEL_EXPORTER_OTLP_ENDPOINT to an OTLP/HTTP collector base URL to enable export from the Next.js API. Set OTEL_EXPORTER_OTLP_HEADERS if the collector requires authentication. Store both as server-side environment variables; never put collector credentials in NEXT_PUBLIC_* values. The API does not require a collector to serve requests.

The TypeScript SDK injects W3C trace context into API requests. A durable enqueue stores traceparent and tracestate on the queued job; the claim response carries them to the worker, which starts an attempt span under the producer trace. Worker applications can opt in to export by calling startWorkerTelemetry() before creating the Ratchet client and calling await telemetry.shutdown() on graceful exit. The worker process needs the same OTLP environment variables and network access to the collector. Applications with their own OpenTelemetry provider can omit this helper and keep their provider.

import { createRatchet, startWorkerTelemetry } from '@getratchet/sdk';

const telemetry = startWorkerTelemetry('my-agent-worker');
const ratchet = createRatchet({ baseUrl: process.env.GETRATCHET_URL!, apiKey: process.env.GETRATCHET_API_KEY! });
// Register handlers and run the worker as usual.
// On graceful shutdown: await telemetry.shutdown();

The Python SDK does not provide startWorkerTelemetry(). Configure your own OpenTelemetry provider in the Python worker if it needs OTLP export.

API metrics include accepted runs and steps, attempt outcomes and duration, retries and scheduled delay, lease expirations, circuit transitions, exhausted steps, recovery item outcomes, queue depth and oldest age, time since the most recent worker heartbeat, and request counts and handler duration by static route, method, and status class. Thrown route errors count as 5xx. Tool and service names are hashed into 128 buckets; no raw names, inputs, outputs, error text, account IDs, API keys, or unbounded URLs are metric labels or span attributes. The queue and heartbeat gauges aggregate across organizations and must not be used to inspect an individual tenant. Request and worker spans are exported only when a collector is configured. The Grafana dashboard template assumes an OTLP collector forwards metrics to a Prometheus-compatible data source. Select that data source after import.

The alert rule guide gives PromQL, evaluation windows, contact-point setup, and an end-to-end retry verification procedure. Use the authenticated /api/v1/health endpoint for tenant-specific queue age and worker heartbeat; a developer-owned worker may be intentionally offline. Dashboard panels remain empty if no OTLP destination is configured or the collector has not received metrics. API handlers schedule a post-response metric flush because short-lived serverless instances may stop before the periodic reader fires. Export can still fail during a collector outage; the database-backed incident history remains the durable source for operational events.