GETRATCHET / GUIDE
PromQL starting points and an end-to-end verification flow.
Grafana alert rules
Import getratchet.json and select the Prometheus-compatible data source that receives the OTLP collector's metrics. Use a contact point owned by the operator. Start with notifications to an internal test address and confirm routing before enabling a production contact point. The thresholds below are starting values, not an SLA.
| Signal | PromQL | Evaluation |
|---|---|---|
| Exhausted tool calls | sum(increase(getratchet_steps_exhausted_total[5m])) > 0 | Every minute, pending 0 minutes |
| Circuit opened | sum(increase(getratchet_circuit_transitions_total{state="opened"}[5m])) > 0 | Every minute, pending 0 minutes |
| Queue age rising | max(getratchet_queue_oldest_age_seconds) > 300 | Every minute, pending 5 minutes |
| Worker heartbeat absent | max(getratchet_worker_latest_heartbeat_age_seconds) > 180 | Every minute, pending 5 minutes; check whether the worker is intentionally stopped |
| Sustained failure ratio | sum(rate(getratchet_attempts_total{outcome="failed"}[10m])) / clamp_min(sum(rate(getratchet_attempts_total[10m])), 0.001) > 0.10 | Every minute, pending 10 minutes; add a minimum-volume condition of 20 attempts in 10 minutes |
| API server error ratio | sum(rate(getratchet_api_requests_total{status_class="5xx"}[10m])) / clamp_min(sum(rate(getratchet_api_requests_total[10m])), 0.001) > 0.05 | Every minute, pending 10 minutes; add a minimum-volume condition of 20 API requests in 10 minutes |
The gauge series can be emitted by more than one serverless instance. Use max, not sum, for queue age and heartbeat age. Use the organization-scoped health API before acting on a global alert. Vercel's short-lived instances may miss periodic metric exports, so set a suitable no-data policy and use database-backed incident email as a second signal.
End-to-end verification
- Configure OTLP/HTTP trace and metric export on the API and one developer worker. Confirm the collector receives both service names.
- Start one durable run with a tool whose test handler fails once with a retryable error, then succeeds. Use a destination-side idempotency key even in this test.
- In the trace backend, follow the producer request span to the API acceptance span and the worker attempt spans using the persisted W3C trace context. Inspect the run page for the same two attempts and retry delay.
- In Grafana, confirm accepted-run and accepted-step counters, two attempt outcomes, retry count and delay, queue age, and attempt duration. Trigger an exhausted SAFE test tool if verifying the exhausted alert.
- Restore the test handler and confirm the circuit and queue return to normal. These live checks require an OTLP collector, a trace backend, Grafana credentials, and a running developer worker; the repository tests verify trace persistence but cannot establish that an external collector received exports.