Advisory analysis, privacy limits, and approval boundaries.

Failure investigation and recovery planning

On a failed run, an admin or developer can request a failure investigation or a dry-run recovery plan. The deterministic investigator uses at most 100 steps, 12 attempts per step, and 100 recent attempts from affected services over seven days. It reports the first event visible within retained history, outcome counts, error categories, hypotheses with uncertainty, checks, and links to the run. It cannot infer events before retention or inspect a destination's own logs.

On the Incidents page, an owner, admin, or developer can investigate a circuit opening or recovery. The service investigator compares up to 100 non-synthetic attempts in the seven days around the event, links failed attempts to their runs, and reports the earliest failed attempt in that bounded sample. A circuit event with no retained attempts produces low-confidence findings. Cross-organization events and attempts are excluded. This view does not approve recovery or change the service policy.

The recovery planner lists each visible step, skips successful and process-bound work, and identifies exhausted durable steps that can be replayed under their existing contract and handler version. A tool marked FORBIDDEN and a parent with dependent branches are excluded. GetRatchet cannot verify idempotency at the destination, so every eligible step carries duplicate-side-effect risk. The plan is a dry run. A separate owner/admin approval requires the exact evidence revision, explicit duplicate-risk acknowledgement, and an additional acknowledgement for any tool not classified SAFE. The server locks and rechecks job state and permissions before creating a recovery job; worker polls recheck each item before scheduling. Model text never determines which steps run.

External advisory text is off by default. An owner or admin may opt in under Settings → AI analysis privacy after an operator configures server-side GROQ_API_KEY. Only bounded statuses, attempt outcomes, durations, circuit state, replay-safety classification, and coarse error categories are sent. No account IDs, names, run or step IDs, tool or service names, prompts, payloads, raw errors, secrets, or API keys are sent. The Groq Responses request sets store: false and offers no tools. Groq's data controls and usage metadata policy still apply. If the key is absent, opt-in is off, the provider times out, or a rate limit or daily quota is reached, deterministic findings and plans remain available.

The integration uses openai/gpt-oss-20b served by Groq, caps model output at 400 tokens, sends at most 4,000 characters of abstract context, and allows at most 10 calls per organization per UTC day. Groq's Free plan limits are separate, apply to the Groq account, and may change. Calls and reserved token ceilings are recorded without storing prompts. Analyses are cached by an evidence revision and removed when the related run expires. Generation and approval actions are audited without raw context. The UI displays when a result is deterministic and when external advisory text was returned.

To verify the live provider path, configure the key in Vercel Production, redeploy, opt in for a test organization, generate a failed SAFE synthetic run or controlled durable run, and inspect the run's analysis. Confirm a second request uses the cache, the daily counter increments only on the external call, and approval creates a recovery job only after a human reviews and acknowledges its plan. The repository tests cover context minimization and deterministic eligibility; they cannot prove a live Groq response without a configured key and opted-in organization.