Running Agents in Production
10 articles in series
Ten essays on AI-native operations — why agent observability looks nothing like service observability, the five-step incident response playbook, blast-radius gates, hallucination rate as a tracked metric, and the on-call runbook for teams running autonomous agents.
The On-Call Runbook for AI-Native Teams
When part of what's on call is autonomous, the rotation needs a different runbook — one built around judgment calls, not just error codes.
When to Kill an Agent Run vs Let It Keep Going
The instinct to let a stuck agent 'figure it out' is usually wrong. Good kill-switch design is a judgment call made in advance, not under pressure.
Version-Controlling Agent Policy Like You Version-Control Code
If your guardrail config is a live setting someone edits in place, you can't answer 'what was the policy last Tuesday' — and eventually someone will ask.
Cost Control for Agents on a Shared Compute Budget
One agent doing the same expensive thing on every bot visit can quietly burn a shared infrastructure budget before anyone notices the trend line.
Audit Logs for Autonomous Systems: What Regulators Will Ask For
Gartner expects AI-related legal claims to top 2,000 by end of 2026, mostly from thin risk guardrails. The audit log is what determines which side you're on.
Hallucination Rate as a Metric You Actually Track
Everyone agrees hallucination is a risk. Almost nobody has a number for it. Here's how to turn a vibe into a dashboarded production metric.
Blast-Radius Gates: Limiting What a Single Agent Action Can Do
You can't guarantee an agent's judgment is always right. You can guarantee that when it's wrong, the damage is small, visible, and reversible.
The Five-Step Incident Response Playbook for a Misbehaving Agent
Detection, classification, containment, root cause, remediation — the same five steps as any incident, applied to a system that can improvise its next move.
Guardrails Block, Observability Explains: Why You Need Both
A guardrail without observability stops the bad action once and tells you nothing. Observability without a guardrail just narrates the damage as it happens.
Observability for Agents Looks Nothing Like Observability for Services
Traditional APM tells you a request was slow. It can't tell you why an agent chose to call the refund tool instead of asking a human first.