AI Agent Error Budget: Pick a Failure Rate You Can Ship

Nobody ships an agent that is right every time, so the only honest question is how often it may be wrong and what happens when it exceeds that. An AI agent error budget turns that from an argument into a number, but the version borrowed straight from web reliability breaks on the first day. It breaks because agent failures are not interchangeable.

What is an error budget, and why borrow it here?#

It is the allowance for failure implied by a reliability target. Google's practice defines it as 1 minus the SLO, so a 99.9 percent service carries a 0.1 percent budget, and when that budget is spent the team freezes changes except urgent fixes until it is earned back or the period resets.

The reason it transfers to agents is that it kills the two useless positions in every launch argument. One side wants the agent to be right every time, which prices nothing and ships nothing. The other says it is good enough, which measures nothing. A budget forces both into a number that can be checked next month.

What transfers less well is the assumption underneath it. Availability is roughly binary: the request served or it did not. An agent has at least 5 distinct ways to fail, they are not equally bad, and a single percentage treats them as though they are.

Exact arithmetic, not an estimate. Percentages hide the count, and the count is what the argument is really about.

The arithmetic is worth doing out loud once, because percentages hide scale. At 10,000 decisions a month, a 99 percent target permits 100 wrong ones. That is 100 real customers, and whether that is fine depends entirely on which 100 and how loudly they failed.

Why is a single accuracy number the wrong target?#

Because two agents with identical accuracy can have completely different costs. One refuses 4 percent of the time and the user retries. The other answers 4 percent of the time with a confident wrong number that a person acts on. The metric is the same and only one of them belongs in production.

The distinction that actually matters is whether a failure is loud or silent. A loud failure announces itself: a timeout, a refusal, an error, an escalation. Someone knows immediately and the cost is usually inconvenience. A silent failure looks exactly like a success and is discovered later, by a customer, or never.

Budget these separately. A generous allowance for loud failures and a very tight one for silent failures describes what you actually want, and a single blended accuracy figure cannot express it.

The last row is the one worth spending real effort on. The rows above it are mostly inconvenience.

This also changes what counts as an improvement. Converting a silent failure into a loud one is progress even when accuracy is unchanged, because a confident wrong answer became a refusal a person can see. Most practical agent hardening is exactly that trade.

What has to exist before you can measure it?#

A per decision record and a definition of wrong that does not require a human to adjudicate every case. Without the first you cannot count anything. Without the second you can count only what somebody complained about, which undercounts silent failures by a wide and unknowable margin.

The cheapest usable signal is downstream correction. If a later step, a reviewer or the customer changed the agent's output, treat that as a failure without asking anyone to grade it. It is imperfect and it is available today, which beats a labelling project that never starts.

Without the first two rows the budget measures complaints, which undercounts silent failures by an unknowable margin.

Sampling covers the rest. Pull 50 decisions a week at random, have a person judge them, and use that to estimate the silent failure rate the corrections never surface. It is a small standing cost and it is the only thing that catches an agent quietly getting worse after a model update.

How do you set the number without guessing?#

Work backwards from the thing the agent replaced. If the process it automated was already wrong some of the time, that rate is your reference point, and matching it is a defensible target. Beating it is a business case. Being told to reach 100 percent is a request nobody made of the humans doing it before.

Then split the budget by mode rather than assigning one figure. Give loud failures room, because they are recoverable and the alternative is an agent so cautious it escalates everything. Keep the silent budget small enough that spending it triggers a real response rather than a shrug.

Step 1 removes the argument about perfection. Nobody asked the previous process to be right every time.

What happens when the budget is spent?#

Whatever you agreed in advance, which is the entire point. In the original practice the team freezes changes except urgent fixes addressing the cause, until the budget is earned back or the period resets. For agents the equivalent lever is usually authority rather than deploys.

Narrowing authority is the fastest response available. An agent that was acting alone starts proposing instead, or its scope shrinks to the categories it handles well, or a confidence threshold routes more cases to people. All three reduce the damage rate immediately without waiting for a fix.

The failure of the whole mechanism is agreeing the number and then not agreeing the consequence. A budget with no attached action is a metric on a dashboard, and it will be exceeded quietly for a quarter before anybody raises it. Write the consequence down next to the number, on the same day.

Narrowing authority acts immediately. Waiting for a fix leaves the damage rate where it is.

When should the silent budget be zero?#

When a single wrong action cannot be undone or is not recoverable at any acceptable cost. Irreversible payments, deletions, external communications sent on your behalf, and anything with a regulatory consequence belong here. For these the answer is not a tighter budget, it is a person in the path.

A zero budget is a design instruction rather than a target. It means the agent may prepare the action but not commit it, and deciding what an agent may do alone is the mechanism. Everything else can carry a real number, and most things should.