When an API error says what to fix, agents use half the tokens

I work at Restless, where we built Agent Recovery, and I wanted to know if it helps agents as much as we hope it does. I kept wondering what an agent does when it gets a generic error back from an API. Does it go read the docs, guess, retry the same call, or give up? And what changes if the error tells it what to fix?

So I came up with a few hypotheses and tested them. I ran four Claude models through the same 15 tasks, 3,600 runs in total. Half the time, errors came back as just the status, like Unprocessable Entity. The other half, the same error had one extra line saying what to change. I logged every call the agents made and checked each hypothesis against the logs.

When the docs were long or missing, that one line made the difference. Every model finished every task, in 29-46% fewer turns and with 45-64% fewer tokens.

Agent Recovery adds that line to your API's errors. An agent that hits one gets the fix right in the response and makes a successful call instead of guessing, and the person who asked it to do the job gets back to building sooner.

What the benchmark found

How I set up the benchmark

Four Claude models worked through 15 tasks on a small shipping API, with short, long, or no docs, and either a generic error or the same error plus one line saying what to change. Each combination ran 10 times. A task passed only when the server state and the agent's final answer were both right, not when the agent said it was done.

The API, tasks, models, and docsShowHide

• The API is a small shipping API with endpoints for rates, shipments, tracking, cancellations, and address validation.
• The 15 tasks are things a user might hand an agent, like "Is this a valid shipping address?", "Buy the cheapest two-day label", or "Cancel the shipment with tracking number 1Z999AA10123456784." Five of them were written so the status code alone doesn't reveal the fix.
• The models are Claude Opus 5.5, Sonnet 5.5, Haiku 5.5, and the older Sonnet 4.5. Each ran at its defaults, with one HTTP tool and a cap of 15 turns per task, and could read only the first 8,000 characters of any response.
• Small docs is a 9-endpoint OpenAPI spec the agent can read in one fetch. Large docs is a 139-operation spec, about the size of a real production API and too long for the agent to read in full. With no docs, the agent knows the endpoint names and nothing else.
• In the generic version, an error body holds only the HTTP status text, like Unprocessable Entity, without the API's own message. In the recovery version, the error body is the same, plus one line telling the agent what to change. That line is what Agent Recovery adds to your error responses.

How the confidence intervals workShowHide

Each combination ran every task 10 times. Runs of the same task tend to pass or fail together, so I tested each difference by resampling whole tasks, 10,000 times. A difference counts only when 95% of those resamples land on the same side of zero.

I built the API and tested only Claude models, and I made the generic errors as bare as possible, so read these numbers as one setup, not a rule. Next, I'm running the same benchmark on OpenAI models, and I'll share those results here too.

Does the fix in the error make agents cheaper?

Hypothesis: an agent that's told what to change spends less effort than one that has to work it out.

Result: with long or missing docs, a next step in the error cut the effort about in half for every model. With a short spec, it made no difference.

Change per task when the error carried a next step:

	Small docs	Large docs	No docs
Errors	No change	↓ 57-72%	↓ 69-77%
Turns	↓ 1-12%	↓ 29-43%	↓ 29-46%
Tokens	No change	↓ 45-64%	↓ 46-61%

Each range spans the four models. With large docs and with no docs, every model's difference held up at the 95% interval.

Tokens per task with the large spec:

Model	Generic error	With fix in the error
Opus 5.5	30,388	16,803 ↓ 45%
Sonnet 5.5	44,357	23,437 ↓ 47%
Sonnet 4.5	31,016	11,277 ↓ 64%
Haiku 5.5	35,608	18,407 ↓ 48%

The gap comes from how agents look for a fix. After a bare Bad Request, an agent goes back to the spec. With the small spec that's one cheap fetch. With the 139-operation spec, it reads the first chunk, often misses the endpoint it needs, and starts guessing. With no spec, it can only guess, and each guess costs a call and a turn of context. When the error says what to change, there's nothing to look up.

With the small and large specs, 58% of the runs that hit a generic error went back to fetch the spec (286 of 491). When the error carried a next step, 6% did (30 of 494).

The table below follows one task in both versions: Sonnet 5.5 creating a UPS Ground shipment with the large spec. The shipments section starts at character 30,568 of the spec, and the agent reads only the first 8,000, so both runs read the spec twice and still missed it.

#	Generic error	With fix in the error
1	Read spec (cut off) · 200	Read spec (cut off) · 200
2	GET /v1/warehouses · 200	GET /v1/warehouses · 200
3	GET /v1/carriers · 200	GET /v1/carriers · 200
4	Read spec again · 200	Read spec again · 200
5	Validate address · 200	POST /v1/shipments · 422 with the fix
6	Validate address · 200	Validate address · 200
7	POST /v1/shipments · 422	Validate address · 200
8	POST /v1/rates · 422	POST /v1/shipments · 201 done
9	POST /v1/rates · 422
10	List shipments to copy their shape · 200
11	POST /v1/shipments · 422
12	POST /v1/shipments · 422
13	GET /openapi.json?path=/v1/shipments · 200
14	POST /v1/rates · 200
15	POST /v1/shipments · 201 done
	5 errors, 10 turns, ~120,000 tokens	1 error, 5 turns, ~45,000 tokens (63% fewer)

With generic errors, the agent needed 15 calls to the recovery version's 8. At step 13 it invented a ?path= filter to ask for just the shipments docs. The API has no such filter, so it got the same truncated spec back. Across all 10 runs of this task, generic errors averaged 10 turns and 109,582 tokens, and errors with a next step averaged 5 turns and 48,286.

What if the agent can search the spec itself? I ran 48 extra runs with two models that also had a terminal. As soon as the spec came back cut off, they used it, and always the same way: download the spec, then print only the endpoints they needed.

With that, a generic error cost no more than one with a next step. Without a spec to search, the terminal didn't help, and the next step still cut errors by two-thirds. So if the agents calling your API are coding agents and your spec is public, they can probably work it out. Many agents, like MCP tools and function-calling apps, only make HTTP requests.

Does the fix in the error help agents finish tasks?

Hypothesis: agents that know what to change fail less often.

Result: agents failed less often with a next step in the error, and older and smaller models gained the most. With long or missing docs, every model finished every task.

Change in task success when the error carried a next step:

Model	Small docs	Large docs	No docs
Opus 5.5	↓ 1%	No change	↑ 7%
Sonnet 5.5	No change	↑ 1%	↑ 1%
Sonnet 4.5	↑ 5%	↑ 15%	↑ 9%
Haiku 5.5	No change	↑ 4%	↑ 18%

Three of these gains held up at the 95% interval: Sonnet 4.5 with large and no docs, and Haiku 5.5 with no docs. The smaller gains each came mostly from one task, so I don't count them. Sonnet 5.5 rarely failed in either version, so there was no gain to measure. The one Opus run that missed with a next step in the error returned an empty answer.

Once the error included a next step, that was the only failed run in the benchmark. With generic errors, eight of the twelve model-and-docs setups had at least one failure.

Most of those failures came from the two address-validation tasks. Sonnet 4.5 with large docs and Haiku 5.5 with no docs also failed rate and shipment tasks, so it isn't only one endpoint, but address validation is where a generic error did the most damage.

Do the newest models need the fix in the error?

Hypothesis: the newest models are good enough to recover from a generic error on their own.

Result: the newest models recovered only when the docs are good. With the small spec, Opus 5.5, Sonnet 5.5, and Haiku 5.5 mostly sent well-formed requests the first time and finished every task. There were so few errors that a next step had little to do, and errors, turns, and tokens barely moved.

Sonnet 4.5 still gained with the small spec, going from 95% to 100% task success.

So a short, clean spec still does the most for an agent. Most APIs don't have one, though, and the agent calling yours may not be the newest model.

Why do agents mistake an error for an answer?

Hypothesis: a generic error can read like a result, and an agent will report it as one.

Result: agents did report an error as an answer, Sonnet 4.5 most often. One task asks: "Is this a valid shipping address: 123 Fake St, Nowhere, XX 00000?" The validation endpoint wants a flat object with street, city, state, and zip. Agents often nested it under address instead, and got back a 422.

To an agent that asked whether an address is valid, Unprocessable Entity reads like "no," and Sonnet 4.5 often took it that way. In 25 of the 27 runs where it failed this task, it never got a successful response from the endpoint, and still told the user something like "No, this is not a valid shipping address. The address validation service returned an error indicating that 123 Fake St, Nowhere, XX 00000 cannot be processed."

The address is fake, so the answer happens to be right, but the check never ran. The 422 meant the request was malformed, and the answer doesn't tell the user that.

The newer models were more careful. When Opus 5.5 and Haiku 5.5 failed this task with no docs, they said they couldn't confirm the address. In one run, Opus tried a known-good address, got the same 422, and concluded that the error said nothing about the user's address. That's more honest, but the user still didn't get an answer. Haiku 5.5, with no docs, also couldn't validate a real address in any of its 10 tries.

In the recovery version, the same 422 came back with a next step. Below is that response, with the part Restless adds highlighted. The generic version got only the error line.

With the next step, every model in every setup fixed the request and got the API's real answer: the ZIP code isn't in the USPS database.

Do agents follow a docs link in the error?

Hypothesis: if the error links to the docs, agents will open it.

Result: agents almost never opened the docs link, though the sentence before it had already told them what to change. Every next step in the benchmark ended with a link to that endpoint's docs. In 818 runs, an agent hit an error and got a next step, and it opened the link in 2 of them, both after several failed retries. In the rest, it acted on the sentence, even in the setup with no spec at all.

How should you write API error messages for agents?

Agents act on the error text and skip the link. In 818 runs where an error said what to change and linked to the docs, agents opened the link twice. They also stopped going back to the spec: after a generic error, 58% of runs fetched it again, and when the error said what to change, 6% did. The sentence in the error is the part an agent reliably reads, so it has to carry the fix. We first made this case in Agent-Ready APIs, and the benchmark data supports it. When you write your error messages:

1. Say what to change. "Send a flat object with street, city, state, and zip; don't nest it under address" gives the agent something to do. "See the address validation docs" doesn't.
2. Name the valid values. If a 404 means the agent passed a tracking number where you expect a shipment id, say what a shipment id looks like and how to look one up.
3. Start with the errors that read like answers. A 422 on a validation endpoint, a 404 on a lookup, or a 403 on a permission check can each pass for a result, and those are the ones agents report as a verdict.

Generic error vs. error with the fix (Agent Recovery)

	Generic error	Agent Recovery
What the agent gets	The HTTP status text, like Bad Request	The same status, plus one line saying what to change
What the agent does next	Searches the spec, or guesses	Changes the request and calls again
Errors per task, large docs	1.2-2.3	0.5-0.7 (57-72% fewer)
Tokens per task, large docs	30,000-44,000	11,000-23,000 (45-64% fewer)
Task success, large or no docs	82-100%	100%
A 422 on a validation check	Can read as "invalid"	Gets fixed, then the real check runs
Changes to your API code	None	Installed Restless SDK

Add next steps to your errors

If every error your API returns already says what to change, agents get most of this benefit today. If not, or if you can't change your errors quickly, Agent Recovery fingerprints every error your API returns and serves the next step you write for it inside the error response.

• Already on Restless? Open Agent Recovery in your dashboard and write next steps for your most common errors first.
• New to Restless? Run npx restless init or sign up here.