Part of an AI agent’s latency and cost buys no quality at all. It hides in four places: path, prefix, prose, and processing, the four Ps of agent waste.
Some waste can be removed with answers guaranteed to remain unchanged, some fixes need a quick replay, and only what remains is a real quality trade-off. Cut in that order, and prove every trade with a test that can detect a drop.
A team moved its agent to a model whose tokens cost 5–6× less. Cost per active user fell only 52%. Average response time rose 88%, from 64 to 120 seconds, because the cheaper model orchestrated its tool calls less efficiently. Users sent 10% fewer messages, and the team kept the original model
The model sets the price of each token. The shape of the work decides how many tokens you buy and how long the user waits for them.
Takeaway: Cheaper tokens do not automatically make an agent cheaper or faster. Measure cost and time per answer, not per token.
Waste or Trade-Off: One Test Decides
The test that separates waste from a quality trade-off is straightforward:
A second or a token is waste if removing it does not change what the agent gets right.
Sort fixes by how much proof they need before shipping. Start with changes that preserve the same inputs and outputs, then test changes that might affect quality.
Takeaway: Sort every fix by its burden of proof and work from low-risk changes to higher-risk ones. Do not change the model until the first category is done.
Two Facts That Create the Waste
1. Generation Is Serial
A model reads the prompt in one parallel pass but writes one token per forward pass. At interactive batch sizes, each pass is often bound by moving weights and cached state out of memory, rather than by arithmetic
One provider’s guide puts the effect into practical terms: halving output tokens roughly halves latency, while halving the prompt saves only a few percent unless the context is very large (OpenAI).
2. Models Are Stateless
Every agent step resends the instructions, tool definitions, conversation, and all tool results so far. All of that is billed as input.
Prices Follow the Physics
On the three major providers’ October 2026 list prices, an output token costs 5× an input token on every current flagship and mid-tier model. Re-reading a cached prefix costs a tenth to a fortieth of a fresh read (OpenAI, Anthropic, Google).
Takeaway: Latency follows what the model writes and how many times it writes in sequence. Cost follows what it has to read again. The fixes below target one or more of these sources of waste.
Path: The Steps Your User Waits Through
A turn’s latency is the sum of its dependent chain. Every link costs a full model call, whether or not it changes the answer.
Common culprits include:
- An orchestrator that only forwards messages.
- A planner that emits the same plan every time.
- A reviewer that approves nearly everything.
- Lookups run one after another even though they do not depend on each other.
Multi-agent designs can multiply the work.
One AI lab’s multi-agent research system used about 15× the tokens of a chat, or roughly 4× those of a single agent. The lab judged that cost worthwhile for broad research that could be split into parallel branches, and described tasks with heavy dependencies between agents as a poor fit (Anthropic, 2025).
Two cuts need no quality proof: remove relay steps and run independent calls in parallel. Parallel execution cut end-to-end latency by up to 3.7× on parallelizable benchmarks (Kim et al., ICML 2024).
Length also affects the slowest responses. A step that is slow 1% of the time makes about 10% of ten-step turns slow (Dean & Barroso, 2013).
Takeaway: Count the serial steps in your ten slowest turns. Every step you delete or parallelize can return latency without changing the model.
Prefix: Paying Again for What the Model Already Read
For a turn with n steps, an initial prompt of S tokens, and new tokens added per step, the input-token total is:
nS+n(n-1)2
The second term grows quadratically. With a 10,000-token prompt and 2,000 tokens added per step, a seven-step turn bills 112,000 input tokens, compared with 90,000 for six steps. One extra step costs 24% more input tokens.
If the bill grows faster than usage, repeated context is one place to investigate.
Exact Prefix Caching Reduces Repeat-Read Costs
Exact prefix caching is the lever here with a documented guarantee: two of the three major providers state that a cache hit does not change how output is generated (OpenAI, Anthropic).
In the six-step example, cached re-reads at a tenth of the price cut the input bill by 70%, before cache-write premiums. Caching lowers the cost curve, though it does not flatten it. In one long coding-agent session, cached re-reads alone accounted for 87% of the cost by the end (Zeyliger, 2026).
Why the Cache Can Break Silently
A team ended its system prompt with the current time to the second. That meant it was paying the cache-write premium without receiving the read discount (Basedash, 2026).
Other cache disruptions can include:
- Switching models, which starts a cold cache.
- Changing the reasoning setting, which does so with at least one provider.
- Cache expiry after a few idle minutes by default.
Takeaway: Put stable content first and changing content last. Track cache hit rate alongside latency. One agent company calls it the single most important metric for a production agent (Manus, 2025).
Prose: Words and Reasoning Nobody Uses
Every generated word is paid for twice: in wait time and at the output-token price. Restating the question, recapping what was just said, and adding unnecessary caveats all sit on the critical path. Hidden reasoning has the same cost, even when users do not see it.
On Princeton’s Holistic Agent Leaderboard, more reasoning produced equal or lower accuracy in 21 of 36 model, agent, and benchmark combinations (Kapoor et al., 2025).
But cutting reasoning indiscriminately can also hurt. One provider lowered the default reasoning level in its coding tool to reduce latency. Users said it felt less intelligent, and the change was reverted a month later (Anthropic, 2026).
Takeaway: Set answer length and reasoning effort by task and evidence. Do not apply a single global default in either direction.
Processing: The Model Doing a Database’s Job
A model is a slow, expensive calculator with no guarantee of getting every digit right. To total 400 rows, it must read every row, reason through the sum, and can still make a numerical mistake. A database query can return the exact answer in milliseconds.
Moving counts, joins, sorts, filters, and date calculations into code removes input tokens, output tokens, and a source of error in one change. The risk shifts to the tool: a wrong query will be wrong every time, so the tool needs its own tests.
Takeaway: If a database or ten lines of code can do the calculation, the model should not. Remove steps that merely relay information; keep steps that compute or make decisions that genuinely need a model.
Why the Waste Survives
Per-call numbers can look reasonable: a few seconds and a few cents each. Waste becomes visible only across a full turn—for example, twelve calls when five would do, each rereading the same instructions, two relaying a message, and one performing arithmetic.
Switching models may take one configuration change. Removing waste requires changing how the agent is built, so teams often reach first for the setting that is easiest to see.
To investigate, trace every call in a turn and record what it read, what it wrote, how long it waited, and whether it changed the answer. See Lyzr’s guide to monitoring and tracing in Agent Studio and the overview of AI agent tracing.
Takeaway: Optimize turns, not individual calls.
Then Trade, With Proof
A smaller model, less reasoning, summarized context, similarity caching, and routing can all change answers. Small evaluations may not detect those changes.
At 70% accuracy, the 95% margin of error is about ±13 percentage points on 50 questions, ±9 on 100, and ±3 on 1,000. Detecting a three-point regression takes on the order of 1,000 questions when comparing results one by one (Miller, 2024).
Even trimming words can affect quality. One provider’s 25-word cap between tool calls passed weeks of internal evaluations. A broader evaluation later found a 3% drop, and the change was reverted (Anthropic, 2026).
How to Evaluate a Change
- Run the old and new versions on the same questions.
- Decide the acceptable quality drop before testing.
- Use an evaluation set large enough to detect the regression you care about.
- Report the upper bound, not just the observed failure rate.
Zero failures in 50 runs is still consistent with a 6% failure rate (Hanley & Lippman-Hand, 1983).
Takeaway: Treat changes to models, reasoning, context, and routing as quality trade-offs until paired evaluations show otherwise.
The Numbers a Business Should Watch
Cost per Correct Answer
Cost per correct answer is total spend divided by the number of answers graded correct. Researchers call it cost-of-pass. It explains why a cheap model that needs retries or human fixes can cost more than an expensive model that succeeds on the first attempt (Erol et al., 2025).
P95 Time to a Correct Answer
Track the 95th-percentile time to a correct answer, not just the average. Slow responses at the tail are more likely to time out.
The human clock matters too: after about 10 seconds, users tend to turn to other things unless they can see progress (Nielsen, 1993). Streaming can shorten the felt wait, but it does not reduce the underlying work.
Removing waste is a one-time engineering cost whose savings recur with every request. A quality trade-off is a recurring risk that needs to be rechecked when the model, data, or traffic changes.
Takeaway: Put cost per correct answer and p95 time to answer on the dashboard. Fund waste removal before model swaps.
The Levers at a Glance
| Lever | Proof needed | What to watch | What it can quietly break |
| Exact prefix caching | None when the input is identical | Cache hit rate | Dynamic content placed early; cache expiry |
| Parallel independent calls | None when the calls are truly independent | P95 time per turn | Calls that were not actually independent |
| Move math and joins into code | Tests on the tool | Numeric correctness | A wrong query, every time |
| Drop relay steps | Replay real requests | Steps per turn | Steps that were doing real work |
| Trim words and reasoning | Paired check | Output tokens per answer | Completeness |
| Use a smaller model or lower reasoning default | Paired evaluation with a set margin | Cost per correct answer | Tool orchestration may require more steps |
| Summarize context or use routing | Paired evaluation | Cost per correct answer | Recall; cache warmth after each switch |
| Similarity caching | Audit cache hits | Wrong-answer rate on hits | Returning an answer to a different question |
Part of an agent’s latency and cost is the price of its structure, not its intelligence. Cut that first, then pay for quality on purpose.
This is the approach to production agents described here, and it extends Lyzr’s guide to cutting LLM costs without changing models.
Frequently Asked Questions
1. What are the four Ps of agent waste?
Path, prefix, prose, and processing: steps the user waits through that do not change the answer; the same prompt reread on every step; words and reasoning nobody uses; and work that a database or plain code should do.
2. Why do AI agent costs grow faster than usage?
Models do not retain memory between calls, so every step resends the prompt and history. Billed input per turn can grow quadratically with the number of steps. Exact prefix caching can reduce the cost of repeat reads to a tenth of the fresh-read price or less, depending on the provider.
3. Does prompt caching change the answers?
Exact prefix caching reuses work on identical input. OpenAI and Anthropic document that it does not change output generation. Similarity caching is different: it may return a stored answer to a similar, rather than identical, question.
4. Should I switch to a smaller model to cut latency?
Only after removing avoidable waste, and only with a paired evaluation. A cheaper model can require more steps. In one live test, tokens that were 5–6× cheaper reduced cost per user by only 52% and made responses 88% slower.
Related Reading from Lyzr
- How to Cut LLM Costs by 60% Without Changing Models
- Monitoring and Tracing in Lyzr Agent Studio: Usage, Performance, and Cost Visibility
- AI Agent Tracing: The Missing Debugging Layer for Production AI Agents
- Small Model Routing: How to Cut LLM Costs Without Losing Quality
- Auto Model Routing in Lyzr: Choosing the Right LLM for Every Request
- One Shot, Every Answer: Multi-Knowledge-Base RAG
Book A Demo: Click Here
Join our Slack: Click Here
Link to our GitHub: Click Here
