The 25% Latency Cut: Why Caching and Model Choice Decide If a Voice Skill Feels Human

There's a cliff in voice interaction. Below a certain response latency, a conversation feels alive — you barely notice the machine. Above it, every reply lands a beat late and the whole thing feels like a phone tree. The gap between those two experiences is measured in milliseconds, and it's the single hardest thing to get right in a voice agent.

That's why the least flashy number in OpenAI's July 6, 2026 Realtime API release is arguably the most important: gpt-realtime-2.1 cut p95 voice latency by at least 25%, driven by improved caching. Not a new voice, not a new capability — just faster. For a voice agent, faster is the capability. This piece breaks down where the time in a voice turn actually goes, the cached-vs-fresh audio economics that decide both your latency and your bill, and when the mini variant is the right call.

Key Takeaways

• gpt-realtime-2.1 cut p95 latency by ≥25% (OpenAI's figure), largely through better caching. Faster p95 means the slow turns — the ones users notice — got noticeably tighter.
• Cached audio input costs ~$0.30 per 1M tokens vs ~$10 per 1M fresh — roughly a 30x difference. Caching is a latency win and a cost win at once.
• The mini variant (gpt-realtime-2.1-mini) adds reasoning at the prior mini price — a cheaper, faster option for turns that don't need the full model.
• Latency is a budget you spend across the turn — capture, reasoning, tool calls, and synthesis each take a slice. Overspend anywhere and the turn feels slow.
• Model choice and caching are the two biggest levers a skill-builder controls. Reasoning effort, the mini variant, and cache hits decide whether your voice skill feels human.

Why p95 is the number that matters

Teams instinctively optimize average latency. For voice, that's the wrong target. What breaks the illusion isn't the average turn — it's the slow turn. The one where the model reasoned a little longer, or a cache missed, and the reply landed a full second late. Users don't average their experience; they remember the pauses.

That's why "p95 latency" is the right frame. p95 is the latency your slowest 5% of turns hit — the tail that produces the awkward silences. Cutting p95 by 25%+ means the turns that used to feel broken now feel merely slow, and the ones that felt slow now feel fine. Improving the tail does more for perceived quality than shaving the average ever could. OpenAI attributes the gain largely to improved caching, which is the thread that runs through this whole piece.

The latency budget of a single voice turn

Think of every voice turn as having a budget — the total time from when the user stops talking to when your agent starts replying. Blow past roughly the human-conversation threshold and the turn feels laggy. That budget gets spent across several stages, and each is a place you can overspend:

Read that table as a checklist of where a turn goes slow. Two of the five stages — audio input processing and reasoning — are directly under your control through caching and reasoning effort. A third — the tool call — is your own infrastructure. The remaining two got structurally faster with the full-duplex, speech-to-speech architecture, which removed the transcribe-then-synthesize round trips of the old pipeline.

The practical implication: you don't fix voice latency in one place. You budget it. If a turn needs a slow tool call, you can't also afford xhigh reasoning on that turn. If you want high reasoning, you'd better have a cache hit on the audio input. The good voice agents spend their budget deliberately.

The economics: cached vs fresh audio

Here's where caching stops being a latency footnote and becomes a business decision. The cost gap between cached and fresh audio input is enormous:

• Cached audio input: ~$0.30 per 1M tokens
• Fresh audio input: ~$10 per 1M tokens

That's roughly a 30x difference. And it's not a separate axis from latency — the same caching that makes gpt-realtime-2.1's p95 faster is what makes cached tokens cheaper. When your audio input hits the cache, you win twice: the turn is faster and it costs a fraction as much.

For a skill-builder, this reframes the whole cost model. A naive voice agent that treats every turn as fresh input will be both slow at the tail and expensive at scale. Say you're running a support line at, hypothetically, 500 conversations a day, each with a dozen turns — the difference between mostly-cached and mostly-fresh audio is the difference between a rounding error and a real line item. Caching isn't an optimization you get to later. For voice at any volume, it's the design.

This is exactly why the consumer-vs-API distinction matters: the consumer GPT-Live app hides these economics because a consumer doesn't pay per token. When you build on the Realtime API, the cache-hit rate and model choice are yours to manage — and they decide your bill.

The mini variant: when cheaper and faster is the right call

The other lever is model choice. gpt-realtime-2.1-mini adds reasoning at the prior mini price point — a smaller, cheaper, faster model that still reasons. For a lot of voice turns, it's the correct default.

The decision is about matching the model to the work:

• Reach for mini on high-volume, conversational, low-stakes turns: routing, status lookups, backchanneling, simple Q&A. These dominate most voice conversations, and they don't need the full model. Mini gives you lower latency and lower cost where it matters most — the common case.
• Reach for the full gpt-realtime-2.1 on the turns that genuinely need it: nuanced reasoning, multi-step planning, high-stakes decisions where a wrong answer is costly.

Combine this with reasoning effort and you have a two-dial system. Model choice (mini vs full) sets your baseline speed and cost; reasoning effort (minimal → xhigh) tunes within it. A well-built voice agent runs mini-at-low-effort for the bulk of turns and escalates to full-at-higher-effort only when a turn earns it. The function-calling deep-dive walks through wiring that per-turn escalation into the agent loop.

A latency-budget checklist for your voice skill

• Measure p95, not average. Instrument the tail. The slow turns are what users feel — optimize those, not the mean.
• Maximize cache hits. Cached audio input is ~30x cheaper and faster than fresh. Structure your sessions so repeated context stays cached. This is your single biggest lever.
• Default to mini and low reasoning effort. Escalate to the full model or higher effort only on turns that need it. Don't pay full-model latency on a status lookup.
• Budget the whole turn. A slow tool call plus high reasoning plus fresh audio will blow past the human threshold. You can afford one expensive stage per turn, not three.
• Keep tool calls off the critical path where you can. The tool-call slice of the budget is your own infrastructure — the model can't cache your slow database. Use the full-duplex acknowledgment trick to keep the line warm while it runs.
• Test on real devices and networks. Lab latency lies. The p95 that matters is the one your users hit on a phone on cellular.

The takeaway

A voice skill feels human or it doesn't, and the deciding factor is milliseconds, not model IQ. gpt-realtime-2.1's 25%+ p95 cut moved the tail closer to the human-conversation threshold — but the rest is on you. The two levers you control are caching (cached audio at ~$0.30/1M vs ~$10/1M fresh — a 30x latency-and-cost win) and model choice (mini vs full, low vs high reasoning effort). Spend your latency budget deliberately across the turn, maximize cache hits, and default to the cheaper, faster model until a turn earns more.

Get that right and the technology disappears — which is the whole point. The user isn't supposed to notice the caching, the model choice, or the reasoning dial. They're just supposed to feel like they're talking to something that's actually listening. That's the bar full duplex set, and latency is how you clear it. Wire your voice agent to the same tools your text agents use, and browse the skill catalog and loop recipes for the automation patterns to put behind it — then make it fast enough to feel alive.