On the phone, silence is a verdict. In face-to-face conversation, a pause of a few hundred milliseconds reads as natural rhythm. Stretch it past a second and something shifts: The caller starts wondering whether they were heard, whether the system is broken, whether they should just say "representative" until a human picks up. Text chat forgives slow responses; a typing indicator buys you time. Voice does not.
That is why the move from scripted phone menus to conversational AI agents is as much a real-time engineering problem as an AI problem. The intelligence of the underlying model matters, but whether a customer trusts a voice agent is decided in the gaps between turns. And those gaps are won or lost in milliseconds.
The latency budget: Where every millisecond goes
Every conversational turn on a voice channel spends from the same budget. The caller’s speech has to be transcribed. The system has to understand the request, decide what to do about it, and often fetch something from a back-end system: a balance, a policy, an order status. Then the response has to be converted back into natural-sounding speech. Each stage is fast on its own; stacked end to end, they can easily add up to several seconds of dead air.
The single most effective technique is not making any one stage faster. It is refusing to run them one after another. When the response is streamed word by word, speech synthesis can begin while the rest of the answer is still being generated. The caller hears the first word well under a second after they stop talking, even if the complete answer takes a few seconds to finish. The total work is the same; the perceived wait collapses.
Voice teams obsess over time-to-first-word the way web teams obsess over time-to-first-byte, because it is the number the customer actually feels.
Why one do-everything agent can’t keep up
The intuitive way to build a voice agent is to give a single AI model one big set of instructions and access to everything: the knowledge base, the customer database, the case system, the transfer button. It works in a demo. It breaks down in production, for three reasons.
First, latency profiles collide. A knowledge-base search and an account lookup have fundamentally different response characteristics; one may take many times longer than the other. Blend them into one agent and every request inherits the worst-case behavior, making the experience unpredictable in exactly the channel where predictability matters most.
Second, choice degrades accuracy. As the number of available actions grows, a language model becomes measurably more likely to pick the wrong one, especially deep into a long conversation. In text chat a wrong turn is a minor annoyance. In voice, it is 10 seconds of confusion followed by a frustrated caller starting over. And accuracy that sounds impressive in isolation becomes dangerous at scale: An agent that is 85% accurate is failing 15% of the time. And when probabilistic decisions are chained together, those failure rates compound.
Third, and most important: security. A question about store hours should not require identity verification. A question about an account balance absolutely should. An agent with standing access to everything violates the principle of least privilege, and in regulated industries, “the model decided it was okay” is not an acceptable compliance posture.
Specialists, working in parallel
The architecture that holds up under real calls looks less like one brilliant generalist and more like a well-run team. Behind a single conversation sit multiple specialized agents: one tuned for general knowledge, one for account data, one for identity verification and escalation, one for completing service requests end to end. Each has its own data access, its own security posture, and its own latency profile.
What makes the team feel like one agent is the coordination layer sitting in front of it. It maintains a single conversational context, routes each request to the right specialist, and dispatches work in parallel wherever it can. When a deeper lookup takes longer, the conversation doesn’t stall. The caller gets a natural acknowledgment while the work continues in the background – and faster requests aren’t held hostage by slower ones.
The payoff shows up in the messiest, most human scenarios: a caller who asks about an outage, then their balance, then store hours, then circles back to a billing question, all in one call. Specialization keeps each answer fast; shared context keeps the conversation whole.
Air traffic control for your AI agents
This is the thinking behind Pega Customer Service™ and Pega Agentic Self-Service™, which turn enterprise workflows into conversational agents across voice and digital channels. Specialized agents draw on Pega GenAI Knowledge Buddy™ for grounded answers to general questions, and on the customer data, cases, and workflows in the Pega Platform™ for secure account service. Coordinating all of it is a deterministic orchestration layer: rules and gates, not a model, decide how each request is handled.
It works like air traffic control. Pilots are skilled, but they don't decide who lands on which runway; air traffic control does. The language model flies the conversation: understanding the caller, choosing the next action, phrasing a natural response. The orchestration layer directs the traffic: routing each request to the right specialist, enforcing identity verification before any account data moves, and controlling which actions even exist at each moment. During verification, account actions are simply not on the runway. The model cannot pick what it cannot see, and it never grades its own security exam.
The split is also where the milliseconds come back in. Every extra model call in a chain costs half a second to a full second, so the orchestration layer routes without adding hops: one model stays in the loop for language while deterministic rules handle routing, retries, and state at machine speed. And where most voice AI stops at a fluent answer, these agents run on the same platform that executes the work, so a request becomes a completed case in the same call. The gap between "the caller asked" and "the work got done" shrinks from a follow-up ticket to the length of the conversation itself, with every routing and security decision captured in auditable rules.
This is the same principle that runs through Pega’s broader architectural vision: AI should be predictable, in its outcomes and in its costs. Creative, exploratory reasoning belongs at design time; at runtime, narrow and targeted AI calls wrapped in deterministic workflow keep behavior reliable, keep consumption predictable, and keep the enterprise in control of the processes it cannot afford to get wrong.
Built for how people actually talk
There is one more milliseconds problem that text channels never face: People interrupt – constantly. Real callers talk over the agent many times in a single call, whether to correct it, redirect it, or jump to a new topic mid-sentence.
Handling that gracefully is its own engineering discipline. The agent has to stop speaking the instant the caller starts, remember exactly how much of the previous answer was actually heard, and treat the new utterance as the conversation, not as an error to recover from. And when an interruption arrives just as a back-end result comes in, that result can’t be thrown away. It is held and woven back in once the caller’s new request is addressed. None of this is glamorous. All of it is the difference between a system that feels human and one that feels like a phone tree with better grammar.
What it adds up to
Put the pieces together (streaming responses, specialized agents running in parallel, deterministic coordination, and interruption handling built for real speech) and the experience changes qualitatively. Callers get instant answers to general questions without signing in, secure account service gated behind verification, multi-topic conversations that never lose the thread, and a warm handoff to a human agent, with context, whenever they want one.
Customers will never see the architecture. They will only feel its absence or its presence: as dead air, or as a conversation. That is why milliseconds matter. They are where trust in voice AI is actually built.
Explore what’s new in Pega Customer Service
These capabilities reflect the direction of Pega Customer Service™ and Pega’s agentic self-service portfolio in the Infinity ’26 generation. To go deeper: