Voice Orchestration BenchmarksVoice Orchestration Benchmarks
Everyone benchmarks the model. We benchmark the stack around it.
An independent benchmark isolating the orchestration layer: turn-taking, interruptions, latency and tool execution. One byte-identical agent, run across 6 platforms. The differences are the platforms, not the model.
Figures are comparisons under one fixed harness · see methodology.
Insights
Per-turn response latency
Cost per minute
How cost is measured⌄
Interruption handling
How Interruption Score is determined⌄
Appropriate end call
How Appropriate End Call is determined⌄
Repetition
How Repetition Score is determined⌄
Voice tone
How Voice Tone is determined⌄
Leaderboard
| Platform | pass^1 | pass^3 | Lat P50 | Lat P95 | Interrupt | End call | Repetition | Voice tone | $/min |
|---|---|---|---|---|---|---|---|---|---|
| Retell | 98.9% | 96.6% | 1.96s | 3.79s | 4.79 | 100.0% | 4.74 | 4.51 | $0.166 |
| Vapi | 98.3% | 94.9% | 2.34s | 2.95s | 4.63 | 98.9% | 4.66 | 4.20 | $0.060 |
| Pipecat | 95.5% | 89.8% | 3.15s | 5.42s | 4.90 | 98.3% | 4.61 | 4.45 | $0.099* |
| LiveKit | 96.0% | 88.1% | 2.96s | 4.43s | 4.81 | 100.0% | 4.62 | 3.17 | $0.088* |
| Synthflow | 90.4% | 81.4% | 3.16s | 5.08s | 4.66 | 95.0% | 4.69 | 4.47 | $0.195* |
| ElevenLabs | 85.3% | 67.8% | 1.73s | 3.19s | 4.77 | 96.7% | 4.61 | 4.60 | $0.107 |
pass^1 = one run meets the success rubric; pass^3= all three runs do; the gap is the consistency signal. Latency is Main-Agent per-turn (ms). Interruption, repetition and voice-tone are on a 0–5 scale (higher is better). $/min is the observed effective rate for measured providers (Retell’s published unit rate is $0.258). * cost estimate (site pricing or component rate-card).
pass^3 by evaluator category
The suite separates providers most in Workflow Complexity & Recovery (tool-failure, multi-step, recovery), a 42.8-point spread, and least on the happy path.
| Category | Retell | Vapi | Pipecat | LiveKit | Synthflow | ElevenLabs |
|---|---|---|---|---|---|---|
| Positive / Core Scheduling | 87.5% | 100.0% | 100.0% | 87.5% | 100.0% | 75.0% |
| Workflow Complexity & Recovery | 95.2% | 85.7% | 95.2% | 90.5% | 76.2% | 52.4% |
| Voice Robustness & Turn-Taking | 100.0% | 100.0% | 80.0% | 84.0% | 80.0% | 76.0% |
| Red Team, Safety & Privacy | 100.0% | 100.0% | 100.0% | 100.0% | 80.0% | 80.0% |
Methodology
The agent
Every platform runs the same agent: Ava, the scheduling assistant for a fictional healthcare practice, Cedarwood Family Clinic. Ava books, reschedules, cancels and looks up appointments over the phone, backed by four tools (lookup_patient, check_availability, book_appointment and cancel_appointment) against fixed mock data. Its prompt carries clinic guardrails: no medical advice, 911 for emergencies, minimal PII, and no disclosure of its own instructions.
What we measure
The orchestration layer: turn-taking, interruption and barge-in, latency, tool execution and call handling, isolated by holding the agent constant. Components (STT/TTS) and model reasoning are measured elsewhere; this isolates the orchestration between them.
What is held constant
The agent is held constant wherever each platform allows it: a byte-identical system prompt (SHA-verified) and first message, the same four tool definitions and mock data, and the same pinned LLM (gpt-4.1, temp 0), STT, TTS and voice. What varies is each platform’s own SDK integration and orchestration stack: the layer under comparison. Where a platform can’t accept a pin, it is documented in Stack parity, below.
Scenarios
59 evaluators across four categories (Positive / Core Scheduling, Workflow Complexity & Recovery, Voice Robustness & Turn-Taking, and Red Team, Safety & Privacy), each run 3 times. Every scenario targets a single expected outcome.
What defines a pass
Each pass is determined by a rubric configured for the scenario. For scenarios where a correct end state determines success, mock tool accuracy is used as a proxy for that state, alongside the order in which the tools are called. In red-teaming scenarios, success is determined by expected outcome, graded by an LLM judge. See examples.
Stack parity
The agent is aligned across all six platforms; the remaining differences are documented platform constraints, not prompt drift.
| Layer | Pinned to | Parity across platforms |
|---|---|---|
| LLM | gpt-4.1 · temp 0 | Pinned across all six. Managed legs use platform model pools; LiveKit via its Inference gateway; Pipecat brings its own Azure. |
| STT | Deepgram nova-3 | Pinned on Vapi, Synthflow, LiveKit and Pipecat. Retell exposes only a coarse mode; ElevenLabs forces Scribe. |
| TTS | ElevenLabs eleven_flash_v2 | Pinned where the platform exposes it. |
| Voice | ElevenLabs Brian | Exact voice ID on all except Retell, which uses a name-proxy. |
| Prompt & tools | Byte-identical | Same system prompt (SHA-verified), first message, four tool definitions and mock data on every platform. |
| Telephony | PSTN / SIP | All scored runs are over the phone; native vs. SIP-bridged path varies by platform. |
Testing conditions
- Every platform is scored by the same evaluators, applied identically, three runs each.
- Vapi, Retell and ElevenLabs are managed platforms that run the agent for us. LiveKit and Pipecat are frameworks: they provide the real-time orchestration under test (the session, SIP/PSTN media, VAD, end-of-turn and turn-taking) while we host the agent worker that wires in the same prompt, tool contract and core stack (gpt-4.1, Deepgram nova-3, ElevenLabs Flash) as the managed providers.
- For LiveKit the agent worker ran locally, bridged into the calls over PSTN/SIP; Pipecat ran on Pipecat Cloud (Daily-backed). A portion of their latency therefore reflects our worker's environment rather than the framework itself.
- Each platform's built-in end-call was disabled; the agent was instructed to end the call from the system prompt, and the testing agent does not end calls. This shapes the End Call and Repetition results.
- The emphasized orchestration metrics (latency, interruption, end-call, repetition) are largely determined by Voice Activity Detection, End-of-Turn detection and other orchestration conditions, rather than the model.
- Telephony path differs by platform (native vs. SIP-bridged); STT and TTS are pinned with per-platform exceptions (see Stack parity).
- Only cascade platforms are included in this version; near-duplex / realtime is a separate track.
How to read this
Every figure is a comparison under one fixed test harness, not a production success rate.
Use it to narrow the field to a few candidates, then validate them on your own traffic.
High-volume support weights cost and reliability; premium CX weights latency and turn-taking; regulated weights accuracy.
See the runs for yourself
Full run collections, with transcripts.