CekuraVoice Orchestration Benchmarks

Voice Orchestration Benchmarks

Everyone benchmarks the model. We benchmark the stack around it.

An independent benchmark isolating the orchestration layer: turn-taking, interruptions, latency and tool execution. One byte-identical agent, run across 6 platforms. The differences are the platforms, not the model. Note: these are platform defaults, expect higher peaks once you tune to your use case.

Performance vs latency
75%80%85%90%95%100%1.60s1.85s2.10s2.35s2.60s2.85s3.10sP50 turn latency · faster ←pass^3 (%) · better ↑↖ upper-left = bestRetell: 96.6% pass^3 at 1.96sRetellVapi: 94.9% pass^3 at 2.34sVapiPipecat: 89.8% pass^3 at 3.15sPipecatLiveKit: 84.7% pass^3 at 2.46sLiveKitSynthflow: 81.4% pass^3 at 3.16sSynthflowElevenLabs: 76.3% pass^3 at 1.73sElevenLabs

Figures are comparisons under one fixed harness · see methodology.

Highest reliability
Retell96.6%
pass^3
Fastest responses
ElevenLabs1.73s
median turn latency
Best interruption handling
Pipecat4.90
out of 5 · interruption score
Most consistent latency
Vapi1.66s-2.95s
P5-P95 turn latency
By metric

Insights

1s2s3s4s5sElevenLabs: p50 1730ms · 1125 turns (p5 1090 · p25 1290 · p75 2260 · p95 3194)ElevenLabs1.73sRetell: p50 1960ms · 1143 turns (p5 1230 · p25 1525 · p75 2480 · p95 3789)Retell1.96sVapi: p50 2340ms · 1205 turns (p5 1660 · p25 2070 · p75 2550 · p95 2950)Vapi2.34sLiveKit: p50 2460ms · 1053 turns (p5 1910 · p25 2130 · p75 3040 · p95 3870)LiveKit2.46sPipecat: p50 3150ms · 1218 turns (p5 1860 · p25 2242 · p75 3660 · p95 5419)Pipecat3.15sSynthflow: p50 3160ms · 1569 turns (p5 1960 · p25 2470 · p75 3910 · p95 5080)Synthflow3.16s
Per-turn latency · line = median (P50), box P25–P75, whiskers P5–P95 · ~1,100–1,570 turns/platform
Latency

Per-turn response latency

With the model, prompt and TTS fixed, per-turn latency reflects the platform’s turn-taking: endpointing, VAD, buffering and network path. ElevenLabs is fastest (median 1.73 s per turn) but carries a long tail (P95 3.19 s, max 10.4 s). Vapi is the most consistent (P5–P95 1.66–2.95 s). LiveKit now sits near Vapi at 2.46 s median; Pipecat and Synthflow remain slower, around 3.15 s.
Average interruption score (0–5) · interruption scenarios only
Interruption

Interruption handling

Turn-taking is set by the platform, not the prompt: Voice Activity Detection and end-of-turn detection decide when the agent yields. Across the interruption scenarios (barge-in, cough, mid-turn silence), Pipecat scores highest at 4.90, with LiveKit close at 4.89. Vapi is lowest at 4.63, and the field is tight overall.
How Interruption Score is determined⌄
Cekura uses voice-activity detection on the stereo recording to find every point where the Main Agent starts speaking while the caller is still talking, then scores 5 × (1 − interruptions ÷ turns), clamped to 0–5, so 5/5 means it never talked over the caller. We report it on the interruption scenarios (those that inject barge-in, coughs, or mid-turn silence) because a whole-suite average is diluted by calls with nothing to interrupt.
Appropriate end-call rate (% of calls)
Call handling

Appropriate end call

Whether a call ends cleanly is handled by the platform, not the model. Retell ended every call correctly; Vapi and LiveKit were next at 99.4%. Synthflow was lowest at 94.9%, usually failing to close after an emergency redirect or a late second request. Native end-call was disabled; ending was specified in the prompt.
How Appropriate End Call is determined⌄
A per-call pass/fail from an LLM judge that reads the transcript and the reason the call ended, and decides whether the Main Agent wrapped up properly: resolving the request before hanging up, and not looping past a clean goodbye. Native end-call was disabled for the benchmark, so this reflects how each platform executes a prompt-driven hangup rather than a built-in one.
Average repetition score (0–5)
Conversation

Repetition

Loops surface when a platform mishandles silence or overlapping speech: a turn-taking failure, not a prompt one. Scores run 4.61 to 4.74, Retell highest. The metric sits near its ceiling, so read the low tail, not the mean.
How Repetition Score is determined⌄
An LLM flags each time the Main Agent re-confirms the same, unchanged information twice or more; the score is 5 × (1 − repetitions ÷ turns), clamped to 0–5, so 5/5 means no unnecessary repetition. Most calls sit near the ceiling; the signal is the low tail, where a platform loops instead of recovering with one clean re-ask or a clean close.
Average voice-tone score (0–5)
Experience

Voice tone

Audio quality is set by the platform’s media pipeline (encoding, jitter, packet handling), not the model. ElevenLabs is highest at 4.60/5; Retell, Synthflow and Pipecat cluster from 4.45 to 4.51, while Vapi and the newer LiveKit run land at 4.20 and 4.04.
How Voice Tone is determined⌄
A dedicated ML model analyzes the Main Agent's audio channel, not the transcript, scoring clarity (how clear and understandable the voice is) and jitter (timing variation that degrades audio). It catches calls that complete the task but still sound noisy, clipped, or degraded.
Full data

Leaderboard

Platformpass^1pass^3Lat P50Lat P95InterruptEnd callRepetitionVoice tone
Retell98.9%96.6%1.96s3.79s4.79100.0%4.744.51
Vapi98.3%94.9%2.34s2.95s4.6399.4%4.664.20
Pipecat95.5%89.8%3.15s5.42s4.9098.3%4.614.45
LiveKit94.9%84.7%2.46s3.87s4.8999.4%4.644.04
Synthflow90.4%81.4%3.16s5.08s4.6694.9%4.694.47
ElevenLabs88.1%76.3%1.73s3.19s4.7796.6%4.614.60

pass^1 = one run meets the success rubric; pass^3 = all three runs do; the gap is the consistency signal. Latency is Main-Agent per-turn (ms). Interruption, repetition and voice-tone are on a 0–5 scale (higher is better).

pass^3 by evaluator category

The suite separates providers most in Workflow Complexity & Recovery (tool-failure, multi-step, recovery), a 42.8-point spread, and least on the happy path.

CategoryRetellVapiPipecatLiveKitSynthflowElevenLabs
Positive / Core Scheduling87.5%100.0%100.0%75.0%100.0%87.5%
Workflow Complexity & Recovery95.2%85.7%95.2%76.2%76.2%61.9%
Voice Robustness & Turn-Taking100.0%100.0%80.0%92.0%80.0%80.0%
Red Team, Safety & Privacy100.0%100.0%100.0%100.0%80.0%100.0%

Methodology

One agent, deployed unchanged on six platforms, called over the phone by the same scenarios and scored by the same evaluators — so what varies between platforms is the layer the numbers measure.

  1. 01

    The agent

    Every platform runs the same agent: Ava, the scheduling assistant for a fictional healthcare practice, Cedarwood Family Clinic. Ava books, reschedules, cancels and looks up appointments over the phone, backed by four tools (lookup_patient, check_availability, book_appointment and cancel_appointment) against fixed mock data. Its prompt carries clinic guardrails: no medical advice, 911 for emergencies, minimal PII, and no disclosure of its own instructions.

  2. 02

    What we measure

    The orchestration layer: turn-taking, interruption and barge-in, latency, tool execution and call handling, isolated by holding the agent constant. Components (STT/TTS) and model reasoning are measured elsewhere; this isolates the orchestration between them.

  3. 03

    What is held constant

    The agent is held constant wherever each platform allows it: a byte-identical system prompt (SHA-verified) and first message, the same four tool definitions and mock data, and the same pinned LLM (gpt-4.1, temp 0), STT, TTS and voice. What varies is each platform’s own SDK integration and orchestration stack: the layer under comparison. Where a platform can’t accept a pin, it is documented in Stack parity, below.

  4. 04

    Scenarios

    59 evaluators across four categories (Positive / Core Scheduling, Workflow Complexity & Recovery, Voice Robustness & Turn-Taking, and Red Team, Safety & Privacy), each run 3 times. Every scenario targets a single expected outcome.

  5. 05

    What defines a pass

    Each pass is determined by a rubric configured for the scenario. For scenarios where a correct end state determines success, mock tool accuracy is used as a proxy for that state, alongside the order in which the tools are called. In red-teaming scenarios, success is determined by expected outcome, graded by an LLM judge. Human-reviewed successes count as effective passes for their configured rubric gates; raw metric scores are retained separately. See examples.

Stack parity

The agent is aligned across all six platforms; the remaining differences are documented platform constraints, not prompt drift.

LayerPinned toParity across platforms
LLMgpt-4.1 · temp 0Pinned across all six. Managed legs use platform model pools; LiveKit via its Inference gateway; Pipecat brings its own Azure.
STTDeepgram nova-3Pinned on Vapi, Synthflow, LiveKit and Pipecat. Retell exposes only a coarse mode; ElevenLabs forces Scribe.
TTSElevenLabs eleven_flash_v2Pinned where the platform exposes it.
VoiceElevenLabs BrianExact voice ID on all except Retell, which uses a name-proxy.
Prompt & toolsByte-identicalSame system prompt (SHA-verified), first message, four tool definitions and mock data on every platform.
TelephonyPSTN / SIPAll scored runs are over the phone; native vs. SIP-bridged path varies by platform.

Testing conditions

  • Same evaluators everywhere

    Every platform is scored by the same evaluators, applied identically, three runs each.

  • Managed vs. framework

    Vapi, Retell and ElevenLabs are managed platforms that run the agent for us. LiveKit and Pipecat are frameworks: they provide the real-time orchestration under test (the session, SIP/PSTN media, VAD, end-of-turn and turn-taking) while we host the agent worker that wires in the same prompt, tool contract and core stack (gpt-4.1, Deepgram nova-3, ElevenLabs Flash) as the managed providers.

  • Where the workers ran

    For LiveKit the agent was deployed on LiveKit Cloud, with LiveKit infrastructure serving all three models (STT, LLM and TTS), bridged into the calls over PSTN/SIP; Pipecat ran on Pipecat Cloud (Daily-backed).

  • End-call handling

    Each platform's built-in end-call was disabled; the agent was instructed to end the call from the system prompt, and the testing agent does not end calls. This shapes the End Call and Repetition results.

  • Orchestration, not the model

    The emphasized metrics (latency, interruption, end-call, repetition) are largely determined by Voice Activity Detection, End-of-Turn detection and other orchestration conditions, rather than the model.

  • Telephony and pins

    Telephony path differs by platform (native vs. SIP-bridged); STT and TTS are pinned with per-platform exceptions (see Stack parity).

  • Cascade only

    Only cascade platforms are included in this version; near-duplex / realtime is a separate track.

Note · Framework baselines

LiveKit and Pipecat were built from the providers’ standard, publicly available agent templates. Their results are a baseline for the default implementation path, not the theoretical ceiling of what each framework can achieve. With deeper tuning, custom orchestration logic and production-specific optimization, teams can likely push several of these numbers higher.

How to read this

01Relative, not absolute

Every figure is a comparison under one fixed test harness, not a production success rate.

02A shortlisting instrument

Use it to narrow the field to a few candidates, then validate them on your own traffic.

03Weight by use case

High-volume support weights reliability and consistency; premium CX weights latency and turn-taking; regulated workflows weight accuracy.

See the runs for yourself

Full run collections, with transcripts.

RetellView runs →VapiView runs →PipecatView runs →LiveKitView runs →SynthflowView runs →ElevenLabsView runs →