CekuraBench

S2S Model Benchmarks

AI voice models answered real phone calls. This is who got them right.

Last updated Methodology by Dileep Chagam

View leaderboardView the agent

Reliable or fast

Best picks
Reliability, pass³ (%)
Median response time (s)
Higher and further left is better. Dotted line: no other model is both more reliable and faster.
Start here

What we tested

We gave every model the same calls: around 60 for a clinic, 23 for a Medicare line.

Healthcare receptionist

59 scenarios · short form

Patients call to book, move or cancel an appointment.

Medicare intake

23 scenarios · long form

Callers ring about Medicare, the US health insurance for older people. The agent takes their consent and details, then passes them to the right team.

Every call is checked onA quick replyNo long silencesDetails saved rightThe right outcomeThe same result on every run
Kinds of call

Results by call

Booking a doctor's appointment is easy, and most models get it right almost every time. A Medicare call asks for far more, and only a few get through half of those calls.

Doctor's appointmentMedicare call
  • GPT Realtime 2.186% → 61%
  • Azure Realtime81% → 43%
  • GPT-Live 1 + Sol low92% → 35%
  • Gemini 3.8 Live Thinking59% → 35%
  • Deepslate Opal73% → 17%
  • Gemini 3.8 Live53% → 13%
  • Gemini 3.1 Flash Live83% → 9%
  • Grok Think Fast 2.092% → 4%
  • Nova 2 Sonic68% → 4%
  • Realtime 2.1 Mini68% → 4%
  • Phonic v193% → 4%
  • Cascade (reference)93% → 57%
0%50%100%
Scenarios that passed on all 3 runs. Cascade: speech-to-text, then an LLM, then text-to-speech, shown for reference.
Leaderboard

How the models stack up

The same caller rings back with the same request, again and again. Models are ranked on how often they get it right every single time.

  • For reference, not ranked: speech-to-text, then an LLM, then text-to-speech
  • Tap a model for the full results.
How to read this table
Reliability (pass³) · Passed all 3 runs
Scenarios the agent passed on every one of its 3 runs.
Success rate · Passed every check
Calls where the agent said the right things and saved the right details.
Data accuracy · Saved exactly right
Calls where every detail the agent saved matched. Calls that save nothing are left out.
Stalled calls · Went silent
Calls where the agent went quiet for 10 seconds or more after the caller spoke. Lower is better.
Response time · Typical reply time
Median time from the caller finishing to the agent starting to reply. Lower is better.
Cost / min · Published price
One minute of call at the vendor's published rates. Blank where no rate is published.
Rankings

No model wins on everything

A caller wants the right answer and a quick reply. You want a low bill. No single model gives all three, so pick the measure that matters most to you.

Scenarios that passed on all 3 runs, pass³. Higher is better.

  • 1GPT Realtime 2.179.3%
  • 2GPT-Live 1 + Sol low75.6%
  • 3Azure Realtime70.7%
  • 4Phonic v168.3%
  • 5Grok Think Fast 2.067.1%
  • 6Gemini 3.1 Flash Live62.2%
  • 7Deepslate Opal57.3%
  • 8Gemini 3.8 Live Thinking52.4%
  • 9Nova 2 Sonic50.0%
  • 10Realtime 2.1 Mini50.0%
  • 11Gemini 3.8 Live41.5%
  • ·Cascade, GPT-4.182.9%
Differences

How each model fails

Two models can share a score and fail for different reasons. Some slip on the details, some go quiet, and some pass a call one run and fail it the next.

  • Plain booking & lookup1598%
  • Several requests in one call7100%
  • Caller corrects or changes mind883%
  • Background noise776%
  • Interruptions & barge-in5100%
  • Silence, cough & packet loss888%
  • Accent & slow speech2100%
  • Tool or system errors4100%
  • Safety & scope694%
  • Medicare compliance & routing2078%

Success rate for each kind of scenario. Darker blue is better. The small number is how many scenarios of that kind.

Models

What we heard on the calls

The numbers say who wins. The calls say why. We went through every call for each model, and this is what stood out.

  • GPT Realtime 2.1

    OpenAI · Reasoning high · service turn detection

    The safest pick when every call has to go right.

    Reliability (pass³)
    79%
    Response time
    1.94s
    Cost / min
    $0.085

    Good at

    • Almost never fails, and rarely twice in a row.
    • Best realtime model on Medicare calls.

    Watch out for

    • Noisy phone lines.
    • Now and then saves a phone number wrong, or forgets something the caller said earlier.

    Most often saved wrong: route_idphonedestination

    See the calls:All Short form Long form

  • GPT-Live 1 + GPT-6 Sol (low thinking)

    OpenAI · Delegates reasoning and tools to Sol · service turn detection

    Talks well, but saves the record wrong.

    Reliability (pass³)
    76%
    Response time
    2.02s
    Cost / min
    $0.055

    Good at

    • Near perfect on bookings, run after run.
    • The conversation always reaches the right outcome.

    Watch out for

    • On Medicare calls it sends the caller to the wrong team or skips a step.
    • Stalls the most, says goodbye without hanging up, and has the lowest voice score.

    Most often saved wrong: destinationroute_statusstate

    See the calls:All Short form Long form

  • Azure Realtime

    Microsoft · No reasoning setting · service turn detection

    Quick and steady, but drops details from Medicare handoff notes.

    Reliability (pass³)
    71%
    Response time
    1.66s
    Cost / min
    $0.080

    Good at

    • Handles a caller correcting themselves better than any other model.
    • Passed every interrupted call and every call where a tool failed, and is among the quickest to reply.

    Watch out for

    • On Medicare calls it often leaves out an ID it was given, most often in the handoff note.
    • Background noise and calls with several requests trip it up more than most, and it now and then goes quiet on a booking call.

    Most often saved wrong: route_idlead_idroute_status

    See the calls:All Short form Long form

  • Phonic v1

    Phonic · Intelligence high · service turn detection

    The fastest talker, best on short calls.

    Reliability (pass³)
    68%
    Response time
    1.60s
    Cost / min
    $0.140

    Good at

    • Quickest to reply, and never left a caller waiting.
    • As good as the best on bookings, run after run.

    Watch out for

    • Misses the strict Medicare check, though the record is usually close.
    • Makes extra tool calls more often than most.

    See the calls:All Short form Long form

  • Grok Voice Think Fast 2.0

    xAI · Reasoning high · service turn detection

    Quick and brief, until the call gets long.

    Reliability (pass³)
    67%
    Response time
    1.62s
    Cost / min
    $0.080

    Good at

    • Shortest replies of the field, and steady on bookings.

    Watch out for

    • On Medicare calls it drops or garbles details from earlier in the call.
    • Every stall happened on a Medicare call.

    Most often saved wrong: lead_idroute_idconsent_id

    See the calls:All Short form Long form

  • Gemini 3.1 Flash Live Preview

    Google · Thinking level high · shared local turn detection

    Steady on bookings, slow everywhere else.

    Reliability (pass³)
    62%
    Response time
    2.88s
    Cost / min
    $0.070

    Good at

    • Reliable on bookings.

    Watch out for

    • Slowest to reply, and after several tool calls replies lag or go missing.
    • Hands Medicare callers over with the wrong details.

    Most often saved wrong: lead_iddestinationroute_id

    See the calls:All Short form Long form

  • Deepslate Opal

    Deepslate · No reasoning setting · service turn detection

    Keeps up with interruptions, but skips steps on long calls.

    Reliability (pass³)
    57%
    Response time
    2.27s
    Cost / min
    $0.112

    Good at

    • Every interrupted call and every call with several requests passed.

    Watch out for

    • Often skips a step on Medicare calls, most often routing the caller or writing the handoff note.
    • The weakest on silences, coughs and packet loss, and goes quiet more often than most on Medicare calls.

    Most often saved wrong: phoneroute_idcallback_phone

    See the calls:All Short form Long form

  • Gemini 3.8 Live Extended Thinking

    Google · Thinking level high · service turn detection

    Strong on long calls, shaky on simple ones.

    Reliability (pass³)
    52%
    Response time
    2.07s
    Cost / min
    —

    Good at

    • One of the strongest realtime models on Medicare calls.

    Watch out for

    • Weakest on plain bookings: every tool call right, outcome still missed.
    • Unpredictable run to run, and can go silent under background conversation.

    See the calls:All Short form Long form

  • Amazon Nova 2 Sonic

    Amazon · Endpointing medium · shared local turn detection

    The best voice, but the details slip.

    Reliability (pass³)
    50%
    Response time
    2.18s
    Cost / min
    —

    Good at

    • Best voice score of the field.

    Watch out for

    • Misses caller names and phone numbers, and rarely passes Medicare calls.
    • Repeats tool calls more than any other model, with the longest replies.

    Most often saved wrong: caller_nameconsent_idcallback_phone

    See the calls:All Short form Long form

  • GPT Realtime 2.1 Mini

    OpenAI · Reasoning high · service turn detection

    The budget pick for simple calls.

    Reliability (pass³)
    50%
    Response time
    2.08s
    Cost / min
    $0.020

    Good at

    • Gets most bookings right, at a fraction of the cost.

    Watch out for

    • Forgets earlier details: not one Medicare record was exact.
    • Mishears phone numbers, and rarely recovers from a tool error.

    Most often saved wrong: route_idzip_codeconsent_id

    See the calls:All Short form Long form

  • Gemini 3.8 Live

    Google · No thinking level · service turn detection

    Faster without extended thinking, but less careful.

    Reliability (pass³)
    41%
    Response time
    1.91s
    Cost / min
    —

    Good at

    • Replies sooner, and more bookings reach the outcome.

    Watch out for

    • Medicare records are wrong far more often.
    • Weakest when a caller corrects themselves, and the least predictable.

    See the calls:All Short form Long form

  • Flux → GPT-4.1 → ElevenLabs Flash

    Cascade baseline, not ranked

    The steady reference: rarely fails, rarely stalls.

    Reliability (pass³)
    83%
    Response time
    2.17s
    Cost / min
    —

    Good at

    • Most scenarios pass every run, and none fails every run.
    • Almost never leaves the caller waiting.

    Watch out for

    • Not the quickest to reply.
    • Now and then forgets something the caller said earlier in a Medicare call.

    Most often saved wrong: route_idintake_intentservice_issue_type

    See the calls:All Short form Long form

Compare

Put two models head to head

Pick any two models and see them side by side. Every pair has its own page with the full numbers.

Compare
GPT Realtime 2.1OpenAI
Flux → GPT-4.1 → ElevenLabs FlashBaseline, not ranked
Reliability
79.3%1st
82.9%
3.7 points higher
Data accuracy
92.0%1st
94.1%
2.1 points higher
Response time
1.94s5th
0.23s sooner
2.17s
Cost / min
$0.0856th
No verified rate
All suites. The small figure is the place among ranked models.All 66 comparisons

The most reliable model against the next six

Every comparison

Methodology

How we tested

Every model ran the same agent on the same calls and was scored the same way.

Only the model changes

One open-source Pipecat agent: the realtime model hears the caller, calls the tools and speaks. Prompt, tools and connection are the same for every model. A cascade (Flux, then GPT-4.1, then ElevenLabs Flash) runs the same agent for reference, unranked.

Live calls, three runs each

A simulated caller works through all 82 scenarios on live calls, three times over. Every call counts, including ones that stalled or ended early.

Words and data both count

A call passes only when the agent reached the expected outcome and every field it saved matched, checked on the Cekura platform.

Every call is public

Each row links to the Cekura report of its calls, with transcripts, tool calls and scores, so any figure here can be checked.

The two kinds of call

  • Appointments (short form)

    59 scenarios · 6 fields · 4 tools

    A patient calling a clinic. Find the patient, check openings, then book, move or cancel a visit.

    • lookup_patient
    • check_availability
    • book_appointment
    • cancel_appointment

  • Medicare (long form)

    23 scenarios · 42 fields · 4 tools

    Someone calling a Medicare insurance line. Record consent first, qualify the caller, route them to the right team and write a handoff note.

    • record_medicare_permissions
    • save_medicare_qualification
    • route_medicare_call
    • create_handoff_summary
Reference agent Suites and tools Test cases
Common questions

Questions

What is the best speech-to-speech model in 2026?

GPT Realtime 2.1 is the most reliable, passing 79.3% of scenarios on every run. Phonic v1 replies fastest, at 1.60s median. GPT Realtime 2.1 Mini costs least per call, at $0.04. The right pick depends on which of the three your voice agent needs most, and all three are measured on the same live calls.

What does the speech-to-speech benchmark measure?

It measures complete realtime voice models on live agent calls, including reliability, task success, saved-data accuracy, stalled calls, response time, and cost.

How is speech-to-speech reliability ranked?

The leaderboard uses pass³, the share of scenarios where all three retained calls passed every required check. A scenario must work repeatedly to count.

Are the models tested under the same conditions?

Yes. Every model runs the same phone agent, prompt, tools, connection, caller scenarios, and scoring rules. Only the realtime model changes.

How does a native speech-to-speech model compare with a cascaded voice agent?

Native speech-to-speech models handle audio directly. A cascade connects speech-to-text, an LLM, and text-to-speech. Cekura runs both through the same agent scenarios so reliability and response-time differences remain comparable.

Can I compare two realtime voice models directly?

Yes. The comparison tool provides head-to-head reliability, accuracy, latency, failure, and cost results for any two tested models.