How we test speech-to-speech models on real human calls

Every model took the same 70 recorded human callers on live phone calls, three times each, ran the same agent and was graded the same way. This page sets out each step.

Last updated Nov 5, 2026Methodology by Prashansa Dadu

The Callers

Real people's recorded voices, played live over the phone.

Each caller is a scenario built around recordings of a real person: 35 banking calls and 35 healthcare calls. The recordings carry what real phone calls carry: hesitation, mid-sentence pauses, interruptions, accents and changes of mind.

Language: 55 callers speak English throughout, 7 switch to Spanish partway through and 8 switch to Hindi.
Hard moments are scripted into the call: background voices, hold music, a bad line that drops words, a child shrieking over a digit. The noise is mixed into the caller's audio, so it is exactly what the agent hears.
Every model hears the same recordings. As in a real call, what the caller says next depends on what the agent just said.

The Agent

Only the speech-to-speech model changes.

Each model runs as the whole voice agent: it hears the caller, decides what to do and speaks the reply. Everything around it is held fixed.

One agent build for every model: the same prompt, the same tools and the same phone connection.
The tools return the same account and clinic data for every model in a given scenario. Some are slow or fail on purpose, to test how the model handles a slow or failed lookup.
Turn detection is each provider's own, as it ships.
A cascade (speech-to-text, a text model and text-to-speech) ran the same calls as a reference point, with English speech-to-text for callers who speak English throughout and multilingual speech-to-text for callers who switch language. It is rated on the same scale but not ranked.

The Runs

Every scenario, three times, for every model.

12 ranked models each took all 70 scenarios three times: 2,730 live calls in all.
Each call was checked for health: that it connected and ran as scripted. A call that failed for reasons outside the model was run again, so every model is scored on a full set.
Results pool all three runs. A model has to handle the same caller well each time to look reliable.

The Skills

Each call is graded on 25 skills in 4 families: Hearing, Turn Taking, Task Handling and Speaking.

A skill is graded only on calls whose scenario tests it, at the moment it is tested. A caller who never interrupts says nothing about barge-in. Each grade is a pass or a fail. A few skills count mistakes across the whole call instead (for example, asking again for something already given); for those, a call passes when it has none.

Hearing · 8 skills

Entity Extraction Accuracy: Gets names, numbers and dates right as the caller says them, and reads them back before using them.
Low-Confidence Confirmation: When part of a number is lost (a cough, a bad line), asks again instead of guessing.
Side-Speech Rejection: Ignores details the caller says to someone else in the room.
Waiting Through Hold: Waits quietly through hold music or the phone being set down, then picks up when the caller is back.
Prosodic Cue Understanding: Hears meaning in tone of voice, such as a doubtful 'Monday?' being a question, not a yes.
Emotion Recognition: Notices when the caller is upset or anxious and still gets the task done.
Degraded-Channel Robustness: Copes when the phone line breaks up and words drop out.
Multi-Speaker Handling: Keeps track when more than one person speaks on the caller's side.

Turn Taking · 6 skills

Endpointing: Waits until the caller has actually finished before replying.
Backchannel vs Barge-in Detection: Keeps talking through 'mm-hmm', but stops quickly when the caller really interrupts.
Overlapping-Speech Recovery: Recovers smoothly when both sides start talking at once.
Tool-Latency Handling: Keeps the caller informed while a slow system lookup runs.
Mid-Utterance Pause Tolerance: Waits when the caller says 'hold on', then picks up when they're back.
Premature-Response Avoidance: Stays quiet during the caller's natural pauses mid-sentence.

Task Handling · 7 skills

Correction Handling: Uses the new detail when the caller corrects something they said earlier.
Human Handoff: Hands over to a person when the caller asks or the request needs one.
Tool Error Handling: Tells the caller honestly when a system action fails, instead of claiming success.
Consent Verification: Gets a clear yes from the right person before acting on the caller's behalf.
Cancellation Handling: Stops or undoes an action when the caller changes their mind, and says truthfully what happened.
Redundant Re-prompt Avoidance: Does not ask again for details the caller already gave (share of calls with none).
Hallucinated-Success Avoidance: Never says an action worked when the system said it failed (share of calls with none).

Speaking · 4 skills

Numeric Read-back: Reads numbers back digit by digit, clearly enough to write down.
TTS Text Normalization: Says things in a form that works out loud: no lists, symbols or markdown read aloud (share of calls with none).
Style Instruction Following: Follows requests about how to talk, such as shorter answers or just the numbers.
Language Switching: Switches language when the caller does, and stays in it.

The Scores

Every score is an average of skills, never of raw call counts.

Skill rate: the share of graded calls in which the model passed that skill. A skill needs at least 3 graded calls to count for a model.
Family score: the average of the family's skill rates, so each skill counts the same whatever the number of calls that test it.
Task Handling: half the average of its skill rates, half the task outcome, which is the share of the scenario's expected results the call achieved (the right booking, refund or handoff).
Call score: each family the call reached counts equally (25% each), out of 100. A call passes at 80.
Pass³: the share of scenarios the model passed in all three runs.
Call-type score: the average of the rates of the skills that call type tests, such as noisy audio or corrections.

The ELO Rating

Head-to-head on the same caller.

For each scenario, a model's score is the average of its three call scores.
Every pair of models that both took a scenario plays one match on it. The higher score wins; scores within 5 points of each other are a draw.
Ratings come from fitting a Bradley–Terry model to all the matches, on the ELO scale: 1000 is the average model, and a 400-point gap means about 10-to-1 odds of winning a match.
The 4 families count equally. No family is weighted up.
95% ranges come from re-drawing the set of scenarios 1,000 times and refitting.
The cascade is placed on the same scale afterwards, with the ranked models' ratings held fixed, so adding it moves no one.

The formula

1Family score

Family = 100 × skills passedskills tested

The share of the family's skills the call passed.

Example: 3 of 4 Hearing skills passed: 75

2Task Handling

Task Handling = ½ × task skills + ½ × task result

Task skills and the task result count half each.

Example: skills 80, task result 60: 70

3Call score

Call = Hearing + Turn Taking + Task Handling + Speaking4

Every family the call tested counts the same.

Example: (75 + 90 + 70 + 85) ÷ 4 = 80

4Scenario score

Scenario = Call₁ + Call₂ + Call₃3

The average of the model's three runs of that scenario.

Example: (80 + 85 + 75) ÷ 3 = 80

5Match

A wins if Scenario_A − Scenario_B > 5

Every pair of models plays one match per scenario. Within 5 points is a draw.

Example: 80 vs 72: A wins · 80 vs 77: draw

6ELO

P(A beats B) = 11 + 10^{(ELO_B − ELO_A) ÷ 400}

The ratings are fitted to all the matches; 1000 is the average model.

Example: 1100 vs 1000: wins 64% · 1400 vs 1000: wins 91%

Show how the ratings are fitted

1Skill grade, per callg = 1 if the call passed the skill, 0 if it failed. Only skills the call tested are graded.
2Family score, per callF = 100 × (skills passed ÷ skills graded in that family)
3Task Handling, per callT = ½ × F(task skills) + ½ × O, where O = 100 × (expected-outcome checks met ÷ checks graded). If no task skill was graded, T = O.
4Call scoreS = Σ w × F ÷ Σ w, over the families the call reached, with w = 0.25 for each family
5Scenario scores(model, scenario) = average of S over the model's three runs of that scenario
6Match, per scenario and pairA wins if s(A) − s(B) > 5; B wins if s(B) − s(A) > 5; otherwise a draw. W(m) = wins + ½ × draws, n(m, j) = matches between m and j.
7Bradley–Terry strengthsStart with p(m) = 1. Repeat 200 times: p(m) ← (W(m) + ½) ÷ (Σ_j n(m, j) ÷ (p(m) + p(j)) + 1 ÷ p(m)), then divide every p by their geometric mean.
8ELOELO(m) = 1000 + 400 × log₁₀ p(m)
9CascadeThe cascade's rating r solves Σ 1 ÷ (1 + 10^((ELO(j) − r) ÷ 400)) = its actual wins + ½ × draws against the ranked models, with their ELOs held fixed.

Latency and Cost

Measured on the same calls.

Latency P50: for each call, the average time from the caller finishing to the agent starting to speak; P50 is the median of that across the model's calls.
Went quiet: the agent stayed silent for more than 10 seconds after the caller finished, and spoke before the caller did.
Cost per minute: the model's API price as each provider publishes it. Models with no published per-minute price are left off the cost chart.

Limitations

What these results do and don't show.

70 scenarios in two domains, banking and healthcare. Other kinds of calls may rank models differently.
The Spanish (7) and Hindi (8) groups are small; read close scores there as a tie.
ELO gaps under about 50 points are within run-to-run noise.
Every model ran one agent build. A model tuned with its own prompt or turn-detection settings may do better.
Prices are list prices at the time of testing and change often.

Test your voice agent the same way.

Cekura runs these simulated calls on your agent and scores every one.

Book a demo Start free