How we rate speech-to-text models head to head
Every model transcribed the same clips. On each clip, every pair of models is compared and the one with fewer errors wins, a wider win counting for more; one fit over all those matches gives each model an ELO rating. This page sets out each step.
Last updated Methodology by Dileep Chagam
The Rating
One number from many head-to-head matches.
Word error rate on its own averages over very different audio. The ELO rating instead asks, clip by clip, which of two models transcribed it better, across clean speech, phone lines, noise, echo, short replies and the key details callers give. A model that wins on the conditions that matter most rates highest.
- 1000 is the average model. A 100-point gap means the higher-rated model is expected to win about 64% of matches; 400 points, about 91%.
- Only transcript quality goes in. Latency and price are shown beside the rating, never inside it.
- Where a model was not run on a condition, it plays no matches there and its rating rests on the other measures.
Measures and Weights
10 measures, each carrying a fixed share of the rating.
Each measure's matches are scaled so that together they carry exactly its weight, however many clips it has. Entities and call-center noise carry the most: a wrong name, date or account number is the mistake a voice agent can least afford, and background talkers on a phone line are what callers most often bring. The clean Ocular turns carry less because the phone line, the noise sets, semantic WER and short replies are all built on the same turns; the phone codec alone barely changes results, and TV and reverb are stress tests.
- Ocular turns WERWord error rate on single-speaker turns cut from the licensed Ocular phone calls.Word error rate on single-speaker turns cut from the licensed Ocular phone calls.1985%
- Pipecat WERWord error rate on the 1,000 public clips from Pipecat's STT benchmark dataset.Word error rate on the 1,000 public clips from Pipecat's STT benchmark dataset.99910%
- Semantic WERThe Ocular turns again, counting only the errors that change the meaning, judged with Pipecat's semantic WER prompt.The Ocular turns again, counting only the errors that change the meaning, judged with Pipecat's semantic WER prompt.19810%
- Phone lineThe Ocular turns sent through a G.711 phone-line codec at 8 kHz.The Ocular turns sent through a G.711 phone-line codec at 8 kHz.1985%
- Call-center noiseThe Ocular turns with call-center chatter 10 dB below the speaker, over a G.711 phone line.The Ocular turns with call-center chatter 10 dB below the speaker, over a G.711 phone line.19820%
- Babble noiseThe Ocular turns with six background voices talking, 5 dB below the speaker.The Ocular turns with six background voices talking, 5 dB below the speaker.19810%
- TV noiseThe Ocular turns with a television playing, 5 dB below the speaker.The Ocular turns with a television playing, 5 dB below the speaker.1985%
- ReverbThe Ocular turns with room echo added (0.6 s reverberation).The Ocular turns with room echo added (0.6 s reverberation).1985%
- Short repliesWord errors per 100 Ocular turns under five words long, such as “yes” or a single number.Word errors per 100 Ocular turns under five words long, such as “yes” or a single number.7010%
- EntitiesShare of the names, numbers, dates and other key details in the Ocular turns written exactly right.Share of the names, numbers, dates and other key details in the Ocular turns written exactly right.9820%
Weights add up to 100%. Clips is the most any one model was scored on; for entities it counts the details checked. A measure with no data at all is dropped and the rest are scaled back up to 100%.
The Matches
Every pair of models, on every clip both were scored on.
- Every clip: score = 100 − WER. The higher score wins; scores within 5 points of each other are a draw.
- A wider win counts for more: a clip won by 50 points counts 2.2 times a 5-point win, and 100 points (or a failed clip) 2.6 times. The scale is logarithmic, so a few garbled clips cannot outweigh the rest. A garbled turn breaks a call in a way one wrong word does not.
- Matches are weighted by the clip's word count, so a long turn counts for more than a short one.
- A clip the model failed on (no transcript, an error or a timeout) loses to any valid one. Clips that failed for reasons outside the model, such as our own pacing or transcript stitching, are rebuilt from the provider's raw events or, where they cannot be, left out; they never count as the model's failure.
- Short replies, under 5 words, are also a measure of their own: decided on the number of word errors, with no draw margin, because a single word wrong is already a large share of a short reply.
- Entities: decided on how many of the clip's names, numbers, dates and other key details the model wrote exactly right, with no draw margin.
Semantic WER
Errors that change the meaning, judged the same way for every model.
Plain WER counts “okay” for “OK” the same as a wrong account number. Semantic WER counts only the errors that change what the caller meant. It uses the judging prompt from Pipecat's STT benchmark word for word, with Claude as the judge, and counts errors with Pipecat's own scoring.
- When the normalized transcript matches the reference exactly, the clip scores zero errors without being judged.
- Every model's transcripts go to the same judge with the same prompt.
The ELO Formula
A Bradley–Terry fit, on the ELO scale.
- Ratings come from fitting a Bradley–Terry model to all the weighted matches: each model gets a strength p, and the chance A beats B is p(A) ÷ (p(A) + p(B)).
- The strength is put on the ELO scale with ELO = 1000 + 400 · log₁₀ p, so 1000 is the average model.
The formula
On every clip, the share of words right.
Example: 2 errors in 20 words: WER 10, score 90
Every pair of models plays one match per clip. Within 5 points is a draw; a failed clip loses to a valid one.
Example: 90 vs 82: A wins · 90 vs 86: draw
A wider win counts for more, on a log scale; a 5-point win counts 1×. Draws, short replies and entities count 1×.
Example: 50-point win: 2.2× · 100 points or a failed clip: 2.6×
Longer clips and wider wins count for more, and each measure's matches are scaled to its weight.
Example: Phone line carries 5% of all match weight, whatever its clip count
A Bradley–Terry fit: the strengths that best explain every weighted win, draw and loss.
Example: p = 2 vs p = 1: wins 2 matches in 3
The strength on the ELO scale; 1000 is the average model.
Example: p = 2: ELO 1120 · p = 0.5: ELO 880
What two ratings predict for one match, as shown on every head-to-head page.
Example: 1100 vs 1000: wins 64% · 1400 vs 1000: wins 91%
Show how the ratings are fittedHide how the ratings are fitted
- 1Clip score
s = 100 − WER, on every clip with words in its reference. A clip the model failed on has no score. - 2Match, per clip and pair
A wins if s(A) − s(B) > 5; B wins if s(B) − s(A) > 5; otherwise a draw. A failed clip loses to a valid one; two failures play no match. - 3Short replies
Clips under 5 words are also scored on their own: s = −(word errors). Fewer errors wins, equal is a draw, no margin. - 4Entities
s = entities written exactly right. More wins, equal is a draw, no margin. - 5Margin of victory
On WER and semantic WER, a decided match counts m = ln(1 + |s(A) − s(B)|) ÷ ln(6) times (a failed clip scores 0); draws, short replies and entities count m = 1. - 6Match weight
w = W(measure) × m × words in the clip ÷ total of m × words over that measure's matches, so every measure adds up to its weight. - 7Wins and games
wins(m) = Σ w × (1 for a win, ½ for a draw); n(m, j) = Σ w over the matches between m and j. - 8Bradley–Terry strengths
Start with p(m) = 1. Repeat 400 times: p(m) ← (wins(m) + ½) ÷ (Σ_j n(m, j) ÷ (p(m) + p(j)) + 1 ÷ p(m)), then divide every p by their geometric mean. - 9ELO
ELO(m) = 1000 + 400 × log₁₀ p(m) - 1095% range
Re-draw the Ocular conversations with replacement (one draw shared by every measure built on them) and the Pipecat clips, refit, and take the 2.5th and 97.5th percentiles of each model's ELO, over 200 draws.
Ranges and Ranks
How sure each rating is, and which places it allows.
- 95% ranges come from re-drawing whole Ocular conversations with replacement, the same draw for every measure built on them, plus the Pipecat clips, and refitting each time. Turns from one conversation share speakers and audio, so they are drawn together; with few conversations, the ranges are wide.
- A model's rank is 1 plus the number of models whose 95% lower bound is above its upper bound, that is, the models certainly ahead of it.
- The rank range runs from that rank to the number of models whose upper bound reaches its lower bound. Models whose ranges overlap share places.
Not in the Rating
Shown beside the rating, never inside it.
- Latency: time to first text and final-text delay are measured on the same streamed turns and plotted against the rating, so speed and accuracy can be traded off in the open.
- Price: each vendor's published rate per hour of audio, as listed on the leaderboard.
- Data: Pipecat dataset and licensed Ocular phone calls. Code: benchmark source.
Test your voice agent the same way.
Cekura runs these simulated calls on your agent and scores every one.