How we rate speech-to-text models head to head

Every model transcribed the same clips. On each clip, every pair of models is compared and the one with fewer errors wins, a wider win counting for more; one fit over all those matches gives each model an ELO rating. This page sets out each step.

Last updated Methodology by Dileep Chagam

The Rating

One number from many head-to-head matches.

Word error rate on its own averages over very different audio. The ELO rating instead asks, clip by clip, which of two models transcribed it better, across clean speech, phone lines, noise, echo, short replies and the key details callers give. A model that wins on the conditions that matter most rates highest.

  • 1000 is the average model. A 100-point gap means the higher-rated model is expected to win about 64% of matches; 400 points, about 91%.
  • Only transcript quality goes in. Latency and price are shown beside the rating, never inside it.
  • Where a model was not run on a condition, it plays no matches there and its rating rests on the other measures.

Measures and Weights

10 measures, each carrying a fixed share of the rating.

Each measure's matches are scaled so that together they carry exactly its weight, however many clips it has. Entities and call-center noise carry the most: a wrong name, date or account number is the mistake a voice agent can least afford, and background talkers on a phone line are what callers most often bring. The clean Ocular turns carry less because the phone line, the noise sets, semantic WER and short replies are all built on the same turns; the phone codec alone barely changes results, and TV and reverb are stress tests.

MeasureClipsWeight
  • Ocular turns WERWord error rate on single-speaker turns cut from the licensed Ocular phone calls.1985%
  • Pipecat WERWord error rate on the 1,000 public clips from Pipecat's STT benchmark dataset.99910%
  • Semantic WERThe Ocular turns again, counting only the errors that change the meaning, judged with Pipecat's semantic WER prompt.19810%
  • Phone lineThe Ocular turns sent through a G.711 phone-line codec at 8 kHz.1985%
  • Call-center noiseThe Ocular turns with call-center chatter 10 dB below the speaker, over a G.711 phone line.19820%
  • Babble noiseThe Ocular turns with six background voices talking, 5 dB below the speaker.19810%
  • TV noiseThe Ocular turns with a television playing, 5 dB below the speaker.1985%
  • ReverbThe Ocular turns with room echo added (0.6 s reverberation).1985%
  • Short repliesWord errors per 100 Ocular turns under five words long, such as “yes” or a single number.7010%
  • EntitiesShare of the names, numbers, dates and other key details in the Ocular turns written exactly right.9820%

Weights add up to 100%. Clips is the most any one model was scored on; for entities it counts the details checked. A measure with no data at all is dropped and the rest are scaled back up to 100%.

The Matches

Every pair of models, on every clip both were scored on.

  • Every clip: score = 100 − WER. The higher score wins; scores within 5 points of each other are a draw.
  • A wider win counts for more: a clip won by 50 points counts 2.2 times a 5-point win, and 100 points (or a failed clip) 2.6 times. The scale is logarithmic, so a few garbled clips cannot outweigh the rest. A garbled turn breaks a call in a way one wrong word does not.
  • Matches are weighted by the clip's word count, so a long turn counts for more than a short one.
  • A clip the model failed on (no transcript, an error or a timeout) loses to any valid one. Clips that failed for reasons outside the model, such as our own pacing or transcript stitching, are rebuilt from the provider's raw events or, where they cannot be, left out; they never count as the model's failure.
  • Short replies, under 5 words, are also a measure of their own: decided on the number of word errors, with no draw margin, because a single word wrong is already a large share of a short reply.
  • Entities: decided on how many of the clip's names, numbers, dates and other key details the model wrote exactly right, with no draw margin.

Semantic WER

Errors that change the meaning, judged the same way for every model.

Plain WER counts “okay” for “OK” the same as a wrong account number. Semantic WER counts only the errors that change what the caller meant. It uses the judging prompt from Pipecat's STT benchmark word for word, with Claude as the judge, and counts errors with Pipecat's own scoring.

  • When the normalized transcript matches the reference exactly, the clip scores zero errors without being judged.
  • Every model's transcripts go to the same judge with the same prompt.

The ELO Formula

A Bradley–Terry fit, on the ELO scale.

  • Ratings come from fitting a Bradley–Terry model to all the weighted matches: each model gets a strength p, and the chance A beats B is p(A) ÷ (p(A) + p(B)).
  • The strength is put on the ELO scale with ELO = 1000 + 400 · log₁₀ p, so 1000 is the average model.

The formula

1Clip score
score = 100 − WER

On every clip, the share of words right.

Example: 2 errors in 20 words: WER 10, score 90

2Match
A wins if scoreA − scoreB > 5

Every pair of models plays one match per clip. Within 5 points is a draw; a failed clip loses to a valid one.

Example: 90 vs 82: A wins · 90 vs 86: draw

3Margin of victory
m = ln(1 + gap)ln(6)

A wider win counts for more, on a log scale; a 5-point win counts 1×. Draws, short replies and entities count 1×.

Example: 50-point win: 2.2× · 100 points or a failed clip: 2.6×

4Match weight
w = Wmeasure × m × words in the clipΣ m × words over the measure's matches

Longer clips and wider wins count for more, and each measure's matches are scaled to its weight.

Example: Phone line carries 5% of all match weight, whatever its clip count

5Strength
P(A beats B) = pApA + pB

A Bradley–Terry fit: the strengths that best explain every weighted win, draw and loss.

Example: p = 2 vs p = 1: wins 2 matches in 3

6ELO
ELO = 1000 + 400 · log10 p

The strength on the ELO scale; 1000 is the average model.

Example: p = 2: ELO 1120 · p = 0.5: ELO 880

7Expected win rate
P(A beats B) = 11 + 10(ELOB − ELOA) ÷ 400

What two ratings predict for one match, as shown on every head-to-head page.

Example: 1100 vs 1000: wins 64% · 1400 vs 1000: wins 91%

Show how the ratings are fitted
  1. 1Clip scores = 100 − WER, on every clip with words in its reference. A clip the model failed on has no score.
  2. 2Match, per clip and pairA wins if s(A) − s(B) > 5; B wins if s(B) − s(A) > 5; otherwise a draw. A failed clip loses to a valid one; two failures play no match.
  3. 3Short repliesClips under 5 words are also scored on their own: s = −(word errors). Fewer errors wins, equal is a draw, no margin.
  4. 4Entitiess = entities written exactly right. More wins, equal is a draw, no margin.
  5. 5Margin of victoryOn WER and semantic WER, a decided match counts m = ln(1 + |s(A) − s(B)|) ÷ ln(6) times (a failed clip scores 0); draws, short replies and entities count m = 1.
  6. 6Match weightw = W(measure) × m × words in the clip ÷ total of m × words over that measure's matches, so every measure adds up to its weight.
  7. 7Wins and gameswins(m) = Σ w × (1 for a win, ½ for a draw); n(m, j) = Σ w over the matches between m and j.
  8. 8Bradley–Terry strengthsStart with p(m) = 1. Repeat 400 times: p(m) ← (wins(m) + ½) ÷ (Σ_j n(m, j) ÷ (p(m) + p(j)) + 1 ÷ p(m)), then divide every p by their geometric mean.
  9. 9ELOELO(m) = 1000 + 400 × log₁₀ p(m)
  10. 1095% rangeRe-draw the Ocular conversations with replacement (one draw shared by every measure built on them) and the Pipecat clips, refit, and take the 2.5th and 97.5th percentiles of each model's ELO, over 200 draws.

Ranges and Ranks

How sure each rating is, and which places it allows.

  • 95% ranges come from re-drawing whole Ocular conversations with replacement, the same draw for every measure built on them, plus the Pipecat clips, and refitting each time. Turns from one conversation share speakers and audio, so they are drawn together; with few conversations, the ranges are wide.
  • A model's rank is 1 plus the number of models whose 95% lower bound is above its upper bound, that is, the models certainly ahead of it.
  • The rank range runs from that rank to the number of models whose upper bound reaches its lower bound. Models whose ranges overlap share places.

Not in the Rating

Shown beside the rating, never inside it.

  • Latency: time to first text and final-text delay are measured on the same streamed turns and plotted against the rating, so speed and accuracy can be traded off in the open.
  • Price: each vendor's published rate per hour of audio, as listed on the leaderboard.
  • Data: Pipecat dataset and licensed Ocular phone calls. Code: benchmark source.

Test your voice agent the same way.

Cekura runs these simulated calls on your agent and scores every one.