AssemblyAI vs OpenAI speech-to-text
AssemblyAI is ahead of OpenAI on time to first text and final-text delay. The Pipecat Dataset and The Ocular Dataset are too close to call.
Each row compares the best AssemblyAI model with the best OpenAI model on that metric, from the same tests.
Last updated Methodology by Dileep Chagam
AssemblyAI and OpenAI, metric by metric
| Metric | AssemblyAI | OpenAI |
|---|---|---|
| Pipecat Dataset WER | 1.77%AssemblyAI Universal 3.6 Pro | 2.16%GPT Realtime Whisper |
| Ocular Dataset WER | 4.23%AssemblyAI Universal 3.5 Pro | 3.53%GPT Realtime Whisper |
| Time to first text | 485msAssemblyAI Universal 3.6 Pro | 1.38sGPT Realtime Whisper |
| Final-text delay | 91msAssemblyAI Universal 3.6 Pro | 543msGPT Realtime Whisper |
- WER · Word error rate
- Share of words the model got wrong: replaced, missing or added. Lower is better.
- Pipecat Dataset
- 1,000 short public clips from Pipecat's STT benchmark dataset.
- Ocular Dataset
- 8 recordings from 4 real phone conversations, licensed from Ocular.
- TTFT · Time to first text
- How long after the audio starts streaming until the first words appear.
- TTFS · Final-text delay
- How long after the caller stops until the full transcript arrives. A voice agent waits for this before it replies.
AssemblyAI has 2 models on Converse-STT: AssemblyAI Universal 3.5 Pro and AssemblyAI Universal 3.6 Pro. Its best on accuracy is AssemblyAI Universal 3.6 Pro (1.77%). Its fastest is AssemblyAI Universal 3.6 Pro (91ms).
OpenAI has 3 models on Converse-STT: GPT-4o Mini Transcribe, GPT-4o Transcribe and GPT Realtime Whisper. Its best on accuracy is GPT Realtime Whisper (2.16%). Its fastest is GPT Realtime Whisper (543ms).
Which to pick
AssemblyAI is the better pick over OpenAI on every measure that separates them: time to first text (894ms sooner) and final-text delay (452ms sooner). For the Pipecat Dataset and the Ocular Dataset, the gap is inside the margin of error, so neither is ahead.
Every model-to-model comparison
- AssemblyAI Universal 3.5 Pro vs GPT-4o Mini Transcribe
- AssemblyAI Universal 3.5 Pro vs GPT-4o Transcribe
- AssemblyAI Universal 3.5 Pro vs GPT Realtime Whisper
- AssemblyAI Universal 3.6 Pro vs GPT-4o Mini Transcribe
- AssemblyAI Universal 3.6 Pro vs GPT-4o Transcribe
- AssemblyAI Universal 3.6 Pro vs GPT Realtime Whisper
Frequently asked questions
Is AssemblyAI or OpenAI better for speech-to-text?
It is too close to call: the gap is inside the margin of error. The best AssemblyAI result is AssemblyAI Universal 3.6 Pro at 1.77%, and the best OpenAI result is GPT Realtime Whisper at 2.16%, on the Pipecat Dataset.
Which is faster, AssemblyAI or OpenAI?
AssemblyAI is faster, 452ms sooner. The best AssemblyAI result is AssemblyAI Universal 3.6 Pro at 91ms, and the best OpenAI result is GPT Realtime Whisper at 543ms, on final-text delay.
Which AssemblyAI and OpenAI models were tested?
AssemblyAI: AssemblyAI Universal 3.5 Pro, AssemblyAI Universal 3.6 Pro. OpenAI: GPT-4o Mini Transcribe, GPT-4o Transcribe, GPT Realtime Whisper. Every model ran the same tests on Converse-STT.