- Reliability
- 75.6%
- Response time
- 2.02s
- Cost / min
- $0.0553
GPT-Live 1 + GPT-6 Sol is ahead on reliability, success rate, agent responses, data accuracy and interruption, and PolyAI Dialog-RSN-1 on calls where the agent went quiet, response time and cost.
Last updated Methodology by Dileep Chagam
Scenarios the agent passed on every one of its 3 runs.
Each model on its own: GPT-Live 1 + GPT-6 Sol and PolyAI Dialog-RSN-1
GPT-Live 1 + GPT-6 Sol is more reliable, by 11.0 points. GPT-Live 1 + GPT-6 Sol passed 62 of 82 scenarios on all three runs, and PolyAI Dialog-RSN-1 passed 53. A scenario only counts when every run of it passed.
GPT-Live 1 + GPT-6 Sol is more reliable, by 11.0 points. GPT-Live 1 + GPT-6 Sol passed 62 of 82 scenarios on all three runs, and PolyAI Dialog-RSN-1 passed 53. A scenario only counts when every run of it passed.
For a voice agent, GPT-Live 1 + GPT-6 Sol, since it gets more scenarios right on every run. GPT-Live 1 + GPT-6 Sol is ahead on reliability, success rate, agent responses, data accuracy and interruption, and PolyAI Dialog-RSN-1 on calls where the agent went quiet, response time and cost.
GPT-Live 1 + GPT-6 Sol, by 16.0 points. GPT-Live 1 + GPT-6 Sol saved everything exactly right on 89.0% of the calls that save data, and PolyAI Dialog-RSN-1 on 73.0%. On Booking calls, 100.0% against 97.7%. On Medicare calls, 60.6% against 9.1%. A call counts only when nothing is missing, wrong, extra or saved twice.
No. PolyAI Dialog-RSN-1 starts replying 0.66s sooner at the median. GPT-Live 1 + GPT-6 Sol takes 2.02s at the median and 2.42s at p90; PolyAI Dialog-RSN-1 takes 1.36s and 1.87s. Both are measured on the call audio, from the caller finishing to the agent starting to reply.
GPT-Live 1 + GPT-6 Sol, by 0.04 on a 0 to 5 scale. GPT-Live 1 + GPT-6 Sol scores 4.89 and PolyAI Dialog-RSN-1 4.85 for how well the agent handled being talked over.
PolyAI Dialog-RSN-1, by 12.2 points. The agent went quiet for 10 seconds or more on 12.2% of GPT-Live 1 + GPT-6 Sol calls and 0.0% of PolyAI Dialog-RSN-1 calls.
PolyAI Dialog-RSN-1 is $0.0147 cheaper a minute. GPT-Live 1 + GPT-6 Sol costs $0.0553 a minute of call and PolyAI Dialog-RSN-1 $0.0406, at each vendor's published rates.
Cekura ran both as the whole agent in the same open-source Pipecat pipeline, with the same prompt, tools and connection. A simulated caller worked through all 82 scenarios on live calls, three times each, 3,936 calls across the benchmark. A call passes only when the agent said the right things and saved the right data.
Cekura runs these simulated calls on your agent and scores every one.