- Reliability
- 58.5%
- Response time
- 2.31s
- Cost / min
- $0.0224
Deepslate Opal is ahead on success rate, data accuracy, interruption and cost, and PolyAI Dialog-RSN-1 on reliability, agent responses, calls where the agent went quiet and response time.
Last updated Methodology by Dileep Chagam
Scenarios the agent passed on every one of its 3 runs.
Each model on its own: Deepslate Opal and PolyAI Dialog-RSN-1
PolyAI Dialog-RSN-1 is more reliable, by 6.1 points. Deepslate Opal passed 48 of 82 scenarios on all three runs, and PolyAI Dialog-RSN-1 passed 53. A scenario only counts when every run of it passed.
PolyAI Dialog-RSN-1 is more reliable, by 6.1 points. Deepslate Opal passed 48 of 82 scenarios on all three runs, and PolyAI Dialog-RSN-1 passed 53. A scenario only counts when every run of it passed.
For a voice agent, PolyAI Dialog-RSN-1, since it gets more scenarios right on every run. Deepslate Opal is ahead on success rate, data accuracy, interruption and cost, and PolyAI Dialog-RSN-1 on reliability, agent responses, calls where the agent went quiet and response time.
Deepslate Opal, by 5.5 points. Deepslate Opal saved everything exactly right on 78.5% of the calls that save data, and PolyAI Dialog-RSN-1 on 73.0%. On Booking calls, 88.3% against 97.7%. On Medicare calls, 53.0% against 9.1%. A call counts only when nothing is missing, wrong, extra or saved twice.
No. PolyAI Dialog-RSN-1 starts replying 0.95s sooner at the median. Deepslate Opal takes 2.31s at the median and 2.88s at p90; PolyAI Dialog-RSN-1 takes 1.36s and 1.87s. Both are measured on the call audio, from the caller finishing to the agent starting to reply.
Deepslate Opal, by 0.09 on a 0 to 5 scale. Deepslate Opal scores 4.94 and PolyAI Dialog-RSN-1 4.85 for how well the agent handled being talked over.
PolyAI Dialog-RSN-1, by 6.1 points. The agent went quiet for 10 seconds or more on 6.1% of Deepslate Opal calls and 0.0% of PolyAI Dialog-RSN-1 calls.
Deepslate Opal is $0.0182 cheaper a minute. Deepslate Opal costs $0.0224 a minute of call and PolyAI Dialog-RSN-1 $0.0406, at each vendor's published rates.
Cekura ran both as the whole agent in the same open-source Pipecat pipeline, with the same prompt, tools and connection. A simulated caller worked through all 82 scenarios on live calls, three times each, 3,936 calls across the benchmark. A call passes only when the agent said the right things and saved the right data.
Cekura runs these simulated calls on your agent and scores every one.