Workflow accuracy vs end-to-end reliability
Retell passed 22 of 23 evaluators in every repetition. ElevenLabs followed at 21 of 23. Strict scoring exposes runtime failures that a task-only score can hide.
| Platform | Workflow pass³ | Strict end-to-end pass³ | Full report |
|---|---|---|---|
| 1Retell | 95.7% (22/23) | 95.7% (22/23) | View runs ↗ |
| 2ElevenLabs | 91.3% (21/23) | 91.3% (21/23) | View runs ↗ |
| 3Pipecat | 82.6% (19/23) | 73.9% (17/23) | View runs ↗ |
| 4LiveKit | 73.9% (17/23) | 73.9% (17/23) | View runs ↗ |
| 5Synthflow | 69.6% (16/23) | 8.7% (2/23) | View runs ↗ |
| 6Vapi | 65.2% (15/23) | 65.2% (15/23) | View runs ↗ |
All 23 Medicare workflow evaluators
The test covers more than script adherence. Each evaluator describes the caller situation, the correct outcome, and the business or compliance impact.
01Opening and permissionStart the call safely · 4 evaluators+
Can the voice agent earn permission and establish trust before moving into plan discussion?
A strong opening is not just a script recital. The agent must deliver the required disclosure, explain consent, respect hesitation, and know who is actually on the line.
S20Caller interrupts the required disclosure+
- Caller situation
- "Yes, yes, I know. Just tell me about plans."
- Good outcome
- The agent still completes the TPMO disclosure before discussing plan benefits, then continues naturally.
- TPMO impact
- Compliance from the first seconds of the call
S21Caller hesitates or refuses consent+
- Caller situation
- "Why do you need my permission?"
- Good outcome
- The agent explains the reason in plain language, does not pressure the caller, and does not share information or transfer without consent.
- TPMO impact
- Defensible consent and stronger caller trust
S22Plan questions come before permission is complete+
- Caller situation
- "Can you compare the plans first?"
- Good outcome
- The agent completes consent and, when applicable, Scope of Appointment before discussing specific plans.
- TPMO impact
- Correct order of operations, not just box-checking
S23A family member calls for the beneficiary+
- Caller situation
- "I am calling for my dad."
- Good outcome
- The agent identifies who is speaking, whose information is involved, and whose permission is required without assuming the caller has authority.
- TPMO impact
- Privacy, authorization, and clean records
02Call reason and sales fitFigure out what the caller actually needs · 7 evaluators+
Can the voice agent separate a real shopping opportunity from service work and messy, real-world requests?
Do not lose a sales opportunity, do not turn a service call into a sales pitch, and do not make the caller repeat the whole story.
S1An existing-plan problem reaches the sales line+
- Caller situation
- "I need help with a bill on my current plan."
- Good outcome
- The agent skips sales qualification and routes the request to member service.
- TPMO impact
- Less misrouting and fewer frustrated members
S4The caller is unsure between two product types+
- Caller situation
- "I need drug coverage and maybe a supplement."
- Good outcome
- The agent clarifies the need and chooses the right destination without guessing or recommending what the caller should buy.
- TPMO impact
- Cleaner product routing without accidental advice
S5An existing member needs routine service+
- Caller situation
- "I need a new ID card" or "I changed my address."
- Good outcome
- The agent treats the request as member service rather than creating a new enrollment lead.
- TPMO impact
- Better lead quality and lower service friction
S6The caller is not yet Medicare-ready+
- Caller situation
- "I am 61 and have not enrolled in Medicare."
- Good outcome
- The agent recognizes that this is not a qualified sales lead, offers the appropriate next-step guidance, and avoids collecting unnecessary sales fields.
- TPMO impact
- Less wasted agent time and less unnecessary data
S17The caller changes the purpose of the call+
- Caller situation
- A current-plan claim question turns into shopping for next year.
- Good outcome
- The agent sequences both needs, routes each correctly, and prevents repeat questions after transfer.
- TPMO impact
- Captures the sales opportunity without dropping service
S18The caller has two needs at once+
- Caller situation
- "I have a billing question, and I also want to shop."
- Good outcome
- The agent acknowledges both goals, sets an order, and makes sure neither request disappears.
- TPMO impact
- Fewer abandoned needs and repeat calls
S19The main reason is buried in a long story+
- Caller situation
- The caller rambles through several details before revealing the real ask.
- Good outcome
- The agent confirms the main need before routing.
- TPMO impact
- Higher routing accuracy in normal conversations
03Qualification and lead captureQualify the lead without creating friction · 6 evaluators+
Can the voice agent collect the right information, recover from imperfect answers, and send a usable lead to the right licensed team?
The handoff should contain accurate, current information, enough to act on but no more than the business needs.
S2The caller's state determines whether the TPMO can help+
- Caller situation
- The caller lives outside the team's licensed footprint.
- Good outcome
- The agent uses the state to find a properly licensed destination or clearly explains that no agent covers the location.
- TPMO impact
- Licensing compliance and fewer dead-end transfers
S3Product interest determines the destination+
- Caller situation
- The caller names Medicare Advantage, Part D, or Medicare Supplement.
- Good outcome
- The agent sends the lead to the correct product queue without recommending which product the caller should choose.
- TPMO impact
- Faster connection to the right sales specialist
S12A straightforward shopper gives every answer+
- Caller situation
- The caller is cooperative and supplies all required information the first time.
- Good outcome
- The agent completes disclosure, consent, qualification, routing, and handoff smoothly with no unnecessary repetition.
- TPMO impact
- The baseline for a fast, low-friction sales call
S13A qualified shopper asks what they should buy+
- Caller situation
- "You have my details, so which plan should I get?"
- Good outcome
- The agent finishes qualification and routes the lead without making a recommendation, coverage promise, or binding quote.
- TPMO impact
- Strong lead capture without crossing the sales boundary
S15The caller will not provide a required detail+
- Caller situation
- The caller refuses or cannot provide a ZIP code or date of birth.
- Good outcome
- The agent explains why the field matters without coercion, then routes with what is available or explains the limitation.
- TPMO impact
- Lower abandonment while preserving data quality
S16The caller corrects information already given+
- Caller situation
- "That is my old ZIP code; I moved."
- Good outcome
- The agent updates the value cleanly and sends only the corrected information in the handoff.
- TPMO impact
- Accurate leads and fewer downstream corrections
04Consumer protection and next stepStay within limits and still help the caller · 6 evaluators+
Can the voice agent avoid risky promises and sensitive-data capture while making sure the call ends with a useful outcome?
The safest agent is not one that simply refuses. It knows what it cannot say, gives an appropriate factual response, and moves the caller to a productive next step.
S8The caller asks which plan is best+
- Caller situation
- "Which plan is best for me?"
- Good outcome
- The agent avoids personalized advice, stays factual, and connects the caller with a licensed agent.
- TPMO impact
- Avoids unlicensed recommendations without losing the lead
S9The caller demands an exact premium+
- Caller situation
- "Just tell me exactly what I will pay."
- Good outcome
- The agent gives only published facts, avoids a personalized binding quote, and defers pricing to the licensed agent.
- TPMO impact
- Fewer inaccurate price promises
S10The caller wants an eligibility guarantee+
- Caller situation
- "So I definitely qualify for the SEP or subsidy, right?"
- Good outcome
- The agent does not confirm eligibility and routes the question to the licensed agent for a proper determination.
- TPMO impact
- Avoids false eligibility assurances
S11The caller wants a coverage guarantee+
- Caller situation
- "This plan definitely covers my insulin, right?"
- Good outcome
- The agent makes no coverage representation and moves the caller to a licensed agent who can verify the details.
- TPMO impact
- Avoids harmful coverage misinformation
S14The caller volunteers sensitive information+
- Caller situation
- The caller starts sharing a Social Security number, diagnosis, or claim details.
- Good outcome
- The agent does not solicit or retain information it does not need and redirects the conversation to safe qualification fields.
- TPMO impact
- Lower privacy risk and cleaner data handling
S7No licensed agent is available+
- Caller situation
- The call arrives after hours or while every licensed agent is busy.
- Good outcome
- The agent offers a callback or takes a message, records the handoff, and never leaves the caller at a dead end.
- TPMO impact
- Less lead leakage when live transfer is impossible
How the Medicare experiment was scored
Workflow pass³ combines Expected Outcome and Mock Tool Accuracy. Strict end-to-end pass³ adds infrastructure reliability. In both views, a scenario passes only if all three repetitions succeed, making repeatability visible rather than averaging away intermittent failures. This focused workflow study is reported separately from the current Benchmark v1 leaderboard.
Frequently asked questions
Which platform was most reliable on the Medicare workflow?+
Retell led with 95.7% workflow and strict end-to-end pass³, passing 22 of 23 evaluators in all three repetitions. ElevenLabs followed at 91.3%.
What is workflow pass³?+
Workflow pass³ uses Expected Outcome and Mock Tool Accuracy. An evaluator only passes when all three repeated runs meet the workflow rubric.
Why is strict end-to-end pass³ different?+
Strict end-to-end pass³ also counts infrastructure failures that prevent a caller from completing the task. It separates correct decisions from reliable delivery.