Doctor calls are part of the product
Kapiva runs a doctor and health-coach programme alongside its products. Doctor calls are guidance, not a sales pitch: they set the right expectations and answer the small questions people won't take to their own doctor.
Doctors work in Care Studio, the tool we built for them: appointments, each patient's history and next stage, and the prescriptions they create and edit.
A ~10% sample can't manage quality
Consultation QA relied on manual sampling of ~10% of consultations, so most consultations were never reviewed at all.
Who the audits serve
Doctors
See their scores, audits and strong and weak points in Care Studio, and can dispute feedback there.
Quality reviewers
Still review 5% of consultations by hand, alongside the AI.
Customers
About a quarter of consultations are in regional languages, so the audits had to hold up beyond one language.
Doctor performance
The effectiveness score feeds the doctors' incentive and performance system.
A 14-parameter rubric, scored by AI, grounded in evidence
Every consultation is audited by AI against a 14-parameter rubric that rolls up into two composite scores: an effectiveness score (CES) and a satisfaction score (CSS).
Every score is grounded: it cites the transcript line that supports it. That makes each output auditable, gives doctors something concrete to act on or dispute, and makes a wrong score easy to catch.
14 parameters
One rubric for every consultation, including medical-knowledge parameters.
Two composite scores
Effectiveness and satisfaction, rolled up from the rubric.
Evidence for every score
Each score cites the transcript line behind it.
Every score shows its evidence, and every doctor can dispute it.
The audit pipeline
The pipeline runs on every consultation, not a sample, and has run in production first on a vendor tool and now on our own system, at 70% lower cost.
Consultation
Every doctor consultation; about a quarter are in regional languages.
Transcript
The model's input, and the source every score cites.
Rubric scoring
The model scores the 14 parameters, with domain vocabulary and few-shot examples from real consultations.
CES and CSS, with citations
Each score is tied to the transcript line that supports it.
Care Studio
Doctors see scores, audits and strong and weak points, and can dispute feedback.
Human-in-the-loop review
People review 5% of consultations: a standing check on production scores.
Incentives and performance
CES feeds the doctors' incentive and performance system.
Evals against real doctors before go-live
Before going live, I validated the model against golden eval sets: consultations scored by real doctors. The evals found CES off on two medical-knowledge parameters, so we fixed those and ran the evals again before shipping.
-
Golden eval sets
Real doctors labelled consultations with rubric scores.
-
Run the evals
The model scored the same consultations; we compared it with the doctors.
-
Error analysis
CES was off on two medical-knowledge parameters.
-
Fix
Reworked their rating scale, expanded the model's medical vocabulary and added real cases as few-shot examples.
-
Regression
Re-ran the golden sets, broken down by language.
Quality controls in production
- Evals broken down by language, not just in aggregate: about a quarter of consultations are in regional languages.
- Grounded outputs: every score cites its transcript line, so a wrong score is visible, not hidden in an average.
- A feedback loop: doctors dispute feedback in Care Studio.
- Human-in-the-loop: people still review 5% of consultations.
- A high bar for use: CES feeds incentives only because it proved accurate enough.
Buy, then build
We started on a vendor tool, auditing every consultation, then moved the audits to an in-house system at 70% lower cost.
-
Buy
AI audits of every consultation on a vendor tool, up from a ~10% manual sample.
-
Build
An in-house system replaced the vendor tool, at 70% lower cost.
-
Trust
CES became accurate enough to feed the doctors' incentive and performance system.
What changed
QA coverage went from ~10% to 100%, the in-house system costs 70% less than the vendor tool, and the effectiveness score is now trusted enough to feed how doctors are rewarded.