A/B testing agents
You changed an agent's prompt and evals say it is better. Real customers may disagree. An A/B experiment sends a share of live conversations to the candidate, keeps the rest on the published version, and measures both with the same Insights data point. When the numbers are in, you know.
Experiments are opt-in per application. Ask your Docana contact to enable Agent branching for it.
What you need
- A published agent with Conversation Insights on and at least one boolean or number data point. That data point decides the winner. A string data point cannot: it describes, it does not compare.
- A candidate branch of the agent. Fork one from the agent's Branches tab or with the API.
- The Docana CLI, logged in.
Start an experiment
docana agents experiments start my-agent \
--branch cmc3kb1x9d0001l608r2vhq7e9 \
--name shorter-greeting \
--traffic 20 \
--win-metric resolved \
--guardrail cost:0.25 \
--guardrail silence:0.5 \
--min-hours 48 \
--max-hours 336
From that moment 20 % of new conversations get the candidate. A conversation stays on its arm for its whole life. Conversations that started before the experiment never enter it.
Both versions are frozen when the experiment starts. Editing the branch or publishing the agent does not change what is being compared. Publishing does end the experiment, because it moves the published version the control arm was pinned to.
Guardrails
A guardrail ends the experiment on its own when the candidate is worse in a way you did not sign up for. cost:0.25 means the candidate may cost up to 25 % more per conversation. The four kinds:
| flag | watches |
|---|---|
cost | USD per conversation |
latency | 95th percentile seconds per turn |
errors | share of turns with errors or that timed out |
silence | share of conversations whose last message is an unanswered customer message |
A guardrail only fires once each arm has 30 conversations (change it with a third part: cost:0.25:50). Until then it reads pending. When the published version has zero errors or zero silences, the candidate needs at least 2 affected conversations before those guardrails fire.
Horizon
--max-hours or --max-threads ends the experiment after that much time or that many conversations. --min-hours does something different: it withholds the recommendation until a full business cycle has passed, because a candidate that wins on Monday morning traffic was only measured on Monday morning.
Read the result
docana agents experiments show my-agent cmc3kb1x9d0001l608r2vhq7e9
shorter-greeting — RUNNING, 20 % to variant
control: 412 assigned, 380 measured
variant: 103 assigned, 91 measured
win metric: Resolved (boolean)
control 61.2 % (n=380) variant 68.4 % (n=91)
P(variant better) 91.3 % expected loss: variant 0.3 %, control 7.5 %
recommendation: inconclusive
guardrails:
costPerThread ok control 0.104 variant 0.111 threshold 0.130 (412/103 threads, floor 30)
silenceRate ok control 0.012 variant 0.010 threshold 0.018 (412/103 threads, floor 30)
horizon: 96.5 h of 336 h, 515 threads assigned
Assigned counts conversations routed to an arm. Measured counts the ones whose Insights data point has been extracted. Measured lags assigned because insights are extracted when a conversation closes, so a fresh experiment shows many assigned and few measured. That is normal.
P(variant better) is the probability, given the data so far, that the candidate has the higher rate. Expected loss is what you give up if you pick that arm and it turns out to be the worse one. The recommendation becomes variant at 95 % and control at 5 %. Between those it reads inconclusive: keep the experiment running. Before the minimum duration, or while either arm has fewer than 30 measured conversations, it reads too-early with the reason.
Conversations that never close
On WhatsApp a conversation has no end. Docana treats a conversation with no message for 24 hours as closed: it extracts its insights and counts it in its arm. Change the window with --idle-hours, from 1 up to 168 hours (a week).
What it costs
Conversations in an experiment always produce insights, even when the agent samples insights below 100 %. An agent sampling 5 % of conversations pays full extraction cost on the experiment's share of traffic while it runs.
Stop an experiment
docana agents experiments stop my-agent cmc3kb1x9d0001l608r2vhq7e9
All traffic returns to the published version at the next turn. Conversations that were mid-way switch back to the published version too, and are left out of the readout, because they saw both versions.
Publishing the agent stops a running experiment the same way. show keeps working after an experiment ends, and keeps counting insights that land during the idle window after it.