Agent Experiments API

Compare pinned agent versions on live conversations. Review setup and results, pause or resume enrollment, stop an experiment, and publish the tested candidate.

📄️Review experiment setup

Returns exact control/candidate pairs, safe node summaries, measurement definitions and verified lineage mapping suggestions. No full spec is returned.

📄️List agent's experiments

Returns every experiment of this agent, newest first, running and ended alike. Both arms are reported as the exact (artifactId, version) pinned when the experiment started, which is what actually served — not whatever the branch or the published pointer holds now.

📄️Start A/B experiment

Starts a live A/B experiment comparing the agent’s published spec (control) against a CANDIDATE branch (variant), routing `trafficPercent` of this agent’s conversations to the variant. Both arms are pinned to the exact (artifactId, version) they hold right now, so neither editing the branch nor publishing the agent moves what is being compared. Assignment is per thread and sticky: a conversation never changes arms mid-way. One running experiment per agent; publishing the agent ends the running one. Optionally names the Insights data point that decides the winner, guardrails that end the experiment when the variant breaches them, a minimum duration before a recommendation, a horizon, and the idle window that closes a thread on channels that never archive.

📄️Read experiment and its readout

Returns the experiment with a readout computed now: threads assigned and measured per arm, the win metric with P(variant better), expected loss and a recommendation (withheld until the minimum duration has elapsed and both arms reach the measured floor), every guardrail’s current state, and horizon progress. For an ended experiment the readout keeps counting snapshots up to endedAt plus the idle window.

📄️Stop running experiment

Stops a running experiment, recording `MANUAL` as the reason. All traffic returns to the published spec immediately. Threads that were still open are stamped as having changed arms mid-conversation, so they belong to neither arm and drop out of the readout. Ending is one-way: an experiment cannot be resumed, only replaced by a new one.

📄️Pause experiment enrollment

Pause new conversation enrollment. Existing assignments retain their tested pins; measurement, horizons and safety sweeps remain active.

📄️Resume experiment enrollment

Resume enrollment for conversations created at or after this resume. Previously unassigned conversations from the pause period stay outside the experiment. Existing assignments retain pins.

📄️Publish tested experiment variant

Requires the four reviewed control/variant pins from the experiment readout. Validates the actual candidate under publication locks, selects it as main, publishes through normal checkpoint validation and ends only this experiment in one transaction. Stale pins and retries after publication return conflict. History publication is refused while another experiment is running. Ordinary branch promotion remains draft-only.