Agent Testing API
Test cases and test runs that check an agent against expected behavior.
List test cases
Returns every test case of the agent, optionally narrowed to one application with `applicationId`. Each entry includes its full `evalSpec` document. Use `POST /api/v1/agents/{agentId}/test-cases/{testCaseId}/run/` to execute one of them.
Create test case
The name must be unique within the agent. When `scheduler` is set and the test case is enabled, the platform also runs it automatically on that cron schedule. Trigger it on demand with `POST /api/v1/agents/{agentId}/test-cases/{testCaseId}/run/`. The application scope comes from `applicationId` in the body, or from the query string when the body omits it, and defaults to the agent's own application when both omit it. An agent that is not part of an application needs an explicit `applicationId`.
Import evals from bundle onto existing agent
Imports the evals from an exported agent bundle onto an existing agent, leaving the agent itself untouched. Accepts the same file as `importAgents` — `application/json` or `multipart/form-data` with a `file` field — and reads its `agent.evals` plus the `agent.evalMockBaseline` those evals rely on. Evals are matched by name, so re-importing the same file updates the test cases rather than duplicating them. Nothing is executed: use `POST /api/v1/agents/{agentId}/test-cases/{testCaseId}/run/` to run one. Fails with 422 when an imported eval references an evaluation criterion the target agent does not have.
Get test case
Returns the full test case, including the `evalSpec` document with its user messages, assertions, context variables, and node mocks.
Update test case
Applies a partial update: only the provided fields change. Pass `null` for `description` or `scheduler` to clear them. Returns the updated test case.
Delete test case
Permanently deletes the test case together with its test runs, which cascade with it. Returns the deleted test case.
Duplicate test case
Copies the eval spec, description, scheduler, and self-improvement settings into a new test case. When `name` is omitted, a unique name is derived from the source name plus `duplicateNameSuffix`, for example `Order status (Copy)` or `Order status (Copy 2)`. The application scope comes from `applicationId` in the body, or from the query string when the body omits it.
Run test case
Queues the evaluation and returns immediately with a stable `runId`. The queued run is visible immediately; the worker atomically claims it before evaluation. The evaluation replays the eval spec steps against the agent and records per-step results. Poll `GET /api/v1/agents/{agentId}/test-runs/{runId}/` to track progress and use `POST /api/v1/agents/{agentId}/test-runs/{runId}/stop/` to cancel. The test case must be enabled and associated with an application. Accepts an optional JSON body `{ repeats?, minPassRate?, target? }` to override the eval spec's own values for this run only and/or evaluate a specific branch instead of the current main draft.
Generate test case spec
Accepts an optional JSON body `{ currentTestCase?: { name?, description?, evalSpec? }, additionalInstructions? }` used to seed and steer the LLM generation.
Generate test cases from flow
Reads the agent's live workflow and derives one skeleton test case per decision and action node, without using an LLM (see `POST /api/v1/agents/{agentId}/test-cases/generate-spec/` for LLM generation). Existing test cases are matched by name: matches are updated in place and the rest are created. The response lists each affected test case with the action taken.
List test-case folders
Returns every test-case folder of the agent as a flat list; build the tree client-side from `parentId`. Folders organize the evals shown by the platform UI. Assign a test case to a folder with the `folderId` field on the test-case endpoints.
Create test-case folder
Creates a folder for organizing the agent's test cases. Folder names must be unique among siblings and must not contain '/'. Pass `parentId` to create a subfolder. The application scope comes from `applicationId` in the body, or from the query string when the body omits it — matching the other test-case endpoints.
Update test-case folder
Applies a partial update: only the provided fields change. Pass `parentId: null` to move the folder to the root. Moving a folder inside itself or one of its subfolders is rejected, as are sibling-name conflicts at the destination. The application scope comes from `applicationId` in the body, or from the query string when the body omits it.
Delete test-case folder
Deletes the folder. Its subfolders and test cases are moved to the deleted folder's parent (or to the root), never deleted.
Get baseline node mocks
Baseline node mocks are the fallback underneath every eval of this agent. When an eval runs, its own `nodeMocks` win per node id and the baseline answers for the rest. An agent without a baseline behaves exactly as it did before baselines existed: a node with no mock performs a real webhook call. Returns 404 when the agent has no baseline.
Replace baseline node mocks
Baseline node mocks are the fallback underneath every eval of this agent. When an eval runs, its own `nodeMocks` win per node id and the baseline answers for the rest. An agent without a baseline behaves exactly as it did before baselines existed: a node with no mock performs a real webhook call. This replaces the whole set rather than patching it. Every node id must exist somewhere in the agent tree, including sub-agent workflows; unknown ids are rejected.
Delete baseline node mocks
Removes the baseline. Evals of this agent fall back to their own `nodeMocks`, and any node neither of them mocks performs a real webhook call again. Succeeds even when there was no baseline.
List test runs
`page` is 1-indexed. `limit` is a legacy alias for `pageSize`; invalid or non-positive values fall back to the defaults (page 1, pageSize 10). `fields=full` opts into the heavy payload (per-step results, untruncated reason); anything else keeps the compact projection.
Get daily score trend of agent’s eval suite
Returns one point per UTC day that has at least one completed, scored run, oldest first, with the mean score and pass rate of that day. Days with runs but nothing scorable produce no point. Advisory evals never feed the average; `advisoryTestCaseCount` reports how many were excluded. `days` defaults to 30 and is capped at 90.
Get test run
Returns the run with the parent test case id and name and an `overallScore` averaged from the smart assertion scores. `responseShape=summary` replaces the heavy `stepResults` with one-line step verdicts — the right shape for polling; the default `full` response embeds the complete `stepResults`. While `status` is `RUNNING`, `passed` and `finishedAt` are null, so poll this endpoint until the run finishes. The `applicationId` query parameter is optional: when omitted, it is derived from the run itself.
Delete test run
Permanently removes the run and its recorded results. The run must belong to the agent in the path. The `applicationId` query parameter is optional: when omitted, it is derived from the run itself.
Stop queued or running test run
Runs with status `QUEUED` or `RUNNING` can be stopped; a queued run is cancelled before the worker ever claims it. Stopping aborts the agent executions of the run thread, then marks the run as `ERROR` with the reason `Cancelled by user`. The terminal write is a compare-and-set: if the run finishes first, the response is `409` and the completed result stands. The `applicationId` query parameter is optional: when omitted, it is derived from the run itself.
Get how much of agent’s workflow its eval suite exercises
Returns coverage across three layers — workflow nodes (also broken down by node type), declared transitions, and decision criteria — as covered/total plus the ids never reached. Advisory evals count, because they execute for real. Criteria on decision nodes whose transitions never route on a named criterion score cannot be attributed to a fired transition and are reported separately, excluded from both sides of the ratio.