Agent Testing

Test cases and test runs that check an agent against expected behavior.

📄️List the test cases of an agent

Returns every test case of the agent, optionally narrowed to one application with `applicationId`. Each entry includes its full `evalSpec` document. Use `POST /api/v1/agents/{agentId}/test-cases/{testCaseId}/run/` to execute one of them.

📄️Create a test case for an agent

The name must be unique within the agent. When `scheduler` is set and the test case is enabled, the platform also runs it automatically on that cron schedule. Trigger it on demand with `POST /api/v1/agents/{agentId}/test-cases/{testCaseId}/run/`.

📄️Get a test case of an agent

Returns the full test case, including the `evalSpec` document with its user messages, assertions, context variables, and node mocks.

📄️Update a test case of an agent

Applies a partial update: only the provided fields change. Pass `null` for `description` or `scheduler` to clear them. Returns the updated test case.

📄️Delete a test case of an agent

Permanently deletes the test case together with its test runs, which cascade with it. Returns the deleted test case.

📄️Duplicate a test case of an agent

Copies the eval spec, description, scheduler, and self-improvement settings into a new test case. When `name` is omitted, a unique name is derived from the source name plus `duplicateNameSuffix`, for example `Order status (Copy)` or `Order status (Copy 2)`.

📄️Run a test case of an agent

Creates a test run in `RUNNING` status, schedules the evaluation in the background, and returns immediately with the `runId`. The evaluation replays the eval spec steps against the agent and records per-step results. Poll `GET /api/v1/agents/{agentId}/test-runs/{runId}/` to track progress and use `POST /api/v1/agents/{agentId}/test-runs/{runId}/stop/` to cancel. The test case must be enabled and associated with an application.

📄️Generate a test case spec from the agent flow

Accepts an optional JSON body `{ currentTestCase?: { name?, description?, evalSpec? }, additionalInstructions? }` used to seed and steer the LLM generation.

📄️Generate or update test cases from the agent flow

Reads the agent's live workflow and derives one skeleton test case per decision and action node, without using an LLM (see `POST /api/v1/agents/{agentId}/test-cases/generate-spec/` for LLM generation). Existing test cases are matched by name: matches are updated in place and the rest are created. The response lists each affected test case with the action taken.

📄️List the test runs of an agent

`page` is 1-indexed. `limit` is a legacy alias for `pageSize`; invalid or non-positive values fall back to the defaults (page 1, pageSize 10). `fields=full` opts into the heavy payload (per-step results, untruncated reason); anything else keeps the compact projection.

📄️Get a test run of an agent

Returns the full run, including `stepResults` with the per-step outcomes, the parent test case id and name, and an `overallScore` averaged from the smart assertion scores. While `status` is `RUNNING`, `passed` and `finishedAt` are null, so poll this endpoint until the run finishes. `applicationId` is a required query parameter.

📄️Delete a test run of an agent

Permanently removes the run and its recorded results. The run must belong to the agent in the path, and `applicationId` is a required query parameter for the access check.

📄️Stop a running test run of an agent

Only runs with status `RUNNING` can be stopped. Stopping aborts the agent executions of the run thread, then marks the run as `ERROR` with the reason `Cancelled by user`. `applicationId` is a required query parameter.