Skip to main content

Test agents in CI

An agent's behavior is a promise: it stays concise, it never invents refund terms, it escalates angry customers. Every prompt tweak and routing change can silently break that promise, and you won't find out from the diff. The JSON looks fine. Evals are how the promise becomes enforceable. Each one pins down a behavior you care about, and from then on, no change ships without proving the behavior survived.

That turns evals into more than a test suite. They're the safety rail that makes every other kind of improvement cheap: humans can refactor prompts boldly, AI assistants can edit specs without you re-reading every line, and Docana's auto-improve can propose fixes, because anything that breaks the contract gets caught before a user sees it.

Write eval files​

Each eval file is one scenario: what the user says, and what must be true of the answer. Keep them next to the spec, in an evals/ directory:

{
  "name": "Stays concise",
  "enabled": true,
  "evalSpec": {
    "steps": [
      { "type": "user", "text": "What are your support hours?" },
      {
        "type": "smartAssertion",
        "expectedOutput": "A short answer that offers further help.",
        "evaluationCriteria": "Concise, no fabricated facts.",
        "minScore": 7
      }
    ]
  }
}

Write them for the behaviors that would embarrass you in production: the unhappy paths, the questions outside the agent's scope, the facts it must never invent.

You don't have to write them all yourself: Auto-generate in the agent's Evals tab reads your workflow and proposes test cases covering its branches. Review, keep the good ones, and pull them into the repo so they're versioned with the spec.

See Testing Agents for what makes a good test case, and docana agents eval-schema for the full spec format.

Mock webhooks, assert on silence, repeat flaky cases​

Real webhooks make evals slow, expensive, and risky: a test run that creates a real ticket is not a test. Mock them instead. nodeMocks maps a workflow node id to the response it should return, and the mock reaches every node with that id, including nodes inside sub-agents your flow transfers to:

{
  "nodeMocks": {
    "emit_invoice": {
      "type": "webhook",
      "mode": "sequence",
      "responses": [
        { "status": 400, "body": { "error": "invalid zip code" } },
        { "status": 201, "body": { "data": { "status": "AUTHORIZED" } } }
      ]
    }
  }
}

mode: "sequence" answers each call in order and repeats the last entry once the list runs out. That makes recovery flows testable end to end: the first call fails, the agent asks the user for the correction, the retry succeeds. For a fixed answer on every call, use mode: "staticResponse" with a single response.

The most common integration failure in production isn't an error status — it's the call that gets no answer at all, because the upstream timed out or refused the connection. Write that as { "noResponse": true } in place of a status response:

{
  "nodeMocks": {
    "check_invoice": {
      "type": "webhook",
      "mode": "sequence",
      "responses": [{ "noResponse": true }, { "noResponse": true }, { "status": 200, "body": { "data": { "status": "AUTHORIZED" } } }]
    }
  }
}

Don't reach for a 503 to stand in for this. They are different branches: an unanswered call leaves the agent with status 0 and an empty body, so a transition testing webhookResponse.status >= 400 does not fire, while a 503 makes it fire. Mocking the wrong one gives you a green test for a path production never takes. Add "error" alongside noResponse to set the message recorded in the trace ("error": "connect ECONNREFUSED"); it is diagnostic only and never affects routing.

Share one set of mocks across the whole suite​

Once you have more than a handful of evals, the same fixtures show up in every one of them. On a real suite of 107 cases we counted 1984 mocks written that were only 49 distinct ones, because every case that walks past a node has to carry that node's response. Nine of the 19 mocked nodes answered with exactly the same thing in every single case.

Set those once, on the agent. Open the agent's Evals tab, click Shared mocks, and paste the responses that hold for the whole suite:

{
  "check_invoice": {
    "type": "webhook",
    "mode": "staticResponse",
    "response": { "status": 200, "body": { "data": { "status": "AUTHORIZED" } } }
  }
}

Every eval of that agent now inherits them. An eval that declares its own nodeMocks for a node still wins for that node, so each case only spells out what makes it different. On that same suite of 107, the specs went from 1984 mocks to 165, and 15 of the cases stopped declaring any mock at all.

The same thing over the API, for a repo-versioned suite:

curl -X PUT "$DOCANA_API_URL/api/v1/agents/$AGENT_ID/eval-mock-baseline/" \
  -H "Authorization: ApiKey $DOCANA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"mocks": {"check_invoice": {"type": "webhook", "mode": "staticResponse", "response": {"status": 200, "body": {}}}}}'

Node ids are checked when you save, against the whole agent tree including sub-agents, so a typo fails there instead of at the next run.

A node that neither the shared mocks nor the eval covers still calls the real webhook. That is the behavior evals have always had, and it is on purpose: some cases are meant to hit the real integration. It also means a suite that relies on shared mocks is only safe if they cover every action node, so cover them all if you want a suite that can never reach outside. Shared mocks travel inside docana agents export, which matters when someone imports your bundle: without them, the imported evals declare only their overrides and every uncovered node reaches out to that company's own systems.

Two more step types cover cases a smartAssertion cannot:

  • { "type": "conversationAssertion", "expectNoResponse": false } fails when the previous turn produced no user-visible reply. Use it to guarantee a conversation never dies silently. With expectNoResponse: true it asserts the opposite, for flows that end without a reply on purpose.
  • { "type": "nodeAssertion", "nodeType": "subAgent", "operator": "eq", "count": 0 } counts executions over a whole node class. This example asserts that no transfer happened, without listing every transfer node.

Judge what a node produced, not only what the agent said​

A smartAssertion reads the reply the user sees. That is the wrong thing to read when the deliverable is assembled in code: a javascript node that builds a report, a webhook payload put together from several lookups. The visible reply is a summary. The product is in the node output, and no judge could see it.

Give the assertion an address and it reads the trace instead:

{
  "type": "smartAssertion",
  "nodeId": "build_report",
  "selector": "last",
  "source": "output",
  "path": "result.findings",
  "evaluationCriteria": "Every finding carries the real headline of the article, not a summary, and the outlet named matches the URL domain.",
  "minScore": 7
}

nodeId picks the node, source picks output (what it returned), input (the context sent into it) or record (execution metadata), and path walks inside that, the same way a nodeAssertion path does. selector decides which run to read when the node ran more than once: last (the default) or first.

Without nodeId nothing changes. Every eval you already have keeps judging the reply.

Two things to know before you reach for it. It is judgment, so it still varies run to run: lock the verdict itself with a nodeAssertion and use this for the questions an operator cannot answer, like whether a title describes the article it links to. And it costs tokens in proportion to the value: point path at the field your criteria is about. Above 20000 characters the assertion fails and tells you the size, rather than judging a truncated value.

The failure messages are written for whoever wrote the path. A node that never ran says so and lists what did run. A path that resolves to nothing lists the keys that are there:

path 'result.findings' on output of build_report "Monta report" resolved no value (undefined) — available at 'result': achados, resumo

Agents are stochastic, so one green run proves little. Set "repeats": 3 at the top of the eval spec to run the whole conversation 3 times, and "minPassRate": 0.67 to pass when at least 2 of the 3 attempts pass. The run report shows the aggregate, like 1/3 attempts passed (minPassRate 0.67).

A case you just wrote, or one you know is flaky, doesn't have to hold up everyone else. Set "advisory": true at the top of the eval spec and the case still runs, still reports, and still shows its score, but it can't fail the suite. It appears under its own heading in the report:

Advisory (reported, never gated):
FAIL  evals/escalates-anger.json  6.03s  score 5.00  assertions 1/2  reason: Evaluation assertions did not pass

Use it as a quarantine, not a parking lot. A case that stays advisory for a month is a case nobody trusts.

Run them locally first​

docana agents eval agents/main.agent.json -e "evals/*.json" -a <dev-application-id>

The command imports the spec, runs every matching eval file, and prints a pass or fail per case:

PASS  evals/stays-concise.json  4.21s  score 8.50  assertions 2/2
FAIL  evals/escalates-anger.json  6.03s  score 5.00  assertions 1/2  reason: Evaluation assertions did not pass

Test Suites: 1 passed, 1 failed, 2 total
Assertions: 3 passed, 1 failed, 4 total
Score: avg 6.75, min 5.00, max 8.50
Time: 10.24s

If any case fails, the command exits non-zero. That exit code is the whole CI integration.

One thing to know: eval imports the spec into the application before running, updating the agent's working version. Point -a at a dev application, not the one your users are on. Agent Projects shows how to keep dev and production side by side.

Decide what "the suite passed" means​

By default every case that isn't advisory has to pass. Once a suite is big enough, that rule starts blocking merges over a single stochastic case, so two flags let you say what you actually require:

docana agents eval agents/main.agent.json -e "evals/*.json" \
  --suite-min-pass-rate 0.9 --max-failures 2

--suite-min-pass-rate takes a fraction from 0 to 1, --max-failures a count. Pass both and both have to hold. Advisory cases count toward neither. The report ends with the verdict and the bound that broke:

Test Suites: 8 passed, 2 failed, 10 total
Advisory: 1 total, 1 failed (not gated)
Gate: FAIL
- suite pass rate 0.80 < required 0.9

The command exits 0 when the gate passes, 1 when it fails, and 2 when a flag value is out of range. On a 2 nothing ran: the values are checked before the agent is imported, so a typo costs you no model calls.

Repeat harder in the nightly run than in the PR​

--repeats and --min-pass-rate override what the eval files ask for, for this run only. Nothing is written back to the test cases, so the same files can be cheap on every pull request and strict once a night:

docana agents eval agents/main.agent.json -e "evals/*.json" --repeats 5 --min-pass-rate 0.8

--repeats takes 1 to 10. When a case ran more than once, its report line carries the pass rate, so you can see which cases are stable and which ones scrape by:

PASS  evals/stays-concise.json  18.62s  score 8.50  assertions 2/2  pass rate 0.80 (4/5)

Read the report as data​

--json writes the same report to a file, with raw numbers instead of formatted columns:

docana agents eval agents/main.agent.json -e "evals/*.json" --json eval-report.json
{
  "cases": [
    {
      "name": "Stays concise",
      "file": "evals/stays-concise.json",
      "passed": true,
      "advisory": false,
      "score": 8.5,
      "passRate": 0.8,
      "repeats": 5,
      "attempts": [
        {
          "attempt": 1,
          "passed": true,
          "score": 9,
          "threadId": "cmc3k9p7f0007l608u3jdx4a1"
        }
      ],
      "durationMs": 18620,
      "reason": null
    }
  ],
  "gate": {
    "passed": true,
    "gated": 10,
    "gatedFailures": 0,
    "gatedPassRate": 1,
    "advisoryFailures": 1,
    "violations": []
  }
}

Upload it as a build artifact and you can answer questions the exit code can't: which case has been drifting down in score for three weeks, which one is slow enough to be worth mocking, which attempt of a flaky case failed and in what thread. Each threadId opens in the platform, so a failed attempt is one link away from its full conversation.

When evals fail, Docana suggests the fix​

A failing eval isn't just a red X. It's input. In the agent's Evals tab, Auto-improve analyzes the failures and proposes concrete changes to the spec: a tightened reasoning, a rewritten instruction, a routing fix. You review each suggestion and apply the ones that make sense, like a code review from someone who just read all your failing tests.

The loop closes back into git: apply a suggestion in the platform, docana pull to bring the improved spec into the repo, and re-run the evals to confirm the fix holds. The agent gets better, and the eval that caught the problem stays behind as a permanent regression guard.

Wire it into GitHub Actions​

Two commands set the secrets (or add them in the repo's Settings → Secrets):

gh secret set DOCANA_API_KEY
gh secret set DOCANA_APPLICATION_ID

Then save this as .github/workflows/agent-evals.yml:

name: Agent evals
on: [pull_request]

jobs:
  eval:
    runs-on: ubuntu-latest
    env:
      DOCANA_API_URL: https://platform.docana.com
      DOCANA_API_KEY: ${{ secrets.DOCANA_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm install -g @docana/cli
      - run: docana validate "*"
      - run: docana agents eval agents/main.agent.json -e "evals/*.json" -a ${{ secrets.DOCANA_APPLICATION_ID }}

docana validate catches schema mistakes in seconds. The eval step catches behavior regressions: the prompt tweak that made the agent stop offering refunds, the routing change that broke escalations. A pull request that changes behavior should also change an eval, and reviewers see both together.

If you started from the template repository, the workflow file is already in the repo. Setting the two secrets is all that's left.

Publish only what passed​

Users talk to the agent's live version. Pushes only update the working version. So let CI gate the promotion: after evals pass on the main branch, push and publish in one step:

docana push -y --publish

A change that fails its evals never reaches a user. That's the whole point.

Watch runs from the terminal​

Evals also run when triggered from the platform. Follow them without leaving your shell:

docana agents eval-run list --agent-id <agent>
docana agents eval-run watch --agent-id <agent> --run-id <run-id>

list shows recent runs with their ids, score and pass rate. watch follows one live until it finishes. A run whose assertions were all pass or fail has no numeric score, and shows a dash in that column.

Next steps​