Skip to main content

Testing agents

An agent that talks to your customers is production software. Test it like production software. Docana lets you write test cases for an agent, run them after every change, and see exactly which step broke.

Find it in your application: Agents, pick an agent, then open its Evals tab.

Test an employee from preparation​

Open Employees, choose an employee, and select Test expectations to run its saved agent scenarios. Results & activity → Expectations shows recorded outcomes and recent runs. These use the same evaluation system described below. See the employee testing walkthrough.

Robin has 3 recorded passing scenarios in this local invoice example. Run the tests again after changing the employee.
Robin has 3 recorded passing scenarios in this local invoice example. Run the tests again after changing the employee.

Create a test case​

  1. Create an eval from the agent's Evals page. In Basic Information, give it a name and describe the behavior to check.
  2. Open Steps, choose Start manually, and enter a User message.
  3. Use Add Step to add node or conversation assertions, or a smart assertion. Each assertion describes a condition the run must satisfy.
  4. Use Runtime Data for initial context and node mocks, then click Create. Mocks provide canned responses for tools so a test does not need to call the real service.

Responses that hold for every test case belong on the agent instead of in each case. Click Shared mocks in the Evals tab and set them once. Every test case inherits them, and a case that declares its own mock for a node still wins for that node, so each one only spells out what makes it different. A node covered by neither still calls the real service.

Name the eval and describe the behavior it should validate. Reliability settings control repeats and the required pass rate.
Name the eval and describe the behavior it should validate. Reliability settings control repeats and the required pass rate.
The Steps tab pairs user messages with assertions. This draft checks that the conversation receives a response.
The Steps tab pairs user messages with assertions. This draft checks that the conversation receives a response.
Runtime Data keeps initial context and node mocks alongside the eval. Configure tool mocks before running a test with external actions.
Runtime Data keeps initial context and node mocks alongside the eval. Configure tool mocks before running a test with external actions.

Run your tests​

Run a single test case or all of them. Each run shows pass or fail per assertion, with the full execution result behind it: what the agent understood, which branches it took, what it answered.

Run your tests after every meaningful change to the agent, the same way you'd run a test suite after changing code.

Generate test cases automatically​

Click Auto-generate, choose how many test cases you want, and Docana proposes cases based on your workflow. Review the assertions before saving, and add missing cases for your business rules and less common paths.

Let Docana suggest fixes​

When tests fail, Auto-improve analyzes the failures and suggests changes to your agent. Review each suggestion and apply the ones that make sense. Run the tests again after applying a change.

Debugging a failure​

The Logs tab keeps every execution, step by step. Open a run and you can trace the exact path: which evaluation fired, what it scored, which branch the agent took, and what each node produced. Compare the trace with the expected path to identify the first step that diverged.

Best practices​

  1. Test the unhappy paths: angry users, missing information, questions outside the agent's scope. Include successful requests too, so a fix for one branch does not break another.
  2. Mock external calls: use node mocks for webhooks and emails so tests are fast and don't spam real systems. Put the responses your whole suite shares in Shared mocks and let each case override only what it needs.
  3. Make assertions specific: "response mentions the refund deadline" catches more bugs than "response is not empty".
  4. Run before you publish: check the updated agent against your saved cases, then review representative answers and their sources.

To compare two versions on live traffic, see A/B testing.

Next steps​