Skip to Content
Baseera Docs
Evaluations

Test suites

Write tests as sentences, group them into sets, mock your tools, run them and read the results.

A set of tests (a suite) groups tests about one topic, such as billing or safety. A test says: when a customer says this, the assistant should do that. Tests never call your real tools and never change your data.

Write a test

Open Evals and press Add a test. Choose A blank test, From a real conversation or Ready-made safety tests.

Under When a customer says... write what the customer says.

Under The assistant should... add what must happen. The test passes when all of it holds. Choose from:

GroupExpectations
ToolsUse a tool, Not use a tool, Ask for approval before changes
ReplySay this text, Not say this text, Reply in a language, and the advanced Reply in a pattern and Not reply in a pattern (regular expressions)
LimitsFinish within a number of steps, Stay within a text size, Stay under a cost, Answer within a time
How it endsFinish with an answer

These checks look only at what the assistant did. No model is called to grade them. For a quality judgement, add a rubric (see Model judge).

Pick the Set of tests, or choose New set of tests... and give it a name and tags. Tags are lowercase letters, digits, - and _, separated by commas.

Press Save test.

Without any expectation, a test passes whenever the assistant answers without an error.

Open More options to set a name, earlier messages in the conversation, who is asking (User role, Locale, Customer account), extra information from the app, tool mocks and personal data masking.

Save a real conversation as a test

Open a session under Sessions, then press Save as test case on the turn you want to check. Review the conversation before saving. It is a real conversation, so use Mask personal data to replace emails, phone numbers, IBANs, card numbers and ID numbers with placeholders such as [email-1], then check the result.

A test cannot be built from a turn when the agent does not store tool data, when the turn's record was cut at its size limit, or when earlier tool data was redacted. Turn on Store tool inputs and results on the Assistant page and record a new session. A thumbs-down turn has Save as test case next to the feedback.

Mock your tools

A test never calls your real tools, so each tool the assistant may call needs a mock. A call to a tool without one fails the test with "No mock for ...".

Mock typeBehaviour
RecordedThe calls from the saved session, answered in order.
FixedEvery call gets the same result.
Matched by inputThe first rule whose input matches answers the call. Add Answer any other input as a fallback.

A mock result is either an Output (JSON) or an Error (the tool fails). For draft and write tools, choose whether the simulated user Approves or Reject with a reason. The default is approve. When a tool changes after you made a mock, the page flags it ("changed since this mock was made").

Run a set

Open the set and press Run.

Choose the Assistant version (production, an earlier version or the draft), the Model and Tries per test. Use 2 to 5 tries to catch answers that change from one try to the next.

Press Review. You see the number of tries, About N on your key, and the judge's share if the set has a rubric.

Press Confirm and start.

Read a run

The run page shows Pass rate, Attempts passed, failed or with errors, Tests passing every attempt, Tokens and Estimated cost. Compared with the previous run lists Newly failing and Newly passing tests.

Under Results, failing tests come first. Each test is Passing, Failing, Error or Flaky (passed on some tries and failed on others). Open one to see every attempt, the checks that failed (Expected against What happened) and Show trace for the steps, tool calls and approvals.

Press Stop run to stop early. The results show what finished.

An attempt that could not finish, for example because of a missing or expired key, is an error and is not the assistant's fault.

What can go wrong

MessageCause and fix
"No mock for ..."Add a mock for that tool in the test.
"A suite with this name already exists for this agent."Choose another name.
"A limit was reached: 100 suites per agent, 500 cases per suite."Delete old suites or tests.
"A run is already in progress for this agent."Wait for it or press Stop run.
"You have reached the limit of 30 runs an hour for this agent."Wait, or lower the number of runs.
"A run is at most 500 tries: tests times tries per test."Lower Tries per test.
"The agent needs a provider key before it can run tests."Add a key on the Assistant page.
"The cost can't be estimated: this model has no known price."The run still works. The estimate is just unavailable.
"Only owners, admins and developers can change tests."Ask for a different role.

Next steps

On this page