Evaluations
Test your assistant before each release and keep an eye on it afterwards, with tests, a judge, live monitoring and human review.
Evaluations answer one question: is my assistant ready? You write tests as sentences, run them on a draft or a version, and read one answer. You then keep watching real use after you publish.
Open Evals in the agent's sidebar. The top tile, Is my assistant ready?, gives one answer taken from the latest run of every test:
| Answer | Meaning |
|---|---|
| Ready | Every test passed in its latest run. Run everything again after you change the assistant. |
| Needs attention | Something went wrong in the latest run. The What went wrong list says what and links to the conversation. |
| Running | Checking now. The page updates by itself, and you can leave. |
| Not tested yet | Some sets of tests have not run. Press Run everything. |
| No tests yet | Add a test to find out. |
Owners, admins and developers can write and run tests. Viewers cannot see them.
The pieces
Test suites
Model judge
Online evaluation
Human review
Compare versions
Run everything
Press Run everything in the bar above the page.
Choose the version in More options if you want something other than the default, and the number of tries. The dialog summarises "N sets of tests and N search tests. N tries in all." Tests use the model the assistant uses now.
Press Run everything. Progress shows on the page and the Is my assistant ready? answer updates when the runs finish.
Test runs call your own provider key and are billed by your provider, not by Baseera. The dialog shows an estimate before anything starts.
Built-in safety tests
Press Add a test and choose Ready-made safety tests, or open the Safety suite that Baseera maintains. It is generated from your agent's own tools and checks approvals, instructions hidden in tool results, other users' data, invented results, reply language and scope, in Arabic and English. You can run it and require it before publishing, but you cannot edit it. After your tools change, press Update the suite. The agent's Overview shows a Safety score from its latest finished run.
Search tests
Search tests check that search finds the right source for a question, before the assistant writes any answer. You save questions from Test retrieval on a knowledge base (see Knowledge), then run them from the Evals page. A run shows Hit rate, Found first, Found in the top 3 and Rank score, and compares itself with the previous run so you can see whether a change to chunking, the embedding model or the sources helped. These runs embed each question once on your embedding key and call no chat model.
Gate publishing on tests
Choose which suites must pass, and the minimum pass rate, on the agent's Deploy page. A required suite must reach that rate on exactly the draft you are publishing. See Preview and publish.
Limits
- 100 suites per agent and 500 tests per suite.
- 30 test runs per agent per hour, and one active run at a time.
- A run is at most 500 tries (tests times tries per test).