Skip to Content
Baseera Docs
Evaluations

Compare versions

Run one set of tests on two to four setups, a version and a model each, and read quality, cost and speed side by side.

Use Compare setups to answer questions like "does the new prompt beat the old one?" or "is the cheaper model good enough?". Each setup is an assistant version (or the draft) on a model of your provider key. The same set runs on every setup, and you read the results side by side.

Owners, admins and developers can compare. It runs tests on your own provider key, so viewers cannot see it.

Run a comparison

On the Evals page, press Compare setups.

Choose the Set of tests. The built-in safety tests can be compared too.

Under Setups to compare pick 2 to 4. For each, choose a version and a model. The same prompt on two models shows what the model changes. Two versions on one model show what the prompt changes. Use Add a setup and Remove setup N.

Press Review the total. You see the number of tries ("N tests × N tries × N setups"), the cost per setup and Total: about N on your key. Nothing is spent until you confirm.

Press Confirm and start. This starts one run per setup.

Every setup must use the same provider as the assistant. Your assistant's saved settings are not changed.

Read the results

The Results table has one row per setup: Pass rate, Judge score, Cost per turn, Average time, Slowest 5% (p95) and Tries. Tries that errored or never ran are counted apart and left out of every average. Best pass rate, then cost marks a suggestion, not a verdict.

Where setups disagree lists tests that passed every try in one setup and failed every try in another. Tests that passed on some tries and failed on others show as Mixed and are not counted as disagreement. Open Open the run for any setup to see its attempts.

Use the winner

Press Use this setup on a finished setup, then Save as draft. Your draft takes that setup's model (and, for an earlier version, its prompt and settings). Any unpublished changes in your current draft are replaced. Nothing is published. You review the draft and publish it from Deploy, where the required tests run first.

What can go wrong

MessageCause and fix
"Two setups are the same. Change the version or the model of one."Make each setup different.
"Every setup must use the same provider as the assistant."One of the versions uses another provider.
"A model here is not one your assistant's provider key offers."Pick a model from the list.
"A comparison is at most 1,000 tries in all: tests times tries times setups."Remove a setup or lower the tries.
"A run is in progress for this agent. A comparison can start when it has finished."Wait for the run.
"The estimate changed since you reviewed it."Check the new total and confirm again.
"Only a finished run can be used."Wait for the comparison to finish before Use this setup.
"Stopped early: the numbers cover the tries that finished."You pressed Stop the comparison.

Next steps

On this page