Skip to Content
Baseera Docs
Evaluations

Model judge

Score answers against a rubric of your own standards, and choose the model that judges them.

Some qualities cannot be checked with a text match: "cites the policy section", "does not promise a refund before approval". For those, add a rubric. A model reads the assistant's answer and gives each criterion a score from 1 to 5 with a short reason.

The judge runs on your own provider key, and its cost is billed to you and included in every run's estimate.

Add a rubric to a set

Open a set of tests and find Rubric.

Under Criteria write one criterion per line, from 1 to 10 criteria of 3 to 300 characters each. Write what a good answer does, for example:

Cites the policy section
Does not promise a refund before approval

Choose the Pass threshold, from 1 to 5. Every criterion must score at least this for the test to pass. 4 is a good start.

Press Save rubric. Leave Criteria empty to remove the rubric.

A single test can have its own Rubric for this test, which replaces the set's rubric for that test only. A test with no rubric of its own uses the set's.

Choose the judge model

Under Judge model, pick the model that scores answers. Same as the assistant is the default. A stronger model judges more reliably and costs more. Only Owners and Admins can change it. If the assistant has no provider key, the picker says "Add a provider key to this assistant to choose a judge model."

Read a judged result

In a run, a rubric appears as a check called Judge's rubric. It shows Rubric met or Rubric not met, "pass at N of 5", the score of each criterion, and "of which the judge" for the cost. The overview reason for a failure reads "The answer scored too low against your rubric."

Trust the judge, but verify

A judge is a model, and models are sometimes wrong. Check it with people. Label a sample of answers in Human review. Agreement with human reviewers shows how often the judge's pass or fail matches the reviewers' once at least 10 answers are labelled.

When you compare setups, one judge scores every setup, so the scores are comparable.

What can go wrong

MessageCause and fix
"The rubric is not valid. Check the number of criteria, their length and the threshold."Use 1 to 10 criteria of 3 to 300 characters and a threshold from 1 to 5.
"You are not allowed to change this rubric."Owners, admins and developers can edit rubrics.
"Only owners and admins can change the judge model."Ask an owner or admin.
"Includes the judge scoring N answers. Its price is not known, so the total can't be estimated."The judge model has no list price. The run still works.

Next steps

On this page