Skip to Content
Baseera Docs
Evaluations

Human review

Label answers as correct, wrong or unsafe, measure how far to trust the judge, and turn mistakes into tests.

People who know your business are the ground truth. In Human review they label answers as correct, wrong or unsafe. The labels show how far to trust the model judge, and a wrong answer with a corrected reply can become a test.

Only Owners, Admins and Developers can open the queue, because it shows what customers wrote. Viewers see "Reviewers only".

What is in the queue

The Waiting for a label queue holds, from the last 90 days:

  • Answers where a customer gave a Thumbs down.
  • Answers the judge scored below its rubric (Low judge score), in live sessions and in test runs.
  • A small random sample of the rest, about 3 percent (Random sample).

Label answers

Open the Evals page and press Review answers in the Human review tile. The tile says how many answers are waiting.

Filter by reason if you like, and between To label and Labelled.

Read What the customer asked, What the assistant answered and, if there is one, What the customer said about it. Press Open the conversation or Open the run for context.

Choose Your label:

  • Correct: right and fine to send. Counts as a pass.
  • Wrong: incorrect, incomplete or off-topic. Counts as a fail.
  • Unsafe: could cause harm by breaking a rule, leaking something private or promising what it must not. Counts as a fail.

Add a Note about why. Other reviewers see it. Do not paste personal data or secrets.

For a wrong or unsafe answer, write the Corrected answer: a phrase a good reply must contain, up to 500 characters.

Press Save label. You can come back and Update label.

Turn a label into a test

On a labelled wrong or unsafe answer with a corrected answer, press Make a test. The new test also checks that the assistant's reply contains your corrected answer, and that check cannot be changed in the test. "A test was already made from this label" appears if you press it twice.

Check the judge against people

On the judge's card, Agreement with human reviewers shows the share of labelled answers where the judge's pass or fail equals the reviewers'. An answer counts as a pass only if every reviewer said correct. It appears once at least 10 answers are labelled. The queue favours answers the judge failed, so read the percentage together with the number of answers.

What can go wrong

MessageCause and fix
"Nothing is waiting for a label."The queue is empty. Check back after more sessions or runs.
"Choose a label, and keep the note and the corrected answer within their limits."Pick one of the three labels and shorten the text.
"You can't review this agent."Your role does not allow review.
"Not enough labels yet (N of 10)."Label more answers.

Next steps

On this page