Online evaluation
Watch quality in real use, set alerts, and optionally judge a sample of live replies.
Tests tell you if the assistant is ready. Online evaluation tells you how it is doing with real users. It lives on the agent's Overview page and calls no model unless you turn on live judging.
Read Quality from real use
The Quality from real use tile shows two numbers over the last 7 days, each against the 7 days before:
- Thumbs down: the share of finished replies that a person rated down.
- Tool errors: the share of tool calls that failed or timed out.
You see "No replies have been rated yet" until your users press the thumbs in the chat, and "No tool calls have ended yet" until tools have run. See Analytics and feedback.
Set alerts
The Alerts tile is on when the latest 24 hours cross a threshold. Alerts are worked out when you open the page. Nothing is sent anywhere, so make opening the Overview part of your routine.
Find Alert thresholds on the Overview page. Each metric has its own row: Thumbs-down rate, Tool error rate, Draft rejection rate and Live judging failure rate.
Set any of:
- Ceiling (%): alert when the latest 24 hours reach this share. Leave it empty for no ceiling.
- Rise (times): alert when the latest 24 hours reach this many times the rate of the 7 days before. For example 2 for a doubling.
- Minimum events: say nothing until at least this many replies, tool calls, decisions or judgments were counted.
Press Save.
With no setting, a metric alerts at 2 times the week before, with at least 20 events. A rate of one in two is not news, so small samples stay quiet. Viewers can read the thresholds but not change them.
Judge live sessions (optional)
Judge live sessions scores a small random sample of finished replies from real users against the rubric of one of this agent's sets of tests. It is off by default.
First make sure one set of tests has a rubric. See Model judge. If none has, the card says "No set of tests has a rubric yet."
Switch on Judge a sample of live sessions.
Set Share of replies (%), from 0.01 to 100. The database picks which replies by a fixed, secret hash of the turn, so the same reply always gets the same answer and nobody can steer the choice.
Set Monthly cap (USD), required, from 0.01 to 10,000 in whole cents. No judgment starts if it could pass the cap.
Choose Rubric from a set of tests, then press Save.
The card shows Judged, Met the rubric, Below the rubric, Not judged, Spent this month, and Recent replies below the rubric. When the cap is reached it says so and no more sessions are judged until next month or until you raise it.
Only the user's message, the reply and the names of the tools called are sent, to your own provider. Only Owners and Admins can turn this on, and if your organization requires two-factor you must have entered your code first.
What can go wrong
| Message | Cause and fix |
|---|---|
| "These numbers could not be loaded. Reload the page to try again." | A temporary failure. |
| "Check the numbers: a ceiling of 0.01 to 100 percent, a rise of 1.1 to 20 times, and at least 1 event." | Fix the thresholds. |
| "A rule that is on needs a ceiling or a rise." | Set at least one. |
| "The share must be from 0.01 to 100 percent." / "The cap must be from 0.01 to 10,000 dollars, in whole cents." | Fix the value. |
| "Choose a set of tests of this agent that has a rubric, and make sure the agent has a model on an active key." | Add a rubric and check the model. |
| "Set up two-factor sign-in first: your organization requires it for admins." | Open Security and enrol. |