What are AI evals?
AI evals are structured tests that measure how well an AI model or agent performs on representative tasks, scoring outputs for accuracy, helpfulness, safety and policy compliance, so teams can compare versions and catch regressions before release.
Types
| Type | How |
|---|---|
| Golden set | Compare to expected answers |
| LLM-as-judge | Another model scores outputs |
| Human review | Experts grade samples |
| Task success | Did the action work? |
| Safety tests | Red-team prompts |
Building an eval set
Collect real tickets, write expected outcomes, include edge cases and risky requests, and rerun on every change.
