---
Title: "What Are AI Evals? How to Evaluate AI Agents"
Url: "https://computer.io/glossary/ai-evals"
Topic: "AI safety"
Description: "AI evals are structured tests that measure how well an AI model or agent performs on real tasks. Types, metrics and how to build an eval set."
---

# What are AI evals?

AI evals are structured tests that measure how well an AI model or agent performs on representative tasks, scoring outputs for accuracy, helpfulness, safety and policy compliance, so teams can compare versions and catch regressions before release.

## Types

| Type | How |
| --- | --- |
| Golden set | Compare to expected answers |
| LLM-as-judge | Another model scores outputs |
| Human review | Experts grade samples |
| Task success | Did the action work? |
| Safety tests | Red-team prompts |

## Building an eval set

Collect real tickets, write expected outcomes, include edge cases and risky requests, and rerun on every change.

## FAQ

### What is LLM-as-a-judge?

Using a model to grade another model's output against criteria.

### How many examples do you need?

Start with dozens to a few hundred representative cases.

### How often should you run evals?

Before every change to prompts, models, knowledge or tools.

## Related terms

- [AI observability](https://computer.io/glossary/ai-observability)
- [AI hallucination](https://computer.io/glossary/ai-hallucination)
- [AI agent builder](https://computer.io/glossary/agent-builder)
- [How do you measure AI agent performance in customer support?](https://computer.io/answers/how-to-measure-ai-agent-performance-in-support)

## How Computer helps

- [Agent studio](https://devrev.ai/agent-studio)

## Further reading

- [Agent studio](https://devrev.ai/blog/agent-studio)