# AI evals

> Tests you write for your own AI app to check it does its job, before and after changes.

- Level: Intermediate
- Topics: AI
- URL: https://tidbit.legios.com.ar/ai-evals/
- In Spanish: [Evals de IA](https://tidbit.legios.com.ar/es/ai-evals/)

**Evals** (short for evaluations) are the tests of an AI product. You collect a set of realistic inputs, run your system on them and score the outputs automatically. Unlike a public AI benchmark, which compares models on general skills, your evals measure the thing you care about: does the support bot answer refund questions correctly, does the summarizer keep the clause that matters.

Scoring comes in three flavors. Some checks are plain code: is the output valid JSON, does it mention the right order number, does the generated code pass its tests. Some use a second model as the grader, known as **LLM-as-a-judge**, with a clear rubric like "does the answer cite the policy and stay polite?". And some need people, who also make sure the automatic graders agree with human judgment.

Evals matter because an LLM's behavior is hard to predict, and a small change can break things far away. A new prompt that fixes one case may quietly ruin ten others; a new model version may be smarter overall but worse at your format. With a good eval set you can change the prompt, the model or the retrieval and know within minutes whether things got better or worse, instead of finding out from users. A useful set can start small, with a few dozen real examples, and grow every time something fails in production.

## An analogy

A chef tasting every dish before service. The recipe worked yesterday, but nobody assumes it still does: someone tastes each plate against what it should be before it goes out to the dining room.

## Keep in mind

- An eval is a set of test cases plus a way to score them, built for your task.
- Graders can be code, another LLM (LLM-as-a-judge) or people.
- Run them after every prompt, model or data change to catch regressions.
- Benchmarks compare models; evals check your product.

## Go deeper

- [Demystifying evals for AI agents (Anthropic)](https://www.anthropic.com/research/demystifying-evals-for-ai-agents)
- [Define success criteria and build evaluations (Claude docs)](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests)
- [Your AI product needs evals (Hamel Husain)](https://hamel.dev/blog/posts/evals/)

## Related terms

- [Artificial intelligence](https://tidbit.legios.com.ar/artificial-intelligence/): Getting computers to do things that, until recently, only people could do.
- [AI benchmark](https://tidbit.legios.com.ar/ai-benchmark/): A standard test that AI models take so their skills can be compared with a score.
- [JSON](https://tidbit.legios.com.ar/json/): The plain-text format programs use to hand each other structured data.
- [LLM](https://tidbit.legios.com.ar/llm/): A neural network trained on huge amounts of text to predict the next word.
- [Prompt](https://tidbit.legios.com.ar/prompt/): Everything you send to an AI model: the question, the instructions, the context.