Evals is short for evaluations. Anthropic gives a plain definition: an eval is a test for an AI system — give the AI an input, then apply grading logic to its output to measure success, with the emphasis on automated evals that can run during development without real users.
What they address is a class of problem traditional testing can't: the same prompt may produce different results today and tomorrow, and swapping a model version or changing one line of retrieval logic can quietly degrade output without raising any error.
Grab It in One Sentence First
Evals turn "is the AI output any good" into a reproducible, comparable score, so before and after a change can be compared.
An everyday analogy is a restaurant's quality standard and tasting routine. The chef changed, the supplier changed, the pan changed — the dish still looks like the same dish, but the taste may already have shifted. Relying on the owner tasting a spoonful now and then isn't reliable; you need a fixed tasting checklist and a scoring sheet to run after every change. Evals are that checklist and scoring sheet for an AI application.
Why This Term Emerged
The typical failure story goes like this: a team builds a promising prototype, tests it with a dozen hand-picked prompts, decides it "looks good," and ships. Problems start after launch, fixing one failure case surfaces another, and the team ends up in a reactive loop with no idea whether overall quality is going up or down.
A fairly common assessment in 2026: the main bottleneck limiting reliable AI deployment isn't model capability but evaluation methodology. Anthropic puts it bluntly in its own guide — teams with evals can complete a model upgrade in days, while teams without face weeks of manual testing.
Agents make this harder. A single question-and-answer only requires looking at one output, whereas an Agent runs many turns, calls tools, modifies state, and adapts based on intermediate results — so "was it right" isn't just about the final sentence, but about what it did along the way.
flowchart LR
Data["Eval set<br/>inputs + expectations"] --> Run["System under test<br/>prompt / model / retrieval"]
Run --> Out["Output and trajectory"]
Out --> Grade["Grading<br/>rules / model judge / human"]
Grade --> Score["Scores and failure cases"]
Score -->|block on regression| CI["CI gate"]
Score -->|new failure modes| DataWhat It Usually Includes
An evaluation setup generally rests on three things. First, an eval set: a collection of samples with inputs and expectations, commonly called a golden dataset. Second, metrics: a clear statement of what's being measured this time — accuracy, format compliance, whether citations actually exist, whether the tool-call sequence was right. Third, a judge mechanism: whoever does the scoring.
Judging falls into roughly three kinds. Programmatic evaluators are deterministic code checks: does the JSON parse, are the fields complete, is the number in range, do the tests pass. These are the cheapest and most reliable, and should be the first choice wherever they apply. Model-based evaluators (LLM-as-judge) have another model score against a rubric, which suits tasks with no single correct answer — summary quality, appropriateness of tone. Human evaluators cost the most but catch subtle issues automated methods miss, and are usually used to sample-calibrate the first two.
One engineering piece is regularly underrated: traceability. A score has to map back to which version of the prompt, which model, and which dataset produced it, or you'll have nowhere to start when the number drops.
The Difference from Traditional Testing
A unit test asserts a deterministic result: input A must produce output B, and anything else fails. Evals deal with probabilistic output, where two runs of the same input may both be correct but worded differently, so the check is usually not "equals" but "does it satisfy certain properties" or "is the score above a threshold."
Another difference is that the eval set itself expires. Production keeps surfacing failure modes the original dataset never covered, and without continuously feeding real cases back in as new samples, the set grows blind spots — full marks on the suite while incidents continue in production. That's quite unlike traditional tests, which stay valid once written.
Its Relationship to RAG, Agents, and Loop Engineering
Evaluating a RAG system usually needs to be split in two: did retrieval bring back the right material, and did generation faithfully use it. Measuring the halves separately is what tells you which end the problem is at.
Evaluating an Agent means looking at the trajectory, not just the result: was the right tool chosen, was there a detour, could it recover after a failure, did it take any action beyond its permissions. Unattended loops of the Loop Engineering kind treat evaluation as the brake outright — a loop without automatic acceptance is just bulk generation of unchecked output, so in those systems evals aren't a bonus but a requirement.
Where It's Easy to Misunderstand
The first misconception is "an LLM judge is enough." Model judges have known biases: preferring longer answers, favoring output from their own model family, being sensitive to ordering. They need sample calibration against human labels and a clearly written rubric — not just "score this answer" and done.
The second is chasing a single aggregate score. A nice-looking total can't hide a structural problem like "99% format compliance but only 60% citation validity." Reading per-dimension is far more useful than reading the total.
The third is treating the eval set as a one-time deliverable. It behaves more like a test suite, needing continuous expansion as new failure cases appear; note too that eval sets saturate as models improve — frontier model scores on SWE-bench Verified went from around 40% to over 80% within a year, so questions that still discriminated six months ago may be full marks for everyone today.
The fourth is getting stuck on tool selection. There are many evaluation platforms, and most comparison articles are written by vendors ranking themselves first — read the definitional material separately from the rankings.
How to Decide Whether to Use It
If an AI feature will keep iterating, it's worth having evals. Priority comes down to three things: how often it changes (prompts, models, or retrieval logic changing frequently makes evals mandatory), how costly a mistake is (money, medical, legal, or anything published externally warrants doing this early), and whether you can currently answer "compared with last week, did quality go up or down?" — if you can't, that's the signal.
Starting small is fine. Begin with twenty or thirty real cases covering typical scenarios and known failure modes, paired with the cheapest programmatic checks; once that runs, add model judging and human sampling, and only then consider wiring it into CI as a release gate. A small eval set that runs every day beats a full platform built up front.