AI Eval / Evaluation Pipeline Guide

Why eval first

Why you build eval first

Improving an LLM with no measurement is steering by feel

"You can't improve what you can't measure." Change a prompt, bump the model, tweak RAG — but if you don't know whether it got better or worse, you're just rolling dice. LLM systems have infinite inputs and probabilistic outputs, so eyeballing a few outputs and saying "looks fine" isn't evaluation. So the real first step isn't building the feature — it's standing up an eval set and a measurement pipeline first. With eval, a change is provably an improvement or a regression, and only then can you iterate fast.

	Without eval	Eval first
Judging a change	"Feels better"	A score tells improvement vs. regression
Iteration speed	A human eyeballs it every time	An automated regression gate, instantly
Deploy safety	Discovered as an incident in prod	Regressions blocked before deploy

Architecture

The 3-tier evaluation architecture

Cheap to expensive — each tier filters, and you use less of the higher ones

1

Automated metrics & benchmarks — cheap and fast

Things with rules or a known answer

Accuracy, regex matching, code execution, retrieval metrics (recall/precision) — anything scored instantly by machine. Near-zero cost, so run it on every change in CI. The workhorse of your first regression gate. Limit: only works on tasks with a "fixed answer."

2

LLM-as-Judge — mid cost, broad reach

Scoring the things with no single answer

A model scores things that don't reduce to one answer — summary quality, tone, helpfulness, groundedness — against a rubric. Strong models reach 80–90% agreement with humans (depends on task and setup). It fills the gap automated metrics can't, but it's biased — see the LLM-as-Judge section.

3

Human evaluation — expensive and slow, but the anchor

Ground truth

Judgment from experts and real users. It's the most expensive and slowest, so spend it sparingly — building the gold set, calibrating the judge (checking it agrees with humans), quarterly quality reviews. To trust tiers 1 and 2, you ultimately anchor here.

The point is a pyramid — the cheap, fast automated metrics (1) the most, LLM-judge (2) in the middle, humans (3) the least. Higher is more accurate but costlier, so only escalate what the lower tier filtered. And periodically calibrate the judge (2) with humans (3) to keep it trustworthy.

Deep dive

LLM-as-Judge — powerful, but biased

Use it knowing it tends to overstate its own accuracy

Bias	Symptom	Mitigation
Position bias	When comparing two answers, the verdict shifts with front/back placement (reported around 10–15%)	Score twice with order swapped and average · allow ties
Verbosity bias	Scores long, wordy answers higher	Control for length or require conciseness in the rubric
Self-preference	Prefers answers written by same-family models	Separate judge from generator · cross-validate
Overconfidence	High-confidence scores even on wrong answers	Measure judge accuracy against a human gold set first

✓

How to use it well

The judge is itself something to evaluate

① A clear rubric — not "good/bad" but concrete scoring criteria. ② CoT + rationale — make it write the reasoning, not just the score, for more consistency. ③ Pairwise comparison over absolute scoring is generally more stable. ④ Validate the judge itself against a gold set — know numerically "how well the judge agrees with humans" before you start. Without that, there's no basis to trust its scores.

Tooling

The tooling landscape

You don't have to build it all — pick by role

Tool	Strength	Mainly used for
DeepEval	pytest-style LLM unit tests	Eval as code in a CI regression gate
Ragas	RAG-specific metrics (faithfulness, context recall, etc.)	Retrieval and groundedness evaluation
Langfuse	Observability (traces) + evaluation	Evaluating on top of production logs
Phoenix (Arize)	Hallucination and drift tracking	Production monitoring and root-cause analysis
Braintrust	Eval + experiment logging/comparison	A/B comparing prompts and models

* The tooling landscape moves fast (June 2026 snapshot). Choosing by role (testing, RAG, observability, experiments) outlasts any tool name.

Anti-patterns

5 Common Anti-patterns

Ways to reach a wrong conclusion even while evaluating

!

Build the feature first, eval later

Develop for weeks with no measurement, then have no basis to judge whether it improved — so you decide by feel.

→ A small eval set first. Improvement starts only once you can measure.

!

Trusting judge scores unvalidated

Taking the LLM-judge's score as truth. You've never measured how well the judge agrees with humans.

→ Measure judge accuracy against a human gold set first, and recalibrate periodically.

!

Ignoring position/verbosity bias

Pairwise results flip just by swapping order, yet you score in one direction only.

→ Average over swapped order, control length, require conciseness in the rubric.

!

Resting on benchmark scores

Assuming a high public-benchmark score means it'll be good on your domain too.

→ Your own eval set, built from your data and tasks, is the real yardstick.

!

Build the eval set once and forget it

The initial gold set drifts away from real traffic distribution and never gets updated (data drift).

→ Keep it alive by periodically absorbing production failures into the eval set.

Evaluation is the instrument panel that drives the "outer loop" of an AI system. The more you automate generation, retrieval, and agents, the more the quality of the eval gate determines whether you can trust that automation. An eval isn't a one-time deliverable — it's a living asset that grows by continually absorbing production failures.

AI Eval Evaluation Pipeline Guide