LLM Evaluations — Evals

Why evals are the most important step

Imagine you changed a prompt to improve one answer. How do you know you didn't break ten others? In regular software development you have tests. In AI Engineering, the equivalent is evals — a collection of test cases that automatically measure the quality of the system's output.

Without evals you develop "by feel": you change something, manually check 2–3 examples, and hope. With evals you know exactly whether a change improved things, broke them, or had no effect — across dozens or hundreds of cases. This is what lets you improve a system with confidence instead of being afraid to touch it.

Types of evals

1. Rule-based / code

The fastest and cheapest. You check objective things in code: is the output valid JSON? Does it contain the required fields? Is the number in range? Great for structured outputs.

2. Reference-based

You have a known "correct answer" and compare against it. Suited to tasks with an unambiguous answer (classification, data extraction). Metrics: accuracy, precision/recall.

3. LLM-as-Judge

For open-ended tasks (writing quality, answer relevance) there's no single "correct answer." The solution: use another model as a judge that scores the output against criteria. Powerful and flexible — more on it below.

4. Human eval

The real gold, but expensive and slow. Humans rate a sample. You use it to calibrate the LLM-judge and make sure it agrees with humans.

Building an eval set — where to start

1. Collect real cases. Take 20–50 real (or realistic) inputs the system is supposed to handle.
2. Include edge cases. Not just the "normal case" — also empty input, mixed language, a manipulation attempt, an out-of-scope question.
3. Define "what good looks like" for each case — an expected answer, or judging criteria.
4. Start small. 20 good cases beat 500 bad ones. Expand over time, mainly from failures you saw in production.

Store the eval set as a file (JSON/CSV) in git, like code. It's a valuable asset that grows over time.

LLM-as-Judge — code example

The idea: a judge model receives the input, the output, and criteria, and returns a score + rationale. It's important to ask for a numeric score + explanation and use temperature 0.

JUDGE_PROMPT = """You are a judge of a support bot's answer quality.
Rate the answer from 1 to 5 by:
- relevance to the question
- accuracy (no wrong information)
- professional tone
Return JSON only: {"score": 1-5, "reason": "..."}

Question: {question}
Bot answer: {answer}"""

def judge(question, answer, client):
    prompt = JUDGE_PROMPT.format(question=question, answer=answer)
    resp = client.chat.completions.create(
        model="gpt-5.6", temperature=0,
        response_format={"type": "json_object"},
        messages=[{"role": "user", "content": prompt}],
    )
    return json.loads(resp.choices[0].message.content)

# run over the whole eval set and average
scores = [judge(c["q"], run_system(c["q"]), client)["score"] for c in eval_set]
print("avg score:", sum(scores) / len(scores))

Critical tip: calibrate the judge against humans on a sample. If it agrees with humans ~85%+ of the time, you can trust it for most cases.

Regression testing & CI

The real power: run the evals automatically on every change. Wire them into CI (e.g. GitHub Actions), and if the average score drops below a threshold — the build fails. That way a change that improves one thing and breaks another is caught immediately.

# pseudo: eval gate in CI
avg = run_evals(eval_set)
THRESHOLD = 4.2
assert avg >= THRESHOLD, f"quality dropped: {avg} < {THRESHOLD}"
print(f"Evals passed: {avg}")

This turns "I think it's better" into "the numbers prove it's better." See also Observability for production measurement (online evals) on real traffic.

Common mistakes

• No eval set at all. The big mistake. Even 20 cases change everything.
• Only "easy cases." If the evals don't include edge cases, they give false confidence.
• An uncalibrated judge. An LLM-judge without a check against humans may give misleading scores.
• High judge temperature. Causes inconsistent scores. Always 0.
• A single metric. One score hides problems. Measure several dimensions (accuracy, tone, safety) separately.
• Not updating. Every production failure should become a new case in the eval set.