#evals#llm-judge#talk-notes

Evals are fuzzy unit tests, and the judge can lie

An eval runs test cases through your agent, and a judge scores each answer against the policy. Before you trust the judge, check that it agrees with humans.

Tejas KumarAI Engineer, IBM · talk notes

October 20, 20268 min read60 min talk

Watch the talk

What you’ll learn

01What an eval isTest cases, a judge, and a score that gates CI.
02Why keyword checks failOne return request, five different judges.
03How judges lieFour biases, and why pointwise beats pairwise.
04How to trust a judge80–90% agreement with humans, and live policy.

Each test case goes to the agent as a prompt, for example “I bought headphones 20 days ago. Can I return them?”
The judge scores one answer at a time against the written policy. It does not compare two answers.
Humans have labeled the same 30 cases pass or fail. The eval counts how often the judge agrees with them.
CI fails when agreement drops below 80%. In the demo the gate passed at 24 of 30.

Fig. 1An eval has four parts: test cases, expected behavior, a judge, and an aggregate score. The aggregate gates the deploy.

Map of the talk

Your link starts at the list of eval techniques

Fig. 2The live demo takes a third of the talk. Click a block to open the video at that point.

Why

Evals catch bad behavior before deploy; harnesses catch it at run time

Fig. 3Meta’s recovery chatbot could edit accounts. Attackers matched the owner’s location with a VPN, then asked it to add their email to the account.

The running example

A keyword check passes a wrong answer; a grounded judge does not

Customer“I bought headphones 20 days ago. Can I return them?”

PolicyReturns only within 14 days. 20 > 14, so the right answer is no.

Wrong pass

Keyword match

The word appears, but in the wrong place.

Wrong pick

Pairwise, no policy

A friendly yes beats a correct no.

Below gate

Pointwise, weak model

A used yoga mat exposed a rule nobody wrote down.

Gate passes

Pointwise + policy

A stronger model, plus a rule against loyalty appeals.

Correct deny

RAG-grounded judge

When the PDF changes, the judge follows.

Fig. 4The same question, five judges. Only the two that read the policy get it right.

How evals lie

LLM judges have four biases that you must test for

Position

Picks the first answer

Sycophancy

Picks the pleasing answer

Self-preference

Picks its own family

Verbosity

Picks the longer answer

Fig. 5Position bias and self-preference come from showing two answers at once. That is why the talk recommends pointwise scoring.

Fig. 6A study cited at about 17:00: pairwise judges are about 39% more open to manipulation than pointwise judges.

Training the judge

Aim for 80–90% agreement with humans, not 100%

Have experts label about 30 question–answer pairs. Run the judge on the same pairs and count agreement. Human support teams agree with each other only about 80% of the time, so a judge near 100% is overfit to your examples.

agreement.test.ts — demo runs

# summary of the demo runs, not literal output run 1 ✓ 27 agree ✗ 3 disagree 90% run 2 ✓ 24 agree ✗ 6 disagree 80% gate ≥ 80% → pass

Fig. 7The same judge on the same 30 cases scored 27, then 24. The gate is a band, so normal run-to-run noise does not break CI.

Staying relevant

Give the judge the same policy search the agent uses

Fig. 8Policies change. If the judge reads them from the same index as the agent, the eval does not go stale.

What to remember

Start cheap and add a judge only where you need one

Climb the cost ladder: deterministic tests, then regex, then trace checks (which tools ran, how long), and a judge last.
Write about 30 realistic cases and have a human label each one pass or fail.
Score answers pointwise against the written policy, not pairwise. Use the cheapest judge model that works, from a different family than the agent, and pin its version.
Run the eval in CI. Fail below 80% judge–human agreement, and treat 100% as a red flag.
Retrieve the policy from one shared source (RAG) for agent and judge. Add new cases from production traffic.

Q&A · 54:32

Three audience questions

How do I grow the dataset before real users arrive?

Synthetic cases and red-team prompts

Generate adversarial prompts, then make more variants of the ones that fail.

Should I write evals for my Claude Code or Cursor skills?

No; write evals when you build a harness

Skills you write yourself are mostly deterministic code.

Should the judge and the agent share the policy?

Yes, one source of truth

Both query it, for example through OpenRAG. Keep the judge as close to the agent as you can.

Sources

hardeep.cv