Data

Which grader is lying

ยท Open

The piece this page is for: a grader whose pass rate looks fine and is not measuring what it claims. The query that found it, the chart, the confound, and what changed as a result.

Not written yet, waiting on P-6, N-1

Evalset, and any grader pass-rate figures from it. No spec or shipped code for evalset was found in the calabrodesign repo or in the linked calabro-mcp pointer as of 2026-09-21, and no figures for the product were supplied or reproduce from anything in scope.

Checked 2026-09-21.

Why a plausible version is not good enough

I know roughly what this story looks like, because it is a common shape: a grader passes at 98%, which reads as a healthy check, and it turns out to be passing because its assertion is satisfied by almost any output, including an empty one. The pass rate was measuring the assertion's looseness, not the model's behaviour.

I could write that up with a chart and it would be entirely believable. It would also be a case study about no particular grader, with numbers I made up, presented on a desk whose entire claim is that the method is shown. One fabricated chart here would make every real one on this site worth less.

What the finished piece needs

  1. The query. Verbatim, runnable, against a named dataset.
  2. The chart. Pass rate per grader over time, with the axis starting at zero and saying so.
  3. The suspicion. What made this one look wrong before the confound was found. Usually it is a rate that is too stable rather than too high.
  4. The confound. What the grader was actually measuring, and the case that proves it.
  5. The action. What changed: the assertion, the fixture set, or the grader being deleted.

Item five is the one that makes it a finding rather than an observation.

Evalset grades deterministically rather than with a model acting as judge, which is the decision that makes this failure mode possible in the first place: a rule can be loose in a way a reader would notice immediately. That decision is written up on the AI desk at no model judge.

A suite of deterministic graders needs a check on the graders themselves, and this page is supposed to be it.