Published rubrics

Written by a person. Last read by a person on 2026-09-04, 4 days ago. Its facts were checked by the eval suite on 2026-09-07.

You want to know what this documentation checks about itself, or you are deciding whether to trust a claim it makes.

These are the rubrics the eval suite enforces. They are published rather than filed as test fixtures because they are the assessment-design artifact, and a rubric nobody can read is an assertion rather than a standard.

What a rubric here contains

Four parts, in this order, every time.

The claim — what is being asserted about the content, stated so it could be wrong.

Observable evidence — what a person or a program would look at to decide. If the evidence cannot be observed, the criterion is an opinion and does not belong in a rubric.

The pass bar — where the line sits, and why it sits there rather than somewhere stricter or looser.

Ambiguous cases — the ones that fall near the line, and how they are treated. This section is the one that makes a rubric usable by someone other than its author, and it is the one usually missing.

The relationship to the tests

The rubric is the source of truth. evals/checks.py implements it. When they disagree, the rubric is right and the check is a bug.

A check that names no rubric is rejected by the suite's own coverage test, so a gate cannot be added quietly without stating what standard it enforces.