Fourteen rules for running, scoring and reporting a comparison, with the evidence for each
Three measurement exercises have been run in this repository. All three produced working machinery and none produced a usable comparison: in every case the instrument returned the same verdict for every subject, and in every case the qualitative signal that was supposed to break the tie was specified and never collected. The pattern is consistent enough to design against.
This document fixes the house standard. Seven of the sixteen rules are already practised
in bakeoff/ and are written down here so they survive the person who wrote
them. Seven are new: five address a specific way one of the three runs failed, and two were
earned by applying the standard to itself. The standard applies to anything published under this masthead, and a run that
departs from it says so in its report.
Three runs, all of them in this repository, all of them recoverable from committed data.
.github/loop/{turns,runs}.jsonl and
bakeoff/results/race-0{1,2}/results.json.| Run | Subjects | Outcome of the automated score | Qualitative signal |
|---|---|---|---|
loop (Aug 4–7) | 89 turns | One signal of three implemented. It took three values and sat at its maximum in 59 of 89 turns. Zero quality scores were produced. | Rubric and adversarial check never built. |
race-01 | 11 runs | 11 of 11 passed the gate. 11 of 11 scored 4 of 4 on primitives. | Judge panel returned null. |
race-02 | 12 runs | 12 of 12 passed the gate. 12 of 12 scored 4 of 4 on primitives. | Judge panel returned null. |
Across the two bake-offs, 23 runs out of 23 passed every automated check. The ranking column in both reports is patch size in bytes, which is not a measure of quality and was never claimed to be. So the comparison those runs were built to make was not made, and the reason was the same in all three cases.
There is a second finding in race-01 worth recording, because it is a
different kind of failure. Two of the eleven entries recorded patchBytes: 0
while also recording hasEntry: true, and their directories contain a 50 KB
index.html and a working field.mjs. The agents did the work; the
patch capture returned nothing. Those two rows then sorted to the top of the report, because
the report sorts ascending by patch size. Nothing in the pipeline noticed.
A gate that everything passes is not proof the gate was badly designed. It is also
consistent with the entries all being adequate, which for a floor is the desired outcome:
bakeoff/CLAUDE.md describes the gate as "a floor, not a grade" and never
claimed it would rank anything. The failure is not that the floor held. It is that nothing
else was collected, so the floor was all there was, and a report with 23 identical rows was
the whole result.
Rules marked established are already practised in bakeoff/. Rules
marked new respond to something in Table 1.
A cell fixes every factor that could move the outcome: harness, model, brief, sample index. Asking which model is better without fixing the harness has no answer, and the converse holds as well. Vary one axis at a time or vary them factorially and say which you did .
Cells run from identical clean checkouts with no shared state. This is what makes the comparison mean anything, and it is worth the cost of the extra runners.
The rubric comes first and the brief is written from it, so that everything scored is something the subjects were told about. A check that exists in the scorer but not in the brief is a rule nobody was given, and any result that turns on it is an artefact of the experimenter's private knowledge.
This is pre-registration in the form that fits a scoring pipeline , and it forecloses the largest family of analytic degrees of freedom .
Two calibration points, both enforced in CI:
Run the floor check before the run, not after. A rig that has stopped measuring produces a clean sweep that reads as success, which is close to what Table 1 shows.
This is the rule that all three prior runs would have failed, and it is the reason this document exists.
Before a report is published, compute how much the instrument separated the subjects: the variance of the score across cells, the number of distinct values it took, and the share of cells at the maximum. Publish those three numbers next to the ranking.
Where every subject lands in the same bucket, the run has not produced a comparison, and the report says so in its first paragraph rather than presenting a table sorted by an incidental column. Reliability collapses as a measure approaches its ceiling, because the true-score variance goes to zero while the error variance does not . A saturated instrument is not a weak result. It is an absent one.
Threshold. If more than 70% of cells sit at the maximum, or the score takes fewer than four distinct values across the grid, the automated tier is reported as non-discriminating and the run's conclusions rest on the human tier alone.
Every cell runs at least twice, independently, from a clean tree. The spread within one cell across two samples can exceed the gap between two cells, and a single draw cannot tell you which you are looking at.
Report the within-cell spread alongside the between-cell difference. Where a cell's samples disagree on the gate, say so and treat any single-run claim about that cell as unsupported.
Where a task has both a correct answer and a matter of taste, report tiers that never sum: a binary gate, a countable checklist, and an unscored tier that a person ranks. Collapsing those into one number ranks whoever best satisfied a checklist and launders a taste judgement through arithmetic until it looks objective.
The tiers are not equal in status. The gate is a floor and excludes; the checklist keeps entries comparable; the human tier decides. Say which tier a claim rests on.
Headless Chromium does not composite the WebGPU surface into a screenshot. Five flag combinations were tried; the ones producing a working adapter still capture the 3D region as blank page background. So the capture step proves an entry is alive and shows its interface, and says nothing about how the game looks.
Every instrument has a boundary of this kind. Find it by testing, write it into the report, and never let a green capture be reported as evidence of something the capture cannot observe.
An early liveness check used pixel-motion magnitude and failed two identical entries differently: peak delta 163 passed and delta 23 failed. The threshold was measuring frame noise. Replacing it with "frames moved or the state clock advanced", where the clock is a contract the brief requires the entry to expose, removed the false discrimination.
Where a heuristic and a contract are both available, the contract is the instrument and the heuristic is a fallback. A threshold that separates two identical subjects is measuring the apparatus.
Existing practice relabels entries and strips every mention of harness and model before review, and no model judges its own entry. Both are kept. Three additions:
A judge panel remains a second opinion rather than a verdict, and each judge keeps one lens rather than producing a score to be averaged.
In all three prior runs the qualitative signal was designed, documented, and never
collected, and in all three the run was reported as having happened. The judge panel returned
null in both bake-offs and no report says so on its face.
The pipeline now asserts that every signal named in the run configuration is present in the results before a report is written, and names the missing ones in the report header where it cannot. A run missing its tie-breaker is a pilot, and calling it one is the honest description.
Two entries in race-01 recorded a zero-byte patch alongside a 50 KB entry
directory, and sorted to the top of a report ordered by that field. The failure was silent
because nothing asserted that the two facts were consistent.
Any field that orders, filters or ranks a report carries a validity check: a patch of zero bytes for an entry that exists is a collection failure and is reported as one. Fields that are neither used nor checked are removed, because an unchecked number in a results file will eventually be read as a finding.
Cells skipped for a missing provider key are listed as skipped with the reason. Runs that errored are listed with their error. Out-of-scope edits are listed even though they are not collected, because they describe how the agent worked.
A results file that contains only the cells that worked is a filtered sample, and the filter is correlated with the outcome.
Generate data from the model the design assumes, at the size the design proposes, and report the sampling distribution of whatever the run is meant to estimate. Where experiments cost money this is ordinary practice , and it is close to free here.
The rule was earned rather than borrowed. A 24-run variance pilot recommended in an earlier revision of the instrument note turned out, on simulation, to return a 95% interval on the intraclass correlation of about [0.00, 0.80] at a true value of 0.5, and to return exactly zero a quarter of the time at a true 0.2. Reallocating the same runs did not help. The full result and the revised design are in WP1.
It has since earned itself a second time, on a design that had not been priced at all. H8 claimed one six-turn chain would measure the attenuation parameter λ. Simulated, that chain returns a 95% interval 0.361 wide on a parameter bounded in [0, 1], and fails to produce an estimate in 92 of 1500 replications. The design that works costs six chains, and the simulation also showed that buying depth beyond six makes the estimate worse rather than better, which no amount of reasoning about geometric decay would have suggested. WP2 §12.
Report the simulation alongside the design, including its seed, and state plainly that its intervals assume the very model the run is partly there to test.
Revised after our own arithmetic refuted the first version. R14 used to mandate one shape: six turns in four stages, no stage wider than two, setup then two paired waves then cleanup. The justification was that each wave is a matched pair and so takes the pairing saving of R5.
That justification argues against the rule. A matched pair is an orbit of size two. The same six turns wired as a star put four turns in one orbit, which is a matched quadruple, and at our measured ρ that is 1.787 effective replicates against the mandated shape's 1.415. The rule forbade the shape its own reasoning preferred, and did so for a year of revisions before anyone computed it.
So the rule is now about declaration rather than prescription. A run states its profile — the layer widths — and reports four invariants of it, all of which the explorer computes:
Two things carry over from the old rule unchanged, because they were right. Turns in one orbit must receive the same state, varying only the condition under test; give them different briefs and the orbit is not an orbit and the saving is gone. And the six-turn scale remains the default unit, because it keeps a run countable in advance. It is now a default rather than a requirement, and the explorer prices the alternatives.
Prices at ρ = 0.413 for the six-turn default: a large effect (d = 0.8) costs 30 runs, or 180 turns. A medium one (d = 0.5) costs 74 runs, or 444. The procedure and both cost tables are at the standard run.
Two turns fed the same upstream products must receive the same brief, byte for byte. Do not tell an agent which shape it sits in, which arm it is, or how many siblings it has. Do not let dispatch order, a shared rate limit or a shared scratch file reach it.
The rule exists because it is what makes pooling legal. Turns are exchangeable when their ancestry digests agree, and the digest is computed from the plan. If the runner adds anything the plan does not contain, then equal digests no longer mean equal inputs and every pooled estimate is quietly wrong. The digest cannot detect the violation; only this rule prevents it.
It is also what makes the catalogue worth running as a set. Under R15 a turn fed by one setup turn is the same experimental condition in a chain, a star and a bottleneck alike, so thirty-six turns of the six-shape catalogue supply 7.65 effective observations in one cell rather than the 1.79 the best single shape supplies. Break R15 and that collapses to nothing.
An edge in a plan is a permission. The absence of an edge is a prohibition, and a prohibition nobody enforces is a drawing. A turn learns things by three channels and the plan controls one of them: the brief it is handed, the files an earlier turn left in its worktree, and the history it can read. Draw whatever arrows you like; if the second and third channels are open the turn sees everything anyway.
| Regime | Effective shapes | Distinct sink ken | What it is |
|---|---|---|---|
isolated | 1960 | 5 | a turn receives its in-edges and nothing else: fresh context, fresh tree, handoff by explicit artefact |
lineage | 16 | 1 | a turn inherits the accumulated worktree of everything upstream. What a per-lane worktree with merges at joins actually gives |
shared | 8 | 1 | one worktree throughout, so a turn reads anything that finished before it started |
Under either sharing regime the ken ratio is 1.000 for every turn of every shape. The effect is not small; the independent variable has no variance, so a shape hypothesis run that way is undefined rather than underpowered. The reason is definitional and was sitting in the definition unnoticed: ken is the in-neighbourhood over the ancestry, and a turn that inherits its ancestry has an in-neighbourhood equal to it. The ratio has content only where the plan withholds, and withholding needs a mechanism.
So a run reports its regime, and reports the leak count: edges the environment
supplies that the plan never declared. A six-turn chain declares five edges; run under sharing
it has fifteen, so it leaks ten and the drawing overstates what was withheld by a factor of
three. chain and briefed, whose contrast is the whole of
H5, become the same fifteen-edge graph with the same sink ken. Zero
contrast, whatever the sample size.
Demonstrating it is the part that cannot be asserted. Plant a marker in a turn that a later turn has no edge from, and check the later turn cannot produce it. A regime is a claim about the harness and the harness is what has to answer for it.
The standard reduces to two lists. Both are short on purpose .
| # | Check | Rule |
|---|---|---|
| 1 | The scorer exists and the brief was written from it. | R2 |
| 2 | The floor fails on the specific check the brief is about. | R3 |
| 3 | A minimal correct implementation scores full marks. | R3 |
| 4 | Every check in the scorer appears in the brief. | R2 |
| 5 | Samples per cell ≥ 2, and the grid is stated. | R5 |
| 6 | The qualitative tier has a named owner and a date. | R10 |
| 7 | Two cells run first to prove the pipeline. | R1 |
| 8 | What the instrument cannot see is written down. | R7 |
| 9 | The design has been simulated and the simulation reported. | R13 |
| 10 | Each wave dispatches the same state to both agents. | R14 |
| # | Check | Rule |
|---|---|---|
| 1 | Discrimination reported: score variance, distinct values, share at maximum. | R4 |
| 2 | If saturated, that is the report's first sentence. | R4 |
| 3 | Within-cell spread reported next to between-cell differences. | R5 |
| 4 | Every signal named in the config is present, or named as absent in the header. | R10 |
| 5 | Judge swap rate reported; agreement given against the human ceiling. | R9 |
| 6 | Every ordering field passed its validity check. | R11 |
| 7 | Skipped and errored cells listed with reasons. | R12 |
| 8 | Which tier each conclusion rests on is stated. | R6 |
Stated so that the gaps are visible rather than discovered later.
The first and third of these are blocking for any published quality claim. The second and fourth are recorded for now.
.github/loop/{turns,runs}.jsonl and
bakeoff/results/race-0{1,2}/results.json. ken.selftest.mjs
recomputes every figure in it on each build and fails if the prose has drifted from the
record. The practices marked established are drawn from
bakeoff/CLAUDE.md, which had most of this right first.