A protocol, deliberately unfinished, whose blanks are the deliverables of the curriculum
A budget-governed agent loop was run for ninety-nine turns and produced no measurement of the quality of its output. This protocol sets out how to run it again so that it does. It is written in the Stage 1 registered-report form, with hypotheses, design and analysis plan fixed before data collection, because the failure mode this programme is most exposed to is a flexible analysis of an expensive run.
It is deliberately incomplete. Six blanks are marked in the text, each requiring a quantity that does not yet exist: a defined construct, an established reliability, a calibrated judge, a powered sample size, a stopping rule, and an outside signal. Each blank is the deliverable of a curriculum unit. The protocol is not registrable, and the run is not fundable, until all six are filled. That ordering is the point of the document.
Restated at revision 16: this document is now Study 2, and it is not the next thing we run.
It was written to make the loop's question answerable, and that question needs all six blanks closed. Sixteen revisions of work went somewhere else, and the somewhere else turned out to be registrable first. Study 1 is H5: does an agent's fidelity to its brief fall with its ken ratio?
Three reasons, in the order that decided it. It is affordable: 180 turns paired at
d = 0.8, against a turn-count study whose sample size cannot even be quoted until B4 closes.
Its manipulation is free: chain against briefed is four
added edges and no added turns, and the two match at every turn but the last. And it needs
no new apparatus — no calibrated judge, no defined construct, no outside signal, so
B1, B3 and B6 are not on its critical path.
What Study 1 does not do is close the blanks. Those still gate this document, and the turn-count curve still needs them. The ordering has changed, not the requirement. Study 1's own protocol is WP2 §7 to §9, and its hypotheses carry their status in the register.
The apparatus exists and is disabled. A ticket graph, a budget governor, a contagion firewall and four chain-reaction workflows were built and run; ninety-nine work orders were issued over roughly fifty-three hours in August 2026, of which eighty-nine recorded a completed turn. The programme's stated deliverable was the curve: quality against turn count, measured, over many artifacts.
Two later bake-off runs met the same fate by a different route, and the house standard was written from all three. This protocol assumes that standard and does not restate it; the rules most load-bearing here are R4 on reporting discrimination, R5 on replication within a cell, and R10 on treating an uncollected signal as a failed run.
No such measurement was produced. The dependent variable was never recorded once. The situation as of the last recorded turn:
.github/loop/{turns,runs}.jsonl. The final row is the finding.| Quantity | Value | Note |
|---|---|---|
| Work orders issued | 99 | over ~53 h |
| Turns with a recorded outcome | 89 | |
| Gate failures | 17 | 6 in the first half, 11 in the second |
| Signals specified | 3 | probes, rubric, adversarial |
| Signals that ever fired | 1 | probes only |
Distinct values probes took | 3 | 0.6, 0.8, 1.0 |
| Turns at the maximum | 59 / 89 | 66%, a ceiling |
| Quality scores produced | 0 | judge uncalibrated; score withheld by design |
Two things follow, and it is important to keep them apart. The first is that the study was never run, in the sense that the quantity of interest was never observed; the pilot is therefore evidence about the apparatus, not about the question. The second is that the one signal which did fire could not have answered the question anyway: a measure taking three values, two thirds of them at its maximum, has almost no variance to explain in the region where the interesting turns live.
A faint negative trend in probes across the run (slope ≈ −0.0014 per turn)
is not statistically meaningful at this sample size, and probes is in any case
a liveness signal rather than a quality measure. The near-doubling of gate failures between
the first and second halves is real but confounded: later work may simply have been harder,
and nothing in the record distinguishes those explanations. Neither observation should be
cited as evidence that quality declined. They are reasons to run the study properly, not
substitutes for having run it.
Stated before the fact, and stated so that they can lose.
H1 (monotone improvement). Mean artifact quality is non-decreasing in turn count over the range studied.
H2 (saturation). Quality rises and then plateaus, with the plateau reached well inside the budgeted range — the "where does unattended work stop paying" hypothesis, and the one the programme was built to test.
H3 (overoptimisation). Quality rises, peaks, and then declines with continued turns, as gate-satisfying effort displaces value-producing effort. This is the reward-overoptimisation shape reported for a closely analogous setting , and predicted here on independent grounds by the multitask result .
H2 and H3 are the live competitors. H1 is included because it is what the system's own gates would have reported, and the study is partly an audit of that report.
The pre-specified discriminator. H2 and H3 are distinguished by the sign of the quadratic term in a quality-on-turn-count regression over the second half of the range, with the decision rule fixed in §4 before any quality data is seen.
The unit is the artifact, not the turn. Turn count is the manipulated variable: artifacts are developed for a randomly assigned number of turns drawn from a pre-specified grid, and quality is measured once, at the end, by raters blind to the turn count that produced it.
This is the design decision that matters most, and it is worth being explicit about why it differs from the obvious approach. Measuring the same artifact repeatedly as it develops confounds turn count with the artifact's own trajectory and invites the rater to reward visible change. Randomising turn count across artifacts and measuring once buys a clean between-subjects contrast at the cost of needing more artifacts — which is precisely the trade Unit II's power analysis is for.
| Arm | Turn allocation | Purpose |
|---|---|---|
| Control | 1 turn | Floor. What a single unattended pass produces. |
| Graded | drawn from a grid across the budgeted range | The curve itself. |
| Restart | equal total turns, split across independent fresh attempts | Separates "more work" from "more continuous work". If restarts match or beat the graded arm at equal cost, continuity is not what is buying quality. |
The restart arm is the one most likely to be cut for budget and the one most likely to produce the study's most interesting result. It should be defended.
Raters see the artifact and nothing else: no turn count, no commit history, no ledger, no gate outcome. Where a model judge is used, the same applies to its prompt, and artifacts are presented in randomised order with pairwise comparisons run in both orders so that position bias can be estimated rather than assumed away .
What follows is the part of the protocol that cannot yet be written. Each blank names the quantity required, the unit that produces it, and the reason the study is not registrable without it.
B6 deserves emphasis because it is the one that cannot be solved by more care. Every other blank is a technique. B6 requires a person who is not us to look at the output and say something, and no amount of internal rigour substitutes for it. The programme's own operator notes have already identified this as the missing signal; the protocol simply refuses to proceed without it.
To be fixed in full once B1–B4 are closed. The skeleton:
Conditions that must hold for the data to be interpretable at all, regardless of which hypothesis it favours. These are checked first, and a failure halts the analysis:
Stated in advance, because a programme that cannot say what would end it is not an experiment:
| Phase | Contents | Closes | Model budget |
|---|---|---|---|
| 0 · Foundation | Units I–II. Reading, both replications. | B1, B2, B4, B5 | none |
| 1 · Theory | Units III–V. The LEN model fitted to our own run, the knowledge hierarchy fitted to our orchestration, and the overoptimisation curve on a synthetic. | predictions for H2/H3 | none |
| 2 · Instrument | Unit VI. Judge built, calibrated, debiased. | B3 | small; calibration only |
| 3 · Oversight | Unit VII. Sandwiching trial; outside signal secured. | B6 | small |
| 4 · Registration & run | Protocol completed and registered, then the run. | the curve | the whole of it |
| 5 · Report | Written and published whichever way it came out. | — | none |
The retrospective analysis of the pilot, meaning the reconstruction of per-turn history from the ledger and an account of why two of three signals never fired, is not a phase. It is a Unit I exercise, and it is listed there deliberately: it is worth doing with the measurement vocabulary in hand and close to worthless without it.
Not a positive result. A curve of any shape, produced under a pre-registered design with a calibrated instrument and an outside signal, would be unusual in this area on methodological grounds alone, since most reported evaluations of unattended agent work supply a number without establishing that it measures the construct , and without a control arm.
The most likely genuinely novel finding is the divergence: gate-pass rate rising while measured quality does not, quantified, on a real system, with the mechanism identified in advance by a model from 1991 . That is a small, checkable, and honest contribution, and it is the one this protocol is designed to be able to detect.
.github/loop/turns.jsonl and runs.jsonl in this
repository.