ken · volume 1 Instrument note

The Harness

A design calculator, and what it says about the runs we have already paid for

ken/lab/design.mjs · 92 known-answer checks · node-only, not served

Abstract

Replication is the only defence against model variance and replication is what costs money, so the useful work happens before any of it is spent. This note describes a small node library that answers three questions offline: how many runs an effect of a given size would need, what the smallest detectable effect is for a budget already fixed, and which design changes make the same claim cheaper.

Applied to the bake-offs, it gives a number worth having early. With two samples per cell, a two-cell contrast could only have detected a difference of about 2.8 standard deviations, which is the gap between total success and total failure. It also produces a result that runs against intuition and against how the existing rig is usually described: for a fixed run budget, repeats do not buy precision. They buy a variance estimate. Beating the noise takes more distinct tasks.

1Scope and coverage

The library sits at ken/lab/design.mjs, imports its statistics from the shared packages/dataviz rather than reimplementing them, and is covered by design.selftest.mjs, which checks 92 values that can be derived by hand or looked up. The standard it is held to is borrowed from bakeoff/CLAUDE.md, which puts it well: a rubric nobody has watched fail is not evidence of anything. So every assertion in that file is an independently known answer rather than a snapshot of what the code returned: z(0.975) = 1.959964, C(8,4) = 70, and a variance decomposition worked out longhand.

Three questions, in the order they are usually needed:

Given a budget, what could this run possibly detect? Nearly always the more useful direction, and the one that stops a study before it wastes anything.
Given an effect you want to detect, what does it cost?
What design change makes the same claim cheaper? This has real answers, and they are worth more than any amount of careful reporting afterwards.

2Power of the bake-off design

Figure 1. Smallest detectable standardised effect against runs per arm, at 5% significance and 80% power, two-sided. The bake-offs ran six cells at two samples each, so a two-cell contrast sits at the far left: d = 2.80, the gap between near-total failure and near-total success. Ten runs per arm still needs d = 1.25; the conventional medium effect of 0.5 needs 63.

So the twenty-three-of-twenty-three result has a second reading alongside the saturated gate in the house standard. Even with a perfectly discriminating instrument, that design had no power to separate the cells. Fixing the gate alone would not have produced a comparison.

What this does not say

Not that the bake-offs were a mistake. Their stated purpose was a floor and an arena for a person to judge, and two samples per cell was chosen so within-cell spread could be seen at all, which §4 shows was right for that purpose. The narrow claim: those runs could not have supported a statistical comparison, and nothing in the reports claims they could.

3The cost of a comparison

Pairing is the lever. Run every condition on the same tasks and whatever makes a task hard cancels out of the difference, cutting required observations by (1 − ρ).

Figure 2. Runs for one medium-effect contrast (d = 0.5) against the correlation between conditions on a shared task. Unpaired costs 126. At a plausible ρ = 0.7 it is 38; at ρ = 0.85, not unreasonable when both conditions attempt the same brief, it is 20 — inside what the bake-offs already spend. The same claim, a sixth of the bill, from a design choice that costs nothing to make.

Small effects stay brutally expensive whatever you do: d = 0.3 needs 350 runs unpaired and 106 at ρ = 0.7, and d = 0.2 needs 786 and 236. A programme that needs to detect d = 0.2 should decide that in advance, not after spending. Hence question one.

4Repeats, tasks, and where precision comes from

The intuition that running the same prompt several times should beat the noise is half right. Write the variance of a condition mean:

Var(mean) = σ²_between ⁄ tasks + σ²_within ⁄ (tasks · repeats) with the budget fixed at R = tasks · repeats, the second term is σ²_within ⁄ R

The second term does not depend on the split, so every repeat is a task forgone. allocate() returns one repeat as the optimum at ICC 0.1, 0.35, 0.6 and 0.9 alike, and the paired two-condition version reaches the same place.

What repeats buy is the variance estimate. At one repeat per task the within-task degrees of freedom are zero and σ²_within is not estimable, so you have no idea how noisy the model is and no input for anything above. allocate() returns the optimum together with a finding string saying what it costs, and flags varianceEstimable: false.

This settles a question the existing rig raised. Two samples per cell is not a way of averaging out noise and would be a poor one. It is a way of seeing the noise, and bakeoff/CLAUDE.md justifies it in exactly those terms.

5The variance pilot, and a correction

Revised after simulation

An earlier version of this section recommended a 24-run pilot as the way to obtain the variance components. Simulating that pilot before running it showed the recommendation was wrong: at a true ICC of 0.5 it returns a 95% interval of about [0.00, 0.80], and at a true 0.2 it returns exactly zero a quarter of the time. Full result and revised design: WP1, The Variance of Unattended Work. What follows is the corrected version.

The first run to buy is still not a comparison, and it is still 24 runs of one condition across several tasks. What changed is the job it is being asked to do. It is a model check, which it does well, rather than a parameter estimate, which it does not do at all.

Table 1. The default pilot from variancePilot(), and what simulation says it can and cannot support.

Quantity	Value	Reading
Tasks	8	Enough groups for a between-task mean square to exist.
Repeats	3	Enough within-task replication to estimate run-to-run noise.
Runs	24	A cost that does not need a decision.
df between	7	Both small, and this is why the ICC interval comes back almost as wide as the parameter range.
df within	16
Detects bimodality at 2 SD	69%	What the pilot is actually for. A gross departure from the model is caught; a subtle one is not.
Detects bimodality at 3 SD	100%
95% interval on ICC	width 0.80	At a true value of 0.5. Not an estimate.
Runs for an ICC interval of ±0.16	144	A separate and deliberate spend.

Feed the result to varianceComponents(), which returns σ²_within, σ²_between and the ICC, clamping a negative between-task moment estimate at zero because a negative variance is not a finding. Report the interval before the point estimate, and report a clamped zero as clamped, since a bare zero reads as a measurement of no task effect when it is often an artefact of a small sample.

6Sequential stopping and exact tests

A run that watches its own results and stops when they look good has an error rate far worse than it believes. Wald's sequential test gives boundaries on the accumulated log-likelihood ratio that permit early stopping while holding the stated error rates, provided the rule was fixed in advance. sprtBounds({ alpha: 0.05, beta: 0.2 }) returns an upper boundary of ln 16 ≈ +2.77 and a lower of ≈ −1.56.

The library also carries the exact randomisation machinery from Unit II's replication: ladyTastingTea() enumerates the null distribution rather than simulating it, and permutationTest() refuses above a stated assignment cap rather than silently switching to sampling, because a p-value from an undisclosed Monte Carlo is a different object. The design lesson falls straight out of the enumeration: at six cups the smallest achievable p-value is exactly 0.05, so six cups cannot produce a significant result at the conventional threshold and eight is the smallest design that can.

7Known gaps

Every ρ here is assumed. Nothing in the repository has measured a task-level correlation between two conditions, so Table 2 is a menu rather than a quotation. The pilot in §5 is what turns it into one.
Normal approximation throughout. Sample sizes run a little light at small n; the exact: true flag adds the conventional correction rather than pretending to more precision than a normal approximation has.
No cost model in tokens or dollars. costToDetect() takes a costPerRun and refuses to assume one. Bake-off cells ran between about 800 and 5,100 seconds of wall clock, which is too wide a spread to convert into a unit price without measuring it.
No runner yet. This is the design half. Executing a design against real cells still goes through bakeoff/, and connecting the two is the next piece of work.

References (0 works cited in this note, —)

ken · instrument note · the harness
Every figure in Tables 1–3 is recomputed from ken/lab/design.mjs by ken.selftest.mjs on each build and compared against the numbers printed here, so the page cannot drift from the library it documents.