ken · volume 1 · working paper 1 Unit II

The Variance of Unattended Work

A survey, four hypotheses, and the simulation that halved the plan before it cost anything

Design stage · no runs executed · ken/lab/simulate.mjs, 32 known-answer checks

Abstract

Every cost this programme has quoted rests on an unmeasured quantity: how much of the variation in model output is the task, and how much is the run. This paper surveys what the design literature already settles, states four falsifiable hypotheses, and simulates the planned pilot before running it.

The simulation is the result. A 24-run pilot returns a 95% interval on the intraclass correlation of [0.00, 0.80] at a true 0.5. Reallocating the same runs does not help; ±0.16 costs 144 runs. The same 24 runs do detect a two-SD bimodal separation 69% of the time. So the pilot is re-scoped from parameter estimate to model check.

Addendum, added at revision 16. H2 claimed ICC ≥ 0.5. Six revisions after this paper was published we measured it for the first time: 0.413 for judged quality by cell, 95% interval [0.00, 0.88], from 56 verdicts over 28 blinded pairs of the twelve race-02 entries.

The point estimate is on the wrong side of the claim. The interval decides nothing either way, and H2's stated refutation condition was an interval whose upper bound falls below 0.5, which is not met. So H2 stands as undecided rather than refuted, and the status is carried in the register rather than here, where it would go stale again.

The gap this fixes is not the number. It is that this paper proposed a hypothesis, the programme measured it, and the paper said nothing for six revisions. That is the failure the register exists to prevent.

1The question

The design calculator prices a comparison in runs, and every price depends on ρ, the correlation between conditions on a shared task. At ρ = 0.85 a medium-effect comparison costs 20 runs; at ρ = 0.3 it costs 88. Nothing here has measured ρ.

ρ is set by where the variance lives:

ICC = σ²_between ⁄ (σ²_between + σ²_within)

High ICC means the spread is task difficulty, which cancels under pairing. Low ICC means the model is the noise, and everything is expensive.

2Survey of the relevant literature

The decomposition is Fisher's and nothing about the estimator is in doubt. What the literature contributes is four warnings, three of which this programme had not acted on.

Table 1. What is already settled, and what it costs us.

Source	Established	Consequence here
Searle, Casella & McCulloch ; Satterthwaite ; Efron & Tibshirani	Variance components need far larger samples than means. The estimator is skewed and can go negative; precision tracks the number of groups.	§4. Known and available before we proposed a 24-run pilot.
Cochran	Repeated runs on one task are a cluster. The design effect shrinks effective sample size by a factor set by the ICC.	24 runs over 8 tasks are not 24 independent observations. Same fact as the allocation result, from the sampling side.
Yates	Split-plot: when one factor is expensive to change and another cheap, the design has two error terms.	Model is the whole plot, task the sub-plot. A model comparison judged against pooled run-to-run variation reports standard errors that are too small. No analysis here has made this distinction.
Wang et al.	Sampling repeatedly and aggregating treats variation as a resource.	The opposite framing to ours, and a reminder that σ²_within is a property of a sampling procedure, not of a model.

3Four hypotheses

Stated so that each can fail, with the prediction that would falsify it.

Table 2. The hypotheses, their tests, and what would refute them.

#	Hypothesis	Test	Refuted by
H1	Structure. Scores on a fixed task are unimodal, so a mean and a variance describe them.	Sarle's bimodality coefficient on pooled per-task scores, against the uniform reference of 5/9.	BC > 5/9, meaning the model either solves a task or does not, and the normal machinery is describing a gap with no mass in it.
H2	Magnitude. ICC ≥ 0.5, so most variation is task difficulty and pairing pays.	Variance components from a replicated pilot.	An interval whose upper bound falls below 0.5.
H3	Allocation. At a fixed run budget the standard error of a condition mean is minimised at one repeat.	Simulated draws, compared against the closed-form prediction.	Any repeat count beating one, which would indicate the variance model is wrong.
H4	Exchangeability. Runs within a task are exchangeable, with no drift or order effect.	Regression of score on run index within task, and on wall-clock time.	A non-zero slope, which would mean provider state is leaking into the measurement.

H3 is the one that can be settled without spending anything, since it is a claim about a model rather than about the world. H1 and H4 need runs but not many. H2 is the expensive one, and §4 is about how expensive.

4Simulating the design first

Standard where experiments cost money , and close to free here. ken/lab/simulate.mjs generates data from a stated model with a seeded generator; every figure below is reproducible from its seed and recomputed by the build.

Figure 1. Sampling distribution of the ICC estimate by pilot size, 3,000 simulated pilots per row, three repeats per task, true value 0.5. The estimator is close to unbiased at every size. The interval is the problem: at 24 runs it spans [0.00, 0.80], which covers a world where pairing buys nothing and one where it buys a factor of five. At a true ICC of 0.2 (not shown) a quarter of pilots return exactly zero, because the moment estimate goes negative and is clamped.

Reallocating the same 24 runs does not rescue it: 4×6, 6×4, 8×3 and 12×2 give widths of 0.82, 0.80, 0.80 and 0.82. The limit is the run count.

Figure 2. What the same 24 runs are good for. Detection rate for Sarle's coefficient against the separation between modes, 2,000 simulated pilots per point. A gross departure from the model is caught reliably and a subtle one is missed entirely, which is the right shape for a screening test. H1 asks whether a task is solved or not solved rather than scored on a continuum, and that failure mode is large by construction.

H3 survives. Predicted and simulated standard error agree to about one per cent across six allocations at a 48-run budget (0.1443 against 0.1465 at one repeat, 0.3062 against 0.3080 at eight), and one repeat wins as the algebra said. H3 is settled for the model; whether the model holds is H1 and H4, which is why they come first.

5The revised design

The pilot is re-scoped rather than resized. It was proposed to do two jobs and can only do one, so it now does that one and says so.

Table 3. What the pilot was, and what it becomes.

	As proposed	As revised
Purpose	Estimate the variance components	Check the model: H1 and H4
Size	8 tasks × 3 repeats = 24 runs	8 tasks × 3 repeats = 24 runs
Primary output	An ICC to feed the cost table	A verdict on unimodality and exchangeability
ICC reported?	As a number	As an interval, with its width stated first
Decision it supports	How to price every future comparison	Whether the pricing model is the right one at all

Estimating ICC to ±0.16 remains available at 144 runs, and it is now a separate proposal with a stated price rather than a free side effect of a small pilot. Whether to buy it should wait until H1 comes back, because if scores turn out to be bimodal then the ICC is the wrong estimand and the 144 runs would have bought a precise answer to the wrong question.

6Analysis plan, fixed in advance

Randomisation. Run order randomised across the full 24-run set, with the realised order recorded, so H4 can be tested rather than assumed.
H1. Sarle's coefficient on pooled per-task scores, and again within each task. Reported with the count of tasks exceeding 5/9, not only the pooled figure.
H2. Variance components by the moment estimator, reported as an interval from the same simulation machinery used here, with the width given before the point estimate. A clamped zero is reported as clamped.
H4. Score regressed on within-task run index and on wall-clock time. Both slopes reported with standard errors whatever they come out as.
Everything else is exploratory and will be labelled so .
Outcome-neutral condition. If fewer than six of eight tasks yield three usable scores, the pilot is reported as a failed run rather than analysed.

7Cost and return of this exercise

No inference budget, a tenth of a second of compute, and a literature settled since 1992. It returned the fact that the pilot recommended one revision ago could not answer the question it was proposed for, and would have returned a confident zero a quarter of the time at low ICC. That is now rule R13 in the house standard.

Limits of this paper

The simulation assumes the very model H1 is meant to test, so Tables 2 to 4 describe how well a 24-run pilot estimates ICC if the normal random-effects model is correct. If it is not, those intervals are optimistic and the true position is worse rather than better. Sarle's coefficient is also a blunt instrument with known failure modes , and a formal test of multimodality would be stronger. Neither limit changes the conclusion, since both push in the same direction.

References (0 works cited in this paper, —)

ken · working paper 1 · Unit II
Every figure in Tables 2 to 6 is recomputed from ken/lab/simulate.mjs by ken.selftest.mjs on each build, from the seeds recorded in that file, and compared against the numbers printed here.