RESEARCH NOTES · REPRODUCIBLE SEARCH
Behind the AIAutoresearch needs a contract, not just a hillclimb
An autoresearch loop can produce exactly the visual you want: a tidy sequence of ideas, dots climbing upward, and annotations celebrating each clever change. It can also reward the first thing that got lucky on three seeds.
Our first collective campaign taught this in the least glamorous way possible: by declining to promote a live policy.
[DISCARDED] Autoresearch v1, fixed historical snapshot. The campaign ran 7,776 valid two-player research games and evaluated 54,514,120 candidate constructions. It generated useful hypotheses and a stronger protocol; it did not establish a live winner.
A lead is not a promotion
The campaign tested changes to the collective Alai, Petra, Ender, and Bean program against frozen controls. It began deliberately in heads-up play, where most actual games were expected to occur, using exact paired maps, seeds, public hands, physical seats, starters, rules, horizons, candidate budgets, and seeded-choice identity.
One idea looked especially tempting: price connected routes to height through the actual public board graph rather than using simple Manhattan proximity. On three untouched seed clusters it showed a +0.84 paired advantage. The 95% interval was wide: −5.17 to +6.84. The right decision was “deepen,” not “ship.”
On 12 new seeds the apparent lead disappeared: −0.35, with a 95% interval of −1.10 to +0.39.
[DISCARDED] Connected-height route cost function. The route topology seam remained interesting; this exact cost function did not survive confirmation.
The story is not that route topology was foolish. It is that the evidence had not yet earned the conclusion.
More compute did not monotonically help
The same lesson arrived through a different door. A measured duel-pressure branch was +0.47 points in a short 3-seed, budget-100 screen. In a deeper 6-seed, budget-200 test it was −0.69. The test also changed seeds and horizon, so we cannot honestly blame “more compute” alone.
What we can say is more useful: additional search and additional reply time gave counterplay room to appear. That is exactly what a promotion gate is for.
The campaign recorded another important non-result. Corrected tactical-frontier search raised depth-two coverage from roughly 50% to 99% and nearly doubled named-AI compute, yet lost 2.23 points on the paired measure. Search breadth is not intelligence when the state signal describes the wrong plan.
The contract
We therefore turned the experiment design into a first-class artifact. A candidate must be frozen and fingerprinted before broad evaluation. Its comparison arm must share the same map, gameplay seed, public hands, home, seat, starter, rules, horizon, candidate budget, and seeded choice. Game outcomes must preserve natural completion, flip-war adjudication, caps, and failures as different classes.
The essential promotion questions are deliberately boring:
| Question | Why it exists |
|---|---|
| Did the exact replay reproduce? | A winning number is meaningless if the game cannot be reconstructed. |
| Did both arms get the same work and schedule? | Otherwise the “policy” may be a budget or role advantage. |
| Did the lead survive fresh seed clusters and longer replies? | A short local advantage may be a trap for the policy itself. |
| Did a cap, loop, or terminal anomaly occur? | A censored outcome cannot be quietly counted as a win. |
| What changed in a move trajectory? | A static weight that never changes an action is a costume, not a mechanism. |
Read left to right. Each card names the claim it is allowed to support.
- QuestionFreeze one mechanism
Name the exact policy change and fingerprint it before the broad run.
No moving target→ - ControlMatch the work
Hold map, seed, hand, home, seat, starter, horizon, and budget equal.
Same game offered twice→ - FalsifyReplay and classify
Recompute moves and separate natural endings, adjudications, caps, and failures.
Bad runs cannot become wins→ - ConfirmUse untouched seeds
Repeat after the mechanism and evaluation rules have stopped changing.
The lead must travel
The protocol also counts deterministic construction work and robust latency. Wall-clock averages stay visible, but they are not allowed to decide a promotion after host-clock discontinuities made unrelated workers appear to stall together.
The embarrassing instrument run
One early run requested budget 60. The live parser’s real minimum was 100, and it normalized the request to the default of 800. The batch ran 864 games and 14,019,220 constructions under a configuration it had not actually requested.
That result was blocked and excluded from every strength estimate.
This was not subtle statistical philosophy. It was a missing contract check. The configuration generator now rejects budgets outside the live parser contract before scheduling a match. Autoresearch is a strange field: sometimes the most valuable model improvement is an if statement that refuses to run.
The hillclimb is a research diary
We still keep the trajectory. Retained, discarded, and invalid trials belong on the same plot because the dead ends explain the surviving protocol. But the line is not a progress meter for intelligence. It is an auditable diary of hypotheses, matched evidence, and reasons not to believe ourselves too quickly.
The next campaign was authorized only after a repaired fairness baseline, adequate candidate coverage, crossed homes and starters, separate termination classes, and replay sentinels. That is slower than declaring a winner after a good week. It is substantially faster than optimizing an illusion for a month.
The separate player guide, A route is not a bank, turns the conversion idea into a table routine without making this experiment carry two audiences.
<details> <summary>Methods and limits</summary>This post describes the frozen v1 campaign rather than a current live profile. All 7,776 valid campaign games were two-player games; any future live-policy promotion also requires a separate 3–6 player robustness panel. The historical KPI was a height-inclusive paired score advantage, not a resolved-game match-point win rate. The campaign itself reports that full-game win-rate claims required longer horizons because many games remained capped.
</details>