Autoresearch did not make the breakthrough. Playtests gave it something real to optimize.

Swyx’s playtests produced the question that the simulation program had been missing: why did the AI miss that?

The prior context: we had already built the measurement machine

Before the playtests changed direction, the project already had deterministic headless games, paired schedules, arena dashboards, reproducible seeds, exact replay, termination checks, and AI receipts. That infrastructure was not wasted effort. It made it possible to tell the difference between a real finding and a broken harness.

But it could not supply the right hypothesis on its own. A leaderboard can say one policy wins more often; it cannot always tell us whether the policy missed a two-dot height capture that a person notices instantly.

1. 01Table signal

   “Petra missed a height capture.”

2. 02Exact record

   OGN, seed, map, and pre-move board.

3. 03Falsifiable test

   Can the engine reproduce the missed alternative?

4. 04Bounded change

   Fix rules or add one public decision feature.

5. 05Paired evidence

   Cross maps, homes, starters, and opponents.

A player’s objection becomes valuable only when it can be replayed, challenged, and measured on fresh schedules.

Research ledger

The useful loop keeps its failed checks in view.

ALLPRELIMINARYDISCARDEDINVALID

1. 01
   A player spots an impossible-to-ignore board error

   The observation becomes a named, replayable question rather than a vague hunch.

2. 02
   The record turns the hunch into a test

   OGN, seed, map, and public pre-state make the alternative inspectable.

3. 03
   A harness anomaly blocks the conclusion

   Fairness, termination, replay, or hidden-state failures invalidate the screen until repaired.

The charts were honest. They were also incomplete.

We built deterministic arenas, replay files, fairness checks, termination audits, paired schedules, confidence intervals, and visual reports. Those are necessary. They catch seat imbalance, cap-driven pseudo-results, accidental hidden information, and policies that improve margin without improving wins.

But a clean metric can still be attached to the wrong question. Several early AI changes moved almost nothing because their character weights only broke ties. Other candidates looked promising until a real game revealed that they were not actually selecting a different action.

Human play supplied the missing question

The most productive inputs were concrete board objections: Petra left an obvious height capture; Bean had a lead but failed to make the board end; an AI appeared to avoid height tiles; games ran far longer than table play. Each observation became a replayable hypothesis instead of a complaint.

That changed the work. We fixed height-award semantics and game endings before trusting more arena results. We added readable OGN, exact replay, and an AI decision audit so a player and a developer could point at the same pre-state. We stopped treating a policy label as evidence of a personality.

These are not retrospective wins claimed from a single game. They are research-quality upgrades: a game can now be replayed, its legality checked, its finish reason inspected, and an AI’s first divergence isolated before an arena result is believed.

The development loop we want

Play. Preserve the exact state. Identify the first move that feels wrong. Reconstruct the public alternatives. State a falsifiable mechanism. Add it behind the shared competence rail. Then measure it on paired games—and keep the counterexamples.

The role of automation is not to replace judgment with an ever-larger tournament. It is to make judgment legible, repeatable, and difficult to fool. Human play supplies the surprise; deterministic replay supplies the evidence; the arena tells us whether the surprise generalizes.

What would count as a breakthrough?

A player observation is a lead, not proof. A real promotion still needs a frozen policy, paired maps and seeds, crossed homes and starters, clean termination and replay checks, a win-rate lift with uncertainty, and a counterexample that survives review. That standard is intentionally slower than a good playtest reaction—and much faster than doing thousands of games around the wrong question.

What we learned about “smarter”

Smarter does not mean more static weights or more random variety. It means an AI can notice a public tactical fact, connect it to a bounded consequence, and explain why it changed course. The new Ender work is deliberately modest on that standard: it records whether it used reply evidence or a public threat proof, rather than inventing a personality story after the game.