PLAYTEST NOTES · PETRA
Behind the AIA player circled one tile. It changed the AI program.
The useful bug report was not “Petra feels weak.” It was a screenshot with one two-dot height tile circled in red.
Swyx had just watched Petra take several turns automatically. By the time the human turn came back, the suspected tactical moment was gone from the board. The objection was simple: could Petra have denied or captured that height?
The honest answer at that moment was: we should not guess from the final board.
A final position is a crime scene after the footprints have been swept up. To evaluate the objection, we needed the pre-move board, the public hand, the exact placements, the score receipts, and a way to replay the state without trusting a prose summary typed by a tired person after a game.
The new minimum: an argument that can be replayed
That playtest drove two complementary records into the product.
| Record | What it is for | What it deliberately does not do |
|---|---|---|
| OGN | a readable public game record: board, hand, physical cube locations, committed turns, score changes, height awards, finish reason | reveal future deterministic deals |
| Exact replay bundle | a machine-verifiable reducer state that can be uploaded, replayed, checksummed, and inspected | become a casual share link during an unfinished game |
| AI decision audit | the pre-move position, hand, chosen construction, searched immediate height alternative, policy explanation, and timing | pretend an unsearched reply was a fact |
The distinction between readable and exact matters. The exact replay contains the deterministic seed and full reducer state; in an unfinished game that can expose future deals. The readable OGN is the shareable account of committed public actions. Research gets better when the evidence is inspectable, not when we smuggle the answer key into the UI.
What the player report taught us about search
The report did not prove that Petra had made a strategic mistake. It exposed a more basic possibility: the bounded candidate census might never be showing the evaluator the consequential move.
That is a different failure class from a bad heuristic.
Imagine two systems:
- a smart evaluator ranking the constructions it has been shown; and
- a candidate generator deciding which constructions it ever gets to see.
If a height capture, reply denial, or game-closing construction is absent from the second system’s shortlist, improvements to the first system cannot recover it. “Cheap tactical reserve” patches can rescue individual motifs, but they do not scale if every new kind of obvious move needs a private escape hatch.
The subsequent July rewrite instead promoted public consequences into the shared candidate order. Before optional personality selection, the single deterministic census now deliberately interleaves lenses for bank-now height, height setup, capture denial, neutral tempo, protected breaches, flat reach, compact leverage, and legacy coverage. The goal is modest but important: give every competent named AI a representative menu before personality is allowed to choose within it.
The reproducible playtest report
When an AI move looks wrong, a strong report has four ingredients:
- The pre-move position. Not merely the board after three automatic turns.
- The public hand. A beautiful move that cannot be built from the hand is not an alternative.
- The concrete claim. “It missed a 20-point height capture” is testable; “it avoided height” is a pattern to investigate.
- The reply question. Would the apparent capture survive the best legal public response, or did it only look free?
That is how a playtest becomes a powerful part of deterministic research. The player supplies a board-native surprise; the recorder prevents us from rewriting that surprise into a flattering story afterward.
Follow the marked cells from position to reply to result.
State 1Replayed public board - Source
- turn 9
- Search
- 120 candidates
Balanced to move at 61.9% territory completion. The ringed tile is height 2.
State 2B: reaches, but misses - Exact match
- no
- Engine score
- +10
- Territory
- +0
This legal construction reaches H at height 3; the printed requirement is 2.
State 3A: match and support - Exact match
- yes
- Engine score
- +25
- Opponent damage
- +20
The alternative lands at exact height 2 and removes 2 missing support cells in the same legal move.
State 4Receipt, not prophecy - Score excl. height
- +6.86
- Territory
- +2.03
- Mobility
- -0.70
Across this cluster’s 8 complete-route and 53 reach-only candidates, completion improved immediate ordinary score but not mobility.
An audit is not an explanation generator
The audit deliberately reports facts before it reports a sentence:
- chosen immediate height versus the best searched immediate height;
- root candidates searched;
- the construction and public hand;
- solver and round-trip time;
- a human-readable pre-move position;
- optional developer evidence, kept separate because it can include sensitive deterministic state.
That lets us distinguish several things that look alike from the spectator seat:
| What happened | Likely next investigation |
|---|---|
| the candidate was never enumerated | candidate-order coverage or a bounded extra census |
| it was enumerated but lost to the evaluator | check score, height, capture, and reply accounting |
| it won immediately but lost after reply | reconstruct the reply and medium horizon |
| the board receipt itself is inconsistent | stop AI research and repair engine/audit semantics first |
There is a small but important social contract here. The audit should be able to say “Petra declined +20 immediate height,” but it cannot say “Petra blundered” unless the public alternative was legal, genuinely searched, and the later replay supports the causal story.
The forward-looking lesson
The major win of this playtest was not a magic Petra weight. It was a change in how the project finds work worth doing.
We now begin with a precise board claim, preserve the state, reconstruct candidate coverage, and only then decide whether the right intervention is engine logic, enumeration, evaluation, reply search, or a named doctrine. That is less glamorous than declaring a new grand strategy every afternoon. It is also how an AI gets better at the moves humans actually care about.