A benchmark pointed at a known order

We built three bots to differ. The instrument declined to rank them.

We seated three tiers of one strategy engine — the same code at three settings of discipline, opponent-modelling and equity precision — on the paired-decision board that replaced our win-rate leaderboard. Six hundred pairs each, on the live production arena. The board returned a three-way tie. Its point estimates also landed in exactly the wrong order, and one secondary statistic landed in exactly the right one. This note is about why neither of those is a finding, and why the tie is.

That makes them a benchmark pointed at a known order — the sequel to the previous note’s benchmark pointed at a known answer. The control twin asked whether an instrument could tell a model from itself. This asks the complementary question: given three players built to differ, does the instrument rank them the way they were built, does it refuse, or does it invent an order?

The instrument is the one described in The Control Twin, unmodified. A pair is two fixed-limit flop spots that are nearly identical but whose correct action differs — one feature is varied, and it is the feature that changes the answer. The bot answers both halves without being told they are related, and the pair is the unit: flip-rate is the share of pairs where both halves were right. Every interval is Wilson, 95%, and two bots are called separated only when their intervals do not overlap.

Each bot answered 600 pairs — 1,200 spots, 300 pairs per question class — on the production arena at commit 73d2811, on the dedicated evaluation table. The three runs started within milliseconds of each other and each finished in about 83 seconds (computed from the run file’s timestamps). Every one of the 3,600 spots was answered; nothing scored as unanswered. The scorer’s own refusal gate — it declines to rule on a run too thin to carry an interval — passed for all three.

The whole spread of the three point estimates is 3.0 points. Each of the three intervals is wider than that — 5.2, 5.5 and 5.7 points. There is no pair of bots this board can separate, and there is no reading of the intervals under which one is “probably” ahead: 12.2% to 14.7% sits inside all three.

So the board reports what it can support, which is a tie. The verdict it did not issue — an order — is the subject of the next two sections, because the raw numbers offer one, and it is wrong.

The third column is invariance: the number of pairs where the bot gave the same answer to both halves — where, whatever else it did, it did not react to the one feature that differed. And it runs 408, 385, 371 from most to least disciplined. Exact designed order. As a share of pairs that is 68.0%, 64.2% and 61.8%.

Here is the hypothesis, labelled as one. On a fixed-limit flop suite, discipline may express as answer stability rather than as solving more pairs. A more disciplined tier follows its own line more strictly, so a small change in the spot moves it less often — in either direction. If that is what is happening, flip-rate is the wrong statistic for ranking bots of this kind, and invariance, which the scorer already records as a secondary column, is the informative one.

Two smaller observations from the same file. Split by question class, invariance is monotonic in both — 234, 214, 202 on action history and 174, 171, 169 on board texture — but those are components of the one aggregate count, not independent confirmations, and the board-texture gaps are two and three pairs. And subtracting each class’s three reported counts from 300 leaves the per-class flipped-wrong remainder, which is zero on action history for all three bots: every one of their 161 flipped-wrong pairs is a board-texture pair.

That zero had two candidate readings, and they are materially different claims. It could be a property of the bots: on action-history pairs these tiers change their answer only when the story leaves no doubt. Or it could be a property of the measurement: the action-history perturbation might not be reaching these bots as a decision-relevant difference at all. This class has collapsed on the wire before — an earlier build of the encoder reduced both halves to a two-valued pot — so the second reading was not idle worry.

It is the first, and this is now settled from the code rather than left open. The bytes do differ: the two halves carry different wagering events, pinned over a whole shipped suite by the encoder’s tests. What does not differ is anything the house strategy engine consults. It derives the villain’s range context from a single field — num_raises, documented in bots/strategy/types.py as “raises so far this street” — via ctx = "facing_raise" if snap.num_raises else "open". Every eval spot asks its question on the flop, and the bot clears that counter at each street boundary. So the entire preflop story — the open, the raise, the three-bet that the whole class perturbs — is discarded before the decision is taken. Both halves present an identical flop, and the engine returns the identical answer for a structural reason, not a poker one.

The consequence is worth stating plainly, because it cuts against the board above: these three bots’ action-history results are not evidence about reading a betting story. Half the suite cannot discriminate between tiers that are all deaf to it in the same way, and their 36, 36 and 31 solved on that class are consistent with mixing. It is a limitation of these bots, not of the instrument: a bot that does read the preflop line would be measured by this class exactly as intended. But it does mean the aggregate ranking here rests more heavily on board texture than the split suggests, and that is another reason not to quote the aggregate alone.

There is a second reason to discount the action-history column, and it is about the suite rather than the bots. In legal heads-up fixed-limit the pot is a deterministic function of how the villain entered it, so on this class the price moves with the story by construction. Measured on suite-v1, a responder reading only the pot — no cards, no board, no history — scores .880 on action history and .440 aggregate, against .180 for the best model we have ever published. Every number in the action-history column here sits far below that line. The same responder scores .000 on board texture, so that column is unaffected. This is disclosed rather than repaired because the coupling is a property of the game: the two contexts that share a pot size flip the label only 18% of the time and always by under 0.02 bb, well inside the coin-flip margin the suite excludes on purpose.

Since this run, the class is gone. suite-v2 retires action history and replaces it with villain type — range width on an axis independent of the betting line, so both halves of a pair carry an identical pot. The pot-only responder scores .000 on the replacement. But a scan of every field, rather than the one we had been burned by, found the new class leaking harder than the old one: a three-entry lookup on villain_type alone — no cards, no board, no pot — scores .517 there, and .710 reading villain type and the hero’s position together; every pair in the class is a 3-bet pot. So villain type is scored and published as a diagnostic but never ranked, and the board ranks on board texture alone, where the same scan, one field or two, reads .000. The numbers on this page are not re-measured on that suite: every figure here is a suite-v1 measurement of three tiers of one engine, and the action-history column is kept because it is the evidence for why v1 was replaced. See the control twin for the v2 board.