Playing log(N)-Questions over Wikipedia Abstracts: Communication Efficiency Between Paired Frontier Models

The report provides a frozen, reproducible protocol for the game over nested document sets from N=4 to N=1024 with prefix-extensible target assignments, and six complete model arms of 408 games on identical documents and targets, making every cross-model comparison paired. The original Potash and Suleman (2019) game trained both agents end-to-end over a Gumbel-softmax discrete channel on small sentence sets; this work runs pretrained frontier models zero-shot, where agents never co-adapt, so it measures whether a protocol already exists rather than whether one can be learned. Six complete arms, 68 games each, on identical documents and targets; nested document sets with bit-reversed target emission for even prefix stratification; code, corpus and all per-game logs released.

Pooling the five leading models, win rate declines monotonically with set size at r=-0.973 and is fit by a single per-round reliability parameter p=0.928 in win=p^{log2 N}; inverting the fit, coin-flip success needs p>=0.933 at ten steps and >=0.986 at fifty. The report attributes the decline to the number of opportunities to fail rather than to harder individual questions, noting that at p=0.93 three rounds are survivable and ten are not, and that improving per-round reasoning quality changes nothing unless it raises p. A fit across nine set sizes spanning three orders of magnitude; every model's individual correlation is negative (-0.88, -0.83, -0.90, -0.82, -0.63, -0.46); the per-round failure rate is flat across the horizon (chi-squared=10.5, df=9, p=0.31) and a failure at one round does not raise the chance of one at the next; the authors state the independence assumption is not tested and that p is a descriptive fit to five models pooled.

By adjudicating only two documents per game, the target and the model's guess, the report reduces the adjudication burden from O(N) to O(1) per round and decomposes losses under three independent judges, 2,931 judgements each, into answer errors (47-55%) and discrimination failures (42-49%), with only 2-4% prediction errors; inter-judge agreement is 97.8-98.7% and a self-preference discount of 40-46% is measured. The report separates the single win/loss bit into three mutually exclusive failures and finds models almost never name a document their own evidence excludes, locating failure upstream rather than at the final inference step. Three judges (Gemini 3.8 Flash, GPT-5.6 Sol, Grok 4.6) over roughly 2,900 shared pairs each; every per-model figure excludes that model's judgement of itself; the authors state that high agreement establishes reproducibility rather than correctness and that no human-adjudicated subset was collected.

Information per question, estimated from answer balance, correlates with win rate at r=+0.88; the only two models to extract a full bit per question are the only two that partition on document titles, a strategy absent below N=32 and used in roughly a quarter of questions above N=64. The report offers a judge-free measure of partition quality computable from logged answers alone, and identifies a mechanism by which a model can beat the granularity limit of semantic categories: a first-letter partition can be made exactly even by counting. Answer balance versus win rate across six arms; at N>=64 GPT-5.6 Sol uses lexical questions in 83/320 and Kimi K3 in 69/320, Grok 4.6 in 10/320, and the other three in none of 320; the authors note the relationship is not exact (Grok reaches 75% on 0.925 bits, better than GLM-5.3 on 0.990) and that six points cannot establish a functional form.

The result applies to a zero-shot communication setting within a single provider, where both roles run on the same model and the questioner receives no feedback, over document sets of 4 to 1024 Wikipedia lead paragraphs with one document set per size and eight games per cell. It is directly usable for designing long-horizon multi-step agent evaluations, for testing communication reliability rather than single-step reasoning quality, and for local ablations on open-weight models; the report also notes the structure can be re-instantiated with other candidate modalities as a multimodal benchmark.

The authors list several open questions: each size uses a single document set, confounding N with set difficulty and weakening claims about the shape of the N-curve; three of six providers do not permit temperature control, so those arms are non-deterministic; prompts were revised four times during a pilot against Claude Opus 5 and then frozen, and that model finished last, which the authors argue runs against rather than produces the headline result but which cannot rule out that prompts shaped around one model's idiosyncrasies fit the other five better or worse; the roughly 126,000-token prompt at N=1024 is a different fraction of each model's trained context (about 13% for Kimi K3, about 63% for GLM-5), so context load is not equalised; architectures are undisclosed for four of six models, so mechanistic accounts are hypothesis rather than explanation; one flagship per provider is an imperfect protocol because tier structures are not commensurable; effort levels are labels rather than a scale; and answer balance is an indirect estimate that conflates question balance with answerer bias. The report also notes that locating refused documents only establishes that removing a document makes the set pass, that whether Z.ai would also refuse the other two documents was never tested, and that the control on the Simon Cheng paragraph cannot separate a sexual-content keyword from a political narrative.