Chess engines can outplay anyone but cannot say why: a training-free method lets language models find reusable motifs and nearly triples explanation scores

Engine moves can now be explained in natural language, and the quality of those explanations has an automatic score rather than resting on whether a reader finds them plausible. Previously an engine reported an evaluation and a preferred move but not the chess ideas behind the choice, and language models writing explanations directly often stayed at surface board features. On 100 starting positions from master games, direct prompting scored 0.031 to 0.173 across eight language models, while BinProp scored 0.174 to 0.296; all eight improved, and the lowest BinProp score exceeded the strongest direct-prompting baseline.

The method first has a model discover motifs that hold more often in winning positions, then uses those motifs to explain why one move beats an alternative. Each motif is written as a Python function that takes a board and returns true or false, so it can be checked mechanically instead of remaining unverifiable prose. Motifs are filtered over 200,000 positions, 100,000 in a winning evaluation band and 100,000 in a losing band; a candidate is kept only if it fires more than one percentage point more often on winning positions and does not overlap too much with existing motifs, yielding 192 chess motifs.

Explanations are framed as a contrast: state the concrete motif that holds after the preferred move but not after the weaker alternative. The current board alone cannot settle whether a move is good, because a move's value depends on the continuations it enables and those it prevents. Scoring shows the model four positives and four negatives, then tests on eight unseen positives, eight negatives and eight nearby negatives that share the earlier moves of a positive line and differ only in the final two plies, which catches motifs that merely capture irrelevant board differences.

The method targets chess analysis settings where a reader needs to understand an engine's choice, and covers chess, Go, draughts and shogi; chess explanation scores are measured on 100 starting positions from master games, and Go, draughts and shogi each use 100 held-out game starting positions. Explanations are built from binary motifs, meaning a motif either holds on a position or does not, rather than from a continuous evaluation. The authors propose extending the same loop to explain agent rollouts spanning hundreds or thousands of steps, but that is future work not yet validated outside games.

The explanation score is an engine-grounded metric built from engine continuations: it measures whether a motif separates positives, negatives and nearby negatives, not whether human players would endorse the explanation as teaching. On draughts not every model improves, and Fable 5's direct-prompting score exceeds its BinProp score. Taking the best motif across models beats any single model, indicating that motif ranking still has room to improve. Go, draughts and shogi reuse the chess discovery settings, but engines, score bands and continuation sampling are set per game.