How many rules can a model hold at once? GPT-4o mini keeps about 37–38 at 95% correct
Synopsis
Instruction-following quality falls as load grows, but there was no load-denominated capacity with an uncertainty. A measurement instrument defines load as single-word replacement rules, making each application a program-checkable Bernoulli trial, then inverts a binned-then-smoothed statistical model to obtain capacity. For GPT-4o mini, 2958 requests and 642430 trials give a capacity of about 37–38 rules at a 0.95 share of right replacements and about 18–19 rules at full replacement density.
Interpretation
Instruction load can now be expressed as one number with an uncertainty: how many rules a model holds simultaneously at a given share of right replacements. Earlier studies reported how many instructions were fulfilled at a few load points chosen by the experimenter, so numbers from different works could not be compared directly. On GPT-4o mini, 2958 requests and 642430 trials span 1 to 256 rules and texts of 20 to 10240 words, and every load factor's binned profile follows a regular shape described by a few parameters.
Load is built from rules that replace one word, and the rules are unlinked by the construction of the task. In existing benchmarks, form requirements share one text and composite constraints are judged by a model, so instruction count is confounded with task difficulty. The text is a meaningless stream of single-token nouns from a fixed vocabulary, the reference answer is produced by one programmatic pass, and line-by-line matching records each replaced word's position and rule number.
The statistical model first measures each load axis's shape with logarithmic bins, then fits smooth laws, and an approximation shift correction restores a reference level that does not depend on the coding choice. A conventional binned model needs a reference bin, and its level changes with that choice, which would make capacities incomparable. Across the five axes the largest pairwise correlation is 0.76, and after merging length and position into a two-dimensional grid all slices share one slanted asymptote.
Answer collapse and instruction capacity are separate limits: collapse is set by text load and progress through the answer, not by how many rules are held. Doubling the rule list lowers the logit of a right replacement by about 0.43, yet at fixed length and density it does not change the probability of losing the text. Of 2958 requests, 995 lost the text, 33.6%; each doubling of text length raises the collapse logit by about 2.4, while rule-count bins stay within one logit of zero.
Perspective
The demonstrated scope is the passport of GPT-4o mini (gpt-4o-mini-2024-07-18) under protocol version 0.1: temperature zero, a fixed seed, an answer limit of 16384 tokens, texts of 20 to 10240 words and 1 to 256 rules. Capacity values refer to the zero point, the beginning of the answer on a short text with sparse replacements; longer texts, later word positions and denser replacements spend the same reserve.
The author states there is no ground to assume the regularities found for GPT-4o mini are universal: larger or differently architected models may show extrema instead of monotonic change, power laws instead of logarithmic ones, or a different set of significant axes. The proposed interpretations of the volume limit, positional limit, density saturation and collapse mechanism are hypotheses requiring separate experimental tests, since the instrument measures the shape of the dependence, not its cause.