Creativity benchmarks are really testing the tail: scoring only a benchmark's rarest items tracks human preference better
Synopsis
Sorting creativity benchmarks by how rare their items are under a reference corpus makes the tail subset's model ranking agree more closely with LM Arena creative-writing human preferences than the peak subset, so creativity evaluation can be read as task performance in the long tail.
Interpretation
Creativity benchmarks sit further into the tail of a reference distribution than standard capability benchmarks. Creativity and standard benchmarks had been treated as separate evaluation families whose task and metric designs differ widely and can yield contrasting conclusions. Across 15 benchmarks in three task domains, writing, mathematical reasoning, and free-form QA, creativity benchmarks show higher input and output novelty than standard benchmarks in the same domain, and pooled tests in the writing domain show the two distributions differ significantly.
The method turns each item's rarity into a score and cuts the benchmark into equal-sized bins along that score. Previously, comparing models on creativity meant averaging over a whole benchmark or building a separate creativity benchmark. Rarity is the fraction of n-grams absent from the Pile pretraining corpus, input and output rarity are combined by a geometric mean into a shared score, items sharing an input stay in the same bin, and models are ranked within each bin by mean task quality.
Within a benchmark, higher-tail bins produce model rankings that agree more closely with human creative-writing preference. Peak bins agree with the human ranking at close to random levels. On Arena-Hard Creative Writing, bins 4 and 5 reach clearly higher agreement with the LM Arena creative-writing ranking than bins 1 to 3, GPQA-Diamond shows the same direction, and at ten-bin resolution the association between bin-level novelty and ranking agreement is statistically significant on both.
Perspective
The analysis addresses evaluation designers and model developers working with existing creativity and standard benchmarks, covering writing, mathematical reasoning, and free-form QA, with the Pile pretraining corpus as the reference distribution and n-gram absence as the rarity measure. The authors suggest that a benchmark serving scientific researchers may benefit from a reference distribution emphasizing literature from the relevant field, and recommend stating the reference distribution, evaluated tail region, and metric choices alongside the benchmark.
The authors state that n-gram statistics capture surface-level rarity and do not directly measure semantic, conceptual, or culturally grounded novelty, and that other reference distributions such as human-authored creative corpora, model self-samples, or domain-specific corpora may lead to different tailness estimates. Model rankings may also depend on the chosen quality estimator, which can exhibit architecture, length, style, and domain biases. The strongest human-alignment analysis relies on the LM Arena creative-writing ranking, which reflects holistic preferences from self-selected users rather than judgments that isolate creativity, and that ranking is itself strongly coupled with overall model capability, so high tail performance may also reflect general capability, domain knowledge, or robustness to rare inputs.