Audit finds a multilingual affective generation benchmark's headline conclusions stem from the measurement instrument, not system differences, and proposes an emoji-affect decodability probe

Re-analysing with annotators as a random factor leaves no significant difference between any pair of the eight systems (F(7,14)=0.59, p=0.76), whereas the conventional analysis declares 19 of 28 pairwise differences significant. The benchmark's headline conclusions treated between-system differences as properties of the systems; this work shows those differences depend on whether annotators are treated as a fixed factor, and reports that annotator identity explains far more rating variance than system identity and that removing any single annotator changes the winning system. An audit over 17,100 Bangla, English and Hindi sentences, eight instruction-tuned models and 6,960 human judgements, reporting F and p values together with a leave-one-annotator-out robustness check.

The ordering that does emerge tracks output length: mean emoji count explains 78.7% of between-system variance, and a within-item length-matched comparison over 2,599 pairs reverses the leaderboard. This reframes an ordering that appears to reflect affective generation quality as a reflection of output length, and supplies a concrete length-matched contrast for testing whether the ordering is robust. Reports a 78.7% variance-explained figure and a within-item comparison over 2,599 length-matched pairs.

Several common analytic choices change the conclusions: cross-provider anisotropy differences vanish under mean-centring, per-language token costs change sign with the normalising unit, and multi-view row-wise splits inflate macro-F1 by 3.1 points and change the top-ranked system. The work links three seemingly technical preprocessing and splitting choices directly to the direction of conclusions, showing that these choices alone can flip comparative results. Evidence comes from before-and-after mean-centring comparisons, sign changes in token cost under different normalising units, and a 3.1-point macro-F1 difference under multi-view row-wise splits.

The authors propose emoji-affect decodability, a reference-based probe, in place of preference scoring, with rankings stable to ±0.003 macro-F1 across seeds. Relative to preference scoring, which is affected by annotators and length, the probe offers an alternative evaluation whose rankings are stable across random seeds. Reports a macro-F1 fluctuation of ±0.003 across random seeds.

The audit targets one multilingual affective generation benchmark, covering three languages (Bangla, English and Hindi), eight instruction-tuned models, 17,100 sentences and 6,960 human judgements, and applies to evaluation settings that rank generative affective summaries by human preference scoring. Its conclusions are aimed at researchers and engineering teams who design or use such benchmarks, to judge whether between-system differences are robust, whether rankings are affected by output length and annotator composition, and whether preprocessing and splitting choices change conclusions. The proposed emoji-affect decodability probe addresses evaluation needs that require rankings stable across random seeds.

A careful reader would still watch how the emoji-affect decodability probe performs beyond the three languages and on other subjective generation tasks; whether the length-matched comparison still reverses the leaderboard at larger sample sizes or under other length measures; and how stable the annotator-randomisation analysis is with more or fewer annotators. In addition, this reading is at summary scope and does not include figures or full methodological detail, so the probe's specific construction, reference source and implementation details, as well as the full settings of each robustness check, remain open questions to confirm against the original text.