# Non-determinism in LLM Evaluation

Holding all other settings fixed and varying only the hidden current date injected into the system prompt across every day of 2024, this study of 9 recent LLMs and 6 datasets finds accuracy swings of up to 6% on multiple-choice QA, 14% on math reasoning, 7% on code generation, and 2.84 BLEU on machine translation, with model rankings reordering, an effect larger than batch size or numerical precision and one that chain-of-thought prompting amplifies rather than reduces.

Holding all other settings fixed and varying only the hidden current date injected into the system prompt across every day of 2024, this study of 9 recent LLMs and 6 datasets finds accuracy swings of up to 6% on multiple-choice QA, 14% on math reasoning, 7% on code generation, and 2.84 BLEU on machine translation, with model rankings reordering, an effect larger than batch size or numerical precision and one that chain-of-thought prompting amplifies rather than reduces.