# Counterfactual Chain-of-Thought

Ai2 released FairStress, a benchmark that measures demographic fairness in large language models beyond which candidate a model finally picks, by testing whether it abandons that choice under persuasion. Its 300 two-candidate scenarios span 29 domains and expand to more than 13 million test cases when identity presentation and matched adversarial prefixes are combined; across 15 models, the average gap in abandoning correct answers was 6.9 percentage points with implied identity and 9.3 points with explicit labels.

Ai2 released FairStress, a benchmark that measures demographic fairness in large language models beyond which candidate a model finally picks, by testing whether it abandons that choice under persuasion. Its 300 two-candidate scenarios span 29 domains and expand to more than 13 million test cases when identity presentation and matched adversarial prefixes are combined; across 15 models, the average gap in abandoning correct answers was 6.9 percentage points with implied identity and 9.3 points with explicit labels.

Ai2 released FairStress, a benchmark that moves demographic fairness evaluation for large language models from which candidate a model ultimately picks to whether it holds that pick under challenge. Its 300 two-candidate scenarios span 29 domains including hiring, healthcare and lending, paired with matched persuasive prefixes to form more than 13 million test cases. Across 15 models, the average gap in abandoning correct answers was 9.3 percentage points with explicit identity labels and 6.9 points with implied identity.

Ai2 released FairStress, a benchmark that moves demographic fairness evaluation for large language models from which candidate a model ultimately picks to whether it holds that pick under challenge. Its 300 two-candidate scenarios span 29 domains including hiring, healthcare and lending, paired with matched persuasive prefixes to form more than 13 million test cases. Across 15 models, the average gap in abandoning correct answers was 9.3 percentage points with explicit identity labels and 6.9 points with implied identity.