Texture-driven visual learning suffers low-frequency shortcuts: pruning low-frequency components raises in-distribution accuracy by up to 10% and out-of-distribution accuracy by up to 40%
This work analyzes shortcut learning in texture-driven visual domains and compares it with a shape-driven standard benchmark, finding that texture-driven domains base most decisions on a few low-frequency components (LFCs) with skewed spectral behavior even though higher-frequency components (HFCs) have higher predictive power; pruning LFCs from training and test sets mitigates the shortcut and yields more balanced spectral behavior, improving in-distribution accuracy by up to 10% and out-of-distribution accuracy by up to 40% under algorithmic and real-world domain shifts, while general-purpose and domain-specific foundation models can also suffer from low-frequency shortcuts.