CodeScan audits code-generation LLMs with black-box, vulnerability-oriented scanning, reporting 97%+ poisoning detection accuracy across 117 models
The work presents CodeScan, a black-box, vulnerability-specific scanning framework for auditing poisoning and backdoor attacks in code-generation LLMs: it identifies attack targets by analyzing structural similarities across multiple generations conditioned on different clean prompts, combining iterative divergence analysis with abstract syntax tree (AST)-based normalization to abstract away surface-level variation and unify semantically equivalent code, then applies LLM-based vulnerability analysis to determine whether the extracted structures contain security vulnerabilities and flags the model as compromised when such a structure is found; evaluated against four representative attacks under both backdoor and poisoning settings across three real-world vulnerability classes, experiments o