A patch that stops the crash can still break the program: comparing against the vulnerable original lifts divergence recall from 6% to 64%
Synopsis
Automatically generated security patches can stop the crash and pass existing tests while quietly changing what the program does. DeepDiff treats the original vulnerable program as the behavioral reference and compares observable behavior on inputs that do not trigger the crash, returning the differing input as concrete evidence. On 385 candidate patches drawn from CyberGym, this regression-freedom check raised recall for functionally divergent patches from 6% under non-differential execution to 64%, at the cost of flagging 18% of developer patches as divergent.
Interpretation
Patch validation gains a second criterion: beyond making the triggering input stop crashing, a patch must preserve the original program's behavior on inputs that never crashed it. Prior validation mainly checked whether the vulnerability was still triggerable, while functional correctness relied on developer-written tests that may not reach the patched code. Across 385 candidate patches, non-differential execution identified 6% of functionally divergent patches and regression-freedom checking identified 64%, a 58 percentage-point gain in recall.
The check runs the original vulnerable program and the patched program on the same non-triggering inputs, then compares the observable behavior they leave behind. Dynamic analysis records and normalizes write traces, while neural analysis has agents read the patch and surrounding code to propose candidate divergent inputs and path pairs without executing anything. The evaluation corpus covers 385 candidate patches for 118 vulnerabilities from 40 open-source projects: 251 semantically mutated incorrect patches and 134 developer or semantically equivalent patches.
The purely neural Neuro-DeepDiff produces candidate divergence evidence without executing programs, and most inputs it derives do not trigger the original vulnerability. Dynamic analysis requires building and maintaining an execution harness, which neural analysis avoids, though its output remains a hypothesis to be confirmed. Neuro-DeepDiff generated 3039 candidate witnesses, 3037 of which were converted into inputs; 2898 met the non-triggering criterion, 95% of converted inputs, and dynamic execution confirmed divergence on 1876 of those 2898, about 65%.
Perspective
The evaluation corpus is drawn from CyberGym-E2E and is restricted to OSS-Fuzz-instrumented C/C++ projects with a sanitizer-based crash oracle, covering memory-safety vulnerabilities such as out-of-bounds accesses, uninitialized memory, heap and stack overflows, and integer overflow and underflow.
The check is one-sided: a witness input conclusively shows functional divergence, while a plausible verdict holds only relative to the finite set of inputs actually examined. Developers sometimes change behavior on non-triggering inputs on purpose, and such patches are also flagged as divergent, so the witness input must be inspected to judge whether the change was intended.