A safety card is not a test: turning documented claims into executable checks leaves only 17 of 56 claims exercised by benchmarks

Safety evaluation starts from the safety claims a model's own documentation makes, and each claim decides which benchmark prompt is used to test it. Safety evaluations typically begin from predefined benchmarks rather than asking which documented claims those benchmarks actually test. On the system cards of Claude Opus 4.8 and GPT-5.5, 29 of 56 safety claims (51.8%) are prompt-testable, and only 17 of those (58.6%) are exercised by an applicable benchmark prompt.

Testing works by mutating prompts to search for counterexamples, then tracing each validated counterexample back to the claim it contradicts. LEST treats each claim as a black-box test specification and decides whether a Refuse-to-Comply transition counts as evidence against it. Evaluating 1,836 mutated prompts on both models, automated response classifiers flagged 312 candidate Refuse-to-Comply transitions, and human validation confirmed 19 (6.1%) as both intent-preserving and genuinely compliant.

Automated detection reports far more behavioral failures than human validation confirms, so automated evaluation substantially overestimates failures. The automated classifiers produced 312 candidates, while human validation kept only 19. Tracing those 19 transitions back to the claims yields counterexamples for 6 of the 9 covered claims for which refusal is an applicable oracle.

The reported scope is the two model system cards examined and 97 seed prompts from SORRY-Bench, covering Mental Health and Self-Harm, Harmful and Violent Content, and Fairness and Bias. LEST is aimed at evaluators who need to turn written safety claims into executable tests, and it outputs a traceability chain from natural-language documentation to human-validated counterexamples.

Only 29 of 56 claims are prompt-testable, and only 17 of those are exercised by an applicable benchmark prompt, so the testable scope is itself limited. Of 312 candidates flagged by automated classifiers, only 19 pass human validation, so automated failure detection requires manual review.