Meta Puts LLM-Written Tests Into Production: 589 Engineers On Board, Privacy Severe Bugs Down 20%

LLM-generated just-in-time tests are no longer a research prototype but part of Meta's infrastructure, generating tests before a code change lands to expose defects. Most AI-based approaches to preventing bugs use LLMs or agentic AI to review code and detect faults from prior experience, much as a human reviewer would; just-in-time tests instead generate executable tests that demonstrate a concrete behavioral difference introduced by the change. In the five months since deployment, 589 engineers at Meta, 64% of outreach, engaged with the system, and these engineers reported genuinely actionable findings with a 60% true-positive rate.

The system splits finding real bugs from controlling false positives into two tandem workflows, then adds two automated assessors and aggregate human oversight. Test generation alone yields many weak catches, meaning failing tests that may be false positives, so strong catches must be separated from weak ones. The dodgy diff workflow treats each diff as a mutant and uses mutation-guided LLM test generation to distinguish the diff's behavior from its parent; the intent-aware workflow infers the diff's intention from the code changes, title and summary, describes risks, builds mutants from them, and generates tests to kill each mutant, with tests failing on the actual diff harvested as weak catches.

False-positive filtering combines a multi-model judge ensemble with a rule-based assessor, and the two produce an overall ranking of weak-catch candidates. The LLM-as-Judge ensemble has rich semantics but non-deterministic, hard-to-verify judgments, while the rule-based assessor gives verified, deterministic judgments. The ensemble aggregates judgments from several state-of-the-art LLMs reviewing the weak-catch test code, the execution outcome, the parent-to-child diff and the inferred intent; the rule-based assessor, Rule-Based False-Positive Killing Environment (RubFake), works alongside it to decide which candidates merit further consideration.

Human involvement shifts from reviewing every output to a human-on-the-loop arrangement in which engineers set criteria and review outcomes at an aggregate level. A human-in-the-loop approach would have an engineer review every output rather than supervising at the aggregate level. The system runs overnight on spare capacity, targets the previous day's diffs deemed risky by a mechanism similar to Diff Risk Score, and then decides whether to reach out to engineers, aiming to keep human labeling consistent while reducing the chance an engineer is brought in to review a weak catch.

The demonstrated scope is a deployment on Meta's internal infrastructure, covering overnight batch runs, risky-diff targeting, two test-generation workflows, automated false-positive filtering and human-on-the-loop review, aimed at engineering teams that need to intercept defects before landing in a large codebase.

The reported 60% true-positive rate and 20% reduction in privacy severe bugs are statistics from Meta's deployment period, and the material does not say whether they hold across other codebases or defect types; the approach also requires more computational steps, and no cost comparison against review-style approaches is given.