Injection probes were reading the conversation, not the injection: matched negatives cut false alarms from 80% to 17%

Linear probes used as cheap monitors for indirect prompt injection can score well on the standard evaluation by tracking where the agent is in the conversation and how much text it has just read, so a high score does not show that the probe detects the injection itself. The standard evaluation takes the other moments of attacked runs, before the injection arrived or after it, as negatives, so positives and negatives also differ in when they occur and in how much text the agent has just read. A logistic regression on seven conversation statistics that never look at the model reaches 0.96–0.98 AUROC on the standard test in seen applications and 0.91–0.96 in new ones, close to the standard probes at 0.97–1.00; the four position numbers alone reach 0.80–0.86.

Adding each attack's clean twin and a harmless look-alike in the same formatting as matched negatives removes most of the shortcut, and the fix changes only the training data. Trained on the standard data, probes flag most clean twins and nearly all harmless look-alikes at the default threshold of 0.5. In applications seen during training, median false alarms across six models fall from 80% to 17% on clean twins and from 98% to 11% on look-alikes; in new applications they fall from 70% to 6% and from 94% to 10%, while the same classifier on the standard data still flags most of both.

With a threshold set on normal traffic, the retrained probe still stops most attacks that fooled the agent while sending far fewer items to the slower second check. At the default threshold, the standard probe sends a median 315k of every million tool outputs to the second check in seen applications. At the training-app threshold, the retrained probe stops a median 95% of successful attacks in seen applications against 79% for the standard probe, at about 74k checks per million outputs, roughly 21 hours against 88 hours at one second per check.

The same matched-negative data also helps a text-only filter, but the probe's advantages are reused computation, plain-text attacks, and a threshold that carries over to new applications. A TF-IDF text classifier trained on the standard data learns the formatting shortcut too, flagging 81% of look-alikes in seen applications. Trained on the matched-negative data, that text classifier flags only 4% of look-alikes, but in new applications with the training-app threshold it stops 38% of successful attacks against 82% for the probe.

The results rest on one benchmark, AgentDojo, covering four applications: banking, an office workspace, Slack, and travel booking, with probes reading hidden states at the last token before the agent's reply in six open models. The intended reader is a team hosting an open model and routing routine tool work to smaller models, using the probe as a fast first filter that sends suspicious tool outputs to a slower second check. In applications seen during training, matched negatives cut false alarms for every model tested; in new applications false alarms stay low, but detection depends on the model and on how the threshold is set.

In new applications, detection depends on the model and on how the threshold is set, and the middle-of-the-model layer rule is not the best layer for every model. Cost estimates assume a second check takes about one second and a 1% attack rate, and are a projection rather than a measurement. The main look-alikes use the formatted template, and other templates are tested only as look-alikes. All runs use a single system prompt, and the labels mark exposure to injected text rather than whether the agent obeys it. Detection in seen applications may partly rest on AgentDojo's reused attacker accounts, and no fresh accounts were tested.