AUROC alone picks the wrong warning probe: Prewarn scores alarms against the onset, and two models warn before most loops

Evaluating a monitor of language-model failure requires scoring whether an alarm comes before the failure begins, not only whether the finished response failed. Earlier monitors were mostly scored with AUROC, accuracy, or recall at a fixed false-positive rate, and these metrics count every unit after the onset as a positive, so they reward flagging the failure rather than flagging it early. In the repetitive-degeneration case only a few percent of responses degenerate and every degenerate response runs to the token limit, so a rule that reads only response length reaches an almost perfect AUROC on Apertus, although its only flag falls after the budget is spent.

Prewarn evaluates a monitor as it would be deployed: the threshold is fixed on validation data, then alarms are scored on test data by their position relative to the onset. The protocol needs two things earlier evaluations often lack, an annotated onset and a threshold fixed in advance at the error rate the application chooses. On Apertus, Llama-3.1-8B-Instruct and Mistral-7B-Instruct, with repetitive degeneration as the case study, thresholds are calibrated on validation data to a false-alarm budget, and the false-alarm rate on test stays within that budget.

Linear probes can warn ahead on some models, while low-rank adaptation mainly widens coverage of the warning window and barely raises how often the first alarm precedes the onset. Frozen probes read the model's own activations and raise their first alarm before the onset on more than half of the loops of Apertus and Llama, with the median first alarm tens of tokens before the onset; on Mistral that share stays below one half and the median alarm comes after the onset. Comparing the frozen probe with the adapted probe at the same layer, adaptation more than doubles warning coverage on every model but raises the early-warning rate little, and none of six untrained baselines warns much more often than random alarms.

The demonstrated setting is repetitive degeneration under stochastic sampling in three open instruction models, with human-verified test labels and thresholds calibrated on validation data to a false-alarm budget; the authors leave other failures such as harmful content and hallucinated entities to future work.

Lead times depend on the onset convention, and placing the onset at the first token of the first loop makes every alarm come earlier relative to it; a response that loops and then emits the end-of-sequence token counts as healthy, which makes false-alarm rates conservative; on Apertus and Llama the onset usually comes after a healthy response has ended, so part of what the probes flag may reflect how long the response has run; the training strategy and selection rule were chosen on degeneration alone.