Monitorability rankings can shift with the task mix: pair matching and post-control risk each close a blind spot
Synopsis
OpenAI's alignment researchers add two monitorability metrics beyond g-mean²: one compares models only on task instances both support, reducing ranking bias from different instance mixtures, and one simulates a monitor as a rejection gate with retries, reporting bad-output probability, deployment probability, and the bad-output share among deployed outputs. In synthetic experiments, a known monitorability ordering is recovered by pair matching as capability spread grows.
Interpretation
Monitorability comparisons can now be restricted to task instances that both models support, instead of scoring each model on its own instance set. Dataset-level g-mean² was computed on each model's own positives, so models supported different instance subsets and scores could reflect different task mixtures. In fixed-seed simulations, model B is easier to monitor on every fixed instance, yet as its capability advantage grows and its positives shift to harder instances, dataset-level g-mean² drifts toward model A while pair matching still favors model B.
Pair matching combines per-pair differences into one ordering while keeping the magnitude of each difference rather than only its direction. An initial matched Bradley–Terry rating recorded only a win, loss, or tie per instance, discarding the size of pairwise differences. Across seven synthetic models with known ordering M7 down to M1, pair-matched HodgeRank stays close to that ordering as capability dispersion increases, while g-mean² and the matched Bradley–Terry prototype degrade.
The second metric treats the monitor as a deployment rejection gate and reports, for a retry budget, how often a request deploys a bad output, deploys any output, and what share of deployed outputs is bad. g-mean² does not reflect how a monitor is used at deployment or how retries after rejection change risk. On historical system card data, gating lowers the primary risk quantity on average across three monitor scopes; on the same instances with at least four valid attempts, moving from one to four attempts increases the chance of a monitor miss and erodes part of that reduction.
Perspective
The metrics target research and engineering settings that compare monitorability across models or assess a monitor as a deployment gate. Pair matching orders models only within connected components of supported comparisons and does not compare across components; post-control risk is reported for outcome and process evaluations, and process results use only positive attempts, so they give a conditional escape rate rather than risk under the model's full output mix. The stated next step is to compare at the level of canonical instance and behavior class and to report behavior-class coverage explicitly.
Intervention evaluations cannot identify which positives the intervention caused, so causal recall is bounded rather than estimated; when the minimum effect fraction is 0.3 and the flag rate is 0.4, the interval can span 0 to 1. Pair matching also does not guarantee that two models produce equally hard or same-class behavior on a shared instance: on one Honeypot task a model openly patched the test harness while another used a short wrapper module, so a higher matched true-positive rate may only mean more legible misbehavior. Post-control risk depends on the model's target-behavior rate and does not replace g-mean².