Can offline evals ship a change on their own? On 183 headline experiments the pooled eval-to-online link was too weak, so every change went to an A/B test
Deciding whether a change can ship on offline evals alone requires that eval gains predict online gains across comparable changes, not that the eval agrees with human judgment; on 183 independent interventions from the Upworthy archive, both sides were measured precisely yet the pooled relationship stayed weak, and the illustrative gate sent every change to an A/B test.