ALICE estimates mutual information zero-shot with one pretrained Transformer, cutting error at least twofold at the 1k-sample budget on the Beyond Normal benchmark and reproducing prior findings in biology, genetics, and neuroscience data
The authors present ALICE, a foundation model trained once on a synthetic distribution family that turns mutual information estimation into in-context estimation of rectified-flow velocity fields for unseen distributions, matching per-distribution neural estimators on the Beyond Normal benchmark with at least twofold lower error at a 1k-sample budget and zero-shot reproducing and extending dedicated estimators' findings on biology, genetics, and neuroscience data the model never saw.