Detector data squeezed into discrete tokens: tracks and topoclusters hold 1% jet resolution at 4–11x

Detector data in high-energy physics can now be represented first as learned discrete tokens and handed to downstream models, instead of always entering them as continuous features. The state of the art in the field operates directly on continuous inputs, and tokenization had been adopted as an ingredient to enable a backbone rather than studied for its own cost. Across four detector modalities, the learned tokenizers reach a better mean absolute error than a per-feature uniform binning baseline at every bit rate, and a better jet resolution where the 1% target is first met.

Tokenization replaces each detector object with a handful of integer indices, from which a decoder reconstructs the object. The method is a residual vector-quantized variational autoencoder, which uses several codebooks to quantize the residual left by the previous stage, mapping continuous features to indices in a finite vocabulary. Training covers four modalities from two simulated detectors: charged-particle tracks from JetSet, and tracks, topoclusters and calorimeter cells from COCOA, with cells handled as patches of per-layer images.

The report is aimed at researchers and engineers working on detector-data modeling and foundation models, and the conclusions rest on two simulated datasets: charged-particle tracks from JetSet, and tracks, topoclusters and calorimeter cells from COCOA. Calorimeter cell tokenizers reach a jet energy interquartile-range-to-median ratio of 1.24% relative to dense 32-bit pixels, still slightly above the 1% target. The tokenizers, scans and evaluations are provided as the open-source heptokens library, and trained models export their token indices and codebooks for downstream reuse.

The four design rules hold only across the set-based modalities and datasets studied here. The tokenized flavor classifier retains 73–93% of the continuous-feature rejection at 60 bits per element, with losses concentrated in the high-rejection tails and growing as bits are removed. The calorimeter cell scan uses 1.6 M events because those events are much larger. Continuous generative decoders have been shown to outperform tokenized objectives at current scale, a comparison not yet made in the large-data and multi-modal regime.