Event forecasting stops ranking on accuracy alone: aggregating a diffusion model's realism beats dedicated predictors on TMS
Synopsis
Event sequence forecasting has long been ranked by prediction accuracy alone, even though generative models can produce many possible futures. A benchmark for heterogeneous event sequences such as card transactions measures accuracy and distributional realism together and finds no method leads on both: diffusion models lead on realism, dedicated predictors on accuracy. A post-hoc aggregation called Wasserstein BaryBooster collapses a diffusion model's samples into one trajectory, reaching the highest TMS on three datasets while keeping non-trivial cDFS.
Interpretation
Forecast quality is split into two separate axes: prediction accuracy measures how close a forecast is to the actual continuation, and distributional realism measures how hard it is to tell generated continuations from real ones. Prior evaluations, in temporal point processes and beyond, focused almost exclusively on prediction accuracy, leaving realism unmeasured. On three card-transaction datasets the two rankings diverge sharply: the latent diffusion model is the only generative model with non-trivial cDFS, while the strongest non-diffusion predictor, DEF, sits near zero, meaning a simple discriminator separates its continuations from real ones almost perfectly.
Two metrics cover an arbitrary number of numerical and categorical features: TMS scores event-by-event accuracy, and cDFS scores how hard a learned discriminator finds it to separate generated from real continuations given the same history. Existing accuracy metrics look one step ahead or, like OTD and T-mAP, leave numerical features aside, while realism metrics ignore the history. TMS replaces hand-chosen edit costs with baseline-rescaled normalization and a single shared Hungarian matching across features; cDFS is a conditional variant of the classifier two-sample test whose discriminator sees the history, and its ranking stays stable across eight discriminator configurations with Spearman correlations from 0.91 to 1.00.
Wasserstein BaryBooster aggregates multiple stochastic samples from any base generator into a single continuation optimal under TMS, improving accuracy without retraining. A generative model predicts many possible futures, but accuracy rewards a single continuation, so that potential went unused. On the three datasets the boosted latent diffusion model reaches the highest TMS at 0.495, 0.436 and 0.522 while retaining cDFS of 0.177, 0.256 and 0.224; boosting also improves metrics it does not optimize directly, such as amount R1 on Alphabattle rising from 0.479 to 0.583.
Perspective
The benchmark covers three card-transaction datasets whose sequence lengths and user counts support long-horizon forecasting, with each event carrying at least a timestamp, a transaction amount and a merchant category code. To probe transfer beyond transactions, evaluation extends to two point-process corpora, Retweet and Amazon, which have far shorter sequences and events carrying only a timestamp and one categorical mark; the accuracy-realism trade-off and the leading models on each axis carry over. Aggregation happens at inference time, and the sampling stage parallelizes across GPUs.
Among public corpora, only transaction data met the size, length and structure criteria simultaneously; no non-financial corpus qualified, a limitation of the current public data landscape. Aggregation requires generating multiple candidate trajectories and solving an iterative barycentric procedure, adding substantial inference overhead that may be prohibitive in latency-sensitive or resource-constrained deployments, though it is an inference-time choice and fewer samples retain most of the TMS gain. cDFS relies on a learned discriminator and is noisier in the intermediate score range, so relative rankings should be read rather than absolute values.