Probability measures get sparse dictionaries: embed once by optimal transport, then train a plain autoencoder
Synopsis
Embedding probability measures into a Hilbert space before learning a sparse dictionary turns training from repeated Wasserstein distance solves into one autoencoder optimization, cutting reconstruction loss by roughly two to three orders of magnitude against Wasserstein dictionary learning under the same time budget.
Interpretation
Measure-valued data can now be given sparse dictionaries much as vectors are: each measure is first mapped to a point in a Hilbert space, and dictionary atoms are sparse nonnegative combinations of those points. Wasserstein dictionary learning instead optimizes directly in the nonlinear Wasserstein-2 geometry, where synthesis can only be solved iteratively by fixed-point methods that repeatedly compute the Wasserstein-2 distance. On Pavia hyperspectral pixels and MNIST, within a fixed 1000-second budget, the linearized methods reach mean reconstruction losses roughly two to three orders of magnitude lower than Wasserstein dictionary learning.
Training is fast because the expensive transport map is computed once, after which everything reduces to finite-dimensional weighted sums. Linearized optimal transport represents each measure as a transport map from a fixed reference measure, so encoder inner products, decoder combinations and reconstruction losses all become finite-dimensional weighted sums. The encoder is LISTA-type, keeping an iterative affine-thresholding form whose weights are not tied to a fixed ISTA update, and is trained end-to-end with the atoms using AdamW and a cosine learning-rate schedule.
One framework covers different dimensions and tasks by swapping how atoms are parameterized. Exact transport maps are available only in one dimension, entropic transport maps give a finite-dimensional parameterization in arbitrary dimension, and Kantorovich potentials replace vector-valued maps with scalar functions. On 1800 three-dimensional point-cloud measures from ModelNet-10, a linear classifier trained on the learned coefficients reaches 96.1% held-out test accuracy.
Perspective
The report covers the full paper: the method is evaluated on a synthetic known-dictionary recovery problem, on Pavia, Botswana and Salinas A hyperspectral images, on ModelNet-10 three-dimensional point clouds, and on Pythia-410M hidden-state distributions, while the theory gives generalization bounds that depend on covering numbers of the atom class. It suits researchers and engineers working on measure-valued representation learning, optimal transport applications or sparse coding. The paper proposes future work on guaranteeing atom identifiability under an LWDL generative model and on understanding the impact of the base measure, potentially by introducing it into the learning process itself.
The paper notes that the rates for OT-map dictionaries are cursed by dimensionality, while the rates for entropic map dictionaries are parametric. In the synthetic experiment, reconstructions capture large-scale features of data points but fail to resolve fine details obscured by local jitter, which the authors posit comes from a base measure that is significantly more diffuse than the data. On The Pile, some atoms are strongly concentrated on a single source, and some pairs of atoms have nearly identical top documents, indicating redundancy among learned atoms that the authors suggest could be mitigated by increasing sparsity or filtering correlated features.