Two-view covariances no longer learned as one blob: CANE separates in-view and cross-view structure and cuts brain-age error to 4.97 years
Synopsis
In two-view data, each view has its own covariance and a cross-view covariance links them, yet covariance neural networks handled only single-view data and could not distinguish these structures once the views were concatenated. CANE splits the model into an in-view covariance filtering stage and a cross-view covariance filtering stage, replacing the whitening and singular-vector selection of canonical correlation analysis with learnable polynomial responses on the covariance spectrum, and trains directly on the downstream task. Across seven brain-imaging and motion-sensing datasets, CANE is best or tied for best on six, reaching a mean absolute error of 4.97 years for age prediction on ADNI1, often with fewer parameters than a two-view VNN.
Interpretation
Covariance neural networks move from single-view to two-view data, with in-view and cross-view covariances learned as two distinct relational priors rather than merged into one matrix. Earlier covariance neural networks learned polynomial filters on a single view's covariance; for two-view data they could only process each view separately or concatenate the views, and although the concatenated covariance holds in-view blocks on the diagonal and cross-view blocks off it, the filter coefficients depend only on the power order and cannot tell the two structures apart. On seven brain-imaging and motion-sensing datasets, CANE matches or outperforms single-view baselines, two-view VNNs, canonical-correlation preprocessing with a predictor, and deep task-aware CCA methods on six, and stays within one standard deviation of the best model on ABIDE.
The model has two stages: a linear graph-convolutional filterbank on each view's covariance graph, then graph-convolutional layers with nonlinearities on graphs built from the cross-view covariance of the first stage's outputs, followed by a readout module for regression or classification. Canonical correlation analysis whitens each view, then projects onto the singular vectors of the cross-view covariance, which requires inverses and a singular value decomposition and ignores the task; CANE covers the same spectral space with learnable polynomial responses of covariance filters, contains the CCA transform as one point in its learnable function space, and trains end-to-end on the task loss alone. Experiments cover four brain-imaging datasets (ADNI1, ADNI2, ADNI3, ABIDE) and three motion-sensing datasets (MHEALTH, REALDISP, DSADS), with patient-age regression and action classification, and hyperparameters chosen by grid search with four-fold cross-validation.
When covariances are perturbed by finite-sample estimation, the output error decays at roughly the square-root rate in the number of samples and is governed by the Lipschitz constants of the learned filters instead of the singular-value gap and inverse covariance norms that control canonical correlation analysis. The CCA projection-matrix error is inversely proportional to the singular-value gap of the cross-view covariance, so a small gap or ill-conditioned covariances inflate projection error and make pipelines that rely on CCA inconsistent when covariance estimates are noisy. With a fixed trained model, covariances re-estimated from progressively smaller training subsets and the model re-evaluated, CANE and VNN embedding perturbations are an order of magnitude smaller than those of CCA preprocessing, downstream performance is barely affected in most perturbation scenarios, and the measured perturbation decay tracks the theoretical rate up to about 60% of the training sample size.
Perspective
The demonstrated scope is two-view brain morphometry and wearable sensor data: patient-age regression on four brain-imaging datasets and action classification on three motion-sensing datasets, with covariances estimated from training samples. The method is designed for two views, and multi-view settings are not part of the experiments.
The stability experiments show that measured perturbation decay matches the theoretical rate only up to about 60% of the training sample size; beyond that the measured decay is faster because subset covariances converge to the full-dataset covariance rather than the true one, so subset estimates are no longer independent. The theoretical bound widens with the number of layers and filterbank size, and stability trades off against discriminability.