# Representation Learning

Marine drill cores no longer need a geologist's rock name as a training target. The usual practice treats shipboard lithology as the ground truth a model should reproduce, yet the six measurements logged on every core (bulk density, magnetic susceptibility, natural gamma radiation and three reflectance channels) are the properties geologists read when they assign that name. A masked variational autoencoder trained on 236,963 observations from 296 boreholes in the LILY database, without labels, separates lithologies as well as a semi-supervised variant (adjusted Rand index 0.237 against 0.247), and raising the label weight a thousand-fold only drops reconstruction from 0.999 to 0.836 while classification barely moves.

Marine drill cores no longer need a geologist's rock name as a training target. The usual practice treats shipboard lithology as the ground truth a model should reproduce, yet the six measurements logged on every core (bulk density, magnetic susceptibility, natural gamma radiation and three reflectance channels) are the properties geologists read when they assign that name. A masked variational autoencoder trained on 236,963 observations from 296 boreholes in the LILY database, without labels, separates lithologies as well as a semi-supervised variant (adjusted Rand index 0.237 against 0.247), and raising the label weight a thousand-fold only drops reconstruction from 0.999 to 0.836 while classification barely moves.

A ViT-B/3D encoder trained on 113,611 thoraco-abdomino-pelvic CT volumes, TAP-JEPA, matches state-of-the-art CT encoders on thoracic and abdominal classification benchmarks; replaying its exact masking, teacher and predictor shows hidden-target prediction lowers mean Skill from 0.388 to 0.271 for pathology but only from 0.921 to 0.873 for anatomy. Pathology moves 34.7% toward matched controls along the teacher direction while controls move 13.6% toward pathology, with larger shifts farther from visible context, and deleting all final-layer pathology tokens changes the decision margin by at most 0.45% in four of seven scan-level probes.

A ViT-B/3D encoder trained on 113,611 thoraco-abdomino-pelvic CT volumes, TAP-JEPA, matches state-of-the-art CT encoders on thoracic and abdominal classification benchmarks; replaying its exact masking, teacher and predictor shows hidden-target prediction lowers mean Skill from 0.388 to 0.271 for pathology but only from 0.921 to 0.873 for anatomy. Pathology moves 34.7% toward matched controls along the teacher direction while controls move 13.6% toward pathology, with larger shifts farther from visible context, and deleting all final-layer pathology tokens changes the decision margin by at most 0.45% in four of seven scan-level probes.

The work formulates dimension-wise posterior collapse in variational autoencoders as a fluctuation theory around partially collapsed states: treating the negative ELBO as an effective free energy, its quadratic expansion gives a Hessian acting as a mass matrix, collapsed directions form an invariant fluctuation sector whose exact mass spectrum follows from a conditional residual operator, and a local reactivation direction lowers the free energy once the decoder variance falls below the residual spectral upper edge, with equality marking marginality; numerical continuation shows latent dimensions lost near these spectral marginalities.

The work formulates dimension-wise posterior collapse in variational autoencoders as a fluctuation theory around partially collapsed states: treating the negative ELBO as an effective free energy, its quadratic expansion gives a Hessian acting as a mass matrix, collapsed directions form an invariant fluctuation sector whose exact mass spectrum follows from a conditional residual operator, and a local reactivation direction lowers the free energy once the decoder variance falls below the residual spectral upper edge, with equality marking marginality; numerical continuation shows latent dimensions lost near these spectral marginalities.

After quantifying that text embeddings and hidden states separate root-cause steps only weakly, the work proposes ReCast: an attribution-aware probe selects layers, a sparse Johnson–Lindenstrauss projection builds complementary pattern and deviation features, and a bidirectional encoder trained with contrastive and ranking objectives converts frozen-LLM hidden states into attribution-oriented step representations; it also releases the ReCast-2K training set and attains the best Hit@1 on Who&When, TraceElephant, AFTraj-2K, and Who&When Pro.

After quantifying that text embeddings and hidden states separate root-cause steps only weakly, the work proposes ReCast: an attribution-aware probe selects layers, a sparse Johnson–Lindenstrauss projection builds complementary pattern and deviation features, and a bidirectional encoder trained with contrastive and ranking objectives converts frozen-LLM hidden states into attribution-oriented step representations; it also releases the ReCast-2K training set and attains the best Hit@1 on Who&When, TraceElephant, AFTraj-2K, and Who&When Pro.

The authors propose CircuitGate, a function-aware representation learning framework for And-Inverter Graphs that explicitly encodes polarity-aware global primary-input (PI) support, modulates fanin updates with support-overlap-aware reconvergence gating, and adds logic-inspired Boolean constraints; across ForgeEDA, EPFL and ITC'99 it outperforms the compared GNN and circuit-specific methods on equivalent-gate identification and signal-probability prediction, reducing MAE by 21.7% and 13.8% on ForgeEDA, and it attains the lowest equivalent-gate MAE under zero-shot ForgeEDA-to-OpenABC transfer.

The authors propose CircuitGate, a function-aware representation learning framework for And-Inverter Graphs that explicitly encodes polarity-aware global primary-input (PI) support, modulates fanin updates with support-overlap-aware reconvergence gating, and adds logic-inspired Boolean constraints; across ForgeEDA, EPFL and ITC'99 it outperforms the compared GNN and circuit-specific methods on equivalent-gate identification and signal-probability prediction, reducing MAE by 21.7% and 13.8% on ForgeEDA, and it attains the lowest equivalent-gate MAE under zero-shot ForgeEDA-to-OpenABC transfer.

The work proposes ΔRepresentation, a visual phenotype representation learning framework for medical VLMs: BaseAnatomy applies geometry supervision to normal anatomical representations using ideal spatial distributions, while Phenotype uses counterfactual reasoning to estimate a normal anatomical reference for each pathological patch and learn phenotype-specific representation increments that cluster by phenotype. On ReXGroundingCT and LIDC-IDRI, it consistently improves lesion grounding and phenotype recognition across six pretrained medical VLMs.

The work proposes ΔRepresentation, a visual phenotype representation learning framework for medical VLMs: BaseAnatomy applies geometry supervision to normal anatomical representations using ideal spatial distributions, while Phenotype uses counterfactual reasoning to estimate a normal anatomical reference for each pathological patch and learn phenotype-specific representation increments that cluster by phenotype. On ReXGroundingCT and LIDC-IDRI, it consistently improves lesion grounding and phenotype recognition across six pretrained medical VLMs.

The authors remove the residual connections around Self-Attention and MLP in the condition encoder of the Decoupled Diffusion Transformer (DDT), stabilize training with zero-initialized output gates and a mixed orthogonal attention initialization, and form the encoder output by aggregating the input patch embedding with the outputs of all encoder blocks; under ImageNet-1k training at the stated resolution, patch size 2, 80 epochs and batch size 1024, DDT-RFE improves overall over DDT on image classification, semantic segmentation, object discovery and semantic correspondence while using fewer encoder blocks (12 vs. 20 at L, 14 vs. 22 at XL), and lowers ImageNet generation FID from 8.98 to 7.68 at L and from 7.27 to 6.81 at XL.

The authors remove the residual connections around Self-Attention and MLP in the condition encoder of the Decoupled Diffusion Transformer (DDT), stabilize training with zero-initialized output gates and a mixed orthogonal attention initialization, and form the encoder output by aggregating the input patch embedding with the outputs of all encoder blocks; under ImageNet-1k training at the stated resolution, patch size 2, 80 epochs and batch size 1024, DDT-RFE improves overall over DDT on image classification, semantic segmentation, object discovery and semantic correspondence while using fewer encoder blocks (12 vs. 20 at L, 14 vs. 22 at XL), and lowers ImageNet generation FID from 8.98 to 7.68 at L and from 7.27 to 6.81 at XL.