DC-SAE splits image tokenization into semantic and pixel branches, hitting 32x compression with 3.37 gFID and 29.79 PSNR on ImageNet and a 4.41x speedup over DC-AE

DC-SAE splits a high-compression tokenizer into two complementary branches: a frozen semantic encoder that supplies a structured latent space easy for diffusion models to learn, and a from-scratch pixel encoder that carries texture, color statistics, local boundaries, and high-frequency detail; the two latents are concatenated along channels, fused by a lightweight projection, and decoded jointly. Earlier representation autoencoders (RAEs) replaced VAE encoders with pretrained semantic encoders to speed up diffusion training but were typically limited to moderate compression, and naively pushing them to a deep-compression regime such as 32 severely degraded reconstruction; DC-SAE adds an unconstrained pixel branch to recover low-level information, extending semantic autoencoders into the deep-compression regime. The paper reports that under the standard compression setting a semantic-only autoencoder reaches about 23.13 PSNR, while adding the pixel branch reaches 29.79 PSNR and 0.17 rFID; masking analysis shows that masking the semantic component barely affects reconstruction while masking the pixel component causes severe degradation, indicating high-fidelity reconstruction comes mainly from the pixel branch.

The paper identifies post-merge compression, which reduces token count after semantic abstraction via average pooling, frozen mergers, or learnable mergers, as discarding spatial and low-level information that is hard to recover, and argues compression should instead happen at the input level (pre-merge) so the semantic encoder forms representations directly under the target token budget. This comparison isolates whether compression happens before or after semantic encoding as a key obstacle to extending semantic autoencoders to deep compression, whereas prior MLLM-style high-compression vision tokenizers mostly extract dense tokens first and merge them afterwards. In Table 4 post-merge variants consistently underperform: Qwen-ViT with a frozen merger gives 23.40 PSNR and 25.96 gFID, average pooling gives 23.07 PSNR and 17.71 gFID, a learnable merger gives 23.85 PSNR and 16.84 gFID, and a DINOv2 learnable merger gives 24.03 PSNR and 8.60 gFID, while pre-merge DINOv2 gives 23.13 PSNR and 3.31 gFID.

Spatial DeMerger expands each compact latent token into a small group of spatial sub-tokens on the decoder side only, so the diffusion model still operates on the sparse compact latent grid while the ViT decoder receives a denser grid for detail recovery. This decouples the latent grid used for diffusion modeling from the token grid used for image decoding, whereas prior high-compression tokenizers typically share one sparse grid for both, creating a decoder-side spatial bottleneck. Under the 32x-compression DINOv2 pre-merge setting, adding Spatial DeMerger consistently improves validation PSNR; in fully frozen QwenViT settings without any pixel branch, the original merger improves from 15.39 to 17.52 and the average-pooling variant from 15.50 to 17.64.

DC-SAE's gain comes from jointly autoencoding the semantic and pixel latent spaces rather than merely exposing DiT to semantic features; joint training converges better than Late-Concat, which first trains a pixel autoencoder alone and concatenates the frozen semantic latent only during DiT training. This ablation separates a jointly learned, compatible semantic-pixel latent representation from post-hoc concatenation, showing that generation friendliness arises from coupling inside the autoencoder rather than from simply injecting semantic features. In Table 6 Late-Concat reconstructs better (31.91 PSNR, 0.09 rFID) but generates worse, with an 80-epoch gFID of 5.64 versus 4.02 for joint training; Late-Concat-E, using an earlier checkpoint whose reconstruction quality is closer to joint training, still reaches 5.40, ruling out a pure reconstruction-generation conflict.

The work targets latent image generation settings that need high-compression latent spaces to cut sequence length and compute, covering class-conditional ImageNet generation and text-to-image generation; its conclusions apply to tokenizers built from a frozen semantic encoder plus a trainable pixel encoder, with a DiT trained in the frozen latent space. For researchers and engineering teams who want to raise the compression ratio without sacrificing reconstruction fidelity, or to shorten diffusion training convergence, DC-SAE offers directly reusable architectural choices: input-level pre-merge compression, joint dual-branch training, and a Spatial DeMerger applied only on the decoder side. The paper also notes its method is complementary to existing techniques such as VFM alignment, self-supervised token losses, and channel masking, and could potentially benefit from combining with them.

The text-to-image study uses one B-parameter DiT configuration with a subset of internal data plus LAION-COCO, so broader validation across training scales, datasets, and generator architectures remains future work. The paper also states that although semantic latents are observed to accelerate diffusion training and input-level pre-merge compression outperforms post-hoc token merging, there is not yet a complete understanding of which properties of semantic encoders matter most for generation, nor a principled explanation of when pre-merge or post-merge compression is preferable. In addition, some numbers in the homepage evidence bundle appear as placeholders in the body text and several table cells are empty, so fully verifying individual comparison numbers still requires returning to the original tables.