Voxel generation stops fixing shape level by level: JoVox lets fine cells help decide coarse ones and cuts training memory

Voxel generation can now let fine cells take part in deciding coarse cells, so a coarse level is no longer a skeleton fixed once and never corrected. Hierarchical methods generate a coarse grid first and refine occupied cells level by level; coarse levels are decided from coarser levels alone and never revisited, so they are often plausible but wrong and finer levels inherit the errors. On image-conditioned generation on Toys4k, the full jointly modeled model reaches voxel Chamfer distance 0.01506 and IoU 0.334, against 0.01941 and 0.264 for a variant that uses two-level prediction but still generates the hierarchy level by level, with the same network architecture, training data and number of training steps.

Generation starts from a fully masked state, unmasks a few cells at any level at each step, and grows the hierarchy as cells are unmasked. Masked discrete flow matching was built for fixed-size sequences, whereas a voxel hierarchy grows as its cells are unmasked; the method extends it to such a growing hierarchy. The network is a transformer that takes one token per cell and processes the tokens of all levels in one sequence with full self-attention; during training the masking fractions of different levels are drawn independently, so intermediate states from any unmasking strategy lie within the training distribution.

Predicting cells two levels finer at once means the network processes only half of the levels. Hierarchical methods predict one level finer at a time, so the network processes every level. In the Toys4k ablation, the two-level variant improves voxel Chamfer distance, IoU, mesh Chamfer distance, LPIPS and Uni3D over the single-level variant; on four A100 GPUs at a fixed batch size, one training step of the two-level variant takes less time than one of the single-level variant.

At inference, the oldest unmasked cells are remasked so they are predicted again once finer cells exist. Cells unmasked in the same step are sampled independently and may be inconsistent with each other; existing samplers either do not remask or choose remasked cells at random or by confidence. In the sampler comparison on image-conditioned generation, this strategy reaches voxel Chamfer distance 0.01107 and IoU 0.4158, better than MDLM at 0.01162 and 0.4035, ReMDM at 0.01115 and 0.4085, ReMDM-conf at 0.01116 and 0.4139, and MaskGIT at 0.01243 and 0.3987.

The demonstrated scope is single-object voxel generation: the image-conditioned task trains on TRELLIS-500K and tests on Toys4k objects unseen during training, and the unconditional task uses the chair, airplane, table and rifle categories of ShapeNet, with shapes represented by surface occupancy at the finest level. The authors propose next applying the framework to large scenes with multiple objects and to long sequences of 3D shapes.

The authors name training-set size as a limitation, training on about K assets, less than a quarter of the roughly K assets used to train TRELLIS.2, and expect more data to improve the model further. They also note that, like other masked discrete flow-matching models, unmasking several occupancy codes independently in one step can make them inconsistent, and the strategy only reduces such inconsistencies.