One freehand sketch, one 3D mesh: keeping local stroke tokens lifts average IoU from 0.616 to 0.628
Synopsis
Reconstructing a 3D mesh from a single freehand sketch usually compresses sketch features into one global vector, which loses fine strokes and thin structures. Locality-Preserving Sketch Token Fusion instead keeps both a global shape representation and spatially arranged local tokens, and lets the global representation query those tokens through cross-attention during decoding. On ShapeNet-Synthetic with ground-truth viewpoints, average IoU rises from 0.616 for SS2M to 0.628; on the human-drawn ShapeNet-Sketch, average IoU is 0.498 and 0.487 under ground-truth and predicted viewpoints respectively.
Interpretation
Sketch reconstruction no longer relies on a single compressed global vector; while generating the mesh, the network can look back at local stroke tokens arranged by spatial position. Earlier sketch reconstruction methods mostly flattened or pooled 2D sketch features into a compact global latent representation to capture shape and viewpoint semantics. On ShapeNet-Synthetic with ground-truth viewpoints, average IoU is 0.628, above the previous best SS2M at 0.616; under predicted viewpoints it is 0.610.
The sketch feature is split into two paths: a global latent representation describing overall shape and viewpoint, and local tokens that retain stroke structure in each region. A token transformer block models long-range relations among 49 spatial tokens with multi-head self-attention and an MLP, then fuses the transformed result with the original features through a learnable gated residual. A local token cross-attention module uses the global shape representation as query and the local tokens as key and value; the retrieved local cues are added residually and passed to the mesh generator, evaluated on synthetic and human-drawn sketches across 13 ShapeNet categories.
Perspective
Evaluation covers synthetic sketches from ShapeNet-Synthetic and 1,300 human-drawn sketch-model pairs in ShapeNet-Sketch, across 13 ShapeNet categories, with meshes produced by deforming a fixed-topology template; ShapeNet-Sketch is used only for testing and not for training.
On ShapeNet-Sketch, Magic3DSketch is slightly better on individual categories such as car, chair and table; minor discrepancies remain in fine-scale regions, which the authors attribute to the inherent ambiguity and incompleteness of freehand sketches.