Accurate monocular 4D hand reconstruction remains challenging. Per-frame discriminative regressors lack temporal context and often produce jittery predictions. Temporal models improve consistency by aggregating information across frames, but they are typically deterministic regressors, making them vulnerable to ambiguous observations caused by occlusion and motion blur. Generative modeling offers a natural alternative by learning a prior over plausible hand motion sequences, enabling coherent hand-state recovery when visual evidence is incomplete or unreliable.
Motivated by this observation, we present HandFlow, a fully generative flow-matching framework for temporally coherent 3D hand pose and shape estimation from monocular video. We formulate 4D hand reconstruction as conditional generation in the MANO parameter space. Given visual and skeletal observations, HandFlow denoises an entire temporal window of joint poses, global translations, and hand shape through a single ODE integration. To capture long-range temporal dependencies, we design a Flux-style dual-stream transformer that jointly attends to latent hand parameters and visual-skeletal conditions across the full sequence, avoiding the sequential bottleneck of autoregressive decoding. To handle noisy or missing observations, we introduce a continuous masking mechanism that blends observed conditional features with learnable mask tokens according to per-frame hand confidence scores.
Experiments on DexYCB and HOT3D show that HandFlow achieves state-of-the-art performance, with particularly large gains in world-space accuracy and temporal smoothness. It reduces world-space pose error by over 30% compared with the strongest baseline and achieves the lowest acceleration error among all evaluated methods, while remaining competitive in per-frame pose accuracy. Moreover, on a single GPU HandFlow reconstructs a 150-frame sequence at ≈47 fps, about 12× faster than the fastest prior video-based method, with reconstruction itself accounting for only a small fraction of the end-to-end latency.
Click image to view vector PDF
HandFlow formulates 4D hand reconstruction as conditional flow matching in MANO parameter space. The pipeline has three stages:
1. Condition Extraction: A frozen HaMeR frontend extracts per-frame hand crops, dense visual tokens (patch features from ViT-Huge), 2D skeletal cues encoded as camera-normalized 3D rays, and detection confidence scores. A continuous masking mechanism interpolates between observed features and learnable mask tokens based on per-frame confidence, so low-confidence frames gracefully fall back to the learned motion prior.
2. Flow-Matching Denoising: A rectified-flow formulation denoises the packed MANO parameter sequence from noise to signal through a single ODE integration. A dual-stream transformer jointly processes noisy MANO parameters ($z$-stream: shape $\beta$, pose $\theta$, translation $t$) and visual-skeletal conditions ($c$-stream) over the full temporal window.
3. Overlapping Window Inference: For long videos, overlapping sliding windows are merged via velocity blending during ODE integration using center-distance weights, avoiding discontinuities of post-hoc averaging.
Applications / Results:
HandFlow leads on every metric on DexYCB, with the largest gains under heavy occlusion.
Qualitative comparison on 6 representative sequences spanning multiple subjects and diverse hand–object interactions.
HandFlow sets a new state of the art on HOT3D in world-space accuracy and temporal smoothness, reducing world-space pose error by over 30% while remaining competitive in per-frame pose accuracy.
Camera + World Space evaluation on 6 representative sequences. Rows: HaMeR / HaWoR / HandFlow (Ours). Columns: Cam Overlay / Cam / Perspective / Topdown / Side.
If you find our work useful, please consider citing:
@misc{xu2026handflow,
title={HandFlow: Fully Generative 4D Hand Recovery with Flow Matching},
author={Mingxi Xu and Bowen Duan and Yi Gu and Zhengyang Shen and Renjing Xu and Yutao Yue},
year={2026},
eprint={2607.11221},
archivePrefix={arXiv},
primaryClass={cs.CV}
}