V-JEPA Hand: Simple, Accurate and Fast 3D Hand and Camera Trajectory Recovery from Egocentric Videos

Ruicheng Li^1,*, Qixiu Li^1,*, Yu Deng², Sicheng Xu², Jiaolong Yang², Baining Guo²

¹ Tsinghua University² Microsoft Research

^* Intern at Microsoft Research

Paper Code Demo

V-JEPA Hand jointly recovers bilateral 3D hand motion and camera trajectories
from egocentric video in a single forward pass.

Abstract

Egocentric video supports diverse applications, including robot learning, augmented reality, and human activity understanding. Recovering metric 3D hand trajectories from such video requires estimating both hand motion and camera egomotion to express the hands in a consistent reference frame. Many existing pipelines recover hand and camera motion in separate stages, combining hand detection, hand reconstruction, camera tracking, and trajectory refinement. We present V-JEPA Hand, a simple feed-forward video model that jointly reconstructs bilateral hand motion and camera motion from egocentric clips. It uses a self-supervised V-JEPA 2.1 encoder to extract spatio-temporal features from full-frame video and a query-based predictor to decode these features into per-frame hand poses, hand presence, and camera motion. We use the predicted camera motion to express the hand trajectories in a common world frame. The model accepts known focal lengths or predicts them when unavailable. It reconstructs each clip in a single forward pass. We train both components end to end in a single stage on a mixture of diverse pseudo-labelled videos and high-precision 3D annotations. A single model achieves the lowest coverage-aware mean per-joint position error (MPJPE) among comparable baselines on ARCTIC, HOT3D, and EgoStandard. It also yields lower world-frame trajectory errors than the compared SLAM-based pipelines. Ablations support the benefits of video pretraining, temporal context, and mixed supervision. V-JEPA Hand achieves a forward-pass throughput of 93.4 FPS on a single RTX A6000. We will publicly release our code, model weights, and training scripts.

Methodology

One video encoder. One predictor. Joint reconstruction. Full-frame video features from V-JEPA 2.1 are decoded into hand pose, hand presence, camera motion, and focal lengths. The predicted cameras place the hands in a common world frame.

V-JEPA Hand architecture: a V-JEPA 2.1 encoder processes an egocentric video clip; a ViT predictor combines its features with per-frame hand-pose queries and an intrinsics token to predict bilateral hand pose, hand presence, camera motion, and focal lengths. — **Overview of V-JEPA Hand.** A V-JEPA 2.1 encoder extracts spatio-temporal features from an egocentric video clip. A ViT predictor jointly processes these features, one learnable hand-pose query per frame, and an intrinsics token. The output queries predict bilateral MANO pose, hand presence probability, and camera motion. The entire clip is reconstructed in a single forward pass; blue and orange meshes denote the reconstructed left and right hands, respectively.

Results

Hand reconstruction

Detection ↑ · Pose and motion errors ↓

Accurate hands across three benchmarks. V-JEPA Hand achieves the lowest coverage-aware MPJPE among comparable baselines on ARCTIC, HOT3D, and EgoStandard.

	Detection			3D Pose				Orient. & Position						Temporal
Method	Acc↑	Rec↑	F1↑	MP-p↓	MP↓	PA-p↓	PA↓	EP-p↓	EP↓	GO-p↓	GO↓	CT-p↓	CT↓	Jit↓
ARCTIC
WiLoR	0.909	0.949	0.971	21.11	15.52	12.74	6.70	65.56	28.84	17.68	11.74	0.077	0.050	25.97
HaMeR	0.873	0.943	0.958	30.78	25.10	16.02	9.44	64.85	28.45	25.14	18.67	0.093	0.063	14.78
WildHands	0.874	0.944	0.958	27.07	21.29	15.99	9.55	65.44	29.58	22.54	16.05	0.070	0.039	12.28
HaWoR	0.853	0.953	0.945	31.52	26.94	14.48	9.07	44.32	16.24	24.38	18.82	0.076	0.050	19.37
ViDiHand	0.997	0.999	0.999	21.67	-	9.82	-	12.41	-	14.64	-	0.047	-	3.18
Ours	0.996	0.998	0.999	15.59	15.36	6.84	6.59	8.99	8.85	12.46	12.18	0.032	0.031	6.09
HOT3D
WiLoR	0.810	0.879	0.922	39.38	26.09	23.10	7.57	79.90	22.11	32.51	16.49	0.103	0.048	17.19
HaMeR	0.719	0.880	0.886	67.09	57.96	25.09	10.26	70.66	15.40	50.01	36.35	0.106	0.055	21.09
WildHands	0.425	0.600	0.609	80.23	42.99	61.89	12.40	275.39	37.14	84.35	38.82	0.250	0.076	25.30
HaWoR	0.733	0.895	0.875	28.79	16.71	20.68	7.64	66.60	8.16	25.01	10.50	0.086	0.039	13.41
ViDiHand	0.948	0.974	0.983	21.51	-	11.38	-	14.95	-	15.83	-	0.040	-	3.74
Ours	0.971	0.979	0.980	14.42	12.04	8.24	5.72	16.81	5.71	9.90	6.52	0.026	0.014	5.74
EgoStandard
WiLoR	0.949	0.971	0.982	35.19	32.16	11.11	7.35	37.09	12.74	25.88	22.13	0.089	0.080	15.53
HaMeR	0.986	0.992	0.994	37.37	36.60	8.27	7.26	18.06	11.45	26.37	25.37	0.093	0.090	18.35
WildHands	0.514	0.658	0.659	69.22	33.83	55.04	12.29	325.25	53.66	74.03	31.84	0.214	0.099	20.10
HaWoR	0.883	0.901	0.937	33.38	23.36	20.11	8.62	91.58	8.00	26.58	12.81	0.089	0.058	13.95
Ours	0.990	0.995	0.997	16.97	16.30	8.01	7.29	11.10	6.51	10.60	9.84	0.045	0.043	4.88

Gray ViDiHand results on HOT3D use a different split and are reference-only.

Evaluation details

WiLoR, HaMeR, and WildHands are single-frame methods; HaWoR, ViDiHand, and ours are multi-frame methods. We follow ViDiHand's evaluation protocol, except that pose metrics without the -p suffix are computed on each method's own intersection of predictions and ground truth (true positives). ViDiHand results are taken from its paper (ViDiHand); its HOT3D row (gray) uses a different split and is shown for reference only, without direct comparison or ranking.

World-coordinate trajectories

W-MPJPE / WA-MPJPE (mm) · RTE (%) ↓

More accurate trajectories in a shared world frame. V-JEPA Hand reduces W-MPJPE, WA-MPJPE, and RTE compared with the evaluated SLAM-based pipelines on all three benchmarks.

	ARCTIC			HOT3D			EgoStandard
Method	W-MP↓	WA-MP↓	RTE↓	W-MP↓	WA-MP↓	RTE↓	W-MP↓	WA-MP↓	RTE↓
WiLoR+SLAM	63.36	34.13	29.78	74.78	28.42	40.21	90.67	41.54	26.27
HaMeR+SLAM	69.05	33.78	28.21	105.10	35.08	47.31	94.86	42.74	27.16
HaWoR	106.05	55.38	48.92	63.08	27.60	51.02	90.23	45.02	35.93
Ours	35.86	19.72	17.58	34.08	16.18	28.15	66.35	32.91	22.37

All four methods are evaluated on the same common true-positive hand-frames.

Evaluation details

All methods are evaluated on the same hand-frames, using the intersection of true positives across the four compared methods. W-MPJPE and WA-MPJPE are in mm; RTE is in %. Lower is better.

Takeaways

Recovering hands in a consistent world frame requires understanding both hand motion and camera motion. V-JEPA Hand learns them together, directly from video.

Hands and cameras, together

A single model predicts bilateral hand motion, camera motion, and intrinsics. It reconstructs world-frame hand trajectories without separate hand detection, SLAM, or test-time optimization.

Learn from video

V-JEPA 2.1 video pretraining provides spatio-temporal features. We fine-tune the encoder and predictor together in one stage on four datasets: Ego4D, in-house recordings, ARCTIC, and HOT3D. Ego4D and in-house videos are pseudo-labelled with VITRA. EgoStandard is held out for evaluation.

Simple, accurate, and fast

The lowest coverage-aware MPJPE among comparable baselines on all three benchmarks, with single-GPU forward throughput of 93.4 FPS on RTX A6000 and 115.1 FPS on A100. Both are measured on the same 100 real-video clips at batch size one with BF16, excluding detection and postprocessing.

A feed-forward approach to turning egocentric video into 3D hand-motion annotations.

Citation

@misc{li2026vjepahandsimpleaccurate,
  title={V-JEPA Hand: Simple, Accurate and Fast 3D Hand and Camera Trajectory Recovery from Egocentric Videos},
  author={Ruicheng Li and Qixiu Li and Yu Deng and Sicheng Xu and Jiaolong Yang and Baining Guo},
  year={2026},
  eprint={2610.26780},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2610.26780},
}