FM-VLA: Force-based Memory for Vision-Language-
Action Models in Contact-Rich Manipulation

Ruicheng Li^1,2^†, Qixiu Li^1,2^†, Ruichun Ma², Yu Deng², Lin Luo², Zhiying Du^3,2^†,

Jianfeng Xiang^1,2^†, Huizhi Liang^1,2^†, Ruicheng Wang^4,2^†, Jiaolong Yang²^*, Baining Guo²

¹Tsinghua University, ²Microsoft Research, ³Fudan University, ⁴USTC

^†Work done during internship at Microsoft Research. ^*Corresponding author.

0:00 / 0:00

FM-VLA equips a vision-language-action model with force-based temporal memory, enabling robots to count contact events, track interaction progress, and complete non-Markovian manipulation tasks that are visually indistinguishable across repetitions.

Abstract

Vision-language-action (VLA) models have achieved impressive generalization in robotic manipulation, and recent memory-augmented VLAs have relaxed the Markovian assumption by conditioning on past images or language summaries.

Vision-based memory approaches address this by conditioning on sampled past image frames, but they are computationally expensive and fundamentally limited when temporal events are visually ambiguous, e.g., pushing a button multiple times with small movements.

We propose FM-VLA, a VLA model with force-based memory, enabling temporal context reasoning for non-Markovian, contact-rich manipulation. We encode force histories into compact force memory tokens with a variational autoencoder (VAE) pretrained with force time series reconstruction. By projecting force latent representations and short state history as additional conditioning tokens to the action expert module, we enable VLAs to leverage accumulated contact event history to guide manipulation.

We evaluate FM-VLA on three memory-dependent tasks, including finding a hidden block, pressing a button, and wiping a dish for a specific number of times. Our lightweight force memory achieves over 80% success rate with minimal inference overhead, significantly outperforming baseline approaches.

Task Demonstrations

All three tasks require the robot to maintain a memory of physical contact events over the course of an episode. Vision alone is insufficient: the scene looks nearly identical before and after each sub-action. FM-VLA uses its frozen Force-VAE memory to count and track contact events reliably.

Task 1
Find a Block Under Two Cups
The robot lifts cups in order with its right hand; when the hidden block is revealed it picks it up with the left hand.

N=1

Task 2
Press a Button N Times
Given an instruction, the robot presses the button exactly N times. Each valid press triggers an audible click.

N=1

Task 3
Wipe a Bowl N Rounds
The robot wipes the bowl interior with a sponge for N back-and-forth rounds, then releases the sponge.

Method

FM-VLA builds on π₀.₅ (a VLM + flow-matching action expert). We add two complementary memory streams:

Long-horizon wrench memory. A pre-trained, frozen Perceiver-IO VAE compresses the entire episode's 6-axis force/torque history into K=8 latent tokens. The KL bottleneck filters noise while preserving contact events (button clicks, wipe passes, cup lifts).
Short-window state history. A lightweight linear projection maps the last ≈0.9 s of joint-state frames to a single token, providing spatial context for pre-contact motion planning.

Both memory tokens are appended to the action-expert suffix after the noisy-action tokens, leaving the original RoPE positions intact and adding only +3 ms latency.

For each task, we visualize the FM-VLA execution process and a selected channel of force readings, with each frame's time marked. FM-VLA correctly memorizes contact events and completes manipulation successfully.

Quantitative Results

Success rates (%) over 18 trials per task. A trial is successful only when the robot completes the exact repetition count specified in the instruction. Each percentage is annotated as its equivalent value over 18 in parentheses.

Method	Task 1: Cups	Task 2: Buttons	Task 3: Wipe	Average
π₀.₅ (no history)	72.2 (13/18)	11.1 (2/18)	0.0 (0/18)	27.8
TA-VLA	50.0 (9/18)	11.1 (2/18)	5.6 (1/18)	22.2
π-MEM (vision memory)	77.8 (14/18)	33.3 (6/18)	50.0 (9/18)	53.7
Modality ablation (VAE)
Force history only	55.6 (10/18)	0.0 (0/18)	22.2 (4/18)	25.9
State history only	100.0 (18/18)	11.1 (2/18)	11.1 (2/18)	40.7
Architecture ablation (Force + Short State)
FM-VLA (GRU)	55.6 (10/18)	38.9 (7/18)	5.6 (1/18)	33.3
FM-VLA (Q-Former)	100.0 (18/18)	16.7 (3/18)	55.6 (10/18)	57.4
🏆 FM-VLA (VAE, Ours)	100.0 (18/18)	72.2 (13/18)	77.8 (14/18)	83.3

Inference Efficiency

Method	Latency (ms)	Δ vs. base (ms)
π₀.₅ (base)	60.7 ± 0.3	—
π-MEM (K=5 frames)	99.8 ± 0.4	+39.1
π-MEM (K=16 frames)	190.0 ± 1.0	+129.3
🏆 FM-VLA (Ours)	64.0 ± 0.4	+3.3

Takeaways

FM-VLA equips a VLA with compact force memory so it can reason over contact history when visual states are ambiguous.

Why It Works

A frozen VAE compresses wrench history into 8 force tokens, while a short state window provides local motion context.

Why It Matters

FM-VLA reaches 83.3% average success with only +3.3 ms overhead, much lower than vision-memory alternatives.

Current Limitation

The fixed 8-token setting still struggles in very long contact sequences. A practical next step is to expand force pretraining with longer and more diverse trajectories.

In short, force memory is a lightweight and effective way to give VLAs reliable temporal reasoning in contact-rich manipulation.

BibTeX

@misc{li2026fmvlaforcebasedmemoryvisionlanguageaction,
      title={FM-VLA: Force-based Memory for Vision-Language-Action Models in Contact-Rich Manipulation}, 
      author={Ruicheng Li and Qixiu Li and Ruichun Ma and Yu Deng and Lin Luo and Zhiying Du and Jianfeng Xiang and Huizhi Liang and Ruicheng Wang and Jiaolong Yang and Baining Guo},
      year={2026},
      eprint={2607.18231},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2607.18231}, 
}

FM-VLA: Force-based Memory for Vision-Language- Action Models in Contact-Rich Manipulation