ForceLDM: A Force-aware Latent Dynamics Model for Contact-Rich Manipulation

Peiyuan Tang*, Yao Li*, Xiaodong Zhang†, Wuyang Zhang,
Qin Xia, Yanyong Zhang, Zijiang James Yang†
*Equal contribution   †Corresponding authors

NeurIPS 2026

Paper Dataset Code Method Rollouts

Motivation

Learning Latent Dynamics through Teacher–Student Distillation.

01

Adding force alone is not enough

When force is simply added as an input, rich visual cues can dominate the sparse force signals. The policy may still learn to copy demonstrated actions without understanding how contact will evolve, limiting its ability to handle new situations.

02

Predicting video can miss what matters for control

A realistic-looking video does not guarantee accurate contact dynamics. Predicting full frames also spends effort on backgrounds, appearance, and sensor noise. Adding force sensing to video models further requires substantial computation and synchronized training data.

03

Learn the motion and contact that matter

Optical flow directly describes how the robot and objects move, while force/torque signals reveal their contact interactions. Together, they capture complementary information for control. Action-guided distillation condenses these future signals into a compact representation, helping the policy learn what matters for choosing actions without reconstructing every sensory detail.

Comparison of force-aware VLA, video-based world action models, and our method.
Figure 1. Comparison with prior methods.

(a) Force-aware VLAs incorporate force feedback but lack explicit modeling of future interaction dynamics. (b) Video-based WAMs model future dynamics through visual prediction, but often lack force/tactile feedback and may fit task-irrelevant visual details. (c) ForceLDM learns compact, action-relevant future dynamics by distilling future motion and force cues from a privileged teacher.

Method

Overview of ForceLDM.
Figure 2. Overview of ForceLDM.

(a) Both teacher and student generate actions from current multimodal observations and language instructions. The teacher additionally receives future force and motion tokens, and its future-aware representations are distilled into the student's future queries. (b) The future dynamics encoder converts future force/torque and optical flow into compact force and motion tokens. (c) Progressive noise injection perturbs the student's future queries with scheduled noise to improve generalization.

Training

A teacher with access to future force/torque and optical flow distills action-relevant dynamics into a student, while progressive noise injection improves generalization.

Inference

Teacher-free and future-free. Only the student branch and current observations are retained at deployment.

Real-World Tasks

Five real-world contact-rich manipulation tasks.
USB Insertion Gear Assembly Box Flipping Board Wiping VGA Plugging

Real-World Rollouts

USB Insertion Default 2× speed

Gear Assembly Default 2× speed

Box Flipping Default 2× speed

Board Wiping Default 2× speed

VGA Plugging Default 2× speed

Probing Latent Dynamics

We freeze the trained ForceLDM policy and extract the layer-12 hidden states of its future force and motion queries. Lightweight probes are trained to decode future contact forces and optical flow while keeping the policy fixed.

USB Insertion

Gear Assembly

Box Flipping

Board Wiping

Results

87.1%
Average ID success rate
68.9%
Average OOD success rate
85.0%
VGA Plugging success rate

Main Results

Table 1. Performance comparison on In-Distribution (ID) contact-rich manipulation tasks. We report the success rate (%) across different tasks. Avg. denotes the average success rate across all ID evaluation settings. Best results are highlighted in bold.
Model USB Gear Box Wiping VGA Avg.
pi0 w/o force44.046.755.060.00.041.1
pi0 w/ force64.083.370.070.05.058.5
TA-VLA80.093.380.075.010.067.7
ForceVLA52.066.765.065.05.050.7
Seer w/ Force76.070.095.070.050.072.2
ForceLDM92.083.395.080.085.087.1

Out-of-Distribution Generalization

Out-of-distribution evaluation settings.

Figure 4. Out-of-distribution generalization settings used in our real-world evaluation.

Table 2. Generalization performance on Out-of-Distribution (OOD) contact-rich manipulation tasks. We report the success rate (%). Avg. denotes the average success rate across all OOD evaluation settings. Best results are highlighted in bold.
Model USB Obj. USB Occ. Gear Height Gear Color Box Mat. Box Shape Avg.
pi0 w/o force20.026.70.040.026.70.018.9
pi0 w/ force40.073.30.046.726.733.336.7
TA-VLA20.066.713.366.713.326.734.5
ForceVLA26.720.00.040.00.016.717.2
Seer w/ Force46.720.053.360.073.340.048.9
ForceLDM86.786.746.760.080.053.368.9

Key Findings

  • Future force and optical flow are complementary: force captures contact evolution, while optical flow captures action-relevant motion and alignment.
  • Distilling latent future dynamics is more effective than predicting future RGB observations for contact-rich control.
  • Progressive noise injection improves robustness when the student must infer future dynamics from current observations alone.

BibTeX

@inproceedings{tang2026forceldm,
  title     = {Force{LDM}: A Force-aware Latent Dynamics Model for Contact-Rich Manipulation},
  author    = {Tang, Peiyuan and Li, Yao and Zhang, Xiaodong and Zhang, Wuyang and Xia, Qin and Zhang, Yanyong and Yang, Zijiang James},
  booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems},
  year      = {2026},
  url       = {https://openreview.net/forum?id=0j9L12vSKx}
}