Tactile world model: ground truth vs. rollout

A tactile-only Diffusion-Forcing world model (tac_only_wan22_fps6act) predicting future GelSight-style tactile frames from 2 latent frames of history plus the robot's action stream. Every clip is ground truth on the left, model prediction on the right, decoded through the Wan2.2 VAE with the same LoRA adapter used to encode the latents.

56 rollouts · 336 clips · 16 samples per short-window cell, 6 per long-rollout cell

Short-horizon prediction generalizes. Held-out 44.15 ± 1.94 dB vs train 43.11 ± 3.42 dB (n = 16 each). The gap is +1.04 dB against a 2×SEM of 1.97 — indistinguishable from noise. No memorization.

Long autoregressive rollout fails immediately, then plateaus. Error jumps to ~0.122 at the first predicted frame and grows only 1.4× over the next 14 seconds. It does not compound — it saturates. That is why 8 s and 16 s score the same.

Short window, held-out

44.15 dB

± 1.94 (n = 16)

Train/test gap

n.s.

+1.04 dB vs 2·SEM 1.97

8 s vs 16 s

n.s.

+0.07 dB vs 2·SEM 0.20

Drift over 14.7 s

1.4×

saturates, does not compound

Drift: where the long rollout actually fails

held-out (test) train

Per-latent-frame MSE across the 16 s rollouts (n = 6 per split, shaded ±1 sd). One latent frame = 0.67 s. The two context frames are given, so their error is 0; the very first predicted frame already carries almost all the error the rollout will ever have. A compounding failure would climb steadily — this does not.

Metrics

held-out (test) train

Mean pixel PSNR over both tactile sensors, error bars ±1 sd across samples. Every train-vs-test pair overlaps. The step down to the long-rollout bars mixes genuine failure with a reconstruction offset — see the caveat below.

Regime	Split	n	Pixel PSNR	± sd	Latent MSE

Long-rollout numbers are not on the same scale as short-window ones. The long-rollout ground truth is stitched from different VAE encode windows than the rollout's seed. The seed frame matches exactly (MSE = 0.0000) but the second context frame — copied verbatim from GT — reads 2.2–4.6e-02. That offset inflates every long-rollout error, so the ~44 → ~32 dB step overstates true failure. It does not affect the train-vs-test or 8 s-vs-16 s comparisons, which are within-regime.

Short window — 2.7 s, 2 latent frames given, 2 predicted

held-out episodes 005/006 · 16 samples per split

Long rollout — autoregressive, predictions fed back

test episodes 005 & 006 · train episodes 004 & 001 · 3 start offsets each

All 56 rollouts

The clips above are a fixed selection. Every rollout is uploaded — pick a cell and sample to load any of the 336 clips (each sample has left/right × gt / rollout / side-by-side).

How it was run

RUN=.../runs/tac_only_wan22_fps6act
CKPT=$RUN/checkpoints/best-epoch=54-val_loss=0.0354.ckpt

# short window, 16 samples per split (latents only — the built-in decoder
# targets 16ch VideoVAE+, wrong for these 48ch Wan2.2 latents)
python -m vm_diffusion.scripts.inference_rollout --ckpt $CKPT --config $RUN/config.yaml \
    --split {val,train} --num_samples 16 --no_decode

# long rollout — --fps 6 is MANDATORY. Without it, fps is derived from the native
# 30fps parquet timestamps and N_lat comes out 5x too large.
python -m vm_diffusion.scripts.long_rollout --config $RUN/config.yaml --ckpt $CKPT \
    --episode <ep>.pt --start <offset> --seconds {8,16} --fps 6

# decode (effecterase env, Wan2.2 VAE + the encode-time LoRA)
python decode_rollout_wan22.py --rollout-dir <dir> --fps 6

Caveats

Long-rollout cells are n = 6 (two episodes × three start offsets); short-window cells are n = 16. All episodes are the same motherboard task domain, so the train/test contrast isolates memorization rather than task difficulty. "n.s." above means the difference is smaller than twice its standard error — with these sample sizes that rules out large effects, not small ones. Latents were encoded with a LoRA adapter and decoded with the same one; the base VAE would give systematically wrong pixels.

Model: 4 latent-frame window, 2 frames of history, pyramid sampling, 100 denoise steps. One latent frame = 4 frames @ 6 fps = 0.67 s.