Reason 1 — no live environment
The benchmark has no simulator to close the loop
VLA and WAM evaluation is always closed-loop: the model acts, the scene changes, the model sees the new state. Datasets like DROID and Open X-Embodiment are fixed recorded videos — the robot cannot actually execute an action and receive a new observation. You can measure offline imitation error (L1 loss) but not task success rate, which is the only meaningful metric for comparing VLA vs WAM.
Reason 2 — no language instruction
WAMs work — VLAs need language to function
VLAs require a natural language instruction at every episode ("pick up the butter and place it left of the plate"). Benchmarks like Meta-World, DMControl, and Atari have no language input — they use reward signals or goal states instead. You can evaluate WAMs that use MPC with a goal image or reward, but VLAs simply cannot run without a language instruction, making head-to-head comparison impossible on these benchmarks.
Reason 3 — no WAM eval code yet
The benchmark exists but WAM integration is missing
Some benchmarks have full VLA eval pipelines but no WAM code yet — SimplerEnv, CALVIN, and RoboCasa all have working VLA evaluation but none have official Cosmos-Policy or Fast-WAM adapters as of May 2026. In principle WAMs could run there, but in practice no published paper has done it yet, making results incomparable to existing literature.