Evaluation Setup

We evaluate eleven current video generation models on both PAW-Calibration and PAW-Coverage. For each scenario, every system receives the same source image and action prompt and produces K=50 independent rollouts using its default inference configuration unless noted otherwise. A scene passes the outcome-readout gate when at least 20 of 50 rollouts yield readable, in-schema outcomes. Scene Pass Rate (SPR) is the percentage of all 25 scenes in a track that pass this gate. This keeps model comparisons on one common sampling and readout contract for all systems.

Benchmark Results

Current video generators remain far from probabilistically aligned world modeling. Cosmos 3 Super I2V has the lowest conditional Calibration TVD, 20.5, but only 80.0% of its Calibration scenes pass the outcome-readout gate. LTX-2.3 has the highest conditional Coverage average, 71.7, but that average is computed over the 72.0% of Coverage scenes that pass. Seedance 2 and LingBot-Video-MoE pass every scene on Calibration and Coverage, respectively, yet neither leads the corresponding conditional metric. Conditional alignment and scene-level reliability must therefore be read together when comparing the eleven generators in Table 1.

Recovering valid futures is not the same as assigning them the right probability. A generator may expose many plausible outcomes while allocating probability mass incorrectly, or closely match the frequencies of observed outcomes while leaving valid alternatives unseen. PAW-Coverage and PAW-Calibration therefore expose complementary failures and remain separate rather than being collapsed into one score. Recovering valid support is necessary, but it cannot establish correct probability allocation within that support under the same condition in PAWBench.

Causal and Non-Causal Controls

A probabilistic world model should change its future distribution when the physical transition changes and preserve it when the transition does not. We test this with paired interventions: causal interventions alter the transition and its reference distribution, while non-causal interventions change only an irrelevant visual or textual cue that leaves the physical process unchanged. The paired design tests both requirements under the same source action.

Model distributions shift incompletely or in the wrong direction under physically causal interventions. Under non-causal interventions, distractor text redirects probability mass even though the reference distribution is unchanged. The models respond to altered inputs, but their response does not reliably track whether the underlying physical transition changed or remained unchanged. This asymmetry persists under the fixed source action and readout rubric.

Robustness of the Evaluation Protocol

The main results could reflect either model behavior or a limitation of the evaluation protocol. We therefore test two measurement explanations for the observed gap: too few rollouts and disagreement between PAWEval and human judgments on clear terminal outcomes. Together, these probes address the sampling and readout stages while preserving the original benchmark target and outcome space used for the main model comparison.

Rollout Budget Sensitivity

Doubling the rollout budget leaves calibration largely unchanged, although coverage continues to rise for three of four models. Additional samples can uncover more valid outcomes without correcting their observed relative frequencies. Across eleven generators, observed TVD averages 31.2; in a matched finite-sample null that preserves each model's passing scenes and readable counts, 99% of simulated averages remain below 9.22, far below the calibration error observed across the evaluated generators under the same outcome-readout protocol.

Agreement with Human Judgments

We collect seven independent human judgments per video using the same scene-specific outcome space as PAWEval. The comparison below includes only videos for which both PAWEval and the human panel provide a clear terminal-outcome label. This gives both evaluators the same finite outcome vocabulary for every comparison.

PAWEval matches the decisive human label on 722 of 888 clear, comparable videos (81.3%). Disagreement on clear outcomes therefore cannot by itself account for the PAWBench gap. The denominator excludes videos without a clear label from either side, so this comparison does not measure all-row accuracy or validate TVD, Coverage, or model rankings. It only tests agreement where both evaluators return a decisive terminal label.

Predicting and Steering Outcome Distributions via Language

Language can steer a video generator by naming a future in its prompt, but aligning repeated rollouts requires both selecting targets with the right distribution and realizing each selected target in video. We evaluate these requirements separately so that an error in future selection is not confused with an error in video realization. A failure at either stage leaves the repeated-rollout distribution misaligned even if the other stage succeeds.

We test three settings. Direct VLM future sampling repeatedly predicts what may happen from the same observation and action, then maps those predictions into the PAWBench outcome space without video generation. Prompt engineering (PE) asks GPT-5.5 to select a possible outcome and write a generator prompt for each rollout. Oracle PE supplies the target outcomes directly, following the reference distribution in PAW-Calibration and balancing valid outcomes in PAW-Coverage, to isolate whether the generator realizes the requested future. This ordering separates target selection from target realization under the same outcome contract for each repeated rollout shown here.

Swipe horizontally to compare all metrics.

Swipe horizontally to compare all metrics.

Direct VLM future sampling is already misaligned: the best Calibration TVD is 34.8, and the best Coverage is 46.6%. GPT-5.5's PE selections score 44.3 Calibration TVD and 35.0% Coverage before video generation. Oracle PE improves conditional TVD and Coverage for every generator, but the generators realize only 37.6–58.1% of requested outcomes. Language can change individual rollouts without reliably aligning their aggregate distribution. Better target selection alone therefore does not align the generator under the fixed observation and action.

Broadening Outcome Exploration via Initial Noise

The language interventions change which future is requested. To test what sampling alone can recover, we instead hold the request and generator fixed and ask whether independent noise draws repeatedly visit the same modes even when other valid outcomes remain accessible. This isolates finite-budget exploration under one unchanged generator and prompt. The generator, prompt, and rollout budget remain unchanged throughout this comparison.

Couple to Control (C2C) introduces negative dependence among the K=50 initial-noise samples while preserving each sample's standard Gaussian marginal. C2C and independent sampling use the same action prompt, generator, rollout budget, and evaluation contract. This matched comparison isolates finite-budget exploration; it does not test a change in the model's learned distribution. Only the dependence among the sampled noise vectors changes.

Swipe horizontally to compare all metrics.

Across the scenes that pass in each condition, C2C lowers mean Calibration TVD and raises mean Coverage for all three generators. The gains vary by mechanism and do not consistently raise SPR. Under this matched setup, C2C helps 50 rollouts explore the model's accessible outcomes more broadly; it does not establish that the model has learned a different probability allocation. The comparison remains finite-budget evidence, not a claim about the model's limiting distribution.

Reshaping the Model's Intrinsic Distribution via Fine-Tuning

Unlike language and noise interventions, changing the training distribution can alter a generator's learned allocation of probability mass. We train five Wan2.2 LoRA models on mixtures containing different proportions of left- and right-falling pencil videos while holding the training recipe and budget fixed. Each model is evaluated on upright and left-leaning pencil scenes with a direction-neutral prompt and K=50 rollouts. The two evaluation scenes therefore demand different state-conditional behavior from the same adapted model under a fixed action.

Swipe horizontally to compare all columns.

Increasing the share of left-falling videos in training raises the generated left-fall frequency in both scenes, but the response is nonlinear and does not track the training proportion one-to-one. The result shows that training composition can reshape a model's outcome distribution, although the actual training distributions of current generators are unknown and are not inferred here. This experiment does not recover or estimate the pretraining distribution.

None of the five adapted models matches both scene-specific references. Mixtures that bring the left-leaning pencil to its 100/0 reference make the upright pencil fall left almost every time, instead of matching its 50/50 reference. The same global adjustment moves both scenes in the same direction. Probabilistic alignment requires the outcome distribution to change with the initial physical state under a fixed action. The adapted model instead applies a similar shift to both states.