Cinecourier
Analysis

SciWorldBench: video models score 1.4% to 24.8% on paired science experiments

A new benchmark of 222 experiment pairs finds ten text-and-image-to-video models rarely get both outcomes of a science experiment right.

The authors of SciWorldBench, a new arXiv benchmark, report that across ten TI2V models Paired Accuracy ranges from 1.4% to 24.8% and the highest Scientific Outcome Accuracy is 53.8%, and that for every model the two videos of a pair frequently show the same outcome even though the conditions call for different results (arXiv, arXiv). The test asks a video model to render two versions of the same experiment that differ in one condition and end differently, so footage that merely looks plausible does not pass. For practitioners who judge generators by visual quality, these figures suggest that scientific correctness is a separate axis on which even the best commercial system scores low.

How the benchmark works

SciWorldBench contains 222 experiment pairs across eight categories. Each pair holds the setup fixed and varies one condition, and the two conditions have mutually exclusive expected outcomes. A model therefore has to show a different result for each condition, not one generic result for both.

According to the paper, videos were generated with a target duration of 8 seconds for all ten models, using the same random seed, 42. Scoring relies on a vision-language judge, doubao-seed-2-1-pro, run at temperature 0, which evaluates the two videos of each pair separately.

The paper’s table caption says PPR and SOA are computed over the 444 videos of each model, and PA over the 222 pairs.

Ten models, low pair scores

Published resultsTable 1 of the paper: Results on SciWorldBench (%)Rows of Table 1 of the paper (arXiv HTML version), copied cell for cell by our code; a two-row header is shown as "group: column"; an empty name cell repeats the name above it. These are the authors' own figures, not an independent test. The table has 7 columns; 6 shown. Not shown: CR ↓.Data read November 18, 2026Source [2]How this format worksDownload the data behind it
Model Type Params PPR ↑ SOA ↑ PA ↑ Source page
Seedance 2.5 Commercial – 94.6 53.8 24.8 https://arxiv.org/html/2611.20899
Kling 3.0 Commercial – 92.6 45.0 15.8 https://arxiv.org/html/2611.20899
Vidu Q3 Pro Commercial – 88.7 44.6 15.3 https://arxiv.org/html/2611.20899
Wan 3.0 Commercial – 92.1 43.2 12.6 https://arxiv.org/html/2611.20899
Cosmos3-Super Open-weight 64B 90.1 37.8 9.0 https://arxiv.org/html/2611.20899
MiniMax-H3 Open-weight 33B 91.7 39.9 7.2 https://arxiv.org/html/2611.20899
LTX-2.3 Open-weight 22B 78.4 32.0 7.2 https://arxiv.org/html/2611.20899
Cosmos3-Nano Open-weight 16B 77.0 30.4 5.4 https://arxiv.org/html/2611.20899
Wan2.2-I2V-A14B Open-weight 14B active 75.0 28.8 3.6 https://arxiv.org/html/2611.20899
Wan2.2-TI2V-5B Open-weight 5B 48.0 16.2 1.4 https://arxiv.org/html/2611.20899

The table below reproduces three of the paper’s metrics for the ten models, as the authors report them. Seedance 2.5, a commercial model, leads with PA 24.8 and SOA 53.8, and it also has the highest PPR, 94.6. Kling 3.0, Vidu Q3 Pro and Wan 3.0, all commercial, follow with PA of 15.8, 15.3 and 12.6. The open-weight model with the highest PA, Cosmos3-Super at 64B parameters, reaches PA 9.0 and SOA 37.8, which is below all four commercial rows on both measures; MiniMax-H3 has a slightly higher SOA of 39.9 but a lower PA of 7.2. PPR alone hides this gap: MiniMax-H3 scores 91.7 on PPR but only 7.2 on PA, and LTX-2.3 has the same PA of 7.2 with a PPR of 78.4. At the bottom, the open-weight Wan2.2-TI2V-5B (5B) scores PPR 48.0, SOA 16.2 and PA 1.4.

Knowing the answer is not showing it

The paper compares text answers with video output for Cosmos3-Super. The model gives correct text answers for both conditions in 63.1% of pairs, yet its video PA is 9.0%. That contrast points to a gap between stated knowledge and what the generator renders, not only a lack of knowledge.

The authors also appended the correct outcome to each condition’s prompt and regenerated videos with the six open-weight models. Per the paper, this improves generation for all six of them. MiniMax-H3 gains most, with its PA rising from 7.2% to 33.3% once the outcome is stated in the prompt. That 33.3% is the highest among the six models in that experiment, and it is above the 24.8% that Seedance 2.5 reaches in the main table without such hints.

By category, pooled over the ten models, optics has the highest PA at 18.1% and chemistry the lowest at 6.2%.

Caveats and availability

The authors acknowledge that the evaluation depends heavily on the VLM judge, whose agreement with human annotators remains imperfect and can introduce scoring errors. Small gaps between neighbouring models, such as the PA of 15.8 for Kling 3.0 and 15.3 for Vidu Q3 Pro, should be read with that in mind.

The authors say they will publicly release the benchmark dataset and open-source the evaluation code. That is a promise rather than a release, so readers should treat availability as a commitment by the authors.

What the sources we read do not state

We check six facts on every story in the full text of each source. The pages listed under Sources do not state a price or hardware requirements.

How we built this

Published results

Table 1 of the paper: Results on SciWorldBench (%)

Rows of Table 1 of the paper (arXiv HTML version), copied cell for cell by our code; a two-row header is shown as "group: column"; an empty name cell repeats the name above it. These are the authors' own figures, not an independent test. The table has 7 columns; 6 shown. Not shown: CR ↓.

arXiv

Sources

The sources this article was written from. Links open the original page.

  1. arXivSciWorldBench: Do Video World Models Really Understand Scientific Experiments?arxiv.orgPrimary source · accessed November 18, 2026
  2. arXivSciWorldBench: Do Video World Models Really Understand Scientific Experiments?arxiv.orgPrimary source · accessed November 18, 2026
  • benchmark
  • world models
  • text to video
  • evaluation
  • seedance