The authors of SciWorldBench, a new arXiv benchmark, report that across ten TI2V models Paired Accuracy ranges from 1.4% to 24.8% and the highest Scientific Outcome Accuracy is 53.8%, and that for every model the two videos of a pair frequently show the same outcome even though the conditions call for different results (arXiv, arXiv). The test asks a video model to render two versions of the same experiment that differ in one condition and end differently, so footage that merely looks plausible does not pass. For practitioners who judge generators by visual quality, these figures suggest that scientific correctness is a separate axis on which even the best commercial system scores low.
How the benchmark works
SciWorldBench contains 222 experiment pairs across eight categories. Each pair holds the setup fixed and varies one condition, and the two conditions have mutually exclusive expected outcomes. A model therefore has to show a different result for each condition, not one generic result for both.
According to the paper, videos were generated with a target duration of 8 seconds for all ten models, using the same random seed, 42. Scoring relies on a vision-language judge, doubao-seed-2-1-pro, run at temperature 0, which evaluates the two videos of each pair separately.
The paper’s table caption says PPR and SOA are computed over the 444 videos of each model, and PA over the 222 pairs.
Ten models, low pair scores
| Model | Type | Params | PPR ↑ | SOA ↑ | PA ↑ | Source page |
|---|---|---|---|---|---|---|
| Seedance 2.5 | Commercial | – | 94.6 | 53.8 | 24.8 | https://arxiv.org/html/2611.20899 |
| Kling 3.0 | Commercial | – | 92.6 | 45.0 | 15.8 | https://arxiv.org/html/2611.20899 |
| Vidu Q3 Pro | Commercial | – | 88.7 | 44.6 | 15.3 | https://arxiv.org/html/2611.20899 |
| Wan 3.0 | Commercial | – | 92.1 | 43.2 | 12.6 | https://arxiv.org/html/2611.20899 |
| Cosmos3-Super | Open-weight | 64B | 90.1 | 37.8 | 9.0 | https://arxiv.org/html/2611.20899 |
| MiniMax-H3 | Open-weight | 33B | 91.7 | 39.9 | 7.2 | https://arxiv.org/html/2611.20899 |
| LTX-2.3 | Open-weight | 22B | 78.4 | 32.0 | 7.2 | https://arxiv.org/html/2611.20899 |
| Cosmos3-Nano | Open-weight | 16B | 77.0 | 30.4 | 5.4 | https://arxiv.org/html/2611.20899 |
| Wan2.2-I2V-A14B | Open-weight | 14B active | 75.0 | 28.8 | 3.6 | https://arxiv.org/html/2611.20899 |
| Wan2.2-TI2V-5B | Open-weight | 5B | 48.0 | 16.2 | 1.4 | https://arxiv.org/html/2611.20899 |
The table below reproduces three of the paper’s metrics for the ten models, as the authors report them. Seedance 2.5, a commercial model, leads with PA 24.8 and SOA 53.8, and it also has the highest PPR, 94.6. Kling 3.0, Vidu Q3 Pro and Wan 3.0, all commercial, follow with PA of 15.8, 15.3 and 12.6. The open-weight model with the highest PA, Cosmos3-Super at 64B parameters, reaches PA 9.0 and SOA 37.8, which is below all four commercial rows on both measures; MiniMax-H3 has a slightly higher SOA of 39.9 but a lower PA of 7.2. PPR alone hides this gap: MiniMax-H3 scores 91.7 on PPR but only 7.2 on PA, and LTX-2.3 has the same PA of 7.2 with a PPR of 78.4. At the bottom, the open-weight Wan2.2-TI2V-5B (5B) scores PPR 48.0, SOA 16.2 and PA 1.4.
Knowing the answer is not showing it
The paper compares text answers with video output for Cosmos3-Super. The model gives correct text answers for both conditions in 63.1% of pairs, yet its video PA is 9.0%. That contrast points to a gap between stated knowledge and what the generator renders, not only a lack of knowledge.
The authors also appended the correct outcome to each condition’s prompt and regenerated videos with the six open-weight models. Per the paper, this improves generation for all six of them. MiniMax-H3 gains most, with its PA rising from 7.2% to 33.3% once the outcome is stated in the prompt. That 33.3% is the highest among the six models in that experiment, and it is above the 24.8% that Seedance 2.5 reaches in the main table without such hints.
By category, pooled over the ten models, optics has the highest PA at 18.1% and chemistry the lowest at 6.2%.
Caveats and availability
The authors acknowledge that the evaluation depends heavily on the VLM judge, whose agreement with human annotators remains imperfect and can introduce scoring errors. Small gaps between neighbouring models, such as the PA of 15.8 for Kling 3.0 and 15.3 for Vidu Q3 Pro, should be read with that in mind.
The authors say they will publicly release the benchmark dataset and open-source the evaluation code. That is a promise rather than a release, so readers should treat availability as a commitment by the authors.
What the sources we read do not state
We check six facts on every story in the full text of each source. The pages listed under Sources do not state a price or hardware requirements.
How we built this
Published resultsTable 1 of the paper: Results on SciWorldBench (%)
Rows of Table 1 of the paper (arXiv HTML version), copied cell for cell by our code; a two-row header is shown as "group: column"; an empty name cell repeats the name above it. These are the authors' own figures, not an independent test. The table has 7 columns; 6 shown. Not shown: CR ↓.
Sources
The sources this article was written from. Links open the original page.