Scaling Video Generation for Reasoning: At What Cost?

Weihang Guo1Xiaoyu Wu2Yifei Wang1Niloofar Mireshghallah2Lydia E. Kavraki1

1 Rice University2 Carnegie Mellon University

Video prediction requires keeping track of things that are temporarily out of view. When an object rotates, for example, its hidden surfaces must return in the right places. We study how well video models learn these changes, and how much training compute they need.

Our experiments use a 2 × 2 × 2 Rubik’s Cube, where every sticker position can be checked against the simulator. We compare model sizes and training budgets and find that lower validation loss does not reliably indicate better state accuracy across models. Although scaling improves accuracy, errors persist over complete trajectories. To understand this difficulty, we examine the roles of representation and visibility: predicting sticker colors directly requires far less compute, while making all six faces visible improves video state accuracy. We also introduce symbolic guidance, which improves video state accuracy at the same training-data budget. Our next step is to find a scalable and generalizable symbolic representation.

Task and evaluation

Each episode starts with a solved 2 × 2 × 2 cube and a prompt containing nine moves. A fixed camera shows three faces, exposing 12 of the 24 stickers. Turns move stickers out of view and bring others back, so predicting a later image requires accounting for changes that were not directly visible.

The solved starting configuration and fixed color convention specify the entire initial state, including the hidden faces. Together with the action sequence, they determine every subsequent configuration. We evaluate the first stationary frame after each move by comparing its visible sticker colors with the simulator state. Unreadable stickers count as errors.

Input
Initial image + all nine prescribed moves
Prediction
80 frames (3.3 s) of future video
Frame accuracy
Fraction of post-action frames with all 12 visible stickers correct.
Full-trajectory accuracy
Fraction of episodes with all 12 visible stickers correct after every one of the nine moves.

This is a prediction task: the moves are given, and the model does not have to find a solution. We train diffusion Transformers from 20M to 1B parameters. Autoregressive (AR) models generate successive chunks using their own earlier predictions; bidirectional models denoise the future sequence jointly. The initial image is the only ground-truth video context supplied at evaluation.

We report training compute in PF-days: one PF-day is 1015 floating-point operations per second for one day, or 8.64 × 1019 operations. Training-video counts measure cumulative presentations to the model.

Extrapolating the compute frontier

For each compute budget, the frontier keeps the highest accuracy observed up to that budget. We fit the remaining error with a power law, 1 − accuracy = A C−α, where C is training compute in PF-days. Fitting the middle or final third of the frontier gives different estimates of the compute required for higher accuracy.

Original extrapolation figure
Compute frontier and power-law extrapolations toward 98 percent accuracy, comparing two fitting windows for AR frame accuracy, AR full-trajectory accuracy, and bidirectional frame accuracy.

Open full-size plot

Gray points show evaluated checkpoints; colored points mark the middle and final thirds of the observed frontier. Solid lines show fits within each window; dashed lines extend those fits. Shading marks the observed compute range, and diamonds mark the projected 98% intersections. Bidirectional full-trajectory accuracy is omitted because it is zero at every evaluated checkpoint.

Predictions during training

The player opens at the last saved checkpoint of the 270M model, after 500,000 training steps. Use Previous, the step selector, or the upper slider to inspect earlier checkpoints. The initial image, action sequence, and displayed video frame stay fixed when the checkpoint changes.

Step 2,048Step 500,000

15 saved checkpoints. The earliest available rollout is at step 2,048; steps 0 and 100 were not saved.

270M model · step 500,000 · 8M training videos · 1.03 PF-days

Loading the saved rollout…

Ground truth VAE reconstruction
Ground-truth cube after action 6.
Generated video AR-k1 · EMA weights
Generated cube after action 6.

The upper slider selects one of 15 saved training checkpoints. The lower slider moves through all 81 video frames. Red outlines mark incorrect predictions at action boundaries. Ground truth is reconstructed through the frozen video autoencoder (VAE).

Validation loss and state accuracy

The training objective is flow matching: the model predicts the velocity used to transform noise into video latents. Validation mean squared error (MSE) measures errors in that velocity prediction. State accuracy measures the sticker configurations reached in the generated video.

Lower validation MSE generally accompanies better state accuracy as a given model trains. Across model sizes, however, similar losses can correspond to substantially different accuracies. The comparison at 1.5M training videos provides one example: the larger model has lower MSE but produces fewer correct states. Validation loss alone therefore cannot rank these models by state prediction ability.

Each point pairs validation loss and free-running accuracy from the same EMA checkpoint. Within a model, lower MSE generally tracks learning progress. Across models, it is not a common scale of reasoning capability.

Training compute and accuracy

We vary model size and training duration to compare their contributions to state accuracy. Smaller autoregressive models use limited compute more efficiently, while larger models reach higher accuracy with further training. A full trajectory fails if any of its nine evaluated states is incorrect, so its accuracy is lower than the average accuracy of individual frames.

All results use the same 100 held-out episodes and generated history. Frame accuracy requires all 12 visible stickers to be correct at a step; full-trajectory accuracy requires this after every one of the nine moves.

Representation and visibility

A

Predicting sticker colors directly

We first replace video prediction with prediction of the 12 visible sticker colors. A small Transformer receives the initial visible configuration and the same kind of action-prompt features used by the video models. Partial observation remains: a sticker can disappear and later return, so the model still has to account for intervening moves.

Same kind of conditioning: initial observation + prescribed action prompt

Video space

Video framesRendered observations
Frozen
Wan VAE
Continuous latents16 × 32 × 32 per latent frame
Prediction modelVideo DiTAR or bidirectional

Vector space

Visible sticker colorsSimulator labels
One-hot
encoding
Categorical vectors12 stickers × 6 colors
Prediction modelState TransformerAR or bidirectional · 2.8M
A comparison of training representations. The vector model predicts visible sticker colors directly, while the video model predicts continuous video latents.

These state-only models predict sequences of 20 actions with high accuracy at much lower compute. Their success shows that the cube’s transitions can be learned efficiently in this representation. The additional difficulty appears when those transitions must be represented and generated through video.

Direct state prediction · 20 actions · 10,000 held-out episodes
State modelTraining PF-daysFinal-state accuracyFull-trajectory accuracy
AR · 2.8M≈ 0.00894.01%92.81%
Bidirectional · 2.8M≈ 0.000499.86%99.85%

Final-state accuracy requires all visible sticker colors to be correct after move 20. Full-trajectory accuracy requires every predicted state to be correct. These are separate state-only experiments.

B

Making all six faces visible

We then keep video prediction and add a complementary camera view. Every settled frame now reveals all 24 stickers. The model must predict both views, but it no longer needs earlier observations to recover the current hidden sticker colors.

The three panels show the same action sequence. The six-face prediction contains both camera views; all panels share the same timeline.

3-face ground truthSimulator
3-face prediction12 visible stickers

6-face prediction24 visible stickers

Loading the paired observation experiment…

A paired rollout from the separate 270M AR-k4 observation experiment, with 7.68M training video presentations per model. Drag through every frame; select a move to compare scored states. Red outlines mark incorrect predictions at action boundaries. The three-face criterion checks 12 stickers; the six-face criterion checks all 24.

Full observation improves state accuracy, particularly for autoregressive generation. This intervention changes both the available state information and the video prediction target. The result supports the role of partial observation in the difficulty of the task, although it does not isolate a single cause of the improvement.

Symbolic guidance

The state-only experiment suggests a more direct learning signal for video training. We train the video model to predict sticker colors as well as future video. Before generating a latent frame, the shared Transformer predicts sticker-state distributions from the initial image, action prompt, and available video history. Those distributions condition a separate video pass using the same Transformer weights.

Simulator states supply training targets. During generation, both the state feedback and the video history come from the model’s own predictions, following the given initial image.

Original paper workflow: video history and the action prompt condition separate state and video passes with shared Transformer weights. Predicted sticker-state probabilities feed back to the video pass. Training combines state cross-entropy and video flow loss.

Open full-size figure

Symbolic-guidance training, from the paper. Predicted sticker-state probabilities condition the video pass; both passes share Transformer weights. Training combines state cross-entropy with the video flow objective.

20M AR-k1 models, each trained on 3M videos and evaluated on 100 paired episodes using EMA weights. Both guided variants supervise all 24 stickers. Front12 feeds back the 12 visible-sticker distributions; Full24 also feeds back the hidden stickers.

At the same training-data budget, both variants improve the correctness of generated states. Front12 shows that hidden-sticker feedback is not required for this improvement when all stickers are supervised. These experiments test state supervision together with predicted-state feedback; they do not separately identify the contribution of each.

Open question: Learning a scalable symbolic representation

Although our targets are discrete sticker colors, our results show that symbolic guidance helps reasoning. The natural next question is: what kind of symbolic representation can generalize across videos, environments, and interactions while capturing how the world changes and how actions update its state?

Here, “symbolic” does not necessarily mean “discrete.” Existing approaches, such as JEPA, offer one direction to investigate. However, designing effective training objectives for such representations remains an open question.

Citation

@misc{guo2026scalingvideogenerationreasoning,
  title={Scaling Video Generation for Reasoning: At What Cost?},
  author={Weihang Guo and Xiaoyu Wu and Yifei Wang and
          Niloofar Mireshghallah and Lydia E. Kavraki},
  year={2026},
  eprint={2609.36599},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2609.36599}
}