We present Video-4M, a multimodal video model that models the world across different modalities, each serving as a representation space (e.g., RGB, depth, optical flow, feature maps, bounding boxes). Video-4M can predict any modality, at any point in time, given an arbitrary combination of input modalities at arbitrary time steps.
Traversing the time-modality grid gives rise to higher-level inference capabilities unavailable to specialist video models, all without any additional training: generation through a chain of modalities, future prediction in any modality, and counterfactual generations with modalities as actions.
Video modeling captures how the world is structured in space and how it evolves over time. In the literature, this spatiotemporal structure is often modeled in a single, familiar representation space, e.g., RGB pixels or semantic features such as DINOv2 or V-JEPA. However, different representations capture different aspects of the underlying world and serve complementary roles. For instance, DINOv2 and V-JEPA excel at high-level semantic understanding, RGB preserves fine-grained detail, and depth maps capture geometry.
So we ask: what if video modeling is not confined to a single representation space? We introduce Video-4M: an any-to-any multimodal model that models the world across different representation spaces (e.g., RGB, depth, optical flow, neural network feature maps). Video-4M can predict any modality, at any point in time, given an arbitrary combination of input modalities at arbitrary time steps.
Traversing the time-modality grid gives rise to higher-level inference capabilities that are not supported by specialist video models:
See how multimodal masked modeling lets a single model traverse time and modalities.
Learn how a task (e.g., caption-to-video) can be decomposed into a chain of simpler sub-tasks via intermediate modalities.
See how Video-4M's predicts the future in any modality, where the modality can be chosen based on the information of interest.
Explore how modalities act as actions, steering generation in any output space with coarse (e.g., captions) to fine-grained (e.g., depth) controls.
Video-4M allows predictions from any input modality to any target modality with a single model. Below, we demonstrate this by showing one-to-all prediction, where Video-4M sequentially predicts each modality from a single input.
Video-4M extends the any-to-any modeling framework of 4M [4][7] to the video domain. It is built on three components: an aligned multimodal video dataset, a unified tokenization scheme that maps all modalities into discrete tokens, and a multimodal masked modeling objective that enables any-to-any prediction across modalities and time. We describe each below.
Training an any-to-any video model requires videos aligned across multiple modalities, which are rarely available together at scale. We therefore construct K600-MM, based on the Kinetics-600 dataset [1]: ~400K videos aligned across 12 modalities: depth, normals, optical flow, poses, feature maps, bounding boxes, captions, transcriptions, etc. These modalities are generated via pseudolabeling using task-specific specialist models. This gives us 34B tokens of aligned multimodal video data for pretraining. K600-MM is publicly available on Hugging Face.
Examples from K600-MM. Below are examples from K600-MM. Each example consists of an RGB video and corresponding aligned modalities.
Click a numbered tile to switch example sets.
Different modalities have different structures (e.g., feature maps vs. RGB vs. bounding boxes) and are not directly compatible with a shared modeling architecture. We eliminate modality-specific design choices by mapping all modalities into discrete tokens, which can be jointly modeled with a single pretraining objective. Unless otherwise specified, all tokenized modalities in each data sample represent a 128×128 resolution video clip covering 17 frames with a 4-second duration, sampled at 4 FPS.
Modalities with a 3D spatiotemporal structure. For modalities such as RGB, depth, normals, optical flow, and neural network feature maps, we use FSQ-VAE [2] video tokenizers that compress inputs both spatially and temporally to map them into discrete tokens.
Modalities that are represented sequentially. We treat captions, transcriptions, and bounding boxes as text tokens and encode them using WordPiece tokenization [3]. Special tokens are added to the vocabulary to represent time step delineators, frame-level boundaries, and modality-specific values, such as quantized bounding box coordinates.
Video-4M is an encoder-decoder transformer trained with a multimodal masked modeling objective [4]. During pretraining, one subset is randomly selected from all modality tokens (across both space and time) of a training sample as input to the encoder and another random subset as target tokens which are predicted by the decoder. For modalities with a 3D spatiotemporal structure, tokens are sampled randomly for input and target, while for sequential modalities, we use span masking [5]. The model is trained with a cross-entropy loss to predict the target tokens. This pretraining objective enables cross-modal prediction at arbitrary time points and scales to any number of modalities.
Beyond any-to-any prediction, Video-4M's ability to traverse modalities and time enables higher-level inference capabilities for temporal modeling. We discuss these below.
By traversing the modality axis, Video-4M allows for chained generations, where the model is iteratively tasked to generate one modality at a time, feeding each prediction back to the model as additional context before predicting the next target modality.
Chaining enables consistent predictions across modalities, e.g., depth that is in agreement with the predicted semantics. Moreover, we can use this property to decompose a one-to-one task (e.g., caption-to-video) into a chain of simpler sub-tasks of predicting intermediate modalities. We demonstrate this in the animation below.
We show generations with a coarse-to-fine chain design: caption → transcription → SigLIP 2 → bounding boxes → depth → RGB. This chain design yields higher-quality generations than direct generation.
Click a numbered tile to switch example sets.
In chained generation, semantic modalities such as SigLIP 2 and bounding boxes define the high-level layout and object-level motion trajectories. These are followed by geometric modalities in the chain, such as depth, which add low-level structural details. This factorization leads to final RGB generations with coherent motion and fine-grained details, in contrast to videos generated directly from the caption. Our paper provides further ablations and analysis on the impact of the order and types of modalities used in the chain.
Predicting how a scene will evolve given present observations (i.e., the current state of the world) is a core ability of any predictive world model and allows an agent to plan and act in the world. Video generation models have received much attention for their potential as world models due to their ability to predict future frames.
However, it remains an open question whether world modeling should happen in raw sensory space, such as RGB video, in a latent space, as in JEPA-style world models, or in other structured abstractions (e.g., bounding boxes, depth).
Rather than settling for an either-or answer, this work proposes joint multimodal modeling as a principled approach.
By treating one underlying physical reality as observable through multiple modalities at once (RGB pixels, semantics, and learned latents, etc.), Video-4M can traverse between modalities and predict the future in any one of them.
Given initial past frames as input, Video-4M predicts the future across different modalities, as shown in the examples below. All videos are sampled at 4 FPS: Video-4M observes the first 5 frames (~1 second) and predicts the next 12 frames (~3 seconds). Blue frames mark the observed input frames, while pink frames are Video-4M's future predictions.
Click a numbered tile to switch examples.
The representation space can be chosen based on the information of interest. For instance, we can predict motion in optical flow space, rather than generating a future video from which motion must then be inferred. The latter is inefficient, since it requires modeling details (e.g., background pixels) in RGB that are unnecessary for motion prediction.
Select the information of interest below to see a representation space that captures it and Video-4M's future prediction in that space.
Just as chaining improves over direct generation for the caption-to-video task, the same idea carries over to future prediction: modalities can be chained together to produce better future rollouts in the target modality space. We compare against direct future prediction below and quantify the comparison further in the energy-score analysis that follows.
For each modality pair below, the left is the direct prediction and the right is the chained prediction.
Click a numbered tile to switch example sets.
Setup. Given the initial observed RGB frames, Video-4M generates the future frames of a chosen target modality. We test both direct prediction of the target modality (e.g., RGB → future depth) and a single-hop chained prediction (e.g., RGB → future DINOv2 → future depth).
Evaluation. There are many valid rollouts into the future given a single past sequence, which poses a problem for evaluation: each video in the test set only provides one realized future. We use Gneiting and Raftery's energy score [6], which quantifies the quality of a predictive distribution against a single realized outcome. Given sampled rollouts $\hat{x}^{(1)},\ldots,\hat{x}^{(K)}$ and the observed rollout $x$, the energy score is calculated as:
\[ \mathrm{ES} = \frac{1}{K}\sum_{i=1}^K d(\hat{x}^{(i)},x) - \frac{1}{2K^2}\sum_{i=1}^K\sum_{j=1}^K d(\hat{x}^{(i)},\hat{x}^{(j)}), \]
where $d$ is a distance metric.
Predicting (a) Depth and (b) Bounding boxes through different modalities. Lower is better. Each line represents the average quality, measured by energy score, of the predictions made through a particular modality across all test samples. The "Best chain" line represents the performance of the oracle-selected best modality for each sample. The shaded area shows the gap between the per-sample best modality chain for prediction and the best single-modality prediction. The shaded area is large relative to the random, retrieval, and static baselines.
Results. Energy scores generally increase with the prediction horizon, as farther-future frames are harder to predict. The static baseline (repeating the last observed frame) does well only in the first few frames before degrading quickly, while the random and retrieval baselines stay roughly flat across the whole horizon. These baselines serve as calibration guidelines to gauge how much difference is meaningful. Takeaway: the best per-sample modality chain substantially outperforms any single fixed chain. This suggests that the optimal modality chain is highly sample-dependent, and that choosing the right representation (i.e., modality) to predict through is important for prediction quality. We refer the reader to the paper for a more comprehensive breakdown.
Different representation spaces capture different aspects of the world and serve complementary roles. Video-4M can use any of these modalities as an action to steer generation in any output space. Starting from an initial observation and given an action (a conditioning modality), Video-4M generates the resulting future; different actions from the same initial observation yield different plausible rollouts.
Conditioning modalities range from coarse (e.g., captions) to fine-grained (e.g., depth or bounding boxes), and the generated output can be in any modality space. Choose a first-frame input, a conditioning modality, and an output space below.
Video-4M's multimodal conditioning enables factorized control of different aspects of videos. By decoupling geometry, motion, and appearance through different modalities, we can edit each component independently. For instance, we can extract depth from the input video and use it alongside another conditioning modality to change attributes like color, while preserving the geometry of the original video. Similarly, we can keep the object-level motion of the input video unchanged while altering its semantics by keeping the bounding boxes and changing their labels.
Changing attributes while keeping the depth map fixed
Changing semantics while keeping motion fixed
We can use Video-4M for any-to-any retrieval in the video domain: retrieving samples in one modality given a query in any other modality. To achieve this, we map all modalities of all samples in the database into the SigLIP 2 embedding space and perform similarity search in that space. This allows retrieval across any pair of modalities, in either direction.
Choose the query modality and the retrieval modality to see retrieval results across different input-output modalities.
While Video-4M demonstrates the benefits of modeling the world across time and modalities, we see several promising directions for future work:
We thank Jason Toskov for discussions and feedback on the pretraining pipeline, Rishubh Singh for discussions on motivating the problem formulation during the early stages of the project, and Kunal Singh for help preparing the any-to-any visualization figure. We also thank Ahmad Jarrar, Saqib Javed, and Muhammad Zakwan for valuable discussion and feedback on the project outside the lab.
We thank our sponsors DVPS, Google, and ELLIOT for their support. This work was supported under project IDs 08, 37, and 43 as part of the Swiss AI Initiative, through a grant from the ETH Domain, and by funding from the Swiss State Secretariat for Education, Research and Innovation (SERI). Computational resources were also provided by the Swiss National Supercomputing Centre (CSCS) under the Alps infrastructure.
