We introduce Video-4M, an any-to-any multimodal video model that is able to predict any modality at any point in time, given any combination of observations from any other modalities. It is thus able to traverse along both the modality and temporal axes, enabling flexible, diverse inference schemes: chained generations, where we decompose a task, e.g., caption-to-video, into a chain of simpler tasks of other intermediate modalities; temporal modeling in abstract modalities, where given a sequence of past observations, the future can be forecasted in any of the modality spaces; and multimodal control & editing where a generation rollout is explicitly constrained by conditioning given as any of the supported modalities specified at any point in time.
TL;DR: We develop an any-to-any multimodal video model called Video-4M. Video-4M is able to predict any modality at any point in time, given any combination of observations from any other modalities. It is thus able to traverse along both the modality and temporal axes and enables flexible, diverse inference schemes.
Video modeling aims to capture spatiotemporal relationships among features of an observation. Most video models focus on modeling this structure using RGB. While this can happen in RGB space, doing so is challenging, as it requires simultaneously learning high-level semantics (e.g., what is happening and where) and low-level appearance (e.g., pixel-level details). It is also often unnecessary: for certain predictive tasks, estimating where an object will be in the future is often sufficient for decision-making, and more actionable than generating its image in that future. While certain models have gone beyond RGB by incorporating other modalities (e.g., depth and segmentation maps), jointly modeling a large number of modalities and studying the consequences for the relationships between them is of fundamental interest.
In this work, we study the role of multimodal modeling for video. We first develop a generic multimodal any-to-any video model, which supports a diverse set of modalities: depth, surface normals, optical flow, bounding boxes, text, and neural network feature maps, etc. This encapsulates multiple specialist models within one model and performs diverse tasks (e.g., depth estimation or caption-to-video). It also enables novel capabilities, such as temporal modeling through any modality, and mappings for which no specialist exists (e.g., depth to caption).
We explore three instantiations of this framework. In chained generations, we decompose a task, e.g., caption-to-video, into a chain of simpler tasks of other intermediate modalities. We find that chained inference yields more consistent and higher-quality generations than direct generation. Secondly, we explore temporal modeling in abstract modalities and find that it can lead to more accurate forecasting of information such as distances (via depth) and object motion (via bounding boxes) than modeling in RGB modality. Finally, we demonstrate the fine-grained and controllable multimodal generation capabilities of our framework.
After pretraining Video-4M, it's any-to-any nature allows traversing along the modality axis with arbitrary selection of input and output modalities. In below visualizations, we demonstrate this by showing one-to-all generation, where Video-4M sequentially generates each modality from a single input. With consistent predictions across different modalities, Video-4M successfully learns the supported multimodal abstract spaces.
Drag the slider for additional video samples. Hover a prediction cell to preview it. Click a cell to pin the pop-up panel.
Video-4M encapsulates multiple video tasks within a single neural network. Given RGB video as input, Video-4M can generate different modalities in a highly consistent manner.
RGB
Drag the slider for additional video samples.
Development of Video-4M entails several components, namely, multimodal video dataset curation, tokenization, and model architecture & pretraining. Below, we briefly discuss each component.
To pretrain Video-4M, we develop K600-MM, a multimodal dataset comprising 12 aligned modalities. K600-MM is built by pseudo labeling Kinetics-600 dataset [1] with specialist models to generate additional modalities. The table below lists the supported modalities and additional details.
| Modality | Pseudo Labeler | Tokenizer |
|---|---|---|
| RGB | N/A | FSQ |
| Geometric modalities | ||
| Depth | VDA-v2 [2] | FSQ |
| Surface Normal | NormalCrafter [3] | FSQ |
| 3D Human Pose | TokenHMR [4] | VQ-VAE |
| Motion modality | ||
| Optical Flow | DPFlow [5] | FSQ |
| Modality | Pseudo Labeler | Tokenizer |
|---|---|---|
| Semantic modalities | ||
| SigLIP 2 | SigLIP 2 [6] | FSQ |
| DINOv2 | DINOv2 [7] | FSQ |
| V-JEPA 2 | V-JEPA 2 [8] | FSQ |
| Detection | Grounding DINO [9] | WordPiece |
| 1D / High-level modalities | ||
| Caption | Gemini 2.0 Flash [10] | WordPiece |
| Class Category | N/A | Embed. |
| Transcriptions | Gemini 2.0 Flash [10] | WordPiece |
Overview of Modalities. We group modalities into four categories based on the type of information they capture. In total, K600-MM spans 12 multimodal abstractions. For class category, we learn a token embedding layer.
Examples. Below we show K600-MM training examples across all 12 aligned modalities.
Drag the slider for additional example sets.
We pretrain Video-4M on tokenized version of K600-MM. Modality-specific tokenizers are used to convert raw modality data into discrete tokens. Below, we discuss tokenization details for different modality types. Each tokenized training sample represents a 17 frames with 4-second duration and 128-resolution video clip sampled at 4 FPS.
Modalities with a dense 2D structure. For modalities with a 2D spatial structure such as RGB, depth, normals, optical flow, and neural network feature maps, we adapt the VidTok tokenizer [11].
Modalities that are represented sequentially. For modalities such as captions, transcriptions, human poses, and bounding boxes, we treat them as text tokens and encode them using WordPiece tokenization [12]. For detections and human poses, we introduce special frame markers — [FRAME_1], [FRAME_2], etc. — to delineate frame boundaries, making the model aware of temporal structure. For transcriptions, we use four special temporal tokens, [SEC_1] through [SEC_4], to mark the start of each one-second transcription segment aligned with the 4-second video clips.
As illustrated above, Video-4M is an encoder-decoder transformer model and uses a multimodal masked modeling objective [13] for pretraining. During pretraining, one subset is randomly selected from all modality tokens (across both space and time) as input to the encoder and another random subset as masked (target) tokens which is predicted by the decoder. For dense modalities, tokens are sampled randomly for input and target, while for sequential modalities, we perform span masking [14]. The model is trained with a cross-entropy loss to predict the output tokens.
By traversing along both the modality and temporal axes, Video-4M allows flexible and diverse inference schemes. We explore three downstream task instantiations from this framework:
The any-to-any flexibility of Video-4M gives rise to a useful generation technique called chained generations, where the model is iteratively tasked to generate one modality at a time, feeding each prediction back to the model as additional context before predicting the next target modality. We illustrate this technique in the video below.
We can use chaining to decompose a task into a sequence of potentially simpler sub-tasks by routing through different abstract spaces. Below, we discuss one such instantiation for the RGB generation task.
The RGB generation task, using input conditions such as class labels or captions, can be viewed as a two-fold sub-task: high-level semantic modeling — such as object positions, identities, and motion trajectories — followed by modeling of low-level details such as geometry-based, pixel-level detail.
Instead of jointly performing these steps (i.e., direct RGB generation), decomposing them into intermediate abstract spaces can make the task more tractable. Given an initial condition such as a caption, Video-4M sequentially generates intermediate abstract spaces before generating RGB in the final step.
Drag the slider for additional example sets.
Qualitative results. In chained generations, intermediate modalities, such as semantic features and bounding boxes, outline the semantics and motion trajectories, while geometric modalities, like depth, contribute in distinguishing objects from each other. We hypothesize that this leads to RGB generations with coherent motion and fine-grained details.
On the other hand, videos generated using the direct generation baseline often exhibit noticeable structural artifacts. While the overall semantics are largely preserved (e.g., a person attempting to perform a side kick when conditioned on this action), direct prediction struggles with low-level details such as object boundaries, particularly in frames where sudden motion occurs.
In our technical report, we present empirical results where chaining overall wins over direct generation. We further provide ablations and analysis on the impact of the order of modalities and the modality types used in the chain.
We often want to predict the state of the world at a future time t + Δ for practical reasons, to use that prediction as a model of the world for planning and acting.
Future prediction (a.k.a. world modeling) is most commonly cast as a video generation problem, where the future is rolled out in RGB pixel space. While this future prediction can happen in RGB, it is often both less accurate and unnecessary. Forecasting in abstract spaces (in the form of other modalities), such as semantics, can be just as effective: knowing where an object will be in the future is often enough, and more actionable, than generating an image of it in the future. This raises a natural question: which space should world modeling happen in? Promising future-prediction models are already emerging for robotics and driving, but they typically commit to rolling out the future in just one or two modalities.
Video-4M takes a different approach: rather than choosing a single modality upfront, it models future states jointly across all of them. The key idea is that a joint multimodal model treats one underlying physical reality as observable through multiple modalities at once: RGB pixels, semantics, and learned latents alike, and can move fluidly between these views instead of committing to any single one. Given initial RGB frames as input, Video-4M can therefore forecast the future across multiple abstract modalities simultaneously, as shown in the demonstrations below.
Drag the slider to switch to different examples.
Setup. We forecast the future by giving Video-4M a short sequence of observed RGB frames and generating future frames in a chosen target modality. We compare direct prediction of the target modality (e.g., RGB → future Depth) against a single-hop chained prediction (e.g., RGB → future DINOv2 → future Depth).
Evaluation. Since many futures are plausible given the same past, a generated rollout cannot simply be compared to the one observed continuation with a standard distance metric. We instead use the energy score [15], a proper scoring rule that evaluates a set of sampled rollouts against the single observed outcome, rewarding predictions that are both accurate and diverse. We use a modality-appropriate distance function for each target: Sqrt-SiLog [16] for depth, L2 in occupancy-map space for bounding boxes, and L2 in DINOv2 / Inception-V3 feature space for DINOv2 and RGB, respectively.
Energy score vs. forecast horizon. Lower is better. Direct and GT-cond. are single fixed forecasting strategies; the shaded band spans all tested modality chains, and Best chain traces the best per-sample chain among them.
Results. Energy scores generally increase with the forecast horizon, as farther-future frames are harder to match to the single observed rollout. The static baseline (repeating the last observed frame) does well only in the first few frames before degrading quickly, while the random and retrieval baselines stay roughly flat across the whole horizon; ground-truth conditioned prediction performs best early on, since past ground-truth frames still tightly constrain the near future. Chained predictions generally outperform direct predictions, reinforcing our chaining results above. More strikingly, the best per-sample modality chain substantially outperforms any single fixed chain — even overtaking the ground-truth-conditioned prediction at later frames, with much slower degradation. This suggests that the optimal modality chain is highly sample-dependent, and that choosing the right abstraction to predict through can meaningfully improve forecast quality; we leave automatically selecting the best chain per sample to future work.
Different modalities capture complementary spatio-temporal information in videos. As Video-4M can accept multiple input modalities as conditions, this enables many exciting video editing and generation possibilities through multimodal input probing. We demonstrate a subset of these capabilities below.
Given an initial observed frame, Video-4M can generate different plausible futures depending on which modality is used to probe it. Below, we probe the model with different hypothetical captions describing what happens next, while keeping the initial frame fixed.
Drag the slider for additional example sets.
Instead of probing from an initial frame, here we condition generation on a full trajectory in another modality space, such as a bounding-box trajectory or a depth map video. Using Video-4M, we can edit an original video in a fine-grained manner by keeping one aspect fixed — such as its motion, depth, or semantics — and editing another. We show three such instantiations below: changing bounding boxes, changing attributes while fixing depth, and changing semantics while keeping motion fixed.
Changing Bounding Boxes
This control is not limited to RGB output either — the same conditioning can generate directly in other abstract output spaces, such as surface normals or optical flow.
Drag the slider for additional example sets.
Changing Attributes While Fixing Depth
Drag the slider for additional example sets.
Changing Semantics While Keeping Motion Fixed
Drag the slider for additional example sets.
Video-4M can be used for any-to-any retrieval tasks in the video domain. Specifically, by predicting the semantic modality (SigLIP2) for the other modalities in the database, we can retrieve similar samples — in the SigLIP2 embedding space — in any desired modality space, given a query sample in any desired modality space. The top-ranked samples are obtained using cosine similarity between the global SigLIP2 embeddings, which we obtain by mean-pooling spatially and temporally. Below, we show retrieval results for bounding boxes, RGB, depth, and captions.
Any to RGB Retrieval.
Drag the slider for additional example sets.
Any to Detection Retrieval.
Drag the slider for additional example sets.
Any to Depth Retrieval.
Drag the slider for additional example sets.
Any to Caption Retrieval.
Drag the slider for additional example sets.
We thank Jason Toskov, Rishubh Singh, and Kunal Singh for their feedback and discussion during the early stages of the project. We also thank Ahmad Jarrar, Saqib Javed, and Muhammad Zakwan for their fruitful discussion about this research outside the lab. This work was supported under project ID a08 as part of the Swiss AI Initiative, through a grant from the ETH Domain and computational resources provided by the Swiss National Supercomputing Centre (CSCS) under the Alps infrastructure. We also thank our funding partners Google, ELLIOT, and DVPS for supporting this project.
