Almost every modern text-to-video model is the same machine: a text encoder sets the target, a denoiser starts from pure noise in a compressed latent space and cleans it step by step, and a decoder turns the final latent back into pixels. The whole field is choices about that loop. This is the map of those choices.
Raw video is far too large to model pixel by pixel, so modern generators work in a compressed latent space. A 3D VAE squeezes a clip along height, width, and time into a small grid of latent tokens (typically around 16x smaller spatially and 4x shorter in time), the model does all its work there, and the decoder expands the result back to pixels at the very end.
Inside that space the recipe is diffusion: start from pure noise and run a denoiser that removes a little noise each step, steered by the text prompt, until a clean latent emerges. Hold that loop fixed and the whole zoo (Sora, Veo, Kling, Wan, HunyuanVideo, CogVideoX, LTX, Mochi) is different fillings for four slots: what the latent looks like, what the denoiser is built from, how its attention is wired, and what objective trained it. One newer question sits on top: are all frames denoised together, or one chunk after another.
Six axes carry almost all the variation, plus one that is the whole cost story. Learn them and you can place any new release the moment its report drops.
Nobody diffuses raw pixels. A 3D causal VAE compresses space and time jointly, so the model reasons over motion rather than independent frames. The compression ratio sets everything downstream.
The engine that predicts the noise. The field walked from convolutional U-Nets to transformers, because transformers scale with compute and handle variable length and long-range motion.
Full 3D attention lets every spacetime token see every other, which gives the best consistency but is quadratic. Factorized or sparse tile attention trades some of that for tractability.
Classic diffusion predicts noise over a curved path. Flow matching and rectified flow learn a straight velocity field from noise to data, which needs fewer sampling steps and now dominates.
A text encoder produces the target, then it enters the denoiser either through cross-attention or by sharing one token stream with the video (MMDiT), often modulated by adaptive LayerNorm.
The live fork. Denoise every frame together (bidirectional, best quality, fixed length) or generate chunk after chunk conditioned on the past (autoregressive, streaming, long, interactive).
A five-second 480p clip is tens of thousands of spacetime tokens, and full 3D attention is quadratic in that count, so cost explodes with resolution and duration. Every efficiency trick in the field points here: heavier VAE compression, sparse and tiled attention, cascaded super-resolution, few-step distillation, and KV-cache compression for the autoregressive branch. When someone says a model is slow or capped at a few seconds, this is why.
Each card is the same denoising loop with one part swapped. Watch the denoiser box and how frames relate to each other in time: that is where a family's identity lives.
The first thing that worked. Take a pretrained image diffusion U-Net and insert temporal layers (attention or convolution across frames) between its existing spatial blocks, so it learns motion on top of a model that already knows how to draw. Cheap, and it bootstrapped the whole field.
The ceiling is real: motion is a bolt-on, temporal windows are short, and long clips drift or flicker. Everything after this is an attempt to make time first-class instead of an add-on.
The move that reset the field. Encode the clip with a 3D VAE, cut the latent into spacetime patches, and treat each patch as a token for a Transformer denoiser (a DiT). Video becomes a sequence problem, which means it scales like an LLM and can absorb variable duration, resolution, and aspect ratio.
Sora popularized this and showed transformers beat U-Nets for long-range temporal dynamics and scalability. Every frontier model since is a DiT.
Where the open field sits today. Text tokens and video tokens share one attention stream (MMDiT), often dual-stream early then merged, with expert adaptive LayerNorm handling each modality. Attention is 3D full so every spacetime token sees every other, which fixed the consistency problems of factorized attention. Trained with flow matching, fed by a 3D causal VAE.
This is CogVideoX (3D full attention, expert adaLN), HunyuanVideo (dual then single stream, LLM text encoder, 13B), and Wan (open suite, MoE variants). Excellent quality, and the quadratic attention is exactly the price.
A tax strategy, not a new engine. Generate at low resolution where attention is affordable, then run a separate few-step super-resolution stage to reach 1080p or 4K, fixing sharpness and detail along the way. Splitting the work keeps the expensive base model small.
Most production stacks have a version of this bolted on, whichever family the base is. HunyuanVideo 1.5 ships an explicit few-step upscaler; LTX pushes toward native 4K in fewer stages.
The other side of axis F. Instead of denoising a whole fixed clip at once, use a causal denoiser that generates the next frame or chunk conditioned on the past, denoising within each chunk. That unlocks streaming, long or infinite length, and real-time interactivity.
The engineering fight is training-time versus test-time mismatch (diffusion forcing, self-forcing, causal forcing) and error accumulation as the rollout drifts. Distillation to a few or one step is what makes it real-time. It has not displaced bidirectional DiT for top-end quality, but it owns long and interactive.
Point the streaming branch at control and you get a world model: the next frame is conditioned not just on the past but on an action, so the model becomes an interactive simulator you can steer. Genie learns action-controllable rollouts from unlabeled internet video; NVIDIA's Cosmos consolidates this for physical AI and robotics.
Cosmos 3 even packs an autoregressive reasoning subsequence and a diffusion generation subsequence into one sequence, optionally emitting joint audio, so a single model reasons, predicts the world, and can act. This is the frontier where video generation meets embodied agents.
The DiT plus 3D-VAE core is settled. The recent motion is at the edges: making it fast, making it long, giving it sound, and turning it into a world. Four eras in a few years:
Temporal layers on a 2D U-Net. Short, flickery, but it moved.
Sora reframes video as spacetime patches. Transformers replace U-Nets.
3D VAE, joint text-video attention, rectified flow, few-step and real-time.
Streaming, action-conditioned, audio-joint, interactive world models.
Rectified flow learns straighter noise-to-data paths, cutting the step count. On top, distillation (distribution matching, self-forcing, causal forcing) collapses dozens of steps into a few or one, which is what turns an offline generator into a real-time one.
The autoregressive branch matured into streaming, minute-scale, interactive generation. Diffusion forcing and its successors close the train-test gap, and KV-cache compression plus sparse attention keep the rollout affordable as it grows.
Video stops being silent. Models now generate synchronized audio and speech jointly with the picture rather than dubbing it afterward, by adding audio tokens to the same generative sequence.
The biggest reframing: a large video generator is a learned simulator. Condition on actions and it supports interactive rollouts for games, robotics, and driving. Perception, prediction, and control start to share the generative backbone.
The highest-quality general models are proprietary. They are known to be DiT-based and latent, and they compete on cinematic physics, character consistency, and now audio, but they do not publish VAE ratios, attention schemes, parameter counts, or objectives. You can place them on the eras strip by behavior and by what the labs say, not by a diagram. The open models (Wan, HunyuanVideo, LTX, CogVideoX, Cosmos-Predict) are where you can actually read the architecture, which is why every card above leans on them.
One-sentence compression: the field went from animating an image model to a DiT denoising spacetime patches in a 3D latent, and is now bending toward fast, long, audible, action-conditioned world models. Place a release on that arc and name its axes, and you are reading it the way the field does.
The single most useful comparison to have loaded, because papers routinely pit one representative of each against the other.
Every frame is refined together with full 3D attention, so the model can enforce global consistency across the entire clip. This is the production default and the quality leader.
A causal denoiser rolls the video forward chunk by chunk, conditioning on the past. This buys streaming, unbounded length, and interactivity, at the cost of drift.
Read left to right and you are reading the axes. Backbone and paradigm are the columns that split the field.
| Model | Latent (VAE) | Backbone | Attention | Objective | Signature |
|---|---|---|---|---|---|
| Stable Video Diff. | 2D + temporal | U-Net | factorized | diffusion | image + motion |
| CogVideoX | 3D causal | MMDiT | 3D full | diffusion | expert adaLN, 3D full attn |
| HunyuanVideo 1.5 | 3D causal 16x/4x | DiT (dual→single) | SSTA sparse | flow match | 8.3B, LLM text enc, SR to 1080p |
| Wan 2.x | 3D causal | MMDiT (MoE) | 3D full | flow match | open suite, frame control |
| LTX-2 | 3D causal | DiT | sparse | flow match | native 4K + audio, efficient |
| Cosmos-Predict | 3D causal | DiT / hybrid | 3D + causal | flow / AR | world model, self-hostable |
| MAGI-1 / SkyReels | 3D causal | causal DiT | causal | diff. forcing | autoregressive, long / infinite |
| Sora 2 | undisclosed | DiT (stated) | undisclosed | undisclosed | closed, +audio |
| Veo 3.1 | undisclosed | DiT (stated) | undisclosed | undisclosed | closed, synchronized audio |
| Kling / Runway / Seedance | undisclosed | DiT (stated) | undisclosed | undisclosed | closed, quality-first |
Same method as your other foundations days: cover the section, then say the answers out loud. Five questions carry most of the ground.