A representation is what an encoder turns pixels into before any task runs. Every one is learned by answering a single question: what should the model be forced to predict? Match a caption, fill in a masked patch, or predict a hidden feature. That one choice decides what the features encode and what they stay blind to. This atlas reads that choice across three regimes: image, video, and 4D geometry.
Before the models, the axes. Every method in this atlas is a set of positions on these six, and reading a new paper is mostly locating it on them. The recurring lesson: the objective decides the properties, so most axes trace back to axis A.
Where the training signal comes from. Human labels, paired text, or the image itself. Self-supervision (from the image alone) is the axis that unlocked scale, because it removes the annotation ceiling.
The target the loss is computed against. A global embedding, the raw masked pixels, or the latent feature of a hidden region. This is the single most predictive axis for what the features become.
Whether the useful signal lives in one vector per image or in a feature per patch. Classification wants global; segmentation, depth, grounding, and detection want dense. Many objectives are good at one and weak at the other.
Contrastive views teach the model to ignore augmentations (invariance), which is great for recognition and can erase the spatial detail dense tasks need. Reconstruction keeps that detail (more equivariant). The trade is real.
How the representation is used downstream. A frozen backbone with a linear or kNN probe is the strict test of representation quality; fine-tuning can rescue a weaker backbone but hides what it actually learned.
What the input carries beyond appearance. An image has space; video adds time; 4D adds geometry that persists and moves. Each added axis introduces a new failure mode, and time is where most current models are weakest.
Four objective families produce four different encoders from the same pixels. Learn what each one makes the features good and bad at, and you can predict a backbone's behavior before you run it. This is the core mental model, and video and 4D are variations on it.
Train image and text encoders so that a matching image-caption pair lands close and mismatches land far. CLIP did this at web scale and got strong zero-shot recognition for free, because the text side hands the image a semantic label space. SigLIP swapped the batch-wide softmax for a pairwise sigmoid loss, which is cheaper and scales better; SigLIP-2 is the current workhorse encoder inside many VLMs.
The catch is on axis C. Language supervision is global by nature, so these features are semantically rich but spatially coarse. Great CLS token, weaker patch tokens for dense tasks.
No text at all. Take two augmented crops of the same image, push their embeddings together, and push other images away. SimCLR needs large batches of negatives; MoCo keeps a momentum queue of negatives instead. The model learns to be invariant to crops, color, and blur.
That invariance (axis D) is exactly the strength and the weakness. It nails instance recognition but is trained to discard the augmented-away detail, so dense localization suffers. This family is the conceptual parent of the self-distillation methods in R4.
Mask roughly 75% of the patches and train the model to reconstruct what is missing. MAE uses an asymmetric design: a heavy encoder sees only the visible patches, a light decoder rebuilds the rest. Because the task is pixel-level, the features carry rich local structure.
The trade flips relative to contrastive. MAE features are excellent once fine-tuned and strong on dense tasks, but weaker under a frozen linear probe, because nothing pushed them toward clean global semantics. The classic finding: MAE features separate instances well, DINO features separate categories well.
A student network is trained to match the output of a teacher that is an exponential moving average of the student, across different views, with no labels and no text. DINO showed this yields attention maps that segment objects for free. DINOv2 added a patch-level (iBOT) loss and careful data curation to become the default frozen backbone across vision.
DINOv3 (2025) scaled to a 7B ViT on 1.7B curated images and added Gram anchoring, a loss that stops dense feature maps from degrading over very long training, plus post-hoc text alignment. The result is the current state of the art in frozen, general-purpose features that are strong globally and densely at once, which is why it now anchors segmentation, depth, tracking, and robotics stacks.
Video reuses the image objectives and adds one hard problem: time has to be encoded, not just tolerated. The open failure mode is appearance bias, where a model scores well by recognizing objects while ignoring the order events happened in. Naming that failure precisely is a live research edge, not a solved detail.
MAE, extended into time. VideoMAE masks space-time tubes at a very high ratio (often above 90%, since video is redundant) and reconstructs them, which forces the model to use motion, not just single frames. VideoMAE V2 scales it to a strong general video backbone.
Architecturally, this rides on space-time attention designs like TimeSformer and ViViT, which factorize attention into spatial and temporal parts so the cost stays manageable. Same trade as image MAE: excellent after fine-tuning, dense-friendly, less strong frozen.
The direction with the most momentum. Instead of reconstructing pixels, predict the latent features of masked space-time regions against an EMA target. V-JEPA 2 (Meta, 2025) scaled this to 1.2B parameters on over a million hours of video and, with a small amount of robot data, became an action-conditioned world model that plans robot motion zero-shot. It leads motion-sensitive benchmarks like Something-Something v2.
V-JEPA 2.1 (2026) fixed the known weakness (dense local features) with one move: supervise the visible tokens too, not only the masked ones, so each token has to carry correct local meaning. A clean lesson that loss design decides what a token is allowed to become. Related: DINO-WM builds world models by predicting future DINO features.
CLIP for clips. Align video segments with captions or ASR text, usually combined with a masking objective. InternVideo2 is the strong open representative, and this family is what most video-language models bolt onto an LLM for video question answering.
It inherits CLIP's profile: strong global video semantics and retrieval, weaker fine temporal grounding. That gap, whether the model knows when and not just what, is exactly the diagnostic territory worth probing.
The furthest regime from the first two, and the one to target for literacy, not mastery: know the vocabulary, the two branches, and one method in each. Definitions first: 3D is static geometry recovered from views; 4D adds time, geometry that moves. Two branches share the space, one for rendering scenes, one for reconstructing structure.
This regime assumes ideas an appearance-only background does not cover. The minimum to read the papers: multi-view geometry (how 2D views relate through camera pose), depth (per-pixel distance), and correspondence (the same world point across frames). Everything below is a way to recover some of these from images without hand-built pipelines.
Point tracking is the correspondence workhorse: TAP-Vid as the benchmark, CoTracker as a strong tracker. Get comfortable here and the foundation models stop looking alien.
NeRF fits an implicit radiance field per scene, so you can render photorealistic novel views, at the cost of slow per-scene optimization. 3D Gaussian Splatting replaces the implicit field with an explicit cloud of Gaussians that renders in real time, which is why it took over.
The 4D move: let those primitives change over time. 4D Gaussian Splatting and dynamic-NeRF variants add a time dimension so a moving scene can be replayed and viewed from anywhere. This branch is about appearance and view synthesis, less about a reusable feature.
The branch a representations team cares about most. DUSt3R (2024) predicts dense pointmaps directly from an image pair, no camera parameters needed, collapsing the classic SfM pipeline into one network. MASt3R sharpens its matching. VGGT (CVPR 2025 best paper) goes feed-forward and multi-view: from one to hundreds of images it regresses camera poses, depth, pointmaps, and 3D point tracks in a single sub-second pass, and its authors frame it explicitly as the GPT-or-DINO moment for 3D.
The 4D frontier extends this to motion: MonST3R handles dynamic scenes, CUT3R keeps a persistent state, and streaming variants reconstruct 4D online. These models double as geometric backbones, which is the bridge from the appearance features above to structure.
All three regimes ride the same current: away from labels, away from reconstructing pixels, toward predicting representations, and toward one backbone that spans image, video, and geometry. Place a new model on this strip and you are reading it the way the field does.
Features are a byproduct of ImageNet classification. Pretrain on labels, transfer the trunk. Annotation is the ceiling.
Learn from the image itself or paired text. SimCLR, MoCo, CLIP. Labels stop being the bottleneck.
Dense features without labels. MAE reconstructs, DINOv2 distills. The frozen backbone era begins.
Predict latents, not pixels; one backbone for image, video, and 4D. DINOv3, V-JEPA 2, VGGT.
Compressed to one sentence: the field moved from labeling images, to contrasting them, to reconstructing them, to predicting their representations, and the same arc is now playing out one axis at a time in video and in geometry.
The single most useful debate to have loaded, because current papers routinely pick one representative from each camp and argue against the other. It runs across all three regimes: when you hide part of the input, do you rebuild the pixels or predict the features?
Predict the raw masked content. The target is unambiguous, the gradient is dense, and the features carry fine local detail. Strong when you can fine-tune, and simple to train without collapse.
Predict the latent feature of the hidden region against a learned target. Ignore unpredictable pixel noise, spend capacity on semantics and dynamics, and get representations robust to nuisance. The bet behind JEPA and, in spirit, self-distillation.
The honest read: reconstruction gives you dense detail and simplicity you fine-tune into shape, while latent prediction gives you semantic, transfer-ready features frozen, at the cost of collapse risk. The 2024–2026 momentum is toward prediction, but the strongest frozen dense features still come from the distillation methods that quietly straddle both. Knowing which camp a paper sits in, and what it therefore trades away, is most of reading it.
Read left to right and you are reading the axes. Objective is the column that decides the rest.
| Method | Regime | Objective | Predicts | Frozen strength | Signature |
|---|---|---|---|---|---|
| CLIP | image | contrastive V-L | match to text | global, strong | zero-shot recognition at scale |
| SigLIP-2 | image | sigmoid contrastive | match to text | global, strong | the workhorse VLM encoder |
| MoCo v3 | image | contrastive I-I | view agreement | global, good | momentum queue of negatives |
| MAE | image | masked pixels | raw patches | dense, tune-first | 75% mask, asymmetric decoder |
| DINOv2 | image | self-distillation | teacher features | global + dense | the default frozen backbone |
| DINOv3 | image | self-distillation | teacher features | state of the art | 7B, Gram anchoring, dense-stable |
| I-JEPA | image | latent prediction | masked features | semantic | predict, don't reconstruct |
| VideoMAE V2 | video | masked tubes | raw patches | dense, tune-first | 90%+ masking forces motion |
| V-JEPA 2 | video | latent prediction | masked features | motion-strong | world model, zero-shot robot plan |
| V-JEPA 2.1 | video | latent prediction | masked + visible | + dense fixed | supervise visible tokens too |
| InternVideo2 | video | contrastive + mask | match to text | clip-global | video-language backbone for LLMs |
| 3D Gaussian Splatting | 4D | per-scene render | novel views | n/a (per-scene) | real-time explicit primitives |
| DUSt3R | 4D | pointmap regression | dense geometry | geometric | no camera params needed |
| VGGT | 4D | feed-forward geometry | pose+depth+tracks | geometric backbone | one pass, GPT-for-3D framing |
| MonST3R | 4D | dynamic geometry | moving structure | geometric | geometry under motion |