Every current VLM is the same three-part machine: a vision encoder turns pixels into tokens, a connector reshapes them, and a language decoder reasons over vision and text together. The whole field is a set of choices about those three boxes. This is the map of those choices.
A text LLM is a stack of decoder blocks. A VLM wraps that stack with two extra jobs: see and merge. The vision encoder maps an image to a grid of feature vectors. The connector projects those vectors into the decoder's embedding space (and often shrinks their count). Then the visual tokens sit in the same sequence as the text tokens, and the decoder's self-attention does the actual reasoning across both.
Hold that skeleton fixed and the entire modern zoo (LLaVA, Qwen3-VL, InternVL3.5, Llama 4 Vision, Flamingo, Fuyu) is just different fillings for three slots, plus one question about how vision reaches the decoder. That last question is the real fork, so keep it in view: does vision join the sequence, or get injected from the side.
In the LLM tour, the axes were attention variant, positional encoding, norm placement, and dense vs MoE. VLMs inherit all of those in the decoder, then add their own. Learn these seven and you can place almost any new model on sight.
What turns pixels into features, and how big it is. A small bolt-on encoder rides along; the InternVL school scales the encoder up until it rivals the LLM.
Fixed square resize throws away aspect ratio and fine text. The field moved to tiling, then to processing images at native resolution and shape.
How visual features enter the decoder's space, and how many survive. This is the main lever on the visual token budget.
The real fork. Either concatenate visual tokens into the sequence and let self-attention mix them, or keep the sequence text-only and inject vision through added cross-attention layers.
Text RoPE is 1D. Images are 2D and video adds time, so modern VLMs extend RoPE across height, width, and frame index.
Bolt-on alignment freezes a pretrained LLM and teaches a projector, then fine-tunes. Native multimodal pretraining mixes image and text from the start.
Every image injects hundreds of tokens into the sequence, and every one of them lives in the KV cache. High resolution and tiling multiply this fast, so a big part of VLM design is quietly a token-compression problem: pixel shuffle, resamplers, and cross-attention all exist to keep this number down. Size a VLM like a text model and you hit OOM under load.
Now walk the gallery. Each card is the same skeleton with one or two slots changed. Watch the connector slot and the fusion arrows: that is where the identity of each family lives.
The simplest thing that works, and the one that won. Run the image through a frozen ViT, push the patch features through a small MLP projector, and drop the resulting visual tokens straight into the sequence next to the text tokens. No new attention machinery: the decoder's own self-attention fuses everything. Cheap to build, scales with data, and almost every current open model is a descendant.
The cost is bluntness: every patch becomes a token in the sequence, so resolution is expensive and the burden of fusion falls entirely on the decoder.
Before concatenation won, the worry was token count. BLIP-2's answer: a small set of learned query vectors that cross-attend to the frozen visual features and distill them into a fixed, tiny number of tokens (often 32), regardless of image size. That bundle is what the LLM sees.
Mostly superseded now (the compression discards detail that OCR and grounding need), but the idea is alive everywhere: a resampler is the same move. Worth knowing as the origin of the token-budget conversation.
The other side of the fork. Keep the LLM sequence text-only and leave the base model's language ability untouched. Insert new gated cross-attention layers between the frozen decoder blocks; those layers attend from text to a compressed set of visual features (Flamingo used a Perceiver Resampler). Vision never enters the main sequence, so image token count is decoupled from sequence length.
Pros: protects text skills, handles many images cleanly, controls KV growth. Cons: extra parameters and a more complex model. This is how Llama's vision models graft sight onto a strong text backbone.
The Qwen school's bet: do not resize the world into a square. The ViT processes images at native resolution and aspect ratio (NaViT-style packing), emitting a variable number of visual tokens. A light MLP merger feeds them in via concatenation, and m-RoPE gives every token a position across height, width, and (for video) time, so the decoder knows where and when each patch sits.
Qwen3-VL sharpens this with a SigLIP-2 encoder and a DeepStack path that mixes multi-level ViT features instead of only the last layer. Strong at OCR, documents, grounding, and long video because nothing gets squashed. The catch is exactly axis G: a big image can mean a lot of tokens.
The opposite instinct to Qwen. If perception is the bottleneck, spend on it: scale InternViT until the encoder rivals the language model in size. Handle high resolution by dynamic tiling (slice the image into 448px crops plus a thumbnail), then tame the token explosion with pixel shuffle, which folds a 2×2 patch block into one token and cuts visual tokens ~4×.
InternVL3 added native multimodal pretraining (train on image and text jointly from the start rather than bolting vision onto a finished LLM), and InternVL3.5 pushed the post-training alignment. The result leans hard on high-fidelity visual features.
The radical branch: do you even need a separate vision encoder? Fuyu says no. Slice the image into patches and linearly project them straight into the decoder, which does all the visual work itself. One model, no CLIP.
Chameleon goes further into early fusion: quantize images into discrete tokens (VQ), then train a single transformer over one interleaved stream of image and text tokens from scratch. Because images are just tokens, the same model can read and generate them. This is the "natively multimodal, one vocabulary" frontier, distinct from the encoder-plus-decoder mainstream.
The tour so far is the settled core. The recent turn, roughly the last year, is that the bridge is dissolving. Vision stops being an adapter bolted onto a finished LLM and gets pretrained into one early-fused stream, and the trunk starts doing more than describe. Four eras in six years:
Contrastive alignment, or a small bridge into a frozen LLM. CLIP, Flamingo, BLIP-2.
A pretrained LLM is the trunk, vision an adapter tuned onto it. LLaVA, Qwen2.5-VL.
Drop the bridge. Modalities enter one early-fused stream, pretrained together from the start.
Predict and act, not just caption: omni agents, vision-language-action, world models.
What was radical for Fuyu and Chameleon (card F6) is now where flagships sit. Image, video, and often audio enter a single stream, the connector shrinks toward nothing, and vision is pretrained in rather than aligned afterward. Generation stays autoregressive text for most, so the live split is now the output axis (text-out vs also generating images), not the input.
Text-image VLMs generalize to text, image, audio, and video in one model. Qwen's omni line uses a Thinker-Talker split (a reasoning module feeding a streaming speech module) on an MoE backbone. NVIDIA's open Nemotron 3 Nano Omni folds vision and audio encoders into one model and, unusually, runs a hybrid Mamba-Transformer-MoE backbone instead of a pure transformer, the only non-standard decoder in this whole map.
Newer models take visual tool use end to end: screenshots and document pages go straight in as tool inputs, and the model reads visual tool outputs (charts, results) back. That needs many images in long context, so aggressive visual-context compression (reported near 20× while holding OCR quality) attacks axis G directly.
The frontier direction turns the trunk into something that predicts and acts, not just answers. NVIDIA's open stack makes it concrete: GR00T pairs an Eagle-based VL backbone with a diffusion-transformer action head for robot manipulation, and Cosmos pairs an autoregressive reasoner with a diffusion generator to unify perception, world generation, and action. Perception, reasoning, and control begin to share one backbone.
The most capable general models are proprietary, and their internals are not published. Google and OpenAI describe their flagships as natively multimodal, trained across modalities from the start, and Anthropic's Claude is strong on documents and vision, but none of the three disclose encoder, connector, resolution, or token-budget details. You can place all three in the early-fusion, natively-multimodal era by stated design and behavior, not by architecture. That distinction is itself worth saying in an interview: for closed models you reason from what they do and what the vendor claims, not from a diagram.
The interview compression of all this is one sentence: the field moved from aligning vision onto a language model to pretraining one multimodal model that early-fuses everything and increasingly acts. Place a new release on the eras strip, name which shift it embodies, and you are reading it the way the field does.
The two strongest open lineages right now disagree about where to spend the compute. This is the single most useful comparison to have loaded, because papers routinely pick one representative from each camp.
Keep the connector minimal, feed the decoder native-resolution tokens, and let position encoding (m-RoPE) plus a strong decoder do the reasoning. Perception stays light; intelligence lives in the language model.
Scale the vision encoder until it rivals the LLM, tile for high resolution, compress with pixel shuffle, and pretrain the whole thing multimodally from scratch. Perception is treated as the hard part.
Read this left to right and you are reading the axes. Fusion is the column that splits the field in two.
| Model | Vision encoder | Resolution | Connector | Fusion | Signature |
|---|---|---|---|---|---|
| LLaVA-1.5 | CLIP ViT-L | fixed 336 | MLP | concat | the baseline everyone forks |
| Qwen3-VL | SigLIP-2 | native dynamic | MLP + DeepStack | concat | m-RoPE, multi-level features |
| InternVL3.5 | InternViT (large) | tiling | pixel shuffle | concat | scaled encoder, native mm pretrain |
| Llama Vision | ViT | tiling | resampler | cross-attn | grafts sight onto a text backbone |
| Flamingo | frozen ViT | fixed | Perceiver | cross-attn | the cross-attention blueprint |
| BLIP-2 | frozen ViT | fixed | Q-Former | concat | fixed 32-token bottleneck |
| Fuyu | none | patch grid | linear | early | no encoder, patches into decoder |
| Chameleon | VQ tokenizer | discrete tokens | shared vocab | early | one stream, reads and generates |
| GLM-4.6V | ViT | native | heavy compression | concat | native visual tool use, ~20× token compression |
| Qwen3.5-Omni | vision + audio | native | Thinker-Talker | early · omni | text·image·audio·video, streaming, MoE |
| Nemotron 3 Omni | C-RADIOv4-H + audio | dynamic | MLP projectors | early · omni | open; hybrid Mamba-Transformer-MoE backbone, 256K |
| Gemini 3 | undisclosed | undisclosed | undisclosed | closed | stated natively multimodal |
| GPT (omni) | undisclosed | undisclosed | undisclosed | closed | stated natively multimodal / omni |
| Claude | undisclosed | undisclosed | undisclosed | closed | strong document + vision, architecture unpublished |
Same method as your other foundations days: cover the topic, then rehearse the questions out loud. Here are the five an interviewer reaches for, with the crisp framing each one wants.