Benchmark · video-language models

RepProbe

Georgetown University · GUCV Lab

RepProbe is a 42K-clip benchmark of real and synthetic videos with exactly k ∈ [1, 20] complete repetitions, controlled for position, coverage, and duration. Counting accuracy collapses at high k across video-language models, yet a linear probe on the last-layer residual still recovers the count: the failure is at generation, not in what the model encodes.

Preprint Code + Data

Real

Synthetic

Construction

Exact k from rest-pose bounds

Real clips are cut from official RepCount rest-pose timestamps. Consecutive L-marks define one cycle; a k-clip is a contiguous window of k cycles closed on those rest poses. The label is the number of intervals in the cut, so it cannot drift from the source annotation.

Source & bounds

RepCount videos with L1, L2, … rest-pose frames from the official CSVs.

Cycle intervals

Pair consecutive L marks into half-open [start, end). One interval is one repetition.

k-windows

First k, centered k, and last k cycles when the source is long enough.

Prompt

The cut clip and “return only an integer” are the two inputs to a video-language model.

Because k comes from the annotation, it needs no human or model recount, and a k-clip cannot bleed into the next motion.

Real split

Windows on one source

Four sit-up cycles from one RepCount video. Adjacent cycles share a rest pose. The k-clip for each k is the first k of those intervals; the cut never falls inside a repetition.

Four sit-up cycles from one source, tiled by repetition, with k=1 through k=4 windows marked along the left. — Clip (left) and corresponding cycle tiling (right). Source: RepCount stu7_56. Windows k = 1–4 share the same rest-pose bounds.

Synthetic split

Ten families, duration held fixed

RepProbe-Synthetic contains 10,000 clips (10 families × 20 values of k × 50), each 64 frames and 4 seconds. Each family varies one attribute periodically. Several families are appearance changes rather than motion, so the count has no fitness prior and no length cue. One k = 4 clip from each family:

Flash · k = 4

Textured flash · k = 4

Bounce · k = 4

Pendulum · k = 4

Scale · k = 4

Color change · k = 4

Multi object · k = 4

Shape change · k = 4

Crossing · k = 4

Orbit · k = 4

Duration does not grow with k. A twenty-rep orbit is the same 4 s as a one-rep flash, so a model cannot read the count from clip length.

Clip counts by k and position, and clip duration versus k, for RepProbe-Real and a synthetic overlay. — RepProbe-Real: 32,233 clips from 1,038 sources, k = 1–20. Duration grows with k. The synthetic overlay holds length fixed at 4 s (500 clips per k).

Diagnosis

Representation vs. output: decodable but poorly reported

On a balanced evaluation set the same Cosmos-Reason2-8B model is queried twice: the integer generated from a count-only prompt, and a linear probe on the last prompt token at the final layer. The comparison localizes error to readout versus representation. Ground truth is the number of rest-pose intervals in the cut.

Generation

Model answer

The model answer.

Same model, same clip. The prompt asks for a count and nothing else.

READOUT 1 · GENERATION

Count the number of repetitions in this video. Return only an integer.

MAE 3.91

measures conversion of a represented count into an emitted number.

Probe

Linear probe

Cosmos-Reason2-8B, layer 35.

A linear probe on the last prompt token at the final layer: the vector the language head reads before it writes a number.

READOUT 2 · LINEAR PROBE

Linear probe on L35, last token → k

MAE 2.49

measures whether the model has formed the count. Decodability peaks at the last layer (sweep 2.91 → 2.51 → 2.49).

k is linearly decodable from the last-layer residual. It is not recovered at generation.

Main result

High-k collapse is a readout failure

Fine-tuning is not required to observe the split. On the same balanced clips, both readouts agree up to k ≈ 10. Past k ≈ 11, Cosmos-Reason2-8B’s generated error more than doubles while a linear probe on its last-layer residual stays approximately flat.

At k = 14–20, generation MAE is 6.34; the probe on the vector the language head reads is 2.75. Generation is biased low: 38.3% of its answers are 10 or less at k ≥ 14, against 10.6% for the probe. Both gaps hold under bootstrap confidence intervals.

6.342.75

MAE at k = 14–20, lower is better
Cosmos generated → L35 probe · −3.59

Evaluation

Zero-shot video-language models

Balanced 10,000-clip set, 500 clips per k. Models are prompted to return only an integer.

MAE mean |pred − k|. Exact pred = k. Off-by-1 |pred − k| ≤ 1. Few / mid / many MAE on k = 1–6, 7–13, and 14–20.

Model	MAE ↓	Exact ↑	Off-by-1 ↑	Few 1–6	Mid 7–13	Many 14–20
Cosmos-Reason2-8B	3.91	13.0%	37.8%	1.91	3.20	6.34
Qwen3-VL-8B	7.88	9.1%	24.2%	2.45	6.97	13.43
InternVL3.5-8B	8.97	6.4%	17.2%	6.41	7.23	12.90
Video-R1-7B	9.74	5.4%	15.1%	7.07	7.52	14.23
VideoChat-R1.5	10.52	5.6%	15.5%	7.44	8.27	15.41
LLaVA-OV-7B	12.41	5.5%	14.9%	21.17	6.86	10.47
LongVILA-R1-7B	24.04	4.6%	13.4%	13.51	24.16	33.02

Best in each column in bold.

Cosmos generation versus linear probe

Same clips. The probe is a linear map on frozen last-layer activations of Cosmos-Reason2-8B, not a generative readout.

Readout	MAE ↓	Exact ↑	Off-by-1 ↑	Few 1–6	Mid 7–13	Many 14–20
Cosmos-Reason2-8B	3.91	13.0%	37.8%	1.91	3.20	6.34
Cosmos L35 probe	2.49	25.0%	50.3%	1.62	2.99	2.75

The probe keeps a lower error at large k; generation MAE more than doubles for k = 14–20.

MAE versus ground-truth k for Cosmos generation and the Cosmos L35 probe.