What video retrieval benchmarks taught us about ground truth

We use public video retrieval datasets to benchmark search quality. The setup looks simple: take a text query, retrieve video moments, and compare the result against the dataset's ground-truth clip.

The benchmark score depends on one assumption.

The dataset ground truth has to be right.

That assumption does not always hold. During manual review, we found cases across MSRVTT, MSVD, VATEX, DiDeMo, and QVHighlights where the dataset annotation looked wrong, too narrow, or too hard to defend as the only valid answer.

Some retrieved clips matched the query visually, but they sat outside the accepted ground-truth window. In a benchmark report, those results look like retrieval misses. When you watch the clips, some of them look like dataset misses.

For example:

• A query like a car is in a wreck can match several crash moments, not just one timestamp.
• A query like a video game is played is broad enough to describe many gameplay clips.
• A query like A woman is doing a big hole in a pumpkin. can match the carving process before the annotated window.
• A query like Most beautiful resort i have ever seen depends on subjective context, not a single visual fact.

This matters because a retrieval score can mix two things: how well the system finds the right moment, and how well the dataset represents the possible right moments.

The second part is easy to miss when we only look at aggregate metrics.

We saw five annotation patterns:

• Narrow windows that capture only part of an event.
• Shifted timestamps where matching evidence appears nearby.
• Missing alternate positives when more than one clip is valid.
• Broad captions that allow several plausible clips.
• Subjective captions that depend on context or judgment.

Methodology

We reviewed the query, the ground-truth clip, and the top retrieved shots side by side.

A case was included only when the retrieved clips had visible evidence matching the query. The useful question in those cases was whether the dataset annotation was complete and correct.

We left out examples that looked better explained by search quality, indexing, clip localization, or retrieved-shot descriptions with limited visual detail.

This was a qualitative review. The goal was to understand where one accepted timestamp can hide reasonable visual alternatives.

Annotation patterns

The examples below use the same vocabulary throughout the note.

Clip evidence

Each example shows the accepted GT clip first, followed by the top 5 retrieved shots.

The clips make the dataset issue easier to inspect.

MSRVTT: a car is in a wreck

• GT: on video9800
• Many returned segments clearly show car crashes or wrecks, but they fall outside or partially outside the annotated GT window. The GT appears narrower than the set of semantically matching segments.

MSRVTT: a video game is played

• GT: on video8027
• Many returned scenes explicitly show video games being played across multiple timestamps. The query is generic, so one accepted GT window captures only part of the visual concept.

MSVD: a delicious japanese dish

• GT: on -wa0umYJVGg_100_115
• The candidate moments show several different Japanese-food preparation scenes, but the benchmark accepts only a narrow annotated answer for a broad and subjective caption.

VATEX: A woman is doing a big hole in a pumpkin.

• GT: on efvOYBo03XM_000351_000361
• Multiple returned clips are clear semantic matches for making a big hole in a pumpkin, but they occur outside the annotated GT window. The action appears to span more than the accepted timestamp.

DiDeMo: head passes in front of camera

• GT: on 53301297@N00_5826898997_0a951bea4f
• Returned clips show heads or faces moving into the immediate foreground at other timestamps. The query is broad, and the accepted GT captures one instance of a repeated visual event.

DiDeMo: castle comes into view

• GT: on 51167579@N06_6829893951_f10a25a3c8
• Several returned windows show the same semantic target, a tower or castle-like structure entering view, but the benchmark accepts a narrow GT span.

DiDeMo: light starts blue

• GT: on 26292851@N04_4223864218_8a531d1c08
• The accepted GT is subtle, while top-ranked clips contain clearer blue-light events outside the GT. The query is visually under-specified for a single narrow target.

QVHighlights: A girl speaking from her car

• GT: on Zhx9Ki9bUkE_360.0_510.0
• The GT is only about two seconds, while the video contains many car-speaking moments that satisfy the query.

QVHighlights: Most beautiful resort i have ever seen

• GT: on lyGaTk4MLVM_60.0_210.0
• The caption is subjective, and many resort scenes can satisfy it outside the small annotated GT windows.

Findings

The strongest examples followed a few repeatable annotation patterns.

Many annotations were wrong because they were too narrow for repeated or extended actions. Car wrecks, gameplay, pumpkin carving, and car-speaking moments can occur across many windows in the same or similar videos.

A single GT span can be brittle for those captions.

Generic captions create multiple plausible answers. Queries like a video game is played, head passes in front of camera, and a delicious japanese dish describe broad visual concepts.

When the benchmark accepts one clip, semantically valid results can still land outside the target window. That points back to the dataset as well as retrieval.

Subjective or context-heavy captions are harder to anchor visually. Most beautiful resort i have ever seen depends on context and judgment.

A clip can match the text while missing the dataset's selected moment. For those queries, the label is not a complete representation of the caption.

Discussion

These examples make the benchmarks more useful when read carefully.

For retrieval tasks, a caption can describe an event, a repeated action, a broad visual category, or a subjective impression.

When that caption is paired with one accepted timestamp, the benchmark becomes sensitive to annotation coverage. If the timestamp is wrong or incomplete, the score can hide a valid retrieval.

That is worth keeping in mind when reading scores. Some misses are retrieval misses. Some are places where the dataset is wrong, even when the dataset is considered a standard benchmark.

Before evaluating a system against a benchmark, the dataset itself needs review. That can be manual review, LLM-assisted review, or both.

In video retrieval, one ground-truth window is not always the only correct answer.

Samuel Alexander, "What video retrieval benchmarks taught us about ground truth",
VideoDB Labs, July 2026.

@article{alexander2026whatVideoRetrieval,
  author = {Samuel Alexander},
  title = {What video retrieval benchmarks taught us about ground truth},
  journal = {VideoDB Labs},
  year = {2026},
  note = {https://labs.videodb.io/engineering/field-notes/video-benchmark-ground-truth-ambiguity},
}