Bag of Words Applied to Computer Vision (Visual Vocabulary / Bag of Visual Words)
Josef Sivic and Andrew Zisserman at the University of Oxford applied the Bag of Words text-retrieval model to visual features in their 2003 ICCV paper 'Video Google', representing image regions as a vocabulary of visual words to enable efficient object retrieval from video.
Background
By the early 2000s, searching for an object in a video was genuinely hard. The standard approach relied on matching exact geometric features between a query image and every frame in the archive. It worked, but it did not scale. A video with thousands of frames meant thousands of comparisons, and even small changes in lighting or viewing angle could break a match entirely.
Text retrieval had already solved a version of this problem. Search engines were indexing millions of documents efficiently by treating each document as a bag of words: not a sequence of words in a particular order, but simply a count of which words appeared and how often. A weighting scheme called TF-IDF, which gives more importance to words that are common in one document but rare across the whole collection, helped rank results by relevance. And an inverted index, the kind of lookup table that maps each word to the documents containing it, made searching fast even at large scale.
Nobody had tried to port that architecture directly into computer vision. The obstacle was that images do not have words. Before you could borrow the text-retrieval machinery, you had to invent a visual equivalent.
What happened
Josef Sivic and Andrew Zisserman, both at the University of Oxford at the time, published their method at the International Conference on Computer Vision in 2003. Their paper was titled “Video Google: A Text Retrieval Approach to Object Matching in Videos.” The core idea was to extract local visual features from image regions, then group similar features together using k-means clustering, a method that sorts data points into a fixed number of groups based on similarity. Each group became a “visual word.” A whole image could then be described as a histogram of visual word counts, its bag of visual words, in direct analogy to how a text document is described by word counts.
Once images were represented that way, all the machinery of text retrieval transferred across cleanly. Sivic and Zisserman applied TF-IDF weighting to down-weight visual words that appeared in nearly every image (the visual equivalent of a stop word like “the”), and they built an inverted index over the resulting vocabulary so that a query image could be matched against a large video archive without scanning every frame. The paper demonstrated this on real film footage, retrieving frames containing a particular face or object with enough speed and accuracy to make the approach practical.
A year later, Gabriella Csurka, Christopher Dance, Lixin Fan, Jutta Willamowski and Cédric Bray at Xerox Research Centre Europe published a related paper showing that the same bag of visual words representation could also support image classification, not just retrieval, by feeding the histograms into a supervised classifier. That 2004 paper helped establish the framework as a general tool rather than a single-task trick.
The two pieces of work together showed that a well-understood model from information retrieval could cross into computer vision with real results, once someone built the bridge.
Why it mattered
Transposing the Bag of Words model from text retrieval into the visual domain gave the computer-vision community a compact, scalable representation for images that did not require exact geometric matching. This enabled practical large-scale image and video retrieval and became a dominant paradigm for image classification benchmarks through the late 2000s, directly influencing the design of recognition pipelines until deep learning representations superseded it.
People
Josef Sivic, Andrew Zisserman, Gabriella Csurka, Christopher Dance, Lixin Fan, Jutta Willamowski, Cedric Bray
Organisations
University of Oxford, Xerox Research Centre Europe
Sources
- Video Google: A Text Retrieval Approach to Object Matching in Videos.Visual Geometry Group, University of Oxford (ICCV 2003).Primary source
- Visual Categorization with Bags of Keypoints (Csurka et al., ECCV 2004 Workshop).Springer.Secondary
Cite this page
AI Achievements. (2003). Bag of Words Applied to Computer Vision (Visual Vocabulary / Bag of Visual Words). Retrieved 2026-08-22, from https://achievements.ai/milestone/bag-of-words-in-computer-vision
@misc{achievements_bag_of_words_in_computer_vision,
title = {Bag of Words Applied to Computer Vision (Visual Vocabulary / Bag of Visual Words)},
author = {{AI Achievements}},
year = {2003},
url = {https://achievements.ai/milestone/bag-of-words-in-computer-vision}
}