29 April 2022

Flamingo: few-shot visual language model for interleaved images, video and text

Researchers introduced Flamingo, a family of Visual Language Models that could handle interleaved images, video and text, achieving state-of-the-art few-shot performance on many benchmarks without task-specific fine-tuning.

Machine perception Deep learning Transformers and attention Multimodal models Foundational method Independently validated

Background

Building a model that works well across many visual tasks has traditionally required large amounts of labelled data for each task separately, making the process expensive and inflexible. What researchers wanted was a model that could pick up a new task from just a few examples, the way a person can.

The separate worlds of vision and language modelling had both advanced considerably by the early 2020s. Pretrained vision models could describe what was in an image; large language models could reason and generate text. The hard part was connecting them in a way that was flexible enough to handle real-world inputs, where images and text appear mixed together in no fixed order.

What happened

On 29 April 2022, Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech and colleagues submitted Flamingo to arXiv. The paper introduced a family of Visual Language Models built around three architectural ideas: bridging powerful pretrained vision-only and language-only models, handling sequences of arbitrarily interleaved visual and textual data, and ingesting images or videos as inputs.

Because Flamingo could handle interleaved text and images, the models could be trained on large-scale multimodal web corpora, the kind of mixed content that actually exists on the internet. That training gave them in-context few-shot learning: the ability to adapt to a new task when shown only a small number of examples at inference time, with no weight updates.

The team evaluated Flamingo across a wide range of tasks. Open-ended visual question-answering asked the model to answer a question about an image. Captioning tasks tested whether it could describe a scene or an event. Close-ended tasks presented multiple-choice visual questions. Across all of these, a single Flamingo model achieved state-of-the-art few-shot results through a combination of architectural design and in-context prompting. On several benchmarks, Flamingo matched or outperformed models fine-tuned on substantially more task-specific data.

The work was later published in the Proceedings of Neural Information Processing Systems (NeurIPS) 2022.

Why it mattered

Flamingo addressed a difficult and active challenge in multimodal machine learning that several contemporaneous efforts were also addressing, bridging pretrained vision-only and language-only models through new architectural innovations. On several benchmarks, Flamingo matched or outperformed models fine-tuned on substantially more task-specific data.

People

Oriol Vinyals, Andrew Zisserman, Karen Simonyan

Sources

Flamingo: a Visual Language Model for Few-Shot Learning. arxiv.org. Primary source
Flamingo: a Visual Language Model for Few-Shot Learning. arxiv.org. Primary source

Cite this page

AI Achievements. (2022). Flamingo: few-shot visual language model for interleaved images, video and text. Retrieved 2026-08-29, from https://achievements.ai/milestone/flamingo-few-shot-visual-language-model-for-interleaved

@misc{achievements_flamingo_few_shot_visual_language_model_for_interleaved,
  title  = {Flamingo: few-shot visual language model for interleaved images, video and text},
  author = {{AI Achievements}},
  year   = {2022},
  url    = {https://achievements.ai/milestone/flamingo-few-shot-visual-language-model-for-interleaved}
}