5 January 2021

OpenAI releases CLIP and DALL-E on the same day

OpenAI announced CLIP and DALL-E together on 5 January 2021. CLIP was trained on 400 million image and text pairs collected from the internet, and matches the accuracy of a ResNet-50 on ImageNet without using any of that benchmark's 1.28 million labelled examples. DALL-E generates images from a written description, using a version of GPT-3 adapted to produce pictures.

Architectures and models Foundational method Demonstrated

Background

Computer vision had been built on labelled data. ImageNet needed around 25,000 people to annotate 14 million images across 22,000 categories, and every new task meant a new set of labels and another round of annotation.

That was the cost of the field. A model could only recognise what somebody had paid to have named.

What happened

OpenAI announced CLIP and DALL-E on 5 January 2021.

CLIP was trained on 400 million image and text pairs collected from the internet, learning to tell which captions belong with which pictures rather than learning a fixed set of classes. Asked to classify images it had never been trained to classify, described in plain words, it matched the accuracy of a ResNet-50 on ImageNet without using any of that benchmark’s 1.28 million labelled examples.

DALL-E generates images from a written description, using a version of GPT-3 adapted to produce pictures. It got the attention, and reasonably: pictures from sentences is a more legible achievement than a scoring function.

What followed

CLIP was the one that lasted. Its text encoder went on to sit inside Stable Diffusion, and the dataset that model was trained on had itself been filtered using CLIP, so the system that put image generation on ordinary hardware two years later rested on this in two places at once.

The wider change is that captions replaced labels. A description written by whoever posted the picture turned out to be enough, and the annotation that computer vision had been built on stopped being the constraint.

Why it mattered

DALL-E took the attention and CLIP did the lasting work. Learning from captions rather than labels removed the need for the annotation that computer vision had been built on, and CLIP's text encoder went on to sit inside Stable Diffusion while the dataset behind it was filtered using CLIP itself.

Sources

Learning Transferable Visual Models From Natural Language Supervision. arxiv.org. Primary source
Zero-Shot Text-to-Image Generation. arxiv.org. Primary source
Contrastive Language-Image Pre-training. en.wikipedia.org. Secondary

Cite this page

AI Achievements. (2021). OpenAI releases CLIP and DALL-E on the same day. Retrieved 2026-08-29, from https://achievements.ai/milestone/openai-releases-clip-and-dall-e

@misc{achievements_openai_releases_clip_and_dall_e,
  title  = {OpenAI releases CLIP and DALL-E on the same day},
  author = {{AI Achievements}},
  year   = {2021},
  url    = {https://achievements.ai/milestone/openai-releases-clip-and-dall-e}
}