10 February 2023

ControlNet adds structured spatial conditioning to pretrained text-to-image diffusion models

Lvmin Zhang, Anyi Rao and Maneesh Agrawala introduced ControlNet, a neural network architecture that adds spatial conditioning controls such as edges, depth, segmentation and human pose to large pretrained text-to-image diffusion models without degrading their existing capabilities.

Architectures and models Deep learning Generative models Convolutional networks Foundational method Demonstrated

Background

Text-to-image diffusion models by 2023 could generate convincing images from written prompts, but getting them to respect specific spatial layouts was hard. A user could describe a scene in words, yet the model might arrange objects however it liked. There was no reliable way to say, for instance, “put a person in this pose” or “follow this edge map” without retraining the whole model from scratch.

Large diffusion models are pretrained on billions of images, and the encoding layers that capture deep visual structure represent a substantial backbone. Any method that forced a researcher to discard that backbone and start again was not going to see wide use.

What happened

Lvmin Zhang, Anyi Rao and Maneesh Agrawala, working in the area of computer vision and graphics, submitted ControlNet to arXiv on 10 February 2023. The idea was to leave the original pretrained model completely untouched and attach a trainable copy of its encoding layers alongside it.

The two halves are connected by what the authors call “zero convolutions”: convolution layers whose weights start at exactly zero. Because the weights begin at zero, the trainable copy adds nothing to the output at the start of training. The pretrained model runs normally, and no harmful noise can enter during fine-tuning. As training proceeds, those weights grow away from zero and the network learns to inject spatial information.

Zhang and colleagues tested this with Stable Diffusion, feeding it conditioning signals including edges, depth maps, segmentation masks and human pose estimates, either alone or in combination, with or without text prompts. The model learned to follow those signals while still producing the image quality the pretrained backbone had always been capable of.

One finding stood out for practical reasons: the training worked with fewer than 50,000 examples and also scaled to more than one million. That range matters because it means researchers do not need a huge labelled dataset to add a new type of control. A modest collection of paired examples is enough to teach the network a new conditioning signal without touching the underlying diffusion model at all.

Why it mattered

Before ControlNet, most approaches to spatial steering required either retraining, substantial architectural changes, or were limited to narrow spatial signal types. By locking the original model's weights and connecting a trainable copy through zero-initialized convolution layers, the method made structured spatial control achievable without degrading the pretrained backbone's general generation quality. The authors showed this worked with datasets smaller than 50,000 images and larger than one million, suggesting the approach was not strictly dependent on large labelled datasets.

Sources

Adding Conditional Control to Text-to-Image Diffusion Models. arxiv.org. Primary source
Adding Conditional Control to Text-to-Image Diffusion Models. arxiv.org. Primary source

Cite this page

AI Achievements. (2023). ControlNet adds structured spatial conditioning to pretrained text-to-image diffusion models. Retrieved 2026-08-29, from https://achievements.ai/milestone/controlnet-adds-structured-spatial-conditioning-to

@misc{achievements_controlnet_adds_structured_spatial_conditioning_to,
  title  = {ControlNet adds structured spatial conditioning to pretrained text-to-image diffusion models},
  author = {{AI Achievements}},
  year   = {2023},
  url    = {https://achievements.ai/milestone/controlnet-adds-structured-spatial-conditioning-to}
}