10 December 2015

Deep residual networks make training at 100+ layers practical with identity shortcuts

Kaiming He and colleagues introduced residual learning, letting networks train at depths of up to 152 layers. Their ResNet won first place on five tracks at the ILSVRC and COCO 2015 competitions, including ImageNet classification, detection, localisation, and COCO detection and segmentation.

Architectures and models Deep learning Convolutional networks Computer vision Foundational method Independently validated

Background

By the mid-2010s, training deeper neural networks had become one of the central problems in computer vision. Depth matters: richer, more layered representations help with recognition. But adding more layers did not reliably produce better results. Optimisation became harder as networks grew, and accuracy could plateau or drop even when more capacity was available.

Going substantially deeper than the networks then in common use seemed to make training worse, not better.

What happened

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun introduced a residual learning framework. Instead of asking each layer to learn a mapping from scratch, the framework reformulates the problem: each layer learns the difference, or residual, between its input and whatever output would be ideal. A shortcut connection passes the input directly to the output of the block, so the layer only has to learn what to add rather than the full transformation.

This sounds like a small change, but the effect on trainability was large. The authors showed that these residual networks are easier to optimise, and that accuracy keeps improving as depth increases rather than stalling.

On ImageNet they evaluated residual nets with a depth of up to 152 layers, eight times deeper than VGG nets, yet still having lower computational complexity. An ensemble of these networks reached 3.57% error on the ImageNet test set, winning first place on the ILSVRC 2015 classification task. They also trained models with 100 and 1,000 layers on CIFAR-10, and reported a 28% relative improvement on the COCO object detection dataset, attributable solely to their extremely deep representations.

The same residual nets formed the basis of submissions that won first place on ImageNet detection, ImageNet localisation, COCO detection, and COCO segmentation at the ILSVRC and COCO 2015 competitions. The paper was submitted to arXiv on 10 December 2015 and later published at CVPR 2016.

Why it mattered

Before this work, making networks substantially deeper tended to hurt rather than help accuracy, because deep networks were too hard to optimise. The residual framework showed that depth can be a reliable source of accuracy gains when identity shortcuts are used, demonstrated by a 3.57% error rate on ImageNet and a 28% relative improvement on COCO object detection from their extremely deep representations.

Sources

Deep Residual Learning for Image Recognition. arxiv.org. Primary source
CVPR 2016 Open Access Repository. openaccess.thecvf.com. Secondary

Cite this page

AI Achievements. (2015). Deep residual networks make training at 100+ layers practical with identity shortcuts. Retrieved 2026-08-29, from https://achievements.ai/milestone/deep-residual-networks-make-training-at-100-layers

@misc{achievements_deep_residual_networks_make_training_at_100_layers,
  title  = {Deep residual networks make training at 100+ layers practical with identity shortcuts},
  author = {{AI Achievements}},
  year   = {2015},
  url    = {https://achievements.ai/milestone/deep-residual-networks-make-training-at-100-layers}
}