Explainer 3 of 6 · Vision · interactive
Skip connections
Why does a deeper network train worse, and what does a shortcut fix?
Watch
The idea
Adding layers should never hurt: the extra ones could just pass their input through. In practice a plain 56-layer network trains worse than a 20-layer one, even on its own training data. The problem is optimisation, not overfitting.
A residual block adds its input back to its output, so each block only has to learn a correction, and the signal has a direct path through the whole stack in both directions.
The measurements on this site show it: in plain-56 the training signal differs many-fold between layers, with shortcuts it stays within a few-fold, and the 56-layer ResNet ends up the most accurate of the four runs.
Try it
plain-56: test accuracy 81.90%, final training loss 0.350, layer-to-layer spread 88×
Start at 56 layers without skip connections, then switch them on. The deeper plain network cannot even fit its training data as well as the 20-layer one, and its gradients differ 88-fold between layers; with shortcuts, the same 56 layers train best of all.