Aditya Sridhar

Model · Oct 2026 · runs in your browser

ResNet on CIFAR-10

Stacking more layers should never hurt: the extra layers could just learn to pass their input through. In practice a deep plain network trains worse than a shallow one. The residual network fixes that with one change, an identity shortcut around every pair of layers. This page reruns the experiment from He et al. (2016) at 20 and 56 layers.

ResNet-56 test accuracy
91.70%
Plain-56 test accuracy
81.90%
Parameters, each 56-layer net
853,018
Model file
3.3 MB, ONNX

Try it

CIFAR-10 test images

Pick an image
32×32 pixels: the whole image, exactly as the model sees it.

Loading the model…

  1. airplane
  2. automobile
  3. bird
  4. cat
  5. deer
  6. dog
  7. frog
  8. horse
  9. ship
  10. truck

Runs in your browser. Nothing is uploaded.

The same test images as the CNN page, so the two can be compared. The heatmap is the class activation map of the last stage, which works here because a ResNet also ends in a global average pool and one linear layer.

The experiment

Training lossResNet-56ResNet-20plain-20plain-56
0.0001.0002.000191725334150
Test accuracyResNet-56ResNet-20plain-20plain-56
25%50%75%191725334150
MetricValueSource
ResNet-56, 853,018 params91.70% test · 8.30% error · train loss 0.022training/resnet_cifar10/resnet-56.csv, epoch 50
ResNet-20, 269,722 params90.12% test · 9.88% error · train loss 0.078training/resnet_cifar10/resnet-20.csv, epoch 50
Plain-20, 269,722 params89.47% test · 10.53% error · train loss 0.114training/resnet_cifar10/plain-20.csv, epoch 50
Plain-56, 853,018 params81.90% test · 18.10% error · train loss 0.350training/resnet_cifar10/plain-56.csv, epoch 50

The plain 56-layer network ends with a higher training loss than the plain 20-layer one (0.350 against 0.114). That is not overfitting, which would show the opposite; the deeper network is harder to optimise. With shortcuts the order flips: ResNet-56 beats ResNet-20, 91.70% against 90.12%, with the same number of parameters as plain-56.

Why: the gradient, layer by layer

Gradient norm per conv layer, first epochResNet-56ResNet-20plain-20plain-56
1e-11e01e11e21102030405055conv layer, input → output

Mean over the first 50 training steps of each run, log scale. Layer 1 is the first convolution after the input.

In the plain 56-layer network the gradient grows on its way back to the input: the first convolution receives a gradient 28× the size of the last one's, and the norms span a factor of 88 across the 55 layers. In ResNet-56 the first-to-last ratio is 3× and the spread 6.

Backpropagation multiplies the signal by one Jacobian per layer. In a deep plain stack with batch norm those products tend to grow with depth (Yang et al., 2019), so a single learning rate is too large for the early layers and too small for the late ones, and optimisation stalls at a higher loss. A residual block computes x + F(x), whose Jacobian is I + F′(x): the gradient at the output reaches every earlier block directly, plus correction terms, which is why the residual curves stay much flatter.

Setup

Architecture. As in section 4.2 of the paper: a 3×3 convolution, then three stages of n residual blocks at 16, 32 and 64 channels, each block two 3×3 convolutions with batch norm. Depth is 6n + 2, so n = 3 gives 20 layers and n = 9 gives 56. When a stage halves the resolution, the shortcut subsamples and pads the new channels with zeros (option A), so a plain network and its ResNet have exactly the same parameters.

Training. All four runs share one recipe and seed: SGD with Nesterov momentum, batch 128, weight decay 0.0001, a one-cycle learning rate peaking at 0.1 over 50 epochs (the paper used 164 with step decay), random crops and flips, bf16 autocast.

Reproduce

Hardware
NVIDIA GeForce RTX 3050 Laptop GPU, PyTorch 2.6.0+cu124
Commit
46cf39a
ONNX test accuracy
91.78% on all 10,000 test images
Time
37 min for all four runs

Environment

pip install torch numpy onnx onnxruntime

All four runs, export ResNet-56

python training/resnet_cifar10/train.py