Explainer 6 of 6 · Training · interactive
Overfitting
Why can a model be right on its training data and wrong on new data?
Watch
The idea
Training data has a pattern and noise. A flexible enough model can fit both, and the noise is exactly what will not repeat in new data.
So training error alone always looks better with more capacity. The honest measure is error on data the model never saw, which falls, bottoms out and rises again as the model starts memorising.
The cures follow: more data, a less flexible model, regularisation, augmentation, or stopping early. The ViT on this site reaches 98.6% on its training images and 82.7% on the test set, a gap a convolution's built-in assumptions largely avoid.
Try it
train 0.120 · unseen 0.264
Grey is the true curve, cyan the fit, dots the noisy training data. Raise the degree: training error keeps falling, but past degree 5 the error on unseen data rises as the curve bends to chase noise. Now add more training points and watch the same high degree behave.