Regularization
A big network can memorise its training data and then fail on new data. Regularization is a set of tricks that stop this. In this lesson you learn weight decay, dropout and early stopping, and you see how to choose between them.
New words
- Overfitting
- The model does well on training data but badly on new data.
- Regularization
- Any trick that keeps a model from memorising the training data.
- Weight decay (L2)
- A small penalty added to the loss for large weights.
- Dropout
- Switching off random neurons during training.
- Early stopping
- Stopping training when validation loss starts to rise.
- Validation set
- Data kept aside to check the model while you train.
- Data augmentation
- Making new training examples by changing the old ones a little.
What you will learn
- Why does a big network memorise?
- What is weight decay (L2)?
- What is dropout?
- What is early stopping?
- Does it help on real data? A curve-fitting test
- Try it: the training curves
- How do you choose what to use?
- Common mistakes
- Explain it to a friend
- Key takeaways
10 quiz questions at the end.
The full lesson, with interactive demos and the quiz, opens after you sign in with Google. It is free.