Optimizers
An optimizer is the rule that moves the weights of a network after each mistake. Plain gradient descent works, but it can be slow. In this lesson you meet momentum and Adam, and you race three optimizers down the same hill.
New words
- Optimizer
- The rule that decides how to change the weights after each step.
- Mini-batch
- A small group of training examples used for one weight update.
- Momentum
- A step rule that keeps part of the last step, like a rolling ball.
- Adaptive step
- A step size that is different for each weight.
- Adam
- A popular optimizer that joins momentum with adaptive steps.
- Learning rate
- The size of one step. It is the most important setting.
- Warm-up
- Starting with a tiny learning rate and raising it slowly.
What you will learn
- Why is plain gradient descent slow?
- What is a mini-batch?
- What is momentum?
- What are adaptive step sizes?
- What is Adam?
- Try it: three balls, one valley
- Is the learning rate still the boss?
- Which optimizer should you pick?
- Common mistakes
- Explain it to a friend
- Key takeaways
10 quiz questions at the end.
The full lesson, with interactive demos and the quiz, opens after you sign in with Google. It is free.