Backpropagation
A network can have millions of weights, but it gets only one error number. Backpropagation is how the network finds which weights caused that error and by how much. In this lesson you do it by hand on a tiny network, then watch it work in a demo.
New words
- Forward pass
- Sending the input through the network, layer by layer, to get a prediction.
- Loss
- One number that tells how wrong the prediction is. Smaller is better.
- Backward pass
- Sending the error back through the network to find each weight's gradient.
- Gradient
- How much the loss changes if one weight goes up a little.
- Chain rule
- Multiply the small changes along a path to get the total change.
- Autograd
- A framework feature that finds all gradients for you automatically.
What you will learn
- Who is to blame for the error?
- The chain rule: a factory line
- A tiny network: the forward pass
- The backward pass: gradients by the chain rule
- Check the gradients, then update
- Forward, backward, update: repeat for every batch
- Try it: step through the network
- Do I write this myself? Autograd
- Common mistakes
- Explain it to a friend
- Key takeaways
10 quiz questions at the end.
The full lesson, with interactive demos and the quiz, opens after you sign in with Google. It is free.