RNNs & LSTMs
Some data comes in a line, one item after another: words, daily temperatures, or the balls of a cricket over. A recurrent neural network (RNN) reads such data one item at a time and keeps a memory. In this lesson you see how an RNN remembers, why it forgets, and how the LSTM fixes this.
New words
- Sequence
- A list of items where the order matters, such as words in a sentence.
- Hidden state
- The running note an RNN carries from one step to the next.
- Vanishing gradient
- The learning signal gets smaller and smaller as it moves back through many steps.
- LSTM
- Long Short-Term Memory: an RNN with a memory cell and three gates.
- Gate
- A number between 0 and 1 that decides how much information passes.
- GRU
- A lighter version of the LSTM with two gates.
What you will learn
- Why does order matter?
- How does an RNN remember?
- The same weights at every step
- The problem: it forgets
- The LSTM: a notebook with three gates
- Where are RNNs used today?
- Try it: a sentence read word by word
- Common mistakes
- Explain it to a friend
- Key takeaways
10 quiz questions at the end.
The full lesson, with interactive demos and the quiz, opens after you sign in with Google. It is free.