Activation Functions
An activation function is a small rule that each neuron applies to its number before passing it on. Without it, a deep network is no better than one simple layer. In this lesson you meet sigmoid, tanh, ReLU and softmax, and you see why some of them make learning slow.
New words
- Activation function
- A rule that changes a neuron's number before it goes to the next layer.
- Non-linear
- Not a straight line. It bends, so layers cannot be merged into one.
- Slope
- How fast the output changes when the input changes a little.
- Vanishing gradient
- The learning signal gets tiny as it travels back through many layers.
- ReLU
- Returns 0 for a negative number and the number itself otherwise.
- Softmax
- Turns a list of scores into probabilities that add up to 1.
What you will learn
- Why do neurons need an extra rule?
- Many layers collapse into one
- Sigmoid and tanh: the smooth curves
- ReLU and Leaky ReLU: the simple ones
- How can you check a slope?
- What is the vanishing gradient problem?
- Try it: pick a function and slide x
- Softmax: scores to probabilities
- Which one should you use?
- Common mistakes
- Explain it to a friend
- Key takeaways
10 quiz questions at the end.
The full lesson, with interactive demos and the quiz, opens after you sign in with Google. It is free.