Attention
Attention lets a model look at every word in a sentence and decide which words matter for each word. It is the main idea inside ChatGPT and other large language models. In this lesson you learn how it works with small numbers.
New words
- Attention
- A way for each word to look at other words and use the useful ones.
- Weight
- A number that says how much one word looks at another.
- Softmax
- A step that turns scores into weights that add up to 1.
- Query, key, value
- Query asks, key says what a word offers, value is what it gives.
- Self-attention
- Words in one sentence look at the other words in the same sentence.
- Head
- One reader inside multi-head attention, which looks for one kind of link.
What you will learn
- Who is "she"?
- For each word: how much should I look at the others?
- Query, key and value: a library
- A tiny example with 3 words
- Self-attention, many heads, and speed
- Try it: see where each word looks
- Trade-offs
- In a real system
- Interview tip
- Key takeaways
10 quiz questions at the end.
The full lesson, with interactive demos and the quiz, opens after you sign in with Google. It is free.