Context Window vs Effective Context
A model may accept 200,000 tokens, but it does not use all of them equally well. The context window is how much text fits. The effective context is how much the model really uses well.
New words
- Token
- A small piece of text that the model reads and counts.
- Context window
- The most tokens a model can take in one call, input plus output.
- Effective context
- The part of the window that the model can really use well.
- Recall
- How often the model finds the right fact in the text.
- Lost in the middle
- Models use the start and end of a long prompt better than the middle.
- Distractor
- A passage that looks like the answer but is wrong.
- KV cache
- The memory where the model keeps notes about every token it has read.
What you will learn
- It fits, but the model still misses it
- Lost in the middle: why the model misses the middle
- Try it: move the key fact around
- Trade-offs: what can go wrong?
- Production practices: what real teams do
- In code
- In a real system
- Interview tip
- Key takeaways
10 quiz questions at the end.
The full lesson, with interactive demos and the quiz, opens after you sign in with Google. It is free.