Model Routing
Most questions that people ask an AI product are easy, like “What are your opening hours?” They do not need the biggest, slowest and most costly model. Model routing sends each request to the cheapest model that can answer it well. The large model is kept for hard requests only.
New words
- Router
- The step in front of your models that chooses which model answers.
- Small model (SLM)
- A cheaper, faster AI model that is good enough for easy requests.
- Classifier
- A small model that puts each request in a group, such as easy or hard.
- Embedding
- A list of numbers that stands for the meaning of a text.
- Cascade
- Try the small model first and call the large model only if the check fails.
- Latency
- The waiting time before the answer comes back.
- Schema
- The list of fields and value types that a reply must have.
What you will learn
- Why pay the top price for an easy question?
- How does the router decide?
- Try it: pick a routing strategy
- Trade-offs: what can go wrong?
- Production practices: what real teams do
- In code
- In a real system
- Interview tip
- Key takeaways
10 quiz questions at the end.
The full lesson, with interactive demos and the quiz, opens after you sign in with Google. It is free.