Design an AI Chat App
You will design a ChatGPT-style app for 10 lakh users each day. You follow the same steps as in an interview. First come requirements and estimates, then the request path, then the parts that save time and money. An AI app is special because every answer is slow and costs real money.
New words
- LLM API
- A service you call over the internet to get an answer from a language model.
- Streaming
- Sending the answer in small pieces, as soon as each piece is ready.
- Context limit
- The most text, counted in tokens, that a model can read at once.
- Rate limit
- A rule that says how many requests one user can send in a given time.
- Message queue
- A waiting line of jobs that workers pick up one by one.
- Time to first word
- The wait before the first word of the answer appears.
- Model routing
- Sending easy questions to a cheap model and hard ones to a strong model.
What you will learn
- The problem: board result night
- Step 1: Requirements
- Step 2: Estimates
- The request path
- Streaming the answer
- Storing the chat history
- The context limit
- A queue for slow jobs, and rate limits
- Try it: switch the parts on and off
- Saving cost and time
- What can go wrong?
- Trade-offs
- In a real system
- Interview tip
- Key takeaways
10 quiz questions at the end.
The full lesson, with interactive demos and the quiz, opens after you sign in with Google. It is free.