S1E2 How LLMs Work
A machine that does one thing, billions of times: guess the next word
What this lesson is about
A large language model works by predicting the next word in a sequence, over and over again. During training it does this billions of times, and along the way it builds a vast internal map of language where words, meanings, and contexts are stored as numbers. In this lesson you will open up that process and follow it from raw training data, through tokenisation and embeddings, to the moment the model picks its next word. The goal is for the inner workings of an LLM to stop feeling like magic and start feeling like something you can explain.
What you'll be able to do
- I can analyse how predictive-text systems generate next-word suggestions.
- I can participate in a structured discussion using evidence from readings and my conversation with the Gem.
A little background
When you type a prompt, the model navigates its internal word map using an architecture called a transformer, paying attention to the parts of your input that matter most. It then uses probability to choose the most likely next token, one piece at a time. The key terms to get comfortable with are training data, tokens, embeddings, the transformer, and self-attention, and you will build your own definition of each.
Core Reading
Further Readings
-
The surprising power of next-word prediction (Large language models explained, part 1) · Center for Security and Emerging Technology (CSET), Georgetown
-
How developers steer language model outputs (part 2) · Center for Security and Emerging Technology (CSET), Georgetown
-
Multimodality, tool use and autonomous agents (part 3) · Center for Security and Emerging Technology (CSET), Georgetown
Sign in with your St Leonards account to open this one.
Where these facts come from
Every source on this page was opened and checked before it went here. Click through and check anything you are about to rely on.
- How Generative AI Works — Financial Times interactive visual explainer covering next-word prediction, tokens, and training.
- CSET AI Explainer Series — In-depth series from the Center for Security and Emerging Technology on how AI language systems are built and deployed. (Optional further reading.)
Key words
- Training data
- The enormous body of text a model learned patterns from, long before it met you.
- Token
- The chunk of text a model actually works in — usually a word, sometimes part of one.
- Embedding
- A list of numbers recording where a token sits on the model's internal map of meaning.
- Transformer
- The architecture that lets a model weigh which parts of your input matter most.
- Self-attention
- The step where the model decides which earlier words bear on the word it is about to choose.