Language Models from the Ground Up
Written with AI. Verified by people. How we make books
Anyone can check this book or correct it. Report an error · How to fix one
How a language model works, built one part at a time: next-word prediction, tokens, embeddings, a neural model, attention and the transformer, training a small model on a CPU, choosing words, what an open model holds inside, and how instruction tuning changes it.
Who it's for For readers comfortable with Python and NumPy who are new to how language models work: a developer who calls LLM APIs and wants to know what happens inside, or a data scientist moving into language work.
Edition history3 editions since October 2026Full changelog
2026.10.12026.10.3
2026.10.3 · Oct 25, 2026Library update, Breaking change59 examples changed
Contents
-
1Predicting the next word
- Load the store's text
- Count which word follows which
- Turn counts into probabilities
- Suggest the next word as a shopper types
- Generate names one word at a time
- Score a model with perplexity
- Unseen word pairs break the model
- Give unseen pairs a little probability
- What goes wrong in production
- What this chapter decided
- 2Tokens
-
3Counting further back
- Longer contexts fit the training text better
- Longer contexts run out of evidence
- Mix short and long contexts
- Count the contexts a token follows
- Smooth with Kneser-Ney
- Autocomplete with three tokens of context
- The baseline every later model is measured against
- What goes wrong in production
- What this chapter decided
-
4Words as vectors
- One-hot vectors know nothing
- Describe a token by the tokens around it
- Weigh co-occurrence by surprise
- Compress counts into dense vectors
- Learn vectors by predicting neighbours
- Compare learned vectors with counted ones
- Analogies, and where they fail
- Find similar names with vectors
- What goes wrong in production
- What this chapter decided
-
5A neural language model
- Predict from a window of embeddings
- Compute gradients by backpropagation
- Take gradient descent steps
- Train with PyTorch
- Stop when validation loss turns up
- Compare with the baseline
- Mix the network with the counts
- More text helps only if it's the same kind
- Suggest from similar contexts
- What goes wrong in production
- What this chapter decided
- 6Attention
-
7The transformer block
- Add each part's output to a residual stream
- Normalize before each part
- Transform each position with a gated feed-forward layer
- Give attention positions with rotations
- Tie the output layer to the embeddings
- Check the model against PyTorch
- Leave out a block to see what it does
- Measure what a mismatched detail costs
- Count where the parameters go
- What goes wrong in production
- What this chapter decided
-
8Training a small model
- Turn products into batches
- Update with AdamW and clipped gradients
- Warm up, then decay the step size
- Read the training curve
- Compare with the baseline on names
- Compare on descriptions
- What width, data and steps buy
- Suggest the next word in the search box
- What goes wrong in production
- What this chapter decided
- 9Choosing the next token
- 10Inside an open model
-
11From predicting to following instructions
- Build the two training losses by hand
- Ask the base model and the tuned model
- Format the request as the model was tuned
- Measure made-up product details
- Check whether confidence warns you
- Find what a longer context loses
- Adapt a model to your text
- Look back at the yardstick
- What goes wrong in production
- What this chapter decided