Rolling Edition

Language Models from the Ground Up

Teaching kit

From Edition 2026.11.8, the book's current edition. It changes when the book does. Use and adapt it freely, with credit to the book.

On this page
  1. Syllabus
  2. Objectives and outlines
  3. Quiz bank

Syllabus

Reading times are for the chapter's text. Students also run its examples, in the Lab or on their own machine.

A chapter a week: 11 weeks

WeekChaptersReadingExamples
11. Predicting the next word20 min8
22. Tokens18 min7
33. Counting further back17 min7
44. Words as vectors18 min8
55. A neural language model18 min10
66. Attention20 min8
77. The transformer block20 min8
88. Training a small model19 min7
99. Choosing the next token17 min7
1010. Inside an open model18 min7
1111. From predicting to following instructions19 min9

In 5 weeks

WeekChaptersReadingExamples
11. Predicting the next word
2. Tokens		38 min15
23. Counting further back
4. Words as vectors
5. A neural language model		53 min25
36. Attention
7. The transformer block		40 min16
48. Training a small model
9. Choosing the next token		36 min14
510. Inside an open model
11. From predicting to following instructions		37 min16

Objectives and outlines

Each chapter's objectives are its first sentence. Each outline follows the chapter's sections, with the terms it defines, the predictions it asks for before an example runs, and the rule each section ends on.

1. Predicting the next word

About 20 minutes to read, 8 examples.

By the end, students can build a complete language model from counts, use it to suggest the next word as a shopper types, generate text with it, and measure with one number how well it predicts text it has never seen.

Lecture outline: 10 sections
  1. Load the store's text
    • Run store-text.py
    • Working rule Split your text into training, validation and test parts before any model sees it, and split by document (here, by product), so near-duplicates of one item never straddle two parts. Compare models on validation text while you work, and touch the test text only for the final report. Learn from the shortest, most regular text you have first; this chapter starts with names, and descriptions come later in the book.
  2. Count which word follows which
    • Key terms bigram
    • Run count-next-words.py
    • Working rule Count pairs in the training split only, and pad every text with start and end markers so the model learns how texts begin and end. Keep the raw counts, not only the probabilities you'll derive from them: the counts tell you how much evidence each probability rests on, and the next sections need them.
  3. Turn counts into probabilities
    • Key terms maximum likelihood estimation
    • Run next-word-probabilities.py
    • Working rule Derive probabilities from counts by division, and check that each word's probabilities sum to one; a sum that doesn't is a bug in the counting. Treat a probability as only as good as the count under it: 0.9 from 1,000 pairs is evidence, while 1.0 from a single pair is an anecdote.
  4. Suggest the next word as a shopper types
    • Run autocomplete.py
    • Working rule For autocomplete, offer the top few next words by probability, and drop tokens that aren't words, such as punctuation. Expect a bigram model's suggestions to be good after words that are almost always followed by the same thing, such as coffee, and poor after words with many possible continuations, such as outdoor. When the earlier words matter, as they do in patio bar, a bigram model can't help, however much text it learns from.
  5. Generate names one word at a time
    • Run generate-names.py
    • Working rule Generate samples whenever you change a model, and read them, because they show failures that a score hides, such as text drifting from one product to another. Use a fixed seed so the samples are reproducible. Never compare models by their samples; compare them by perplexity on validation text.
  6. Score a model with perplexity
    • Key terms Perplexity
    • Run perplexity-by-hand.py
    • Working rule Score every language model by its perplexity on validation text, computed exactly as above, with natural logarithms and every predicted word counted, the end marker included. Compare perplexities only between models that use the same tokens and the same validation text, since splitting text differently changes N and what counts as a word. Report training perplexity only beside validation perplexity, to show how much the model memorized.
  7. Unseen word pairs break the model
    • Run perplexity-on-new-names.py
    • Working rule Never trust a language model that gives zero probability to anything; check for infinite or undefined perplexity on validation text before you compare models. When you see one, count the unseen pairs, as above, to know how much of the text the model can't handle.
  8. Give unseen pairs a little probability
    • Run add-one-smoothing.py
    • Working rule Never ship a counted model without smoothing, and never ship one with add-one smoothing: measure validation perplexity and expect add-one to lose to every method in chapter 3, which compares them on the same tokens. Use add-one only as this chapter does, to show that zero probabilities are fixable and that the fix needs care.
  9. What goes wrong in production
  10. What this chapter decided

2. Tokens

About 18 minutes to read, 7 examples.

By the end, students can build a byte-pair encoding tokenizer from scratch, choose its vocabulary size by measuring what each size costs, and read the tokenizer of an open language model well enough to apply its merges by hand.

Lecture outline: 9 sections
  1. Words leave gaps
    • Run unseen-words.py
    • Working rule Measure the out-of-vocabulary rate of your text on held-out data before choosing word tokens. As a rule of thumb, if more than a percent or two of held-out words are unseen, as in the store's names, don't use words as tokens. Word tokens remain reasonable for text with a closed vocabulary, such as a fixed set of commands.
  2. Characters cover everything
    • Run character-tokens.py
    • Working rule Use characters as tokens only for small experiments, or as a fallback inside a larger scheme, as the next sections do. Check how many characters per word your text has, since that factor multiplies every sequence length and every training step.
  3. Merge the most frequent pair, again and again
    • Run merge-pairs-by-hand.py
    • Working rule Learn merges from the training split only, as you would any other model, and keep the order of the merges, since encoding depends on it. Break ties between equally frequent pairs by a fixed rule, as the example does, so the same text always gives the same tokenizer.
  4. Train the store's tokenizer
    • Run store-tokenizer.py
    • Working rule Train the tokenizer on the same text your model will learn from, here names and descriptions together, so frequent words in that text become single tokens. Build it once and use it for every model you compare; the book's models all read the store's text through lib.bpe.store_tokenizer().
  5. Choose how many merges
    • Run vocabulary-size.py
    • Working rule Choose the number of merges by measuring, on your training text, the share of rare token types, and on held-out text, the tokens per word; stop where tokens per word stops falling much and rare types start climbing. Scale the vocabulary with the training text; as a rule of thumb, a small model on a few million words wants thousands of merges, not tens of thousands.
  6. Read an open model's tokenizer
    • Run smollm2-tokens.py
    • Working rule When you use an open model, use its tokenizer exactly as shipped; a model only understands the tokens it was trained on. Count tokens with that tokenizer, not words, whenever you budget for context length or cost. When you train your own model on narrow text, train your own tokenizer on that text.
  7. Apply SmolLM2's merges by hand
    • Run merges-by-hand.py
    • Working rule Treat a tokenizer as part of a model: version it with the model, load it from the model's own files, and test it on your text, as this section does, whenever you upgrade either one. When two tokenizers disagree on even one token for the same text, find out why before you trust anything a model built on them says.
  8. What goes wrong in production
  9. What this chapter decided

3. Counting further back

About 17 minutes to read, 7 examples.

By the end, students can build an n-gram language model that looks several tokens back, smooth it with Kneser-Ney so it handles text it hasn't seen (smoothing gives some probability to what the counts never saw, as chapter 1 showed), and measure the baseline that every neural model in this book is measured against.

Lecture outline: 9 sections
  1. Longer contexts fit the training text better
    • Run ngram-memorizes.py
    • Working rule Never judge an n-gram model, or any language model, by its perplexity on its own training text: it falls toward 1 as the model memorizes. Choose the order by perplexity on held-out text, as the rest of this chapter does.
  2. Longer contexts run out of evidence
    • Ask first Before you run it, predict: for 4-grams, what share of the validation names' 4-grams never occur in the training names?
    • Run unseen-ngrams.py
    • Working rule Before choosing an n-gram order, measure the share of held-out n-grams your training text never contains, as above. As a rule of thumb, where it nears a third, as it does here for trigrams, a model of that order depends mostly on how it handles the unseen, so its smoothing matters more than its order.
  3. Mix short and long contexts
    • Run interpolate-orders.py
    • Working rule Never use a single n-gram order alone; always combine it with lower orders, so every token keeps some probability. Prefer methods that set each context's weight from its counts, such as Kneser-Ney below, over fixed weights. If you do use fixed weights, choose them on held-out text, not the text you counted.
  4. Count the contexts a token follows
    • Run continuation-counts.py
    • Working rule When you back off from a long context to a short one, weight tokens by how many different contexts they follow, not by how often they occur. Use raw counts only at the highest order, where the context has actually been seen.
  5. Smooth with Kneser-Ney
    • Run kneser-ney-by-hand.py
    • Run kneser-ney-by-order.py
    • Working rule Smooth n-gram models with interpolated Kneser-Ney and a discount of 0.75 unless held-out text says otherwise. Raise the order only while validation perplexity keeps falling noticeably, and stop when a step barely helps or hurts, as from 4-gram to 5-gram here. A variant, modified Kneser-Ney, uses three discounts instead of one and is the one toolkits such as KenLM build; it would likely lower the baseline a little, and this book keeps the single discount because it's easier to follow by hand.
  6. Autocomplete with three tokens of context
    • Run ngram-autocomplete.py
    • Working rule Use the longest context your model can support for suggestions, and check suggestions for contexts that are rare in training, such as outdoor bar, separately from common ones: that's where a counted model falls back to generic guesses.
  7. The baseline every later model is measured against
    • Working rule Before building a learned model, build the strongest counted model you can and record its held-out perplexity as the baseline. Report every later model against it, on the same text and with the same tokenizer, and treat a learned model that doesn't beat it as a result to explain, never as a success. Later chapters find exactly that, and earn the learned models their place another way.
  8. What goes wrong in production
  9. What this chapter decided

4. Words as vectors

About 18 minutes to read, 8 examples.

By the end, students can turn every token of the store's text into a short vector of numbers, so that tokens used in similar ways get similar vectors, by two methods:

  • compressing counts of which tokens appear together, training a small model to predict each token's neighbours
Lecture outline: 10 sections
  1. One-hot vectors know nothing
    • Run one-hot-vectors.py
    • Working rule Don't feed one-hot vectors to a model that needs to relate tokens to each other. Use them only as an index, as the next sections do, to look up a learned vector for each token.
  2. Describe a token by the tokens around it
    • Run cooccurrence-counts.py
    • Working rule Count co-occurrence within a small window, two to five tokens, in text the model will learn from, and never across document boundaries. Don't use the raw counts as vectors: reweight them, as the next section does, so frequent neighbours stop dominating.
  3. Weigh co-occurrence by surprise
    • Run ppmi-neighbours.py
    • Working rule Reweight co-occurrence counts with positive PMI before using them as vectors. Expect PMI to overrate pairs of rare tokens, whose few co-occurrences look far above chance; that's one reason to keep only frequent tokens, as this chapter does, or to require a minimum count.
  4. Compress counts into dense vectors
    • Run svd-neighbours.py
    • Working rule Compress PPMI with SVD to between 50 and 300 dimensions, and check the result the way this section does: list a handful of tokens' nearest neighbours and read them. If the neighbours are mostly frequent function words, the reweighting went wrong; if they're unrelated, the window or the vocabulary is too small.
  5. Learn vectors by predicting neighbours
    • Run skipgram-step.py
    • Working rule When you train embeddings, start the second set of vectors at zero and the first at small random values, as lib.skipgram.train does, and draw noise tokens in proportion to their frequency raised to the power 0.75, which Mikolov et al. found works better than plain frequency. Check one update by hand, as here, before running millions: a probability moving the wrong way means the gradient's sign is wrong.
  6. Compare learned vectors with counted ones
    • Run skipgram-neighbours.py
    • Working rule Treat counted embeddings (PPMI and SVD) and learned ones (skip-gram) as close alternatives: on a few million tokens they give similar neighbours, and counted ones are deterministic and fast. Prefer learned embeddings when they're part of a larger model you're training anyway, as in every later chapter of this book. Never read a high similarity as synonymy without checking for opposites.
  7. Analogies, and where they fail
    • Run analogies.py
    • Working rule Don't judge embeddings by analogies; judge them by how much they help the task they're for, as the next section and every later chapter do. When an analogy fails, check whether the answer is in the vocabulary and whether the words mean what you assume in your text.
  8. Find similar names with vectors
    • Run name-vectors.py
    • Working rule Average token embeddings when you need a quick, order-free representation of short texts, and expect it to find related items but rank them loosely. For real semantic retrieval, use a model trained for it, as the Primer does. For language modeling, give the model the embeddings themselves and let it learn how to combine them, which chapter 5 starts doing.
  9. What goes wrong in production
  10. What this chapter decided

5. A neural language model

About 18 minutes to read, 10 examples.

By the end, students can build a neural language model that predicts the next token from a window of earlier tokens, compute its gradients by hand and check them, train it with PyTorch, and measure it fairly against the counted baseline.

Lecture outline: 11 sections
  1. Predict from a window of embeddings
    • Run window-model-by-hand.py
    • Working rule Build the forward pass of any model you train by hand at least once, and print the shape of every intermediate array, as this example does: a wrong shape is a common bug in neural code, and the shapes say exactly what the model computes. Expect the output layer to hold most of the parameters whenever the vocabulary is large.
  2. Compute gradients by backpropagation
    • Run gradients-by-hand.py
    • Working rule Check hand-written gradients against nudged estimates on a tiny model before trusting them, as here. When you use a library that computes gradients for you, as the PyTorch section does, check the library's forward pass against your own instead.
  3. Take gradient descent steps
    • Run gradient-descent-by-hand.py
    • Working rule Start any new model by running a few steps on a handful of examples and checking that the loss falls, as here: a loss that won't fall on three examples means a bug, not a hard problem. Choose the step size by trying a few: too small and the loss crawls, too large and it jumps around or grows.
  4. Train with PyTorch
    • Run same-as-pytorch.py
    • Working rule Train with a library that computes gradients for you, and keep a short, independent forward pass of your own to check its outputs, as lib.neural does here. Record the library's outputs for a few inputs whenever you save weights, so the check can run anywhere, long after training.
  5. Stop when validation loss turns up
    • Run training-curve.py
    • Working rule Always record validation loss per epoch and keep the best epoch's weights, never the last. When the curves part early, as they do here after a few epochs, the model has more capacity than its training text supports: regularize it, use a smaller model, or get more text of the same kind.
  6. Compare with the baseline
    • Ask first Before you run the comparison, predict which model wins.
    • Run neural-vs-baseline.py
    • Run seen-and-unseen-contexts.py
    • Working rule Always compare a learned model with a strong counted baseline on the same text and tokens before claiming it helps, and split the comparison by seen and unseen contexts to see where each model wins. Don't conclude from one small dataset that neural models don't work, or that they always do.
  7. Mix the network with the counts
    • Run mix-neural-and-counts.py
    • Working rule When a learned model and a counted one are close, try mixing their probabilities before choosing one, and choose the share on held-out text. Keep the share small when one model is much stronger, as here, and confirm the gain on a separate test set before relying on it.
  8. More text helps only if it's the same kind
    • Run more-text-of-another-kind.py
    • Working rule Before adding training text, check that it resembles the text you'll predict; train on both and measure on the target text, as here, rather than assuming more is better. When the extra text differs a lot, train on the target text alone, or lay out each text so the model can tell its kinds apart, as chapter 8 does with a separator between a product's name and its description.
  9. Suggest from similar contexts
    • Run neural-autocomplete.py
    • Working rule Look at a learned model's predictions on contexts the training text lacks, with control prefixes beside them, but judge it by measurements over many contexts, not by examples. For autocomplete in production, use the mix, and test it on rare prefixes as well as common ones.
  10. What goes wrong in production
  11. What this chapter decided

6. Attention

About 20 minutes to read, 8 examples.

By the end, students can compute attention by hand:

  • score how well each token's query matches every earlier token's key
  • turn the scores into weights
  • average the values those weights select
Lecture outline: 10 sections
  1. Look back with a weighted average
    • Run attention-by-hand.py
    • Working rule Write out attention's four lines once, as this example does, and print the weight matrix: its rows say exactly where each position looks. Keep queries, keys and values as separate matrices while you're learning. Libraries often keep them in one matrix, as PyTorch's MultiheadAttention does with its in_proj_weight (source), and split the result, which computes the same thing.
  2. Hide the future with a causal mask
    • Run causal-mask.py
    • Working rule Apply the mask to scores before softmax, never to weights after it, so each row still sums to one. Test it the way this example does: check that the weight on later positions is exactly zero, and that changing a later token leaves earlier outputs unchanged. If a model's training loss falls far faster than expected, check the mask first.
  3. Scale the scores
    • Ask first Before you run it, predict the largest weight with 256 numbers and no division:
    • Run scaling-matters.py
    • Working rule Divide attention scores by the square root of the query's size, as the transformer paper does. A few models, such as T5, leave the division out and fold the factor into how the query weights start instead (source), so check which a model does before you load its weights. If you change how queries and keys are computed or initialized, print the average largest weight of an untrained model, as here, and expect it to be well below 1.
  4. Several heads look for different things
    • Run attention-heads.py
    • Working rule As a rule of thumb, keep each head at least 32 numbers wide. In the transformer paper's experiments, 16 heads of 32 did as well as 8 heads of 64, while 32 heads of 16 did worse, and reducing the size of keys hurt (Vaswani et al., 2017, table 3). Check the weight tensor's shape, heads by positions by positions, whenever you change the model: a head axis in the wrong place mixes heads together without any error.
  5. Read what trained heads attend to
    • Run trained-attention.py
    • Working rule When you inspect a trained model, start with specific tokens in texts you understand, where you know what the model should look at, as here. Treat what the weights suggest as a hypothesis, and test it by changing the input, as the next section does.
  6. Test what the model copies
    • Run copy-from-the-name.py
    • Working rule Test a claim about what a model uses by changing that input and measuring the output, as here, and include a change that should leave the output alone. Read attention weights for hypotheses, and use tests like this one for conclusions.
  7. Some heads attend to the previous token
    • Run previous-token-heads.py
    • Working rule Describe a trained head by measured averages, such as its weight on the previous token, the current token or the start marker, over a few hundred texts. Compare each with what an even spread would give, averaged the same way, so a number such as 0.3 can be read as large or small.
  8. Reach past the window
    • Run attention-reaches-back.py
    • Working rule Measure how far back a trained model attends, as here, before deciding how long its context needs to be, and compare with an even spread so the number means something. Report the weight on markers separately, so marker weight doesn't look like long-range attention, and keep the first tokens when you cut a model's context to save space.
  9. What goes wrong in production
  10. What this chapter decided

7. The transformer block

About 20 minutes to read, 8 examples.

By the end, students can assemble a transformer block from its parts, joined by residual connections: attention, normalization, a gated feed-forward layer and position information.

Lecture outline: 11 sections
  1. Add each part's output to a residual stream
    • Run residual-stream.py
    • Working rule Write every block as additions to the residual stream, never as replacements, and keep the stream's width the same through the whole model so the additions line up. When you debug a deep model, measure what each part adds, as here: a part that adds almost nothing is doing almost nothing.
  2. Normalize before each part
    • Run rms-norm-by-hand.py
    • Working rule Normalize before each part of the block, not after, and normalize the stream once more before the output layer. Use RMSNorm unless you're matching a model that uses layer normalization. When you load another model's weights, check which normalization it uses, where, and with what ε, before anything else: the mismatched-details example shows what a small mismatch costs.
  3. Transform each position with a gated feed-forward layer
    • Run gated-feed-forward.py
    • Working rule Use a gated feed-forward layer with SiLU. Set its width so its three matrices hold as many parameters as a plain layer four times the stream's width, the original transformer's ratio, 2,048 for 512 (Vaswani et al., 2017, section 3.3): that's about 8/3 times the width. SmolLM2-360M uses exactly 8/3, 2,560 for 960. The store's model rounds 8/3 of its width up to a multiple of 32, a rule of thumb that keeps matrix sizes convenient for hardware. When you load another model, check whether its feed-forward layer is gated, since a gated layer has three matrices where a plain one has two.
  4. Give attention positions with rotations
    • Run rope-by-hand.py
    • Working rule Use rotary position embeddings, applied to queries and keys, never values. When you load another model's weights, check how its RoPE pairs the numbers and what base it uses. Some implementations pair neighbouring numbers; others pair the first half with the second, as lib.transformer.rope does and as SmolLM2's configuration states with "rope_interleaved": false. SmolLM2 uses a base of 100,000 where the store's model uses 10,000.
  5. Tie the output layer to the embeddings
    • Working rule Tie the output layer to the embeddings in a small model, and check whether a model you load ties them before you look for a separate output matrix. Keep your own forward pass short enough to read whole, as lib.transformer is, so that every detail the next sections test is visible.
  6. Check the model against PyTorch
    • Run transformer-same-as-pytorch.py
    • Working rule Check your own forward pass against the training library's outputs on a full text, not a single position. Where the library has its own implementation of a part, train with it, so the check compares two independent implementations; where it doesn't, as with RoPE here, test the part's properties by hand.
  7. Leave out a block to see what it does
    • Ask first Before you run it, predict which block's removal hurts most:
    • Run skip-a-block.py
    • Working rule To learn what a part of a model does, remove it and measure, as here, rather than inferring from its weights. Expect the first block to matter most, since everything later is built on it; beyond that, measure, because here the second and last blocks mattered about equally, and more than the third. Read a small loss from removing a part with care: the remaining parts were trained alongside it, so the measurement says what the trained model relies on, not what the part could do alone.
  8. Measure what a mismatched detail costs
    • Run mismatched-details.py
    • Working rule When you load another model's weights, compare its configuration with your code line by line: normalization and its ε, RoPE's pairing and base, the feed-forward layer's gating, tied outputs, and how many heads share keys and values. Then score text you know, as here, against the original implementation's outputs; a small loss with no error is the case to fear.
  9. Count where the parameters go
    • Run transformer-parameters.py
    • Working rule Count parameters by part whenever you design or load a model, as here. At small sizes, the embedding table is a large share of the total, so tie it to the output layer, and choose the vocabulary size with that cost in mind, as chapter 2's choice of merges did.
  10. What goes wrong in production
  11. What this chapter decided

8. Training a small model

About 19 minutes to read, 7 examples.

By the end, students can train a small transformer from start to finish and judge the result.

Lecture outline: 10 sections
  1. Turn products into batches
    • Key terms inputs, targets
    • Run training-batches.py
    • Working rule Build batches as inputs and shifted targets, mask padding out of the loss, and shuffle the texts each epoch with a fixed seed, so the run can be repeated. Measure the share of padding in your batches, as here; if it's large, pack texts or group them by length before you buy a bigger machine.
  2. Update with AdamW and clipped gradients
    • Working rule Use AdamW with a modest weight decay, and clip gradients by their combined length. Keep the training step short enough to read whole, as here, so that every choice in it is visible: what the loss ignores, what the optimizer decays and what gets clipped.
  3. Warm up, then decay the step size
    • Run learning-rate-schedule.py
    • Working rule Warm the step size up over the first few hundred steps, then decay it, and match the schedule's length to the run's. Save the step size as training runs, as the training script does, and check it against the schedule you meant: a schedule bug looks like a training problem, and is far easier to find this way.
  4. Read the training curve
    • Run transformer-training-curve.py
    • Working rule Record validation loss per epoch on each kind of text you care about, and keep the best epoch's weights. Read the gap between training and validation as the warning, not the training loss alone. Never compare perplexities across different kinds of text.
  5. Compare with the baseline on names
    • Ask first Before you run it, predict whether the transformer, alone, beats the counted baseline that chapter 5's window model couldn't.
    • Run transformer-on-names.py
    • Working rule Compare a new model with the strongest baseline you can build on exactly the same predictions, including how each marks the end of a text. When a learned model and a counted one are close, mix them, and choose the share on held-out text.
  6. Compare on descriptions
    • Run transformer-on-descriptions.py
    • Working rule Measure a model on the kind of text it will predict, against a baseline that sees the same text, and expect the ranking to change from one kind of text to another. Split the comparison by seen and unseen contexts to see where each model wins. Try a mix before concluding that either model is enough: here, the mix gained more than any change to either model that this chapter measured.
  7. What width, data and steps buy
    • Run width-data-and-steps.py
    • Working rule Get more text of the kind you'll predict before repeating what you have more than a few times; it costs nothing per step. With computation to spare, try a wider model for fewer epochs before training a narrow one longer, and measure both. Change one thing per run and hold the number of steps fixed where it matters, or the comparison measures two things at once.
  8. Suggest the next word in the search box
    • Run transformer-autocomplete.py
    • Working rule Serve the mix for autocomplete, and test it on rare prefixes as well as common ones, with control prefixes beside each, as here. Treat a single striking suggestion as an illustration of a measured gain, never as the evidence for it.
  9. What goes wrong in production
  10. What this chapter decided

9. Choosing the next token

About 17 minutes to read, 7 examples.

By the end, students can turn a language model's probabilities into text, and choose how:

  • always the likeliest token
  • the likeliest whole text by beam search
  • a sample shaped by a temperature
  • a top-k or top-p cut
  • a repetition penalty and a minimum length
Lecture outline: 9 sections
  1. Choose the likeliest token
    • Run greedy-description.py
    • Working rule Use greedy decoding when you want the same output every time and short text, such as a search-box suggestion, where the likeliest next word is the point. For anything longer than a phrase, expect it to repeat itself, and choose one of the methods in the sections that follow.
  2. Search for the likeliest text
    • Ask first Before you run it, predict the welcome sign's likeliest description:
    • Run beam-search.py
    • Working rule Use beam search for short completions where the likeliest phrase is the goal, such as search queries, ranked by total probability. For longer text, rank finished texts per token, since total probability favours the shortest, and even then prefer the sampling methods of the next sections, which write text closer to what people write.
  3. Sample, with a temperature
    • Run temperature-by-hand.py
    • Working rule Treat temperature as the dial between predictable and varied text. Keep it at or a little below 1 for text that should stay close to what the training text usually says, such as product descriptions. A lower temperature makes text more predictable, not more accurate.
  4. Sample, and cut the tail
    • Run sample-descriptions.py
    • Working rule For varied text, cut the tail with top-p around 0.9 or top-k around 20, with a temperature at or a little below 1, and fix the random seed when you need to reproduce a result. Don't expect any setting to make the text true: decoding chooses among what the model finds plausible, and checking facts is a separate job.
  5. Measure what each setting writes
    • Run measure-decoding.py
    • Working rule Compare decoding settings on measures like these, against real text of the same kind, before reading samples. Decide which measure matters most for the job: for product descriptions, staying on the product and not repeating come first. Then read a sample of the outputs anyway, for what no measure catches, and check every factual claim against the product's data.
  6. Penalize repetition
    • Run repetition-penalty-by-hand.py
    • Working rule Add a repetition penalty around 1.2 to 1.3 when you decode greedily or at a low temperature, and check what it does to length, since it shortens text. As a rule of thumb, if descriptions come out clipped, leave common function words and punctuation out of the penalty.
  7. Forbid an empty description
    • Run minimum-length.py
    • Working rule Set a minimum length whenever the training text includes empty or very short examples of what you're generating. Combine it with the other settings in this chapter, not instead of them, and expect the text it forces out to need the same checking as any other.
  8. What goes wrong in production
  9. What this chapter decided

10. Inside an open model

About 18 minutes to read, 7 examples.

By the end, students can read an open language model with what this book has built:

  • count where its parameters go
  • work out what it costs to run
  • score text with it fairly against a model with a different tokenizer
  • find the attention patterns of chapter 6 in its heads
  • test what it copies
Lecture outline: 10 sections
  1. Count the parameters at scale
    • Run open-model-parameters.py
    • Working rule Count an open model's parameters by part from its weights before you use it, as here, and read the shapes for its design choices: tied embeddings, grouped keys and values, a gated feed-forward layer. Expect the blocks, not the embeddings, to hold most of a large model's parameters.
  2. Compare a family's sizes
    • Run open-model-family.py
    • Working rule Count a model's parameters from its configuration, and check the count against its weights, as here; a mismatch means you've misread a detail. Compare a family's sizes to see how its makers scaled the design, and don't assume a larger model differs from a smaller one only in size.
  3. Count what it costs to run
    • Run open-model-memory.py
    • Working rule Budget a model's memory as its weights plus one cache per text in progress, computed from its configuration, as here. Prefer models that group keys and values when you serve long texts or many at once, and cap the text length you allow to what you've budgeted for.
  4. Score the store's text with a general model
    • Ask first Before you run it, predict which model scores the store's names better per word, and by how much.
    • Run open-model-on-the-store.py
    • Working rule Compare models with different tokenizers per word, or per character, on exactly the same text and events, never per token. Before choosing a large general model over a small one trained on your own text, measure both on your text, as here; size doesn't make up for text unlike anything the model learned from.
  5. Find the same heads at scale
    • Run open-model-heads.py
    • Working rule Describe a large model's heads by the same measured averages you'd use on a small one, compared with an even spread. Expect a few specialized heads and many that put their weight on the first token, and keep that token when you cut a model's context to save memory.
  6. Test copying at scale
    • Run open-model-copying.py
    • Working rule Test a behaviour by changing the input and watching the output, as here, and run the same test on every model you compare. When two models of very different sizes share a behaviour, it's a basic one; look for what differs elsewhere.
  7. Suggest the next word at scale
    • Run open-model-autocomplete.py
    • Working rule Don't put a general model in front of shoppers without measuring its suggestions on your own prefixes against what you have. Where a model trained on your text suggests better, keep it, and measure a general model on the jobs where its broader knowledge could matter.
  8. Set the two models side by side
    • Working rule Set every model you're choosing between side by side on the measures that matter for your jobs, as here, with the same text and the same tests in every column. Choose a model per job, not one model for everything.
  9. What goes wrong in production
  10. What this chapter decided

11. From predicting to following instructions

About 19 minutes to read, 9 examples.

By the end, students can explain how a model that predicts the next token becomes one that follows requests, build the two training losses that do it, and measure what that change does and doesn't fix.

Lecture outline: 10 sections
  1. Build the two training losses by hand
    • Run instruction-loss-by-hand.py
    • Run dpo-loss-by-hand.py
    • Working rule Mask the request out of the loss when you tune a model on requests and responses, and keep the reference model's log-probabilities beside the tuned model's when you train on preferences. Both are small changes to a loop you already have.
  2. Ask the base model and the tuned model
    • Run base-and-tuned.py
    • Working rule Use a tuned model when you need a request followed, and a base model when you need text continued in the style of what came before. Don't read fluency as knowledge: a tuned model writes a better-shaped description from the same lack of facts.
  3. Format the request as the model was tuned
    • Run chat-template.py
    • Working rule Always format requests to a tuned model with its own chat template, applied by the library from the model's tokenizer files, never assembled by hand. Check the template's default system message, since the model reads it on every request, and record the full formatted prompt with every answer you keep.
  4. Measure made-up product details
    • Ask first Before you run it, predict whether the tuned model makes up fewer materials than the base model where the data names none:
    • Run made-up-materials.py
    • Working rule Test a model on questions its input can't answer, not only on ones it can, and count made-up answers separately. Check every factual claim a model makes about a product against the product's data, automatically where you can, as lib.claims does for materials.
  5. Check whether confidence warns you
    • Run answer-confidence.py
    • Working rule Don't use a model's confidence alone to decide whether to trust an answer. Measure how its confidence relates to being right on your own questions, as here, and if you use a threshold, choose it from those measurements and count what it lets through.
  6. Find what a longer context loses
    • Run answer-position.py
    • Working rule Put the details a model most needs at the start or the end of its context, not in the middle, and give it as little unrelated text as the task allows. Test retrieval from a longer context at each position, as here, before trusting a model to find details among many.
  7. Adapt a model to your text
    • Run lora-parameters.py
    • Working rule To teach a general model your text, fine-tune it on that text, with LoRA when memory or storage is tight, and measure the result on held-out text of the same kind. Don't expect instruction tuning or a cleverer prompt to supply knowledge the model never learned.
  8. Look back at the yardstick
    • Run test-names.py
    • Working rule Keep one fixed yardstick for comparing models, the same text, tokenizer and measure every time, as this book did. Then measure the job itself, with tests built for it, before choosing a model to serve it: a lower perplexity is a reason to test a model, not a reason to ship it.
  9. What goes wrong in production
  10. What this chapter decided

Quiz bank

Questions on every chapter, with answer keys, for verified teachers with the Educator plan. Each answer is tied to its section. Ask for the answer keys