Glossary
The 134 terms Language Models from the Ground Up defines, each in the book's own words, with the section that defines it.
From Edition 2026.11.16, the book's current edition. Where a definition depends on a package, it says the version this edition pins and runs its examples on.
#
- <unk> also unknown token
-
A word the training text doesn't contain at all is counted as one more vocabulary entry, <unk>, for unknown, so the probabilities still sum to one.
- Part of: vocabulary
Defined in Give unseen pairs a little probability, chapter 1. In the book since Edition 2026.11.16. Permanent link to <unk>
A
- ablation
-
Leaving a part out allows a simple experiment, called an ablation: remove one part, and measure what the model loses.
Defined in Leave out a block to see what it does, chapter 7. In the book since Edition 2026.11.16. Permanent link to ablation
- Adam
-
The training script's optimizer is Adam, a refinement of gradient descent (Kingma and Ba, 2015).
- A kind of: gradient descent
Defined in Train with PyTorch, chapter 5. In the book since Edition 2026.11.16. Permanent link to Adam
- AdamW
-
AdamW, the Adam of chapter 5 with weight decay applied apart from the gradient
- A kind of: Adam
Defined in Update with AdamW and clipped gradients, chapter 8. In the book since Edition 2026.11.16. Permanent link to AdamW
- add-one smoothing
-
The simplest smoothing method, add-one smoothing, pretends that every possible pair occurred once more than it did.
P(v | w) = (count(w v) + 1) / (count(w) + V)- A kind of: smoothing
- Computed from: vocabulary
Defined in Give unseen pairs a little probability, chapter 1. In the book since Edition 2026.11.16. Permanent link to add-one smoothing
- analogy
-
In vectors, the answer to "a is to b as c is to what?" is the token nearest to b − a + c, leaving out the three tokens in the question
Defined in Analogies, and where they fail, chapter 4. In the book since Edition 2026.11.16. Permanent link to analogy
- attention
-
Attention lets every position look at every earlier one, and decide from their contents how much each matters.
- Contrasts with: input window
Defined in Attention, chapter 6. In the book since Edition 2026.11.16. Permanent link to attention
- attention score
-
The score of position j for position i is the dot product of i's query with j's key, divided by the square root of d, the size of each query and key, meaning how many numbers it holds.
Defined in Look back with a weighted average, chapter 6. In the book since Edition 2026.11.16. Permanent link to attention score
- attention sink also attention sinks
-
Xiao and colleagues found that language models put large weights on the first tokens of a text whatever they are, calling them attention sinks (Xiao et al., 2024).
Defined in Reach past the window, chapter 6. In the book since Edition 2026.11.16. Permanent link to attention sink
B
- backpropagation
-
Backpropagation computes all the gradients at once, by applying the chain rule of calculus backwards through the forward pass
Defined in Compute gradients by backpropagation, chapter 5. In the book since Edition 2026.11.16. Permanent link to backpropagation
- base model
-
the base model, the one trained only to predict the next token
Defined in Inside an open model, chapter 10. In the book since Edition 2026.11.16. Permanent link to base model
- baseline
-
a baseline, a fixed yardstick: one model, one text, one tokenizer and one measure that every later model is compared against
Defined in The baseline every later model is measured against, chapter 3. In the book since Edition 2026.11.16. Permanent link to baseline
- batch
-
Each step of real training is computed on a batch of examples, a few hundred at a time, instead of all of them, so each step is cheap and the steps vary a little.
- Part of: gradient descent
Defined in Take gradient descent steps, chapter 5. In the book since Edition 2026.11.16. Permanent link to batch
- beam search
-
Beam search looks further, keeping the k likeliest partial texts, the beam, scored by their total log-probability
- Contrasts with: greedy decoding
Defined in Search for the likeliest text, chapter 9. In the book since Edition 2026.11.16. Permanent link to beam search
- bigram
-
A bigram is a pair of adjacent words, such as coffee table.
Defined in Count which word follows which, chapter 1. In the book since Edition 2026.11.16. Permanent link to bigram
- byte
-
in UTF-8, the usual text encoding, every character in every language is stored as one to four bytes, and there are only 256 possible byte values, so a vocabulary that starts from all 256 can represent any text at all
Defined in Characters cover everything, chapter 2. In the book since Edition 2026.11.16. Permanent link to byte
- byte-pair encoding also BPE
-
Byte-pair encoding, or BPE, starts from characters and builds bigger tokens out of the pairs that occur most often.
- Implemented by: tokenizers 0.23.3
Defined in Merge the most frequent pair, again and again, chapter 2. In the book since Edition 2026.11.16. Permanent link to byte-pair encoding
C
- calibrated also calibration
-
A model whose confidence matches how often it's right is calibrated
Defined in Check whether confidence warns you, chapter 11. In the book since Edition 2026.11.16. Permanent link to calibrated
- causal mask
-
the causal mask: before softmax, set every score for a later position to minus infinity.
- Part of: attention
Defined in Hide the future with a causal mask, chapter 6. In the book since Edition 2026.11.16. Permanent link to causal mask
- chat template
-
A tuned model's chat template is the layout it learned: special tokens that mark where a system message, instructions placed before the user's request, the request itself and the model's reply begin and end.
Defined in Format the request as the model was tuned, chapter 11. In the book since Edition 2026.11.16. Permanent link to chat template
- co-occurrence window
-
count, for every pair of tokens, how often one appears within a small window of the other, here two tokens either side, and never across the gap between two products' texts
- Comes before: pointwise mutual information
Defined in Describe a token by the tokens around it, chapter 4. In the book since Edition 2026.11.16. Permanent link to co-occurrence window
- confidence per token
-
a confidence per token can be computed for each answer, as a geometric mean
confidence per token = exp((log p₁ + … + log pₙ) / n)Defined in Check whether confidence warns you, chapter 11. In the book since Edition 2026.11.16. Permanent link to confidence per token
- context
-
The tokens before the one being predicted are its context.
Defined in Longer contexts fit the training text better, chapter 3. In the book since Edition 2026.11.16. Permanent link to context
- continuation count
-
a token's continuation count, the number of different tokens it follows
- Comes before: Kneser-Ney smoothing
Defined in Count the contexts a token follows, chapter 3. In the book since Edition 2026.11.16. Permanent link to continuation count
- cosine decay
-
After the warm-up, the step size falls along half a cosine curve, to a tenth of the peak by the last step
rate = peak × (0.1 + 0.9 × ½ (1 + cos(π × progress)))- Part of: schedule
Defined in Warm up, then decay the step size, chapter 8. In the book since Edition 2026.11.16. Permanent link to cosine decay
- cosine similarity
-
To measure how alike two vectors are, this book uses cosine similarity: the cosine of the angle between them, 1 when they point the same way, 0 when they're at right angles.
Defined in One-hot vectors know nothing, chapter 4. In the book since Edition 2026.11.16. Permanent link to cosine similarity
- cross-entropy
-
The cross-entropy of a text is the average surprise over every word in it
cross-entropy = −(1/N) Σ log P(next word | previous word)- Computed from: surprise
Defined in Score a model with perplexity, chapter 1. In the book since Edition 2026.11.16. Permanent link to cross-entropy
D
- declined answer
-
An answer is declined if it says the product's data doesn't say
Defined in Measure made-up product details, chapter 11. In the book since Edition 2026.11.16. Permanent link to declined answer
- decoding
-
Turning a model's probabilities into text, one chosen token at a time, is called decoding
- Kinds: greedy decoding, beam search, sampling
Defined in Choosing the next token, chapter 9. In the book since Edition 2026.11.16. Permanent link to decoding
- Direct Preference Optimization also DPO
-
Direct Preference Optimization, DPO, does preference tuning without a reward model, with a simple loss on the preference pairs themselves
- A kind of: preference tuning
Defined in From predicting to following instructions, chapter 11. In the book since Edition 2026.11.16. Permanent link to Direct Preference Optimization
- discount
-
At each order, Kneser-Ney subtracts a fixed discount d from every count it has seen, and gives the probability it took to the next order down
D = n₁ / (n₁ + 2 n₂)- Part of: Kneser-Ney smoothing
Defined in Smooth with Kneser-Ney, chapter 3. In the book since Edition 2026.11.16. Permanent link to discount
- DPO loss also DPO's loss
-
DPO's loss works on a pair of responses to one request, one preferred and one rejected.
loss = −log σ(margin)- Part of: Direct Preference Optimization
- Computed from: reference model
Defined in Build the two training losses by hand, chapter 11. In the book since Edition 2026.11.16. Permanent link to DPO loss
E
- early stopping
-
the simplest defence is early stopping: measure validation loss after every epoch, one full pass over the training text, and keep the weights from the best epoch.
Defined in Stop when validation loss turns up, chapter 5. In the book since Edition 2026.11.16. Permanent link to early stopping
- embedding also token embedding, word embedding
-
an embedding, a dense vector that stands for a token
- A kind of: vector
Defined in Compress counts into dense vectors, chapter 4. In the book since Edition 2026.11.16. Permanent link to embedding
- embedding table
-
Each token is looked up in an embedding table, a matrix with one row of numbers per token in the vocabulary, as in chapter 4, except that this table is learned for predicting the next token rather than computed from counts.
- Part of: neural language model
Defined in Predict from a window of embeddings, chapter 5. In the book since Edition 2026.11.16. Permanent link to embedding table
- epoch
-
epoch, one full pass over the training text
- Comes before: early stopping
Defined in Stop when validation loss turns up, chapter 5. In the book since Edition 2026.11.16. Permanent link to epoch
- even spread
-
an even spread, which puts 1 / (i + 1) on each of the i + 1 tokens position i can see
1 / (i + 1)Defined in Some heads attend to the previous token, chapter 6. In the book since Edition 2026.11.16. Permanent link to even spread
F
- feed-forward layer
-
Attention moves information between positions; the feed-forward layer transforms it, at each position separately.
- Part of: transformer block
- Contrasts with: attention
Defined in Transform each position with a gated feed-forward layer, chapter 7. In the book since Edition 2026.11.16. Permanent link to feed-forward layer
- fine-tuning
-
The way to teach a general model the store's text is fine-tuning: continue training it on that text, the same training loop as chapter 8, starting from the general model's weights instead of random ones.
- Contrasts with: instruction tuning
Defined in Adapt a model to your text, chapter 11. In the book since Edition 2026.11.16. Permanent link to fine-tuning
- full fine-tuning
-
Full fine-tuning updates every weight, which for a model of hundreds of millions of parameters takes time and memory.
- A kind of: fine-tuning
Defined in Adapt a model to your text, chapter 11. In the book since Edition 2026.11.16. Permanent link to full fine-tuning
G
- gated feed-forward layer also SwiGLU
-
A gated feed-forward layer computes two widened vectors from the input, passes one through a function called SiLU, and multiplies the two number by number, so the first decides how much of the second gets through
FFN(x) = (SiLU(x W_gate) ⊙ (x W_up)) W_down- A kind of: feed-forward layer
Defined in Transform each position with a gated feed-forward layer, chapter 7. In the book since Edition 2026.11.16. Permanent link to gated feed-forward layer
- generation also generate
-
To generate, start at <s>, draw the next word at random in proportion to its probability, append it, and repeat from the word you drew, until the model draws </s>.
Defined in Generate names one word at a time, chapter 1. In the book since Edition 2026.11.16. Permanent link to generation
- gradient
-
To lower the loss, each weight needs a gradient: how much the loss would change if that weight changed a little, written ∂L/∂w.
- Comes before: backpropagation, gradient descent
Defined in Compute gradients by backpropagation, chapter 5. In the book since Edition 2026.11.16. Permanent link to gradient
- gradient clipping also clipping
-
The training script also clips the gradients: when their combined length, measured as if every gradient were one long vector, exceeds 1.0, it scales them all down to that length
Defined in Update with AdamW and clipped gradients, chapter 8. In the book since Edition 2026.11.16. Permanent link to gradient clipping
- gradient descent
-
Gradient descent moves every weight a small step against its gradient
w ← w − η · ∂L/∂wDefined in Take gradient descent steps, chapter 5. In the book since Edition 2026.11.16. Permanent link to gradient descent
- greedy decoding also greedy
-
The simplest decoder is greedy: at every step, take the single likeliest token.
- Contrasts with: sampling
Defined in Choose the likeliest token, chapter 9. In the book since Edition 2026.11.16. Permanent link to greedy decoding
- grouped-query attention also GQA
-
grouped-query attention, which shares each key and value head among several query heads
- A kind of: multi-head attention
Defined in Count the parameters at scale, chapter 10. In the book since Edition 2026.11.16. Permanent link to grouped-query attention
H
- hallucination
-
Making up a detail the input doesn't support is one kind of what's called hallucination
Defined in Measure made-up product details, chapter 11. In the book since Edition 2026.11.16. Permanent link to hallucination
- head also heads, attention head
-
several attention computations, called heads, side by side, each with its own query, key and value matrices
- Part of: multi-head attention
Defined in Several heads look for different things, chapter 6. In the book since Edition 2026.11.16. Permanent link to head
I
- induction head also induction heads
-
an induction head: a head in an earlier layer copies each token's predecessor into its position, and the induction head uses that to find earlier positions that followed the current token, and attends to them
- A kind of: head
Defined in Read what trained heads attend to, chapter 6. In the book since Edition 2026.11.16. Permanent link to induction head
- input window
-
This chapter's model looks at a fixed window of the previous tokens, four here, the same context a 5-gram counts.
- Part of: neural language model
Defined in Predict from a window of embeddings, chapter 5. In the book since Edition 2026.11.16. Permanent link to input window
- inputs
-
The inputs are a text's tokens without the last
- Contrasts with: targets
Defined in Turn products into batches, chapter 8. In the book since Edition 2026.11.16. Permanent link to inputs
- instruction tuning also supervised fine-tuning, instruction-tuned
-
Instruction tuning, or supervised fine-tuning, continues training a model on examples of requests paired with good responses, so that predicting the next token of such a pair means answering the request.
- Comes before: preference tuning, reference model
Defined in From predicting to following instructions, chapter 11. In the book since Edition 2026.11.16. Permanent link to instruction tuning
- interpolation
-
Interpolation falls back smoothly: the probability of a token is a weighted mix of what the trigram, bigram and unigram counts say
P(token | a b) = λ₃ · P₃(token | a b) + λ₂ · P₂(token | b) + λ₁ · P₁(token)- A kind of: smoothing
- Comes before: Kneser-Ney smoothing
- Contrasts with: add-one smoothing
Defined in Mix short and long contexts, chapter 3. In the book since Edition 2026.11.16. Permanent link to interpolation
K
- key
-
a key, a vector for what this position offers to the positions that look at it
- Part of: attention
Defined in Look back with a weighted average, chapter 6. In the book since Edition 2026.11.16. Permanent link to key
- key and value cache also KV cache
-
The key and value cache, every layer's keys and values for the earlier tokens, grows with the length of the text
cache per token, in bytes = layers × 2 × key and value heads × head size × 2Defined in Count what it costs to run, chapter 10. In the book since Edition 2026.11.16. Permanent link to key and value cache
- Kneser-Ney smoothing also Kneser-Ney, interpolated Kneser-Ney
-
Kneser-Ney smoothing combines interpolation of orders with continuation counts.
P(w | h) = max(c(h w) − d, 0) / c(h) + d · N(h) / c(h) · P(w | shorter h)- A kind of: smoothing
Defined in Smooth with Kneser-Ney, chapter 3. In the book since Edition 2026.11.16. Permanent link to Kneser-Ney smoothing
L
- language model
-
A language model is a program that assigns a probability to every word that could come next, given the words so far.
Defined in Predicting the next word, chapter 1. In the book since Edition 2026.11.16. Permanent link to language model
- layer normalization
-
layer normalization, which subtracts each vector's mean and divides by its standard deviation
Defined in Normalize before each part, chapter 7. In the book since Edition 2026.11.16. Permanent link to layer normalization
- logistic function also sigmoid
-
the logistic function, σ(s) = 1 / (1 + e⁻ˢ), which turns a score into a probability
σ(s) = 1 / (1 + e⁻ˢ)Defined in Learn vectors by predicting neighbours, chapter 4. In the book since Edition 2026.11.16. Permanent link to logistic function
- logits also logit
-
one score per token in the vocabulary, called the logits
- Comes before: softmax
Defined in Predict from a window of embeddings, chapter 5. In the book since Edition 2026.11.16. Permanent link to logits
- LoRA also low-rank adaptation
-
LoRA, low-rank adaptation, freezes the model's weights and trains a small correction beside some of them.
W + BA- A kind of: fine-tuning
- Contrasts with: full fine-tuning
Defined in Adapt a model to your text, chapter 11. In the book since Edition 2026.11.16. Permanent link to LoRA
- loss
-
To measure how badly a model predicts, use the loss: L = −log P(next token), the negative log probability the model gave to the token that actually came next, the same surprise chapter 1 averaged into perplexity.
L = −log P(next token)- Comes before: gradient
Defined in Compute gradients by backpropagation, chapter 5. In the book since Edition 2026.11.16. Permanent link to loss
M
- made-up answer
-
An answer is made up if it names a material that the product's name and description don't contain.
- A kind of: hallucination
Defined in Measure made-up product details, chapter 11. In the book since Edition 2026.11.16. Permanent link to made-up answer
- maximum likelihood estimation also maximum likelihood, MLE
-
Estimating probabilities from the fraction of times something happened is called maximum likelihood estimation
P(v | w) = count(w v) / count(w)- Computed from: bigram
Defined in Turn counts into probabilities, chapter 1. In the book since Edition 2026.11.16. Permanent link to maximum likelihood estimation
- minimum length
-
a minimum length: forbid the end marker, by setting its logit to minus infinity, until the description has some number of tokens
Defined in Forbid an empty description, chapter 9. In the book since Edition 2026.11.16. Permanent link to minimum length
- modified Kneser-Ney
-
A variant, modified Kneser-Ney, uses three discounts instead of one
- A kind of: Kneser-Ney smoothing
Defined in Smooth with Kneser-Ney, chapter 3. In the book since Edition 2026.11.16. Permanent link to modified Kneser-Ney
- multi-head attention
-
Multi-head attention runs several attention computations, called heads, side by side, each with its own query, key and value matrices, and each in a share of the vector's numbers
- A kind of: attention
Defined in Several heads look for different things, chapter 6. In the book since Edition 2026.11.16. Permanent link to multi-head attention
N
- n-gram model also n-gram, n-gram language model
-
An n-gram model predicts each token from the n − 1 tokens before it
- A kind of: language model
Defined in Longer contexts fit the training text better, chapter 3. In the book since Edition 2026.11.16. Permanent link to n-gram model
- neural language model
-
A neural language model reads the previous tokens as vectors, combines them with learned arithmetic, and produces a probability for every token in the vocabulary.
- A kind of: language model
Defined in A neural language model, chapter 5. In the book since Edition 2026.11.16. Permanent link to neural language model
- noise token
-
For a few random noise tokens drawn from the vocabulary, the model nudges the probability down.
- Part of: skip-gram with negative sampling
Defined in Learn vectors by predicting neighbours, chapter 4. In the book since Edition 2026.11.16. Permanent link to noise token
- normalization
-
Normalization rescales a vector to a standard size before a part reads it, so the part's weights work the same whether the stream is short or long.
- Kinds: layer normalization, RMSNorm
- Part of: transformer block
Defined in Normalize before each part, chapter 7. In the book since Edition 2026.11.16. Permanent link to normalization
O
- one-hot encoding also one-hot, one-hot vector
-
a one-hot encoding, in which each token gets a vector as long as the vocabulary, with a 1 in its own position and 0 everywhere else
- Contrasts with: embedding
Defined in One-hot vectors know nothing, chapter 4. In the book since Edition 2026.11.16. Permanent link to one-hot encoding
- out-of-vocabulary also OOV
-
any word that first appears after training is out of reach: an out-of-vocabulary word
Defined in Words leave gaps, chapter 2. In the book since Edition 2026.11.16. Permanent link to out-of-vocabulary
- output layer
-
An output layer then computes z = hU + c from the hidden layer's output h
z = hU + c- Part of: neural language model
Defined in Predict from a window of embeddings, chapter 5. In the book since Edition 2026.11.16. Permanent link to output layer
- output projection
-
The heads' outputs are joined side by side and mixed back together by one more learned matrix, the output projection
- Part of: multi-head attention
Defined in Several heads look for different things, chapter 6. In the book since Edition 2026.11.16. Permanent link to output projection
- overfitting
-
A validation perplexity that turns up while training perplexity keeps falling is overfitting
Defined in Stop when validation loss turns up, chapter 5. In the book since Edition 2026.11.16. Permanent link to overfitting
P
- packing
-
packing, several texts end to end in each row, with the attention kept from crossing between them
- Contrasts with: padding
Defined in Turn products into batches, chapter 8. In the book since Edition 2026.11.16. Permanent link to packing
- padding also padding token
-
padding, which fills out short texts in a training batch
Defined in Count where the parameters go, chapter 7. In the book since Edition 2026.11.16. Permanent link to padding
- parameter also parameters
-
The window model has 1,394,732 parameters, the numbers it learns.
Defined in Predict from a window of embeddings, chapter 5. In the book since Edition 2026.11.16. Permanent link to parameter
- perplexity
-
Perplexity is the cross-entropy turned back into a count
perplexity = exp(cross-entropy)- Computed from: cross-entropy
Defined in Score a model with perplexity, chapter 1. In the book since Edition 2026.11.16. Permanent link to perplexity
- perplexity per word also per-word perplexity
-
A perplexity per word divides a model's total log-probability of a text by the number of words in it, instead of by its number of tokens.
perplexity per word = exp(−(log p₁ + … + log pₙ) / W)- A kind of: perplexity
- Computed from: total log-probability
Defined in Score the store's text with a general model, chapter 10. In the book since Edition 2026.11.16. Permanent link to perplexity per word
- pinned revision also revision
-
a pinned revision, a fixed version of its repository
Defined in Inside an open model, chapter 10. In the book since Edition 2026.11.16. Permanent link to pinned revision
- pointwise mutual information also PMI
-
Pointwise mutual information, or PMI, measures how far the real count departs from chance: the log of the ratio between how often two tokens co-occur and how often they would by chance
PMI(a, b) = log [ P(a, b) / (P(a) · P(b)) ]Defined in Weigh co-occurrence by surprise, chapter 4. In the book since Edition 2026.11.16. Permanent link to pointwise mutual information
- positive PMI also PPMI, positive pointwise mutual information
-
positive PMI, or PPMI, replaces every negative value with zero.
- Computed from: pointwise mutual information
Defined in Weigh co-occurrence by surprise, chapter 4. In the book since Edition 2026.11.16. Permanent link to positive PMI
- pre-tokenizer
-
A pre-tokenizer splits text into pieces before any merging
- Part of: tokenizer
- Implemented by: tokenizers 0.23.3
Defined in Apply SmolLM2's merges by hand, chapter 2. In the book since Edition 2026.11.16. Permanent link to pre-tokenizer
- preference tuning
-
Preference tuning then trains a model to prefer the better of two responses to the same request.
Defined in From predicting to following instructions, chapter 11. In the book since Edition 2026.11.16. Permanent link to preference tuning
Q
- query
-
a query, a vector for what this position is looking for
- Part of: attention
Defined in Look back with a weighted average, chapter 6. In the book since Edition 2026.11.16. Permanent link to query
R
- reference model
-
DPO compares how much more likely the model being tuned makes each response than the reference model it started from, the model after instruction tuning, and rewards it for raising the preferred one more.
Defined in Build the two training losses by hand, chapter 11. In the book since Edition 2026.11.16. Permanent link to reference model
- regularization
-
regularization, which is anything in training that discourages memorizing
Defined in Stop when validation loss turns up, chapter 5. In the book since Edition 2026.11.16. Permanent link to regularization
- repetition
-
repetition: the share of four-token phrases that repeat an earlier one in the same description
Defined in Measure what each setting writes, chapter 9. In the book since Edition 2026.11.16. Permanent link to repetition
- repetition penalty
-
Keskar and colleagues proposed a repetition penalty: divide the logit of every token already generated by a factor θ above 1, so it becomes less likely
Defined in Penalize repetition, chapter 9. In the book since Edition 2026.11.16. Permanent link to repetition penalty
- residual connection
-
Adding a part's output to its input, instead of replacing the input, is a residual connection.
x ← x + attention(x), then x ← x + feed-forward(x)- Part of: transformer block
- Comes before: residual stream
Defined in Add each part's output to a residual stream, chapter 7. In the book since Edition 2026.11.16. Permanent link to residual connection
- residual stream
-
Elhage and colleagues call this running sum, each position's vector with every part's output added to it, the residual stream
Defined in Add each part's output to a residual stream, chapter 7. In the book since Edition 2026.11.16. Permanent link to residual stream
- RMSNorm
-
RMSNorm drops the mean: it divides each vector by its root mean square, the square root of the average of its squared numbers, and multiplies each dimension by a learned gain
RMSNorm(x) = x / √(mean(x²) + ε) · g- Computed from: root mean square
- Contrasts with: layer normalization
Defined in Normalize before each part, chapter 7. In the book since Edition 2026.11.16. Permanent link to RMSNorm
- root mean square
-
root mean square, the square root of the average of its squared numbers
Defined in Normalize before each part, chapter 7. In the book since Edition 2026.11.16. Permanent link to root mean square
- rotary position embeddings also RoPE
-
Before scoring, RoPE rotates each query and key by angles that grow with its position.
(a, b) → (a cos pθᵢ − b sin pθᵢ, a sin pθᵢ + b cos pθᵢ)- Part of: transformer block
Defined in Give attention positions with rotations, chapter 7. In the book since Edition 2026.11.16. Permanent link to rotary position embeddings
S
- sampling
-
Sampling, which chapter 1 used to generate names, uses all of the distribution: draw the next token at random, in proportion to its probability, so a token with probability 0.2 is chosen one time in five.
Defined in Sample, with a temperature, chapter 9. In the book since Edition 2026.11.16. Permanent link to sampling
- schedule also learning-rate schedule
-
changes the step size after every batch, following a schedule, which sets the step size for each step of the run
Defined in Warm up, then decay the step size, chapter 8. In the book since Edition 2026.11.16. Permanent link to schedule
- SiLU
-
SiLU passes large positive numbers almost unchanged and turns large negative ones to nearly zero, smoothly.
SiLU(z) = z · σ(z)- Comes before: gated feed-forward layer
Defined in Transform each position with a gated feed-forward layer, chapter 7. In the book since Edition 2026.11.16. Permanent link to SiLU
- singular value decomposition also SVD
-
the singular value decomposition, SVD, which finds the 64 directions that together reconstruct the PPMI rows best, and weights each by how much of the rows it explains
- Computed from: positive PMI
Defined in Compress counts into dense vectors, chapter 4. In the book since Edition 2026.11.16. Permanent link to singular value decomposition
- skip-gram loss
-
The skip-gram loss for one pair is the surprise at the real neighbour plus the surprise at each noise token's not being one
L = −log σ(v·u) − Σₙ log σ(−v·uₙ)- Part of: skip-gram with negative sampling
- Computed from: logistic function, noise token
Defined in Learn vectors by predicting neighbours, chapter 4. In the book since Edition 2026.11.16. Permanent link to skip-gram loss
- skip-gram with negative sampling also skip-gram, SGNS
-
Skip-gram with negative sampling, the method behind word2vec, gives each token two vectors, one for when it's the centre token and one for when it's a neighbour, and trains them on one task: given a token, tell its real neighbours apart from random tokens
- Contrasts with: singular value decomposition
Defined in Learn vectors by predicting neighbours, chapter 4. In the book since Edition 2026.11.16. Permanent link to skip-gram with negative sampling
- SmolLM2-360M
-
SmolLM2-360M, an open model released by Hugging Face under the Apache 2.0 licence
Defined in Inside an open model, chapter 10. In the book since Edition 2026.11.16. Permanent link to SmolLM2-360M
- SmolLM2-360M-Instruct
-
SmolLM2-360M-Instruct is the base model of chapter 10 after both steps: supervised fine-tuning on public and curated datasets, then DPO on UltraFeedback
Defined in From predicting to following instructions, chapter 11. In the book since Edition 2026.11.16. Permanent link to SmolLM2-360M-Instruct
- smoothing
-
smoothing, which gives unseen pairs some probability by taking a little away from the pairs the model has seen
Defined in Unseen word pairs break the model, chapter 1. In the book since Edition 2026.11.16. Permanent link to smoothing
- softmax
-
Softmax exponentiates every logit and divides by the sum, so the results are positive and sum to one, and a higher score means a higher probability
P(token i) = exp(zᵢ) / Σⱼ exp(zⱼ)- Comes before: attention
Defined in Predict from a window of embeddings, chapter 5. In the book since Edition 2026.11.16. Permanent link to softmax
- special token
-
Open models mark the start or end of a text, or a turn in a conversation, with special tokens written as strings such as <|endoftext|>.
Defined in What goes wrong in production, chapter 2. In the book since Edition 2026.11.16. Permanent link to special token
- step size also learning rate
-
η, the step size, also called the learning rate, sets how far each step goes
- Part of: gradient descent
Defined in Take gradient descent steps, chapter 5. In the book since Edition 2026.11.16. Permanent link to step size
- supported answer
-
An answer is supported if every material it names is in the product's name or description
- Contrasts with: made-up answer
Defined in Measure made-up product details, chapter 11. In the book since Edition 2026.11.16. Permanent link to supported answer
- surprise
-
If the model gave the actual next word probability p, its surprise at that word is −log p: zero when it was certain (p = 1) and larger the less it expected the word.
- Comes before: cross-entropy
Defined in Score a model with perplexity, chapter 1. In the book since Edition 2026.11.16. Permanent link to surprise
- system message
-
a system message, instructions placed before the user's request
- Part of: chat template
Defined in Format the request as the model was tuned, chapter 11. In the book since Edition 2026.11.16. Permanent link to system message
T
- tanh
-
tanh, a function that squashes any number into the range from −1 to 1
- Part of: hidden layer
Defined in Predict from a window of embeddings, chapter 5. In the book since Edition 2026.11.16. Permanent link to tanh
- targets
-
the targets are the text's tokens without the first, so position i is trained to predict token i + 1
Defined in Turn products into batches, chapter 8. In the book since Edition 2026.11.16. Permanent link to targets
- temperature
-
Before sampling, the distribution can be made sharper or flatter with a temperature, T: divide every logit by T before the softmax.
P(token i) = exp(zᵢ / T) / Σⱼ exp(zⱼ / T)Defined in Sample, with a temperature, chapter 9. In the book since Edition 2026.11.16. Permanent link to temperature
- test split
-
a test split kept for final reports
Defined in Load the store's text, chapter 1. In the book since Edition 2026.11.16. Permanent link to test split
- token
-
A token is the unit a language model reads and predicts: whatever the text is cut into before the model sees it.
- Comes before: vocabulary
Defined in Tokens, chapter 2. In the book since Edition 2026.11.16. Permanent link to token
- token type
-
The different tokens a text uses are its token types.
Defined in Choose how many merges, chapter 2. In the book since Edition 2026.11.16. Permanent link to token type
- tokenizer
-
The list of merges, in the order they were learned, is the tokenizer.
- Implemented by: tokenizers 0.23.3
Defined in Merge the most frequent pair, again and again, chapter 2. In the book since Edition 2026.11.16. Permanent link to tokenizer
- top-k also top-k sampling
-
Top-k keeps only the k likeliest tokens and renormalizes their probabilities to sum to one
Defined in Sample, and cut the tail, chapter 9. In the book since Edition 2026.11.16. Permanent link to top-k
- top-p also nucleus sampling, top-p sampling
-
Top-p, also called nucleus sampling, keeps the smallest set of likeliest tokens whose probabilities reach at least p
- Contrasts with: top-k
Defined in Sample, and cut the tail, chapter 9. In the book since Edition 2026.11.16. Permanent link to top-p
- total log-probability
-
total log-probability, the sum of the logarithms of a text's token probabilities, where higher means likelier
- Comes before: beam search
Defined in Search for the likeliest text, chapter 9. In the book since Edition 2026.11.16. Permanent link to total log-probability
- training split
-
a training split that models learn from
Defined in Load the store's text, chapter 1. In the book since Edition 2026.11.16. Permanent link to training split
- transformer block
-
The transformer block adds parts around attention that transform each position, give it word order and keep a deep stack from degrading
Defined in The transformer block, chapter 7. In the book since Edition 2026.11.16. Permanent link to transformer block
V
- validation split
-
a validation split for comparing models while you build them
Defined in Load the store's text, chapter 1. In the book since Edition 2026.11.16. Permanent link to validation split
- value
-
a value, a vector for what this position passes on when it's looked at
- Part of: attention
Defined in Look back with a weighted average, chapter 6. In the book since Edition 2026.11.16. Permanent link to value
- vector
-
a vector, a list of numbers
Defined in Words as vectors, chapter 4. In the book since Edition 2026.11.16. Permanent link to vector
- vector length
-
A vector's length here is the square root of the sum of its squared numbers
Defined in Add each part's output to a residual stream, chapter 7. In the book since Edition 2026.11.16. Permanent link to vector length
- vocabulary
-
A model can only predict tokens in its vocabulary, the set of tokens it learned from training text
- Comes before: out-of-vocabulary
Defined in Words leave gaps, chapter 2. In the book since Edition 2026.11.16. Permanent link to vocabulary
W
- warm-up
-
For the first 200 steps, the step size rises in a straight line from nearly zero to its peak, 0.003: a warm-up
- Part of: schedule
Defined in Warm up, then decay the step size, chapter 8. In the book since Edition 2026.11.16. Permanent link to warm-up
- weight decay
-
Weight decay reduces each weight at every step by its value times the step size times a decay factor
- A kind of: regularization
- Comes before: AdamW
Defined in Update with AdamW and clipped gradients, chapter 8. In the book since Edition 2026.11.16. Permanent link to weight decay
- weight tying also tied embeddings
-
weight tying: the output layer reuses the embedding table, turned on its side, instead of learning a matrix of its own.
- Computed from: embedding table
Defined in Tie the output layer to the embeddings, chapter 7. In the book since Edition 2026.11.16. Permanent link to weight tying