Glossary

The 134 terms Language Models from the Ground Up defines, each in the book's own words, with the section that defines it.

From Edition 2026.11.16, the book's current edition. Where a definition depends on a package, it says the version this edition pins and runs its examples on.

#

<unk> also unknown token

A word the training text doesn't contain at all is counted as one more vocabulary entry, <unk>, for unknown, so the probabilities still sum to one.

Defined in Give unseen pairs a little probability, chapter 1. In the book since Edition 2026.11.16. Permanent link to <unk>

A

ablation

Leaving a part out allows a simple experiment, called an ablation: remove one part, and measure what the model loses.

Defined in Leave out a block to see what it does, chapter 7. In the book since Edition 2026.11.16. Permanent link to ablation

Adam

The training script's optimizer is Adam, a refinement of gradient descent (Kingma and Ba, 2015).

Defined in Train with PyTorch, chapter 5. In the book since Edition 2026.11.16. Permanent link to Adam

AdamW

AdamW, the Adam of chapter 5 with weight decay applied apart from the gradient

Defined in Update with AdamW and clipped gradients, chapter 8. In the book since Edition 2026.11.16. Permanent link to AdamW

add-one smoothing

The simplest smoothing method, add-one smoothing, pretends that every possible pair occurred once more than it did.

P(v | w) = (count(w v) + 1) / (count(w) + V)

Defined in Give unseen pairs a little probability, chapter 1. In the book since Edition 2026.11.16. Permanent link to add-one smoothing

analogy

In vectors, the answer to "a is to b as c is to what?" is the token nearest to b − a + c, leaving out the three tokens in the question

Defined in Analogies, and where they fail, chapter 4. In the book since Edition 2026.11.16. Permanent link to analogy

attention

Attention lets every position look at every earlier one, and decide from their contents how much each matters.

Defined in Attention, chapter 6. In the book since Edition 2026.11.16. Permanent link to attention

attention score

The score of position j for position i is the dot product of i's query with j's key, divided by the square root of d, the size of each query and key, meaning how many numbers it holds.

Defined in Look back with a weighted average, chapter 6. In the book since Edition 2026.11.16. Permanent link to attention score

attention sink also attention sinks

Xiao and colleagues found that language models put large weights on the first tokens of a text whatever they are, calling them attention sinks (Xiao et al., 2024).

Defined in Reach past the window, chapter 6. In the book since Edition 2026.11.16. Permanent link to attention sink

B

backpropagation

Backpropagation computes all the gradients at once, by applying the chain rule of calculus backwards through the forward pass

Defined in Compute gradients by backpropagation, chapter 5. In the book since Edition 2026.11.16. Permanent link to backpropagation

base model

the base model, the one trained only to predict the next token

Defined in Inside an open model, chapter 10. In the book since Edition 2026.11.16. Permanent link to base model

baseline

a baseline, a fixed yardstick: one model, one text, one tokenizer and one measure that every later model is compared against

Defined in The baseline every later model is measured against, chapter 3. In the book since Edition 2026.11.16. Permanent link to baseline

batch

Each step of real training is computed on a batch of examples, a few hundred at a time, instead of all of them, so each step is cheap and the steps vary a little.

Defined in Take gradient descent steps, chapter 5. In the book since Edition 2026.11.16. Permanent link to batch

beam search

Beam search looks further, keeping the k likeliest partial texts, the beam, scored by their total log-probability

Defined in Search for the likeliest text, chapter 9. In the book since Edition 2026.11.16. Permanent link to beam search

bigram

A bigram is a pair of adjacent words, such as coffee table.

Defined in Count which word follows which, chapter 1. In the book since Edition 2026.11.16. Permanent link to bigram

byte

in UTF-8, the usual text encoding, every character in every language is stored as one to four bytes, and there are only 256 possible byte values, so a vocabulary that starts from all 256 can represent any text at all

Defined in Characters cover everything, chapter 2. In the book since Edition 2026.11.16. Permanent link to byte

byte-pair encoding also BPE

Byte-pair encoding, or BPE, starts from characters and builds bigger tokens out of the pairs that occur most often.

  • Implemented by: tokenizers 0.23.3

Defined in Merge the most frequent pair, again and again, chapter 2. In the book since Edition 2026.11.16. Permanent link to byte-pair encoding

C

calibrated also calibration

A model whose confidence matches how often it's right is calibrated

Defined in Check whether confidence warns you, chapter 11. In the book since Edition 2026.11.16. Permanent link to calibrated

causal mask

the causal mask: before softmax, set every score for a later position to minus infinity.

Defined in Hide the future with a causal mask, chapter 6. In the book since Edition 2026.11.16. Permanent link to causal mask

chat template

A tuned model's chat template is the layout it learned: special tokens that mark where a system message, instructions placed before the user's request, the request itself and the model's reply begin and end.

Defined in Format the request as the model was tuned, chapter 11. In the book since Edition 2026.11.16. Permanent link to chat template

co-occurrence window

count, for every pair of tokens, how often one appears within a small window of the other, here two tokens either side, and never across the gap between two products' texts

Defined in Describe a token by the tokens around it, chapter 4. In the book since Edition 2026.11.16. Permanent link to co-occurrence window

confidence per token

a confidence per token can be computed for each answer, as a geometric mean

confidence per token = exp((log p₁ + … + log pₙ) / n)

Defined in Check whether confidence warns you, chapter 11. In the book since Edition 2026.11.16. Permanent link to confidence per token

context

The tokens before the one being predicted are its context.

Defined in Longer contexts fit the training text better, chapter 3. In the book since Edition 2026.11.16. Permanent link to context

continuation count

a token's continuation count, the number of different tokens it follows

Defined in Count the contexts a token follows, chapter 3. In the book since Edition 2026.11.16. Permanent link to continuation count

cosine decay

After the warm-up, the step size falls along half a cosine curve, to a tenth of the peak by the last step

rate = peak × (0.1 + 0.9 × ½ (1 + cos(π × progress)))

Defined in Warm up, then decay the step size, chapter 8. In the book since Edition 2026.11.16. Permanent link to cosine decay

cosine similarity

To measure how alike two vectors are, this book uses cosine similarity: the cosine of the angle between them, 1 when they point the same way, 0 when they're at right angles.

Defined in One-hot vectors know nothing, chapter 4. In the book since Edition 2026.11.16. Permanent link to cosine similarity

cross-entropy

The cross-entropy of a text is the average surprise over every word in it

cross-entropy = −(1/N) Σ log P(next word | previous word)

Defined in Score a model with perplexity, chapter 1. In the book since Edition 2026.11.16. Permanent link to cross-entropy

D

declined answer

An answer is declined if it says the product's data doesn't say

Defined in Measure made-up product details, chapter 11. In the book since Edition 2026.11.16. Permanent link to declined answer

decoding

Turning a model's probabilities into text, one chosen token at a time, is called decoding

Defined in Choosing the next token, chapter 9. In the book since Edition 2026.11.16. Permanent link to decoding

Direct Preference Optimization also DPO

Direct Preference Optimization, DPO, does preference tuning without a reward model, with a simple loss on the preference pairs themselves

Defined in From predicting to following instructions, chapter 11. In the book since Edition 2026.11.16. Permanent link to Direct Preference Optimization

discount

At each order, Kneser-Ney subtracts a fixed discount d from every count it has seen, and gives the probability it took to the next order down

D = n₁ / (n₁ + 2 n₂)

Defined in Smooth with Kneser-Ney, chapter 3. In the book since Edition 2026.11.16. Permanent link to discount

DPO loss also DPO's loss

DPO's loss works on a pair of responses to one request, one preferred and one rejected.

loss = −log σ(margin)

Defined in Build the two training losses by hand, chapter 11. In the book since Edition 2026.11.16. Permanent link to DPO loss

E

early stopping

the simplest defence is early stopping: measure validation loss after every epoch, one full pass over the training text, and keep the weights from the best epoch.

Defined in Stop when validation loss turns up, chapter 5. In the book since Edition 2026.11.16. Permanent link to early stopping

embedding also token embedding, word embedding

an embedding, a dense vector that stands for a token

Defined in Compress counts into dense vectors, chapter 4. In the book since Edition 2026.11.16. Permanent link to embedding

embedding table

Each token is looked up in an embedding table, a matrix with one row of numbers per token in the vocabulary, as in chapter 4, except that this table is learned for predicting the next token rather than computed from counts.

Defined in Predict from a window of embeddings, chapter 5. In the book since Edition 2026.11.16. Permanent link to embedding table

epoch

epoch, one full pass over the training text

Defined in Stop when validation loss turns up, chapter 5. In the book since Edition 2026.11.16. Permanent link to epoch

even spread

an even spread, which puts 1 / (i + 1) on each of the i + 1 tokens position i can see

1 / (i + 1)

Defined in Some heads attend to the previous token, chapter 6. In the book since Edition 2026.11.16. Permanent link to even spread

F

feed-forward layer

Attention moves information between positions; the feed-forward layer transforms it, at each position separately.

Defined in Transform each position with a gated feed-forward layer, chapter 7. In the book since Edition 2026.11.16. Permanent link to feed-forward layer

fine-tuning

The way to teach a general model the store's text is fine-tuning: continue training it on that text, the same training loop as chapter 8, starting from the general model's weights instead of random ones.

Defined in Adapt a model to your text, chapter 11. In the book since Edition 2026.11.16. Permanent link to fine-tuning

full fine-tuning

Full fine-tuning updates every weight, which for a model of hundreds of millions of parameters takes time and memory.

Defined in Adapt a model to your text, chapter 11. In the book since Edition 2026.11.16. Permanent link to full fine-tuning

G

gated feed-forward layer also SwiGLU

A gated feed-forward layer computes two widened vectors from the input, passes one through a function called SiLU, and multiplies the two number by number, so the first decides how much of the second gets through

FFN(x) = (SiLU(x W_gate) ⊙ (x W_up)) W_down

Defined in Transform each position with a gated feed-forward layer, chapter 7. In the book since Edition 2026.11.16. Permanent link to gated feed-forward layer

generation also generate

To generate, start at <s>, draw the next word at random in proportion to its probability, append it, and repeat from the word you drew, until the model draws </s>.

Defined in Generate names one word at a time, chapter 1. In the book since Edition 2026.11.16. Permanent link to generation

gradient

To lower the loss, each weight needs a gradient: how much the loss would change if that weight changed a little, written ∂L/∂w.

Defined in Compute gradients by backpropagation, chapter 5. In the book since Edition 2026.11.16. Permanent link to gradient

gradient clipping also clipping

The training script also clips the gradients: when their combined length, measured as if every gradient were one long vector, exceeds 1.0, it scales them all down to that length

Defined in Update with AdamW and clipped gradients, chapter 8. In the book since Edition 2026.11.16. Permanent link to gradient clipping

gradient descent

Gradient descent moves every weight a small step against its gradient

w ← w − η · ∂L/∂w

Defined in Take gradient descent steps, chapter 5. In the book since Edition 2026.11.16. Permanent link to gradient descent

greedy decoding also greedy

The simplest decoder is greedy: at every step, take the single likeliest token.

Defined in Choose the likeliest token, chapter 9. In the book since Edition 2026.11.16. Permanent link to greedy decoding

grouped-query attention also GQA

grouped-query attention, which shares each key and value head among several query heads

Defined in Count the parameters at scale, chapter 10. In the book since Edition 2026.11.16. Permanent link to grouped-query attention

H

hallucination

Making up a detail the input doesn't support is one kind of what's called hallucination

Defined in Measure made-up product details, chapter 11. In the book since Edition 2026.11.16. Permanent link to hallucination

head also heads, attention head

several attention computations, called heads, side by side, each with its own query, key and value matrices

Defined in Several heads look for different things, chapter 6. In the book since Edition 2026.11.16. Permanent link to head

hidden layer

A hidden layer computes h = tanh(xW + b) from its input x: it multiplies x by a matrix of weights W, adds a vector of biases b, and passes each result through tanh

h = tanh(xW + b)

Defined in Predict from a window of embeddings, chapter 5. In the book since Edition 2026.11.16. Permanent link to hidden layer

I

induction head also induction heads

an induction head: a head in an earlier layer copies each token's predecessor into its position, and the induction head uses that to find earlier positions that followed the current token, and attends to them

Defined in Read what trained heads attend to, chapter 6. In the book since Edition 2026.11.16. Permanent link to induction head

input window

This chapter's model looks at a fixed window of the previous tokens, four here, the same context a 5-gram counts.

Defined in Predict from a window of embeddings, chapter 5. In the book since Edition 2026.11.16. Permanent link to input window

inputs

The inputs are a text's tokens without the last

Defined in Turn products into batches, chapter 8. In the book since Edition 2026.11.16. Permanent link to inputs

instruction tuning also supervised fine-tuning, instruction-tuned

Instruction tuning, or supervised fine-tuning, continues training a model on examples of requests paired with good responses, so that predicting the next token of such a pair means answering the request.

Defined in From predicting to following instructions, chapter 11. In the book since Edition 2026.11.16. Permanent link to instruction tuning

interpolation

Interpolation falls back smoothly: the probability of a token is a weighted mix of what the trigram, bigram and unigram counts say

P(token | a b) = λ₃ · P₃(token | a b) + λ₂ · P₂(token | b) + λ₁ · P₁(token)

Defined in Mix short and long contexts, chapter 3. In the book since Edition 2026.11.16. Permanent link to interpolation

K

key

a key, a vector for what this position offers to the positions that look at it

Defined in Look back with a weighted average, chapter 6. In the book since Edition 2026.11.16. Permanent link to key

key and value cache also KV cache

The key and value cache, every layer's keys and values for the earlier tokens, grows with the length of the text

cache per token, in bytes = layers × 2 × key and value heads × head size × 2

Defined in Count what it costs to run, chapter 10. In the book since Edition 2026.11.16. Permanent link to key and value cache

Kneser-Ney smoothing also Kneser-Ney, interpolated Kneser-Ney

Kneser-Ney smoothing combines interpolation of orders with continuation counts.

P(w | h) = max(c(h w) − d, 0) / c(h) + d · N(h) / c(h) · P(w | shorter h)

Defined in Smooth with Kneser-Ney, chapter 3. In the book since Edition 2026.11.16. Permanent link to Kneser-Ney smoothing

L

language model

A language model is a program that assigns a probability to every word that could come next, given the words so far.

Defined in Predicting the next word, chapter 1. In the book since Edition 2026.11.16. Permanent link to language model

layer normalization

layer normalization, which subtracts each vector's mean and divides by its standard deviation

Defined in Normalize before each part, chapter 7. In the book since Edition 2026.11.16. Permanent link to layer normalization

logistic function also sigmoid

the logistic function, σ(s) = 1 / (1 + e⁻ˢ), which turns a score into a probability

σ(s) = 1 / (1 + e⁻ˢ)

Defined in Learn vectors by predicting neighbours, chapter 4. In the book since Edition 2026.11.16. Permanent link to logistic function

logits also logit

one score per token in the vocabulary, called the logits

Defined in Predict from a window of embeddings, chapter 5. In the book since Edition 2026.11.16. Permanent link to logits

LoRA also low-rank adaptation

LoRA, low-rank adaptation, freezes the model's weights and trains a small correction beside some of them.

W + BA

Defined in Adapt a model to your text, chapter 11. In the book since Edition 2026.11.16. Permanent link to LoRA

loss

To measure how badly a model predicts, use the loss: L = −log P(next token), the negative log probability the model gave to the token that actually came next, the same surprise chapter 1 averaged into perplexity.

L = −log P(next token)

Defined in Compute gradients by backpropagation, chapter 5. In the book since Edition 2026.11.16. Permanent link to loss

M

made-up answer

An answer is made up if it names a material that the product's name and description don't contain.

Defined in Measure made-up product details, chapter 11. In the book since Edition 2026.11.16. Permanent link to made-up answer

maximum likelihood estimation also maximum likelihood, MLE

Estimating probabilities from the fraction of times something happened is called maximum likelihood estimation

P(v | w) = count(w v) / count(w)

Defined in Turn counts into probabilities, chapter 1. In the book since Edition 2026.11.16. Permanent link to maximum likelihood estimation

minimum length

a minimum length: forbid the end marker, by setting its logit to minus infinity, until the description has some number of tokens

Defined in Forbid an empty description, chapter 9. In the book since Edition 2026.11.16. Permanent link to minimum length

modified Kneser-Ney

A variant, modified Kneser-Ney, uses three discounts instead of one

Defined in Smooth with Kneser-Ney, chapter 3. In the book since Edition 2026.11.16. Permanent link to modified Kneser-Ney

multi-head attention

Multi-head attention runs several attention computations, called heads, side by side, each with its own query, key and value matrices, and each in a share of the vector's numbers

Defined in Several heads look for different things, chapter 6. In the book since Edition 2026.11.16. Permanent link to multi-head attention

N

n-gram model also n-gram, n-gram language model

An n-gram model predicts each token from the n − 1 tokens before it

Defined in Longer contexts fit the training text better, chapter 3. In the book since Edition 2026.11.16. Permanent link to n-gram model

neural language model

A neural language model reads the previous tokens as vectors, combines them with learned arithmetic, and produces a probability for every token in the vocabulary.

Defined in A neural language model, chapter 5. In the book since Edition 2026.11.16. Permanent link to neural language model

noise token

For a few random noise tokens drawn from the vocabulary, the model nudges the probability down.

Defined in Learn vectors by predicting neighbours, chapter 4. In the book since Edition 2026.11.16. Permanent link to noise token

normalization

Normalization rescales a vector to a standard size before a part reads it, so the part's weights work the same whether the stream is short or long.

Defined in Normalize before each part, chapter 7. In the book since Edition 2026.11.16. Permanent link to normalization

O

one-hot encoding also one-hot, one-hot vector

a one-hot encoding, in which each token gets a vector as long as the vocabulary, with a 1 in its own position and 0 everywhere else

Defined in One-hot vectors know nothing, chapter 4. In the book since Edition 2026.11.16. Permanent link to one-hot encoding

out-of-vocabulary also OOV

any word that first appears after training is out of reach: an out-of-vocabulary word

Defined in Words leave gaps, chapter 2. In the book since Edition 2026.11.16. Permanent link to out-of-vocabulary

output layer

An output layer then computes z = hU + c from the hidden layer's output h

z = hU + c

Defined in Predict from a window of embeddings, chapter 5. In the book since Edition 2026.11.16. Permanent link to output layer

output projection

The heads' outputs are joined side by side and mixed back together by one more learned matrix, the output projection

Defined in Several heads look for different things, chapter 6. In the book since Edition 2026.11.16. Permanent link to output projection

overfitting

A validation perplexity that turns up while training perplexity keeps falling is overfitting

Defined in Stop when validation loss turns up, chapter 5. In the book since Edition 2026.11.16. Permanent link to overfitting

P

packing

packing, several texts end to end in each row, with the attention kept from crossing between them

Defined in Turn products into batches, chapter 8. In the book since Edition 2026.11.16. Permanent link to packing

padding also padding token

padding, which fills out short texts in a training batch

Defined in Count where the parameters go, chapter 7. In the book since Edition 2026.11.16. Permanent link to padding

parameter also parameters

The window model has 1,394,732 parameters, the numbers it learns.

Defined in Predict from a window of embeddings, chapter 5. In the book since Edition 2026.11.16. Permanent link to parameter

perplexity

Perplexity is the cross-entropy turned back into a count

perplexity = exp(cross-entropy)

Defined in Score a model with perplexity, chapter 1. In the book since Edition 2026.11.16. Permanent link to perplexity

perplexity per word also per-word perplexity

A perplexity per word divides a model's total log-probability of a text by the number of words in it, instead of by its number of tokens.

perplexity per word = exp(−(log p₁ + … + log pₙ) / W)

Defined in Score the store's text with a general model, chapter 10. In the book since Edition 2026.11.16. Permanent link to perplexity per word

pinned revision also revision

a pinned revision, a fixed version of its repository

Defined in Inside an open model, chapter 10. In the book since Edition 2026.11.16. Permanent link to pinned revision

pointwise mutual information also PMI

Pointwise mutual information, or PMI, measures how far the real count departs from chance: the log of the ratio between how often two tokens co-occur and how often they would by chance

PMI(a, b) = log [ P(a, b) / (P(a) · P(b)) ]

Defined in Weigh co-occurrence by surprise, chapter 4. In the book since Edition 2026.11.16. Permanent link to pointwise mutual information

positive PMI also PPMI, positive pointwise mutual information

positive PMI, or PPMI, replaces every negative value with zero.

Defined in Weigh co-occurrence by surprise, chapter 4. In the book since Edition 2026.11.16. Permanent link to positive PMI

pre-tokenizer

A pre-tokenizer splits text into pieces before any merging

  • Part of: tokenizer
  • Implemented by: tokenizers 0.23.3

Defined in Apply SmolLM2's merges by hand, chapter 2. In the book since Edition 2026.11.16. Permanent link to pre-tokenizer

preference tuning

Preference tuning then trains a model to prefer the better of two responses to the same request.

Defined in From predicting to following instructions, chapter 11. In the book since Edition 2026.11.16. Permanent link to preference tuning

Q

query

a query, a vector for what this position is looking for

Defined in Look back with a weighted average, chapter 6. In the book since Edition 2026.11.16. Permanent link to query

R

reference model

DPO compares how much more likely the model being tuned makes each response than the reference model it started from, the model after instruction tuning, and rewards it for raising the preferred one more.

Defined in Build the two training losses by hand, chapter 11. In the book since Edition 2026.11.16. Permanent link to reference model

regularization

regularization, which is anything in training that discourages memorizing

Defined in Stop when validation loss turns up, chapter 5. In the book since Edition 2026.11.16. Permanent link to regularization

repetition

repetition: the share of four-token phrases that repeat an earlier one in the same description

Defined in Measure what each setting writes, chapter 9. In the book since Edition 2026.11.16. Permanent link to repetition

repetition penalty

Keskar and colleagues proposed a repetition penalty: divide the logit of every token already generated by a factor θ above 1, so it becomes less likely

Defined in Penalize repetition, chapter 9. In the book since Edition 2026.11.16. Permanent link to repetition penalty

residual connection

Adding a part's output to its input, instead of replacing the input, is a residual connection.

x ← x + attention(x), then x ← x + feed-forward(x)

Defined in Add each part's output to a residual stream, chapter 7. In the book since Edition 2026.11.16. Permanent link to residual connection

residual stream

Elhage and colleagues call this running sum, each position's vector with every part's output added to it, the residual stream

Defined in Add each part's output to a residual stream, chapter 7. In the book since Edition 2026.11.16. Permanent link to residual stream

RMSNorm

RMSNorm drops the mean: it divides each vector by its root mean square, the square root of the average of its squared numbers, and multiplies each dimension by a learned gain

RMSNorm(x) = x / √(mean(x²) + ε) · g

Defined in Normalize before each part, chapter 7. In the book since Edition 2026.11.16. Permanent link to RMSNorm

root mean square

root mean square, the square root of the average of its squared numbers

Defined in Normalize before each part, chapter 7. In the book since Edition 2026.11.16. Permanent link to root mean square

rotary position embeddings also RoPE

Before scoring, RoPE rotates each query and key by angles that grow with its position.

(a, b) → (a cos pθᵢ − b sin pθᵢ, a sin pθᵢ + b cos pθᵢ)

Defined in Give attention positions with rotations, chapter 7. In the book since Edition 2026.11.16. Permanent link to rotary position embeddings

S

sampling

Sampling, which chapter 1 used to generate names, uses all of the distribution: draw the next token at random, in proportion to its probability, so a token with probability 0.2 is chosen one time in five.

Defined in Sample, with a temperature, chapter 9. In the book since Edition 2026.11.16. Permanent link to sampling

schedule also learning-rate schedule

changes the step size after every batch, following a schedule, which sets the step size for each step of the run

Defined in Warm up, then decay the step size, chapter 8. In the book since Edition 2026.11.16. Permanent link to schedule

shared module

the book's shared modules: short files in examples/lib/ that hold setup code many examples need, which you can open and read

Defined in Load the store's text, chapter 1. In the book since Edition 2026.11.16. Permanent link to shared module

SiLU

SiLU passes large positive numbers almost unchanged and turns large negative ones to nearly zero, smoothly.

SiLU(z) = z · σ(z)

Defined in Transform each position with a gated feed-forward layer, chapter 7. In the book since Edition 2026.11.16. Permanent link to SiLU

singular value decomposition also SVD

the singular value decomposition, SVD, which finds the 64 directions that together reconstruct the PPMI rows best, and weights each by how much of the rows it explains

Defined in Compress counts into dense vectors, chapter 4. In the book since Edition 2026.11.16. Permanent link to singular value decomposition

skip-gram loss

The skip-gram loss for one pair is the surprise at the real neighbour plus the surprise at each noise token's not being one

L = −log σ(v·u) − Σₙ log σ(−v·uₙ)

Defined in Learn vectors by predicting neighbours, chapter 4. In the book since Edition 2026.11.16. Permanent link to skip-gram loss

skip-gram with negative sampling also skip-gram, SGNS

Skip-gram with negative sampling, the method behind word2vec, gives each token two vectors, one for when it's the centre token and one for when it's a neighbour, and trains them on one task: given a token, tell its real neighbours apart from random tokens

Defined in Learn vectors by predicting neighbours, chapter 4. In the book since Edition 2026.11.16. Permanent link to skip-gram with negative sampling

SmolLM2-360M

SmolLM2-360M, an open model released by Hugging Face under the Apache 2.0 licence

Defined in Inside an open model, chapter 10. In the book since Edition 2026.11.16. Permanent link to SmolLM2-360M

SmolLM2-360M-Instruct

SmolLM2-360M-Instruct is the base model of chapter 10 after both steps: supervised fine-tuning on public and curated datasets, then DPO on UltraFeedback

Defined in From predicting to following instructions, chapter 11. In the book since Edition 2026.11.16. Permanent link to SmolLM2-360M-Instruct

smoothing

smoothing, which gives unseen pairs some probability by taking a little away from the pairs the model has seen

Defined in Unseen word pairs break the model, chapter 1. In the book since Edition 2026.11.16. Permanent link to smoothing

softmax

Softmax exponentiates every logit and divides by the sum, so the results are positive and sum to one, and a higher score means a higher probability

P(token i) = exp(zᵢ) / Σⱼ exp(zⱼ)

Defined in Predict from a window of embeddings, chapter 5. In the book since Edition 2026.11.16. Permanent link to softmax

special token

Open models mark the start or end of a text, or a turn in a conversation, with special tokens written as strings such as <|endoftext|>.

Defined in What goes wrong in production, chapter 2. In the book since Edition 2026.11.16. Permanent link to special token

step size also learning rate

η, the step size, also called the learning rate, sets how far each step goes

Defined in Take gradient descent steps, chapter 5. In the book since Edition 2026.11.16. Permanent link to step size

supported answer

An answer is supported if every material it names is in the product's name or description

Defined in Measure made-up product details, chapter 11. In the book since Edition 2026.11.16. Permanent link to supported answer

surprise

If the model gave the actual next word probability p, its surprise at that word is −log p: zero when it was certain (p = 1) and larger the less it expected the word.

Defined in Score a model with perplexity, chapter 1. In the book since Edition 2026.11.16. Permanent link to surprise

system message

a system message, instructions placed before the user's request

Defined in Format the request as the model was tuned, chapter 11. In the book since Edition 2026.11.16. Permanent link to system message

T

tanh

tanh, a function that squashes any number into the range from −1 to 1

Defined in Predict from a window of embeddings, chapter 5. In the book since Edition 2026.11.16. Permanent link to tanh

targets

the targets are the text's tokens without the first, so position i is trained to predict token i + 1

Defined in Turn products into batches, chapter 8. In the book since Edition 2026.11.16. Permanent link to targets

temperature

Before sampling, the distribution can be made sharper or flatter with a temperature, T: divide every logit by T before the softmax.

P(token i) = exp(zᵢ / T) / Σⱼ exp(zⱼ / T)

Defined in Sample, with a temperature, chapter 9. In the book since Edition 2026.11.16. Permanent link to temperature

test split

a test split kept for final reports

Defined in Load the store's text, chapter 1. In the book since Edition 2026.11.16. Permanent link to test split

token

A token is the unit a language model reads and predicts: whatever the text is cut into before the model sees it.

Defined in Tokens, chapter 2. In the book since Edition 2026.11.16. Permanent link to token

token type

The different tokens a text uses are its token types.

Defined in Choose how many merges, chapter 2. In the book since Edition 2026.11.16. Permanent link to token type

tokenizer

The list of merges, in the order they were learned, is the tokenizer.

  • Implemented by: tokenizers 0.23.3

Defined in Merge the most frequent pair, again and again, chapter 2. In the book since Edition 2026.11.16. Permanent link to tokenizer

top-k also top-k sampling

Top-k keeps only the k likeliest tokens and renormalizes their probabilities to sum to one

Defined in Sample, and cut the tail, chapter 9. In the book since Edition 2026.11.16. Permanent link to top-k

top-p also nucleus sampling, top-p sampling

Top-p, also called nucleus sampling, keeps the smallest set of likeliest tokens whose probabilities reach at least p

Defined in Sample, and cut the tail, chapter 9. In the book since Edition 2026.11.16. Permanent link to top-p

total log-probability

total log-probability, the sum of the logarithms of a text's token probabilities, where higher means likelier

Defined in Search for the likeliest text, chapter 9. In the book since Edition 2026.11.16. Permanent link to total log-probability

training split

a training split that models learn from

Defined in Load the store's text, chapter 1. In the book since Edition 2026.11.16. Permanent link to training split

transformer block

The transformer block adds parts around attention that transform each position, give it word order and keep a deep stack from degrading

Defined in The transformer block, chapter 7. In the book since Edition 2026.11.16. Permanent link to transformer block

V

validation split

a validation split for comparing models while you build them

Defined in Load the store's text, chapter 1. In the book since Edition 2026.11.16. Permanent link to validation split

value

a value, a vector for what this position passes on when it's looked at

Defined in Look back with a weighted average, chapter 6. In the book since Edition 2026.11.16. Permanent link to value

vector

a vector, a list of numbers

Defined in Words as vectors, chapter 4. In the book since Edition 2026.11.16. Permanent link to vector

vector length

A vector's length here is the square root of the sum of its squared numbers

Defined in Add each part's output to a residual stream, chapter 7. In the book since Edition 2026.11.16. Permanent link to vector length

vocabulary

A model can only predict tokens in its vocabulary, the set of tokens it learned from training text

Defined in Words leave gaps, chapter 2. In the book since Edition 2026.11.16. Permanent link to vocabulary

W

warm-up

For the first 200 steps, the step size rises in a straight line from nearly zero to its peak, 0.003: a warm-up

Defined in Warm up, then decay the step size, chapter 8. In the book since Edition 2026.11.16. Permanent link to warm-up

weight decay

Weight decay reduces each weight at every step by its value times the step size times a decay factor

Defined in Update with AdamW and clipped gradients, chapter 8. In the book since Edition 2026.11.16. Permanent link to weight decay

weight tying also tied embeddings

weight tying: the output layer reuses the embedding table, turned on its side, instead of learning a matrix of its own.

Defined in Tie the output layer to the embeddings, chapter 7. In the book since Edition 2026.11.16. Permanent link to weight tying