Glossary

The 100 terms Small Open Models defines, each in the book's own words, with the section that defines it.

From Edition 2026.11.9, the book's current edition. Where a definition depends on a package, it says the version this edition pins and runs its examples on.

A

accuracy

Accuracy is the messages whose team equals the gold label, divided by all the messages in the sample.

Defined in Score the router on a fixed sample, chapter 1. In the book since Edition 2026.11.9. Permanent link to accuracy

accuracy on messages not handed off also accuracy on kept messages

Accuracy on messages not handed off is the correct messages among those the router keeps, divided by the messages it keeps.

Defined in Score the router on a fixed sample, chapter 1. In the book since Edition 2026.11.9. Permanent link to accuracy on messages not handed off

adapter also LoRA adapter

the trained LoRA matrices are the adapter

Defined in Adapters train a small part of the model, chapter 5. In the book since Edition 2026.11.9. Permanent link to adapter

adapter version

The version the route reports is the suffix of the adapter's name.

Defined in What goes wrong in production, chapter 8. In the book since Edition 2026.11.9. Permanent link to adapter version

agreement with the teacher

Agreement with the teacher says how much of the teacher the student has learned, whether the teacher is right or wrong.

Defined in Measure agreement, not only accuracy, chapter 6. In the book since Edition 2026.11.9. Permanent link to agreement with the teacher

answer log-probability

the log-probability of the answer, which is the sum of the answer's token log-probabilities

Defined in Ask a model for one team, chapter 1. In the book since Edition 2026.11.9. Permanent link to answer log-probability

B

BANKING77

The messages and their teams come from BANKING77, a public dataset of customer messages to a bank, published under CC BY 4.0.

Defined in Build a router for the bank's inbox that hands off when unsure, chapter 1. In the book since Edition 2026.11.9. Permanent link to BANKING77

banking77-distilled-v1 also student adapter

The student's adapter is banking77-distilled-v1.

Defined in Train the student on the teacher's labels, chapter 6. In the book since Edition 2026.11.9. Permanent link to banking77-distilled-v1

banking77-gold-1000-v1 also gold-label control

The control, banking77-gold-1000-v1, is the student's recipe trained on gold labels, and its rows are the rows the teacher labelled.

Defined in Measure agreement, not only accuracy, chapter 6. In the book since Edition 2026.11.9. Permanent link to banking77-gold-1000-v1

banking77-lora-v1 also recorded adapter, human-labelled adapter

The working router uses banking77-lora-v1: the adapter from the full recorded run, version 1, written by recordings/lora_full.py.

Defined in Use the recorded adapter, chapter 5. In the book since Edition 2026.11.9. Permanent link to banking77-lora-v1

base model

A base model is a model already pretrained on a large amount of text, which you adapt to one job.

Defined in Which candidates we keep, chapter 3. In the book since Edition 2026.11.9. Permanent link to base model

baseline

The baseline is the stand-in's score on test_1001, the sample that the final comparison in chapter 10 uses.

Defined in The baseline from recorded answers, chapter 2. In the book since Edition 2026.11.9. Permanent link to baseline

break-even

The break-even is the volume at which the hosted model's monthly bill and the machine's are equal.

break-even messages per month = machine bill per month ÷ hosted price per message

Defined in Know the break-even, chapter 10. In the book since Edition 2026.11.9. Permanent link to break-even

C

changed-prediction count

The changed-prediction count uses the same 200 messages and asks a question the three measures do not: did the two copies agree on each one?

Defined in Count the predictions that change, chapter 7. In the book since Edition 2026.11.9. Permanent link to changed-prediction count

chat form

The chat form wraps the router prompt's text in the model's chat template, which adds a default system prompt.

True for transformers 5.23.0, as this edition pins it.

Defined in The prompt, settled before any recording, chapter 4. In the book since Edition 2026.11.9. Permanent link to chat form

classification head also score head, classifier head

A classification head works differently from the stand-in: it keeps the language model's layers and adds one output layer with 77 outputs, one for each team.

Defined in A head that scores the teams, chapter 5. In the book since Edition 2026.11.9. Permanent link to classification head

classifier

A classifier reads the message once and returns a score for each team.

Defined in Measure what one message costs, chapter 3. In the book since Edition 2026.11.9. Permanent link to classifier

confidence

The confidence is a number from zero to one that says how sure the router is of that team.

confidence = exp(sum of log p(token))

Defined in What the router returns, chapter 1. In the book since Edition 2026.11.9. Permanent link to confidence

confidence

In this chapter, the confidence is the softmax probability of the chosen team.

Defined in A head that scores the teams, chapter 5. In the book since Edition 2026.11.9. Permanent link to confidence

D

dev baseline

The baseline is the hand-off rate on the dev split at the adapter's threshold.

Defined in Watch the hand-off rate, chapter 9. In the book since Edition 2026.11.9. Permanent link to dev baseline

dev split

the dev split, the training messages held back for choosing settings

Defined in What the router returns, chapter 1. In the book since Edition 2026.11.9. Permanent link to dev split

distillation also knowledge distillation

Distillation trains a small model to reproduce a large one.

Defined in What distillation copies, chapter 6. In the book since Edition 2026.11.9. Permanent link to distillation

drift

A window drifts when its hand-off rate is more than a stated number of points from the baseline, in either direction.

Defined in Watch the hand-off rate, chapter 9. In the book since Edition 2026.11.9. Permanent link to drift

E

end-of-answer token also <|im_end|>

The end-of-answer token is the model's <|im_end|>, which is also its pad token (id 2).

Defined in A token trie allows only team names, chapter 4. In the book since Edition 2026.11.9. Permanent link to end-of-answer token

error

With no threshold applied, the error is one minus accuracy

Defined in Where the labels collide, chapter 2. In the book since Edition 2026.11.9. Permanent link to error

F

forward pass also forward passes

the forward passes, each one call of the network

Defined in Measure what one message costs, chapter 3. In the book since Edition 2026.11.9. Permanent link to forward pass

free decoding also unconstrained decoding

the model decoded freely: greedy, taking the highest-scoring token at each step, as in this chapter, but with no mask on the tokens it could pick

Defined in Why free text fails, chapter 4. In the book since Edition 2026.11.9. Permanent link to free decoding

full fine-tuning

Full fine-tuning changes every weight in the base model.

  • Contrasts with: LoRA

Defined in Adapters train a small part of the model, chapter 5. In the book since Edition 2026.11.9. Permanent link to full fine-tuning

G

generative model

A generative model reads the prompt once and then writes its answer one token at a time, and each token it writes is one more pass through the network.

Defined in Measure what one message costs, chapter 3. In the book since Edition 2026.11.9. Permanent link to generative model

greedy answer

Here the teacher's greedy answer plays the part of the mode: it is the one team the teacher returns for each message.

Defined in What distillation copies, chapter 6. In the book since Edition 2026.11.9. Permanent link to greedy answer

greedy decoding

greedy decoding, which takes the most likely token at each step

Defined in Ask a model for one team, chapter 1. In the book since Edition 2026.11.9. Permanent link to greedy decoding

H

hand-off

A hand-off gives the message to a person who can route it

Defined in What the router returns, chapter 1. In the book since Edition 2026.11.9. Permanent link to hand-off

hand-off flag

The hand-off flag is true when the confidence is too low to send the message on, and a person takes the message instead.

Defined in What the router returns, chapter 1. In the book since Edition 2026.11.9. Permanent link to hand-off flag

hand-off rate

Hand-off rate is the messages whose confidence is below the threshold, divided by all the messages.

Defined in Score the router on a fixed sample, chapter 1. In the book since Edition 2026.11.9. Permanent link to hand-off rate

hand-off rule

handoff(confidence, threshold) returns true when the confidence is below the threshold, and false when it is equal to the threshold or above it.

Defined in Hand off the unsure messages, chapter 1. In the book since Edition 2026.11.9. Permanent link to hand-off rule

hand-off watch

A watch needs three things, a window, a baseline and a rule for when a window has moved too far from the baseline.

Defined in Watch the hand-off rate, chapter 9. In the book since Edition 2026.11.9. Permanent link to hand-off watch

hosted bill

A hosted model's monthly bill is the number of messages times what one message costs, and one message costs its input tokens at the input price plus its output tokens at the output price.

hosted cost per month = messages × (input tokens × input price + output tokens × output price)

Defined in Count the bill and the machine, chapter 10. In the book since Edition 2026.11.9. Permanent link to hosted bill

I

inbox

Customer messages to a bank land in one support queue, which this book calls the inbox.

Defined in Build a router for the bank's inbox that hands off when unsure, chapter 1. In the book since Edition 2026.11.9. Permanent link to inbox

int8 copy also banking77-lora-v1-int8

The int8 copy, named banking77-lora-v1-int8 in the recordings, is the model the helper returns: a derived model built from the adapter, not a second adapter.

Defined in One call to quantize the linear layers, chapter 7. In the book since Edition 2026.11.9. Permanent link to int8 copy

Int8WeightOnlyConfig also weight-only int8, int8 weight-only quantization

The scheme is torchao's Int8WeightOnlyConfig, which the torchao source at v0.18.0 describes as "int8 weight-only symmetric per-channel quantization to linear layers".

True for torchao 0.18.0, as this edition pins it.

Defined in One call to quantize the linear layers, chapter 7. In the book since Edition 2026.11.9. Permanent link to Int8WeightOnlyConfig

intra-op threads also set_num_threads

The set_num_threads setting covers the threads used inside one operation, the intra-op threads the documentation names.

True for torch 2.15.0+cpu, as this edition pins it.

Defined in Load the model once at startup, chapter 8. In the book since Edition 2026.11.9. Permanent link to intra-op threads

K

kept message

The keeps are the messages whose answers the route returned

Defined in Sample handed-off messages for people, chapter 9. In the book since Edition 2026.11.9. Permanent link to kept message

kept-message sample

The check on kept messages is a second sample, drawn in the same way, and it is not a by-product of the sample of handed-off messages.

Defined in Sample handed-off messages for people, chapter 9. In the book since Edition 2026.11.9. Permanent link to kept-message sample

L

label scoring

For each team, label scoring sums the log-probability of the team's tokens after the prompt, where a log-probability is the logarithm of the probability the model gives a token.

Defined in Why a prior is not a router, chapter 4. In the book since Edition 2026.11.9. Permanent link to label scoring

layer

a layer is one of the stacked blocks that a token passes through

Defined in Load each model at its pinned revision, chapter 3. In the book since Edition 2026.11.9. Permanent link to layer

linear layer also nn.Linear

A linear layer is a weight matrix that maps one vector to another.

Defined in One call to quantize the linear layers, chapter 7. In the book since Edition 2026.11.9. Permanent link to linear layer

log-probability

a log-probability is the logarithm of the probability the model gives a token

Defined in Why a prior is not a router, chapter 4. In the book since Edition 2026.11.9. Permanent link to log-probability

LoRA also low-rank adaptation

The paper that introduced LoRA, Hu et al. (2021), says it freezes the pre-trained weights and injects trainable low-rank matrices into the layers of the Transformer, and the trained LoRA matrices are the adapter.

Defined in Adapters train a small part of the model, chapter 5. In the book since Edition 2026.11.9. Permanent link to LoRA

LoRA rank

The rank, r, is the inner size of each added matrix, and it is kept small.

Defined in Adapters train a small part of the model, chapter 5. In the book since Edition 2026.11.9. Permanent link to LoRA rank

lora_alpha

lora_alpha sets how strongly the update counts

self.scaling[adapter_name] = lora_alpha / r

True for peft 0.21.3, as this edition pins it.

  • Part of: LoRA
  • Implemented by: peft 0.21.3

Defined in Adapters train a small part of the model, chapter 5. In the book since Edition 2026.11.9. Permanent link to lora_alpha

M

machine bill

A machine's monthly bill is its hours times its hourly rate.

machine cost per month = hours × machine cost per hour

Defined in Count the bill and the machine, chapter 10. In the book since Edition 2026.11.9. Permanent link to machine bill

machine capacity also machine capacity per month, capacity

The number of messages a machine can route in a month is its throughput, in messages per second, times the seconds in a month, times its utilisation

machine capacity per month = messages per second × seconds per month × utilisation

Defined in Count the bill and the machine, chapter 10. In the book since Edition 2026.11.9. Permanent link to machine capacity

machine cost per message

The machine's cost per message at capacity is its fixed bill divided by its capacity

Defined in Know the break-even, chapter 10. In the book since Edition 2026.11.9. Permanent link to machine cost per message

median

The median is the lower of the two middle values, so it is a whole number of tokens.

Defined in Measure what one message costs, chapter 3. In the book since Edition 2026.11.9. Permanent link to median

merging also merge, merge_and_unload

Merging folds the adapter's update into the base weights, so each layer is one weight matrix, quantized once, and the layer count describes the router.

True for peft 0.21.3, as this edition pins it.

  • Comes before: int8 copy
  • Implemented by: peft 0.21.3

Defined in One call to quantize the linear layers, chapter 7. In the book since Edition 2026.11.9. Permanent link to merging

O

open-weights licence also open-weights license

An open-weights licence such as Apache-2.0 governs the model's files

Defined in A teacher whose terms allow it, chapter 6. In the book since Edition 2026.11.9. Permanent link to open-weights licence

P

parameter

A parameter is one of the numbers the model learns during training

Defined in Load each model at its pinned revision, chapter 3. In the book since Edition 2026.11.9. Permanent link to parameter

per-row scale

A per-row scale lets each output row use the full range of the 8-bit integers for its own weights.

Defined in One call to quantize the linear layers, chapter 7. In the book since Edition 2026.11.9. Permanent link to per-row scale

pinned revision

A revision pinned by a full commit hash names one snapshot of the files

True for huggingface_hub 2.4.0, as this edition pins it.

Defined in Load each model at its pinned revision, chapter 3. In the book since Edition 2026.11.9. Permanent link to pinned revision

pins file also models.toml

the pins file, examples/models.toml, records the SHA-256 of every file in that snapshot, which the runtime checks

Defined in Load each model at its pinned revision, chapter 3. In the book since Edition 2026.11.9. Permanent link to pins file

placeholder threshold

A placeholder threshold has no rule behind it: it is a number that lets the rule run.

Defined in Hand off the unsure messages, chapter 1. In the book since Edition 2026.11.9. Permanent link to placeholder threshold

poorly calibrated

Guo et al. (2017) report that their networks are "poorly calibrated", meaning their confidence does not match how often they are right.

Defined in Choose the threshold on dev, chapter 5. In the book since Edition 2026.11.9. Permanent link to poorly calibrated

prefix_allowed_tokens_fn

Transformers applies the rule through a callback, prefix_allowed_tokens_fn, which returns the allowed tokens for each position.

True for transformers 5.23.0, as this edition pins it.

  • Implemented by: transformers 5.23.0

Defined in A token trie allows only team names, chapter 4. In the book since Edition 2026.11.9. Permanent link to prefix_allowed_tokens_fn

prior

If the top team is the same for every message, the score is the model's prior and says little about any one message.

Defined in Why a prior is not a router, chapter 4. In the book since Edition 2026.11.9. Permanent link to prior

Q

quantization also quantizing

Quantizing a layer stores each of its weights as an 8-bit integer, with a scale that maps the integers back to approximate float values.

Defined in One call to quantize the linear layers, chapter 7. In the book since Edition 2026.11.9. Permanent link to quantization

quantization scale

a scale that maps the integers back to approximate float values

Defined in One call to quantize the linear layers, chapter 7. In the book since Edition 2026.11.9. Permanent link to quantization scale

R

raw form also raw prompt

The raw form passes the router prompt's text to the model as it stands.

Defined in The prompt, settled before any recording, chapter 4. In the book since Edition 2026.11.9. Permanent link to raw form

recording

A recording fixes the answers to one model at one revision, so a reader who runs the same example later sees the same numbers.

Defined in The baseline from recorded answers, chapter 2. In the book since Edition 2026.11.9. Permanent link to recording

recording manifest also manifest

The manifest names the script that made the recording, the model and its pinned revision, the hash of the prompt, the decoding settings and the seed, and the hash of the answers file.

Defined in The baseline from recorded answers, chapter 2. In the book since Edition 2026.11.9. Permanent link to recording manifest

rough router

A router is rough when its threshold is a placeholder and its model is a stand-in

Defined in Build a router for the bank's inbox that hands off when unsure, chapter 1. In the book since Edition 2026.11.9. Permanent link to rough router

route

A route is a function that a web framework calls when a request arrives at a path with a method.

Defined in One route that returns the hand-off, chapter 8. In the book since Edition 2026.11.9. Permanent link to route

router

The router is the whole system: a model, a prompt, a threshold and the code that sends a message on.

Defined in Build a router for the bank's inbox that hands off when unsure, chapter 1. In the book since Edition 2026.11.9. Permanent link to router

router prompt also routing prompt

The router's prompt holds the 77 team names, the message and a request for the team

Defined in Measure what one message costs, chapter 3. In the book since Edition 2026.11.9. Permanent link to router prompt

router_lib

router_lib, the book's shared module

Defined in A token trie allows only team names, chapter 4. In the book since Edition 2026.11.9. Permanent link to router_lib

S

sample of handed-off messages also handed-off sample

A person would label a sample of the handed-off messages, drawn with a fixed seed, and the labels would say which team the person chooses for each.

Defined in Sample handed-off messages for people, chapter 9. In the book since Edition 2026.11.9. Permanent link to sample of handed-off messages

selective classification also reject option

Letting a router say when it is unsure has a name in the research literature: selective classification, or the reject option.

Defined in What the router returns, chapter 1. In the book since Edition 2026.11.9. Permanent link to selective classification

SmolLM2

The first candidate is SmolLM2, a family of models published on Hugging Face.

Defined in Which candidates we keep, chapter 3. In the book since Edition 2026.11.9. Permanent link to SmolLM2

SmolLM2-135M-Instruct also core model

SmolLM2-135M-Instruct is the smallest model in the family that this book keeps, and it is the core model for the rest of the book.

Defined in Show what a small model does first, chapter 1. In the book since Edition 2026.11.9. Permanent link to SmolLM2-135M-Instruct

soft targets

Hinton, Vinyals and Dean's paper, which gave distillation its name, proposed training a small model on the probabilities that a large model gives every class, as soft targets (Hinton et al., 2015, arXiv:1503.02531).

Defined in What distillation copies, chapter 6. In the book since Edition 2026.11.9. Permanent link to soft targets

softmax

Softmax turns the scores into 77 probabilities that sum to one.

Defined in A head that scores the teams, chapter 5. In the book since Edition 2026.11.9. Permanent link to softmax

spawned worker

A spawned worker is a new Python process.

True for uvicorn 0.54.0, as this edition pins it.

Defined in What goes wrong in production, chapter 8. In the book since Edition 2026.11.9. Permanent link to spawned worker

stand-in also Qwen3-4B-Instruct-2507

This book doesn't call a hosted model, so it uses a stand-in, a larger open model: Qwen3-4B-Instruct-2507.

Defined in Measure the routing task before you choose a model, chapter 2. In the book since Edition 2026.11.9. Permanent link to stand-in

suggestion

The recorded answer of the stand-in, Qwen3-4B-Instruct-2507, may travel with a handed-off message as a suggestion, labelled as the stand-in's, and the person decides whether to use it.

Defined in Fall back to a person, chapter 9. In the book since Edition 2026.11.9. Permanent link to suggestion

support staff

The people who read messages are the bank's support staff, and a message the router cannot send on goes to one of them.

Defined in Build a router for the bank's inbox that hands off when unsure, chapter 1. In the book since Edition 2026.11.9. Permanent link to support staff

T

teacher's agreement with gold also teacher agreement

The teacher's agreement with gold on the training rows, measured above, is the share of the teacher's training labels that are right.

Defined in When the teacher is wrong, chapter 6. In the book since Edition 2026.11.9. Permanent link to teacher's agreement with gold

teacher's recording

the teacher's recording, teacher-labels/manifest.json, which names the teacher's model and version, its settings and the rows it labelled

Defined in Train the student on the teacher's labels, chapter 6. In the book since Edition 2026.11.9. Permanent link to teacher's recording

team also intent

A team is a label the router can return, such as card_arrival, and it is never a group of people.

Defined in Build a router for the bank's inbox that hands off when unsure, chapter 1. In the book since Edition 2026.11.9. Permanent link to team

test split

The test split is the source's own held-out file.

Defined in Dev, test and fixed samples, chapter 2. In the book since Edition 2026.11.9. Permanent link to test split

test_1001

test_1001, a fixed sample of 1,001 test messages

Defined in Ask a model for one team, chapter 1. In the book since Edition 2026.11.9. Permanent link to test_1001

test_200

The first 40 messages of the shuffled test sample are test_40, the first 200 are test_200, and the whole sample is test_1001.

Defined in Dev, test and fixed samples, chapter 2. In the book since Edition 2026.11.9. Permanent link to test_200

test_40

test_40, a fixed sample of 40 test messages

Defined in Score the router on a fixed sample, chapter 1. In the book since Edition 2026.11.9. Permanent link to test_40

threshold

the threshold is the value below which a message is handed off

Defined in Three measures, each with its denominator, chapter 2. In the book since Edition 2026.11.9. Permanent link to threshold

token also tokens

A token is a piece of text the model reads and writes.

Defined in What the router returns, chapter 1. In the book since Edition 2026.11.9. Permanent link to token

tokenizer

The tokenizer turns the prompt's text into tokens, the units a model reads.

Defined in Measure what one message costs, chapter 3. In the book since Edition 2026.11.9. Permanent link to tokenizer

top team

The team with the highest sum is the model's top team for the message.

Defined in Why a prior is not a router, chapter 4. In the book since Edition 2026.11.9. Permanent link to top team

trie also token trie

a trie: a tree whose paths spell out each team's tokens, with the team's index at the node where its path ends

Defined in A token trie allows only team names, chapter 4. In the book since Edition 2026.11.9. Permanent link to trie

truncation also max_length

A message longer than 128 tokens is cut at 128 before the model reads it.

Defined in Know what the route cannot score, chapter 8. In the book since Edition 2026.11.9. Permanent link to truncation

U

uniform guess

A uniform guess gives each team the same probability, one in 77, so its loss is the natural log of 77.

Defined in Thirty steps are not enough, chapter 5. In the book since Edition 2026.11.9. Permanent link to uniform guess

utilisation also utilization

utilisation, which is the share of the month it is busy

Defined in Count the bill and the machine, chapter 10. In the book since Edition 2026.11.9. Permanent link to utilisation

V

version check

The version check pins the SHA-256 of each adapter's manifest.json in examples/adapter-pins.json, and make_app refuses to build on any mismatch or a moved base pin.

Defined in The decisions this chapter made, chapter 9. In the book since Edition 2026.11.9. Permanent link to version check