Small Open Models

Teaching kit

From Edition 2026.11.9, the book's current edition. It changes when the book does. Use and adapt it freely, with credit to the book.

Syllabus

Reading times are for the chapter's text. Students also run its examples, in the Lab or on their own machine.

A chapter a week: 10 weeks

In 5 weeks

Objectives and outlines

Each chapter's objectives are its first sentence. Each outline follows the chapter's sections, with the terms it defines, the predictions it asks for before an example runs, and the rule each section ends on.

1. Build a router for the bank's inbox that hands off when unsure

About 17 minutes to read, 4 examples.

By the end, students can build a router that returns a team, a confidence and a hand-off flag for one message, score it on a fixed sample by three measures, and see why an untrained small model, prompted as it stands, can't be the router's first version.

Lecture outline: 6 sections
  1. What the router returns
    • Working rule Return the flag with every team, and send a message to a person whenever the flag is true. Return the team to the caller only when the flag is false. Keep the flag even when removing it would make a router look more complete, because the hand-off measure counts it.
  2. Ask a model for one team
    • Run record-your-own-answers.py
    • Working rule Score every model on the same fixed sample, and record its answers with record_answers before you compare it with the stand-in. Say which sample each score is on, and set two scores side by side only when their samples match.
  3. Hand off the unsure messages
    • Run route-one-message.py
    • Working rule Choose the threshold on the dev split, never on the test sample, and report the hand-off rate beside accuracy every time. While the threshold is a placeholder, say so in each output, and keep its accuracy out of any claim about the router's result.
  4. Score the router on a fixed sample
    • Key terms Accuracy, Hand-off rate, Accuracy on messages not handed off
    • Run route-recorded-answers.py
    • Working rule Report accuracy, the hand-off rate and the accuracy on messages not handed off together, each with its count, on the same fixed sample, and compare two routers only when both were measured on that sample.
  5. Show what a small model does first
    • Run unconstrained-first-try.py
    • Working rule Before a model routes anything, count its answers that are team names on a fixed sample, and compare the count with the sample's size. If the model does not return a team for every message, do not route with it as it stands. Constrain its answer to the teams, or train it for the task. A hand-off rule placed on an answer that is not a team protects a team that does not exist.
  6. Where a rough router breaks

2. Measure the routing task before you choose a model

About 17 minutes to read, 3 examples.

By the end, students can say what each of the book's samples is for, compute the three measures the router is scored on with their denominators, and read a baseline that was recorded once and is reused for every comparison.

Lecture outline: 5 sections
  1. The data and its licence
    • Working rule Keep the licence file beside the data, cite the paper in the data card, and print counts, never message text. If you reuse a dataset of real customer messages, read its card before you print anything from it, and cite the source where the data is documented.
  2. Dev, test and fixed samples
    • Run count-the-splits.py
    • Working rule Draw dev from training rows only, team by team, with a fixed seed, and commit its row numbers. Draw each test sample once, commit its order, and read it only to report. Choose every threshold, prompt and setting on dev. When dev is too small to decide, draw more dev messages from the training file, never from the test file.
  3. Three measures, each with its denominator
    • Working rule Print each of the three measures with its count and its denominator, and round every rate to one decimal place. Report accuracy on the messages not handed off beside the hand-off rate, never alone. When no message is kept, print "not defined" rather than zero.
  4. The baseline from recorded answers
    • Run baseline-measures.py
    • Working rule Record the baseline once, on test_1001, the fixed sample that every candidate's prefixes come from. Store the model's pinned revision, the prompt's hash and the decoding settings in a manifest, and check the answers file against its hash before you read a line. Compare each candidate only with the baseline scored on the same messages, whether that is test_40, test_200 or test_1001. When your own hosted model is the baseline, record its answers the same way, and say in the write-up that its scores differ from the stand-in's.
  5. Where the labels collide
    • Run confusions-in-the-baseline.py
    • Working rule Count the wrong answers and the most frequent pairs for every candidate, on the same sample, and print each pair with its count. Read the pairs as a list of where to look. Print the three most frequent pairs rather than only the first, because one pair invites over-reading, and a list shows whether the errors cluster or scatter. Check any fix on the dev split before the test split is read again.

3. Choose a base model by counting what one message costs

About 16 minutes to read, 3 examples.

By the end, students can name the two base models this book keeps, load each one at its pinned revision, and count what one routing message costs in tokens and parameters before you commit to a model.

Lecture outline: 5 sections
  1. Which candidates we keep
    • Working rule Start with the smallest model in the family that passes the measures you set on the dev split, the training messages held back for choosing (chapter 2). Move up one size only when the smaller model misses a measure, and only after you have measured the larger model on the same samples. This is a rule of thumb. It holds until your own measurements say otherwise.
  2. Load each model at its pinned revision
    • Run load-both-models-offline.py
    • Working rule Pin every model by its commit in examples/models.toml, and load it with that commit and the offline flags. Record the parameter and layer counts next to the pin, so that a change in a model's size shows up as a changed output line and not as a surprise in a later chapter.
  3. Measure what one message costs
    • Run count-forward-passes.py
    • Run count-prompt-tokens.py
    • Working rule Count the prompt's tokens with the tokenizer of the model you will run, before you choose a design, and count again whenever the prompt changes. The count belongs to that tokenizer, so a count from another model's tokenizer answers a different question.
  4. Why 135M is the core
    • Working rule Fine-tune the smaller model first, and measure it on the dev split against the measures you set before you looked at the test split. Keep the larger model as a comparison until the smaller one has missed a measure on the dev split, and do not switch models on the test split.
  5. What a bigger model would need
    • Working rule Before you choose a larger model, train it with the same recipe you used for the smaller one, and measure it on the same dev and test samples, on your own machine. Judge it by its measures and by what it costs to run on that machine, and not by a published score.

4. Prompt a small model and constrain its answers to the teams

About 18 minutes to read, 5 examples.

By the end, students can write a router prompt for a small open model, constrain its answer so that it can only be a team name, and measure how many of those answers are right.

Lecture outline: 7 sections
  1. The prompt, settled before any recording
    • Run constrained-decode-135m-chat.py
    • Working rule Use one prompt for every model you compare, in the form the comparison needs. Before you adopt a chat template, measure it on the same sample as the raw form, and report a small gap as a small gap.
  2. Why free text fails
    • Working rule Treat an answer that is not a team as a failed route. Count such answers on a fixed sample before a model routes anything, and do not match a near-miss to the closest team. Constrain the answer at decode time, and keep the count, so that a broken constraint shows up as a number.
  3. A token trie allows only team names
    • Run constrained-decode-135m.py
    • Working rule Constrain the answer to the labels with a token trie built from the label names, and count the answers that are labels on a fixed sample. The constraint makes every answer a team. It does not make any answer right, so count the right answers as a separate measure.
  4. Decode greedily so every count repeats
    • Working rule Use greedy decoding, fix the seeds and the thread count, and compare each printed line with a recorded copy on every run. Do not describe a result as reproducible beyond the versions and the kind of machine it was run on.
  5. Measure the constrained answers
    • Run constrained-decode-360m.py
    • Run prompting-on-test-200.py
    • Working rule Report the count of right answers beside the count of valid answers, on one fixed sample, for every variant you compare. A router whose answers are all valid can still be wrong on most messages, and only the second count shows it.
  6. Why a prior is not a router
    • Run label-scoring-prior.py
    • Working rule Use label scoring to see which team a model favours before training, and do not route with it. Do not route messages with a prompted model of this size on this task; the 200-message recording is the evidence. Train a head on the labelled messages instead, as chapter 5 does.
  7. What this chapter decided

5. Fine-tune a classification head with LoRA

About 16 minutes to read, 5 examples.

By the end, students can train a classification head on the labelled messages with LoRA, count the parameters that training changes, choose a hand-off threshold on the dev split by a stated rule, and record that threshold beside the results it produced.

Lecture outline: 6 sections
  1. A head that scores the teams
    • Working rule Read the team and the confidence from one forward pass, and take the team as the output with the highest probability. Keep both in one function, so that a hand-off decision and the team it protects come from the same answer. Do not set a head's confidence beside a stand-in's as though they measured the same thing.
  2. Adapters train a small part of the model
    • Working rule Set the rank, the scaling and the target layers in one configuration, and count the trainable parameters from the loaded model before training starts. Print the total with the head in it. A count that leaves out the head reports part of what training changes, and it will not match what the optimizer updates.
  3. Thirty steps are not enough
    • Run lora-thirty-steps.py
    • Working rule Run a short training to check that the pipeline works from start to finish, and do not route with its head. Judge a fine-tuned head on the dev split, with the measures set in chapter 2, after a run long enough to reach them. Do not read a loss curve of a few points as evidence that the head learns.
  4. Train on the training rows, never on the test rows
    • Working rule Train only on training rows that are outside dev. Record the seed, the batch size, the step count and the data commit in the manifest. When dev is too small to decide a threshold, draw more messages from the training file, never from the test file, as chapter 2 says.
  5. Use the recorded adapter
    • Run lora-recorded-adapter.py
    • Run compare-with-standin.py
    • Working rule Compare two models only on the same messages, and name the threshold each one used beside its count. When one threshold is a placeholder, say so beside its count, and do not read its hand-off count as the model's own result.
  6. Choose the threshold on dev
    • Run choose-threshold-on-dev.py
    • Run calibration-bands-on-dev.py
    • Working rule Choose the threshold on the dev split by the stated rule, and write its value and the rule into the adapter's manifest. Report the hand-off rate beside accuracy on the messages kept, each with its count, and show at least one setting either side of the chosen value. Check calibration on dev before you trust the confidence as a cut, and when the mix of messages changes, choose the threshold again on a fresh dev draw from the training file.

6. Distill a larger model's answers into a small one

About 16 minutes to read, 3 examples.

By the end, students can train a small student on the labels a larger teacher gives your training rows, measure how often those labels agree with the gold labels, and decide from the same messages whether the student is worth using beside a student trained on gold labels.

Lecture outline: 6 sections
  1. What distillation copies
    • Working rule Before you train a student on a teacher's answers, measure how often the teacher's answer matches the gold label on rows that you have labelled, and read some of the disagreements. Expect the student to reproduce the teacher's errors, and count them on messages the student has not trained on.
  2. A teacher whose terms allow it
    • Working rule Read the teacher's licence and its terms of use on the day you train. Name the teacher, its licence and its revision in the chapter, and point the student's manifest to the teacher's recording. Do not train on the outputs of a model whose terms forbid it. A licence that places no restriction on outputs answers the question for that licence, so read its text before you assume the answer for another model.
  3. Label training rows with the teacher
    • Run teacher-agreement.py
    • Working rule Measure the teacher's agreement with gold on a labelled sample of rows before you train a student on the labels. Label only rows outside dev and test, and record the teacher's model, its version and its settings in the teacher's recording manifest, and point the student's manifest to it, so each label can be traced.
  4. Train the student on the teacher's labels
    • Working rule Choose a student's threshold on dev by the same rule as the human-labelled adapter, and when no threshold reaches the target, say that the threshold is the fallback. Report the hand-off rate the threshold costs beside accuracy, with both counts, before you compare the student with anything.
  5. Measure agreement, not only accuracy
    • Run student-recorded-adapter.py
    • Run student-vs-gold-control.py
    • Working rule Compare two adapters only on the same messages and the same rows. Count the messages each one gets right that the other gets wrong, and print both counts with their denominator. Do not set accuracies taken at different thresholds side by side as though they were the same measure.
  6. When the teacher is wrong
    • Working rule Train a student on a teacher's labels only when you have too few gold labels for the task and a sample of gold labels shows that the teacher agrees with them well enough. Compare the student with the human-labelled adapter, and with the same recipe on gold labels, on the same messages, before you choose it. When the student loses, ship the adapter and say so in the write-up.

7. Quantize the router and measure what changes

About 16 minutes to read, 2 examples.

By the end, students can store the fine-tuned router's backbone weights as 8-bit integers in one call, count the predictions that change on a fixed sample, and say what that count does and does not show.

Lecture outline: 5 sections
  1. One call to quantize the linear layers
    • Run int8-layers-and-bytes.py
    • Working rule Quantize only the layers you can name, leave the output layer in float32, and count the bytes from the shapes before you claim any saving. Report the saving as a ratio of weight bytes, with the count of layers beside it, and never as a file size or as a speed.
  2. Count the predictions that change
    • Run int8-changed-predictions.py
    • Working rule Count the changed predictions on the same messages, at the same threshold, for both copies, and print each count with its denominator beside the accuracy and the hand-off count. When the count is zero, report it as zero on that sample, and name the sample and the adapter it was taken from.
  3. Keep the score head in float32
    • Working rule Keep the score head in float32 unless a count on the dev split, at the chosen threshold, shows that a quantized head changes no team and no hand-off that you rely on. Say which threshold you counted at.
  4. Measure speed in a recording
    • Working rule Read a speed ratio only from a recording that names the machine it ran on, and never infer one from the weight bytes. The ratio above describes its own machine only, so run the recording on the machine you would serve from before you read it as your speed.
  5. Decide with the hand-off in view
    • Working rule Serve the float32 adapter by default. Consider the int8 copy only when a recording on the serving machine names a speed gain, a measured memory need favours it, and then count its changed predictions on the dev split at the chosen threshold before it serves a message.

8. Serve the router on a CPU, with the hand-off inside the route

About 16 minutes to read, 2 examples.

By the end, students can serve the router as one POST /route endpoint that returns a team, a confidence and a hand-off flag for each message.

Lecture outline: 5 sections
  1. One route that returns the hand-off
    • Run serve-route-testclient.py
    • Working rule Compute the flag in the route, from the same confidence as the team, with handoff, and return a team only when the flag is false. Do not add a second route, a field or an option that returns a team for a message the flag has marked. If a caller needs a team for every message, the answer is a person for the messages the flag marks, not a team from the route.
  2. Load the model once at startup
    • Working rule Build the app once per process, with the thread count that process should use, and before the first request. On a two-vCPU budget, as this book uses, keep the default of two threads, and choose another count only after you have measured your own machine with your own messages.
  3. Know what the route cannot score
    • Run truncation-count.py
    • Working rule Count the messages longer than the cut before you serve a set of messages, with the tokenizer of the adapter's base model and the same cut the route uses. The route cuts a longer message at 128 tokens and scores only what is left, so when the count is above zero, send those messages to a person outside the route, and report the count with the hand-off rate.
  4. Test the route in the same process
    • Working rule Test the route through the test client on a committed sample, and print the counts that tie the flag to the team: the number of handed-off messages, the number of those with no team, and the number of team names outside the list. A test that checks only the status code does not show that the hand-off rule holds. Each claim the test makes should have a printed line behind it, because a line is what the book can quote and a reader can rerun. A claim without a line is a claim the reader has to take on trust.
  5. What goes wrong in production
    • Working rule Run one model per worker process, built with the thread count you have measured for your machine. Keep two threads per worker, as this book does, until your own measurement says otherwise. Count the worker processes before you count the model's memory on the machine, and keep --reload and --workers apart. Watch the hand-off rate the route returns, beside its errors, from the first day the route serves messages.

9. Version the router, watch its hand-offs and fall back to people

About 19 minutes to read, 4 examples.

By the end, students can name a router by the parts that make its answers, check those parts before the route loads them, count the hand-off rate for each window of messages against a stated baseline, and say where each handed-off message goes.

Lecture outline: 8 sections
  1. Name the router by its parts
    • Run adapter-manifest-check.py
    • Working rule Name a router by its base revision, its adapter's name with the version in that name, its data commit, its threshold and its recipe hash, and write all five into the adapter's manifest before the route serves it. Compare the base revision in the manifest with the pin in examples/models.toml each time you load the adapter. A pin that has moved means a different base model, and the adapter needs a new check before it serves.
  2. Check the adapter's files against its manifest
    • Working rule Run the check before any route loads an adapter. Pin the SHA-256 of the manifest with the code, count the matching settings and files, print each count with its total, and refuse the load when any count falls short or the manifest digest does not match. Take the hashes from the files on disk, and check and load from the same files, with nothing writing to them in between.
  3. Watch the hand-off rate
    • Run handoff-windows.py
    • Working rule Count the hand-offs in each window of the same size, print each count with its denominator, and compare each rate with the dev baseline at the adapter's threshold. Set the drift rule, and its number of points, before you read the windows, and write it into your report. Treat a drifting window as a reason to read its messages and their teams, not as a verdict on the router.
  4. Sample handed-off messages for people
    • Working rule When you build the sample, draw it with a fixed seed, print how many messages were labelled, and report how often the router's team and the person's team agree, with its count. Label a sample of kept messages on the same schedule, and read the router's accuracy on kept messages only from that sample. Never read the kept rate off the handed-off sample.
  5. Fall back to a person
    • Run fallback-to-larger-model.py
    • Working rule Send each handed-off message to a person. A recorded answer from the stand-in may come with it, labelled as a suggestion. Never return a team for it, and never put a suggestion in the route's response shape without a field that says who answered. Count the suggestions that are right, with the count and its denominator, before a reader decides how much weight to give them.
  6. Retrain on a trigger
    • Run retrain-trigger.py
    • Working rule Retrain when the trigger you stated is met, and not on one drifting window. Write the trigger, its number and the rule beside the windows that met it. A retrained adapter is a recording named as the next version, with its manifest digest pinned and its threshold chosen again on dev. Keep the previous adapter until the new one passes the check in make_app.
  7. What goes wrong in production
    • Working rule Print each control's count where a reader can see it at startup and in each window, and keep the counts that would reveal a failure in the same report as the rate they explain.
  8. The decisions this chapter made
    • Working rule Write each decision above into the report with its number and whether it is the book's choice or a sourced figure. Change a decision only by changing the rule it names, and rerun the examples that print its counts.

10. When a small model is the wrong choice

About 19 minutes to read, 5 examples.

By the end, students can say where a small router still loses to the stand-in and where a hosted model is expected to do better, count each option's monthly cost from inputs you state with their dates, find the volume at which a machine of your own becomes the cheaper route, and name the requirements that decide the question before any cost does.

Lecture outline: 6 sections
  1. Where the small model still loses
    • Run router-on-test-1001.py
    • Run where-the-router-still-loses.py
    • Working rule Before you route with a small model, run it and the hosted model on the same fixed sample. Count the messages each gets right that the other gets wrong, and the wrong answers that would reach a caller. Count a prompted model's answers against the team list, and count how often its most common answer is the gold team. Do not route a team that has no labelled messages of its own, and do not train a student on a teacher's labels before you have counted how often the teacher agrees with the gold labels.
  2. Count the bill and the machine
    • Run cost-ratio-from-recording.py
    • Run monthly-bill-from-labelled-inputs.py
    • Working rule Compare the two bills with inputs you have checked and a machine you have priced. Write each price with its source and the date you read it. Count the input tokens with the tokenizer of the hosted model you compare with, and take the throughput from a recording on the machine you would buy. Where a value is an example, say so beside it, and replace it before you decide.
  3. Keep the data inside the building
    • Working rule Settle the data requirement before the cost. Write it down as a requirement, with the people who own it, and do not count it as a saving or as a cost. When the messages must stay on the bank's machines, choose a model that runs there, and then count its accuracy and its hand-offs as in the first section of this chapter.
  4. Choose with a table
    • Working rule Answer every row from your inbox before you choose. Where a row asks for a count, take it on a fixed sample and print its denominator. Where a row names a requirement, let the requirement decide before the counts do, and write it into the decision as a requirement.
  5. Know the break-even
    • Run break-even-from-recorded-ratio.py
    • Working rule Compute the break-even with your machine's bill and the hosted price you read, with that price's date and source written beside it. Check that the volume you expect sits inside the capacity you measured on your machine before you read the break-even as a saving. Above that capacity, count a second machine and compute again. Read the break-even as a cost point, not a decision. The requirements and the accuracy counts from the earlier sections still decide whether the router goes live.
  6. The decisions this book made for the inbox
    • Working rule Write this list again from your inbox, with your numbers, before you choose. A verdict that does not come from your counts, dates and requirements is not the book's verdict for your inbox.

Quiz bank

Questions on every chapter, with answer keys, for verified educators with the Educator plan. Each answer is tied to its section. The Educator plan