Distill a larger model's answers into a small one
About 16 minutes to read, with 3 examples in 6 sections.
Written with AI. Verified by people. How we make books
This book is open to contributions, so anyone can check it or correct it. Its repository · Report an error · How to fix one
Labs
Learn this chapter by doing it
Run every example exactly as the book did, change the code to see what happens, and try your own ideas on the book's code and data. Nothing to install, and your work is saved. Opens in about 30 seconds.
Run every example exactly as the book did, change the code to see what happens, and try your own ideas on the book's code and data. Nothing to install, and your work is saved. Labs runs it for you: 40 hours a month.
Your browser can translate this page from its menu, and the code stays as the book has it.
Or read it with your AI assistant:
By the end of this chapter, you can train a small student on the labels a larger teacher gives your training rows, measure how often those labels agree with the gold labels, and decide from the same messages whether the student is worth using beside a student trained on gold labels.
A router learns to name the team from labelled messages. This chapter asks whether a larger model can do that labelling for the training rows, so that a small model learns from its answers. The gold labels come from BANKING77. Chapter 5 trained a head on the gold labels of the training rows, and the adapter it saved, banking77-lora-v1, is the human-labelled adapter. Its test result is the bar for everything in this chapter, and its threshold is the one its manifest records, chosen on dev by the rule chapter 5 gave.
Gold labels cost people time to make, and distillation is the attempt to get around that: a larger model's answers stand in for the gold labels. The larger model here is the stand-in. It is larger than SmolLM2-135M-Instruct, the base model this book's router is built on, and it answers the same question the router asks.
What distillation copies
Distillation trains a small model to reproduce a large one. Hinton, Vinyals and Dean's paper, which gave distillation its name, proposed training a small model on the probabilities that a large model gives every class, as soft targets (Hinton et al., 2015, arXiv:1503.02531). Those probabilities carry more information than a single label, the paper argues, and a small model trained this way can often learn from far less data than the large model needs. That is the paper's own claim, from its own experiments, and it is not a rule about every task.
This chapter does not use soft targets. The stand-in returns one team for each message, the team it judges most likely, and the student sees only that team. Kim and Rush used the same simplification for translation, where they replaced the teacher's distribution with its most likely output: "Our simplest approximation is to replace the teacher distribution q with its mode" (Kim and Rush, 2016, arXiv:1606.07947). Their setting is translation, not classification. We borrow the simplification, and we measure it on this task instead of assuming it carries over. Here the teacher's greedy answer plays the part of the mode: it is the one team the teacher returns for each message.
Two things follow from training on one label per message. A student learns the teacher's errors along with its correct answers, because the training loop treats a wrong label as a label, the same as a right one. And the student's errors can differ from the teacher's, which is the reason to measure the student against gold labels as well as against the teacher.
Working rule: Before you train a student on a teacher's answers, measure how often the teacher's answer matches the gold label on rows that you have labelled, and read some of the disagreements. Expect the student to reproduce the teacher's errors, and count them on messages the student has not trained on.
A teacher whose terms allow it
The teacher has to be a model whose terms allow training on its outputs. The stand-in's GGUF build in this book is the repository unsloth/Qwen3-4B-Instruct-2507-GGUF, file Qwen3-4B-Instruct-2507-Q4_K_M.gguf, at the revision the teacher's manifest records. The model is Qwen3-4B-Instruct-2507, from Qwen, and the build is by unsloth. We credit both. The model card of the build declares license: apache-2.0 and names Qwen/Qwen3-4B-Instruct-2507 as its base model (model card at the pinned revision). The base model's licence, at a pinned commit, is Apache License 2.0. The Apache License 2.0, in its second section, grants the right "to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works" (Apache License 2.0).
We read Apache-2.0 as placing no restriction on using a model's outputs. Its text governs the Work and such Derivative Works, as section 2 above sets out, and it does not mention outputs. This is our reading of the licence, not a quotation from it, and it holds for this model's licence only. The adapters this book ships are distributed under Apache-2.0, and the notice beside them gives the licence and the attribution.
Licences differ. An open-weights licence such as Apache-2.0 governs the model's files; a hosted model is governed by its provider's terms of service, which can forbid using the service or its outputs to train a competing model. Terms change, so read the current terms of any model you would train on, on the day you use it. This book never uses a hosted model as a teacher.
The stand-in is an open model, so its licence can be read and its file can be fixed by a hash, which the teacher's manifest records. A hosted model's answers can change when its provider changes the model, and its terms may forbid this use. Those are reasons to prefer an open teacher whose file is fixed, even before its terms are read.
Working rule: Read the teacher's licence and its terms of use on the day you train. Name the teacher, its licence and its revision in the chapter, and point the student's manifest to the teacher's recording. Do not train on the outputs of a model whose terms forbid it. A licence that places no restriction on outputs answers the question for that licence, so read its text before you assume the answer for another model.
Label training rows with the teacher
The teacher labels training rows only. Dev rows are kept for choosing thresholds, and test rows are kept for reporting, as chapter 2 set out. The rows to label are drawn from the training file, outside dev. The labelled rows are the first 1,000 of a seeded random draw of 2,000 such rows, a random subset of them, and the recording sorts them by index before labelling. The labelling run was planned for all 2,000, but it could not finish that many within its time limit, so the book reports the 1,000 rows it labelled.
The draw is seeded. Labels already saved for a chosen row are kept and not labelled again, so a run that stops and continues adds only the rows it has not labelled. We did not test whether a resumed run gives the same labels as one full run.
The stand-in answers each message with the router prompt from chapter 1. It decodes greedily, at temperature zero. In a probe on 40 test messages, two runs gave identical answers, and that probe is the only check we made of repeatability. Its answer is constrained by a grammar to the 77 team names, so each label is a real team.
1"""Measure how often the teacher's label matches gold on the training rows it labelled, and its accuracy on test_2002from its recorded answers. The point rests on the agreement line; the example reads recorded answers only."""34import os56os.environ["HF_HUB_OFFLINE"] = "1"7os.environ["TRANSFORMERS_OFFLINE"] = "1"89import json # noqa: E4021011import torch # noqa: E4021213torch.set_num_threads(2)1415import router_lib # noqa: E4021617router_lib.setup_runtime()18RECORDED = router_lib.HERE / "data" / "recorded"1920train = router_lib.read_rows("train")21teacher_rows = [json.loads(line) for line in (RECORDED / "teacher-labels" / "labels.jsonl").read_text().splitlines()]22# row["index"] is the row's position in the training file; field 1 of a row is its gold team23agree = sum(row["label"] == train[row["index"]][1] for row in teacher_rows)24n = len(teacher_rows)25print(f"training rows labelled by the teacher: {n}")26print(f"teacher agrees with gold on training rows: {agree}/{n} ({agree / n:.1%})")27print(f"teacher disagrees with gold on training rows: {n - agree}/{n} ({(n - agree) / n:.1%})")2829test = router_lib.read_rows("test")30sample = router_lib.read_sample("test_200")31standin = router_lib.standin_answers()32# sample holds test row positions; field 1 of a test row is its gold team, field 0 its text33golds = [test[i][1] for i in sample]34teacher_preds = [standin[i] for i in sample]35correct = sum(p == g for p, g in zip(teacher_preds, golds))36print(f"messages on test_200 with a teacher answer: {sum(i in standin for i in sample)}/{len(sample)}")37print(f"teacher accuracy on test_200: {correct}/{len(sample)} ({correct / len(sample):.1%})")
python teacher-agreement.pytraining rows labelled by the teacher: 1000
teacher agrees with gold on training rows: 549/1000 (54.9%)
teacher disagrees with gold on training rows: 451/1000 (45.1%)
messages on test_200 with a teacher answer: 200/200
teacher accuracy on test_200: 109/200 (54.5%)
The teacher's label matches the gold label on 549/1000 (54.9%) of the 1000 training rows it labelled, and it disagrees on 451/1000 (45.1%). A student trained on those disagreeing rows learns the wrong label as the answer to each one. The last two lines of the output come from the stand-in's recorded answers on test_200, the 200 test messages this chapter reports on. There the teacher is right on 109/200 (54.5%) messages.
Read the line teacher agrees with gold on training rows before any student is trained. The line teacher accuracy on test_200 is the teacher's own accuracy on messages no one has used for training, and it is the reference for every copy measure below.
Counts do not show what the disagreements look like. Three training rows, each labelled card_arrival in the gold file, show the pattern. Each message is about a card that has not arrived, described here without its text, since messages are never quoted in this book. The teacher named card_delivery_estimate for two of them and lost_or_stolen_card for the third, so the teacher names a different team from the gold label each time.
Working rule: Measure the teacher's agreement with gold on a labelled sample of rows before you train a student on the labels. Label only rows outside dev and test, and record the teacher's model, its version and its settings in the teacher's recording manifest, and point the student's manifest to it, so each label can be traced.
Train the student on the teacher's labels
The student uses the recipe of chapter 5. It has the same base model, the same LoRA targets, the same head, the same seed and the same batch size. Its labels come from the teacher instead of the gold file. The rows and steps differ from chapter 5 as well: the student trains on 1000 rows for 125 steps, and the human-labelled adapter trained on 9000 rows for 1125 steps. The control in the agreement section keeps the student's rows and steps and changes only the labels. The student's adapter is banking77-distilled-v1. Its manifest records its settings and points to the teacher's recording, teacher-labels/manifest.json, which names the teacher's model and version, its settings and the rows it labelled, so anyone can trace where each label came from.
The recipe is held fixed on purpose. If the student changed its recipe and its labels at once, a gap in the results could come from either, and a reader could not say which. Keeping the recipe fixed makes the labels the only difference that the results can point to.
A student's threshold is chosen on dev by the same rule as the human-labelled adapter's. The rule takes the lowest threshold, on a grid of hundredths, at which the messages not handed off are at least 0.90 accurate on dev. The rule has a fallback for a student that cannot reach that target. The fallback takes the threshold with the highest accuracy on the messages not handed off, the lowest of any tie, and it skips any threshold that hands off every message. The output says that the fallback was used. This is the book's rule for a result that misses, so it is reported as a fallback and not as a student that met the target. A fallback threshold is chosen from the messages a threshold keeps on dev, and at a high threshold that can be a handful of messages. Treat a fallback value as provisional, and report it with the hand-off rate it costs on test, as this chapter does.
The student's threshold is chosen on part of the dev split, 500 of 1003 messages, because a run over every dev message did not fit the time budget of two CPU cores. Both student examples use that same part, and the control uses it for its own threshold, so the two thresholds are chosen on the same dev messages. The human-labelled adapter's threshold, from chapter 5, was chosen on all 1,003 dev messages.
Working rule: Choose a student's threshold on dev by the same rule as the human-labelled adapter, and when no threshold reaches the target, say that the threshold is the fallback. Report the hand-off rate the threshold costs beside accuracy, with both counts, before you compare the student with anything.
Measure agreement, not only accuracy
Three measures run through this chapter, and each answers its own question. The teacher against gold on the training rows asks whether the teacher is right on the rows it labels, and it is the one above. The student against the teacher on test_200 asks how much of the teacher the student copies. The student against gold on test_200 asks how often the student names the right team, which is the figure a router would be judged on. Each is printed with its denominator: the first is read on the 1,000 training rows, and the other two on the same 200 test messages.
Agreement and accuracy answer different questions. Agreement with the teacher says how much of the teacher the student has learned, whether the teacher is right or wrong. Accuracy against gold says how often the student names the right team. A student can agree with a wrong teacher and be wrong, and it can disagree with a wrong teacher and be right. Printing both lets a reader see which of the two is happening, instead of inferring it from one figure.
1"""Score the distilled student banking77-distilled-v1 on test_200 beside the human-labelled adapter, with the student's2threshold chosen on dev. The point rests on the student's accuracy line and its copy lines; both adapters are loaded3and scored live, and nothing is trained."""45import os67os.environ["HF_HUB_OFFLINE"] = "1"8os.environ["TRANSFORMERS_OFFLINE"] = "1"910import json # noqa: E4021112import torch # noqa: E4021314torch.set_num_threads(2)1516import router_lib # noqa: E4021718router_lib.setup_runtime()19RECORDED = router_lib.HERE / "data" / "recorded"2021tokenizer, student = router_lib.load_classifier(RECORDED / "distill-student" / "adapter")22DEV_N = 500 # the first 500 of the 1,003 dev messages: the run's time budget under two CPUs23# reached is False when no threshold meets the target, and the threshold returned is then the fallback24threshold, reached = router_lib.dev_threshold(student, tokenizer, target=0.90, n=DEV_N)25print(f"dev messages used: {DEV_N} of {len(router_lib.read_sample('dev'))}")26print(f"dev threshold reaching 0.90 accuracy not handed off: {'yes' if reached else 'no'}")27print(f"student threshold chosen on dev: {threshold:.2f}")2829test = router_lib.read_rows("test")30sample = router_lib.read_sample("test_200")31# sample holds test row positions; field 1 of a test row is its gold team, field 0 its text32golds = [test[i][1] for i in sample]33texts = [test[i][0] for i in sample]34student_preds, student_confs = router_lib.classify(student, tokenizer, texts)35router_lib.print_copy_report("banking77-distilled-v1", student_preds, student_confs, golds, threshold)3637# chapter 5 chose the adapter's threshold on dev, and its manifest records the value used here38lora = json.loads((RECORDED / "lora-full" / "manifest.json").read_text())39_, lora_model = router_lib.load_classifier(RECORDED / "lora-full" / "adapter")40lora_preds, lora_confs = router_lib.classify(lora_model, tokenizer, texts)41router_lib.print_copy_report("banking77-lora-v1", lora_preds, lora_confs, golds, lora["handoff_threshold"])4243teacher = router_lib.standin_answers()44teacher_preds = [teacher[i] for i in sample]45matches = sum(s == t for s, t in zip(student_preds, teacher_preds))46# positions within sample where the student's answer is not the gold team47wrong = [k for k in range(len(sample)) if student_preds[k] != golds[k]]48wrong_like_teacher = sum(student_preds[k] == teacher_preds[k] for k in wrong)49print(f"student answers equal to teacher answers on test_200: {matches}/{len(sample)} ({matches / len(sample):.1%})")50print(f"student errors on test_200: {len(wrong)}/{len(sample)} ({len(wrong) / len(sample):.1%})")51print(f"student errors equal to the teacher's answer: {wrong_like_teacher}/{len(wrong)}")
python student-recorded-adapter.pydev messages used: 500 of 1003
dev threshold reaching 0.90 accuracy not handed off: no
student threshold chosen on dev: 0.68
banking77-distilled-v1 accuracy on test_200: 69/200 (34.5%)
banking77-distilled-v1 handed off: 150/200 (75.0%)
banking77-distilled-v1 accuracy on messages not handed off: 28/50 (56.0%)
banking77-lora-v1 accuracy on test_200: 172/200 (86.0%)
banking77-lora-v1 handed off: 33/200 (16.5%)
banking77-lora-v1 accuracy on messages not handed off: 156/167 (93.4%)
student answers equal to teacher answers on test_200: 100/200 (50.0%)
student errors on test_200: 131/200 (65.5%)
student errors equal to the teacher's answer: 36/131
The student is well behind the human-labelled adapter, and the gap is not only in hand-offs, as the overall figures below show. On test_200, the student is right on 69/200 (34.5%) messages and hands off 150/200 (75.0%), while the adapter is right on 172/200 (86.0%) messages and hands off 33/200 (16.5%). The student's threshold is the fallback described above. Each run's accuracy on the messages it keeps is at its own threshold, and the output gives both beside this paragraph.
The copy measures come next. The student's answer equals the teacher's on 100/200 (50.0%) messages, and the teacher itself is right on 109/200 (54.5%). A student that copies the teacher closely therefore copies the teacher's wrong answers too. The student makes 131/200 (65.5%) errors on test_200. Of those errors, 36/131 repeat the teacher's answer. The rest differ from it, so a student's mistakes are not only the teacher's mistakes.
The comparison with the adapter alone mixes two things. The adapter learned from the gold labels of the full training split, and the student learned from the teacher's labels on a smaller set of rows. A gap between them could come from the labels, from the number of rows or from both. To separate them, the book records the same recipe on the same rows with gold labels. The control, banking77-gold-1000-v1, is the student's recipe trained on gold labels, and its rows are the rows the teacher labelled.
1"""Put the student banking77-distilled-v1 beside its control banking77-gold-1000-v1, which has the same recipe and the2same 1,000 training rows but gold labels, on the same test_200 messages. The point rests on the two counts of messages3each adapter gets right that the other gets wrong, which need no threshold; both adapters are scored live, and nothing4is trained."""56import json7import os89os.environ["HF_HUB_OFFLINE"] = "1"10os.environ["TRANSFORMERS_OFFLINE"] = "1"1112import torch # noqa: E4021314torch.set_num_threads(2)1516import router_lib # noqa: E4021718router_lib.setup_runtime()19RECORDED = router_lib.HERE / "data" / "recorded"20DEV_N = 500 # the first 500 of the dev messages, as student-recorded-adapter uses: the time budget under two CPUs21print(f"dev messages used: {DEV_N} of {len(router_lib.read_sample('dev'))}")2223test = router_lib.read_rows("test")24sample = router_lib.read_sample("test_200")25# sample holds test row positions; field 1 of a test row is its gold team, field 0 its text26golds = [test[i][1] for i in sample]27texts = [test[i][0] for i in sample]2829runs = {}30for name, folder in [("banking77-distilled-v1", "distill-student"), ("banking77-gold-1000-v1", "gold-same-rows")]:31 tokenizer, model = router_lib.load_classifier(RECORDED / folder / "adapter")32 threshold, reached = router_lib.dev_threshold(model, tokenizer, target=0.90, n=DEV_N)33 print(f"{name} dev threshold reaching 0.90 accuracy not handed off: {'yes' if reached else 'no'}")34 print(f"{name} threshold chosen on dev: {threshold:.2f}")35 preds, confs = router_lib.classify(model, tokenizer, texts)36 router_lib.print_copy_report(name, preds, confs, golds, threshold)37 runs[name] = [p == g for p, g in zip(preds, golds)]3839# dict order is insertion order: the student first, then the control40student, control = runs.values()41n = len(golds)42print(f"test_200 messages the student gets right and the control gets wrong: {sum(s and not c for s, c in zip(student, control))}/{n}")43print(f"test_200 messages the control gets right and the student gets wrong: {sum(c and not s for s, c in zip(student, control))}/{n}")4445# Recipe sizes from the recorded manifests: the student and its control share rows and steps;46# the human-labelled adapter does not. Its manifest names its row count train_rows, not rows, which is why rows_key differs.47for name, folder, rows_key in [48 ("banking77-distilled-v1", "distill-student", "rows"),49 ("banking77-gold-1000-v1", "gold-same-rows", "rows"),50 ("banking77-lora-v1", "lora-full", "train_rows"),51]:52 settings = json.loads((RECORDED / folder / "manifest.json").read_text())["settings"]53 print(f"{name} training rows: {settings[rows_key]}")54 print(f"{name} training steps: {settings['steps']}")
python student-vs-gold-control.pydev messages used: 500 of 1003
banking77-distilled-v1 dev threshold reaching 0.90 accuracy not handed off: no
banking77-distilled-v1 threshold chosen on dev: 0.68
banking77-distilled-v1 accuracy on test_200: 69/200 (34.5%)
banking77-distilled-v1 handed off: 150/200 (75.0%)
banking77-distilled-v1 accuracy on messages not handed off: 28/50 (56.0%)
banking77-gold-1000-v1 dev threshold reaching 0.90 accuracy not handed off: no
banking77-gold-1000-v1 threshold chosen on dev: 0.97
banking77-gold-1000-v1 accuracy on test_200: 77/200 (38.5%)
banking77-gold-1000-v1 handed off: 197/200 (98.5%)
banking77-gold-1000-v1 accuracy on messages not handed off: 3/3 (100.0%)
test_200 messages the student gets right and the control gets wrong: 24/200
test_200 messages the control gets right and the student gets wrong: 32/200
banking77-distilled-v1 training rows: 1000
banking77-distilled-v1 training steps: 125
banking77-gold-1000-v1 training rows: 1000
banking77-gold-1000-v1 training steps: 125
banking77-lora-v1 training rows: 9000
banking77-lora-v1 training steps: 1125
The control also has no threshold that meets the dev target, so it uses the fallback too, and its threshold is provisional for that reason. It hands off 197/200 (98.5%) of test_200, and it is right on 3/3 (100.0%) of the messages it keeps. Three messages are not a measure. Overall accuracy needs no threshold, because it counts every message whatever the threshold, so the control's 77/200 (38.5%) sits directly beside the student's 69/200 (34.5%).
The fair comparison is on the answers themselves, and it needs no threshold, because it compares the teams. The control is right and the student wrong on 32/200 messages, and the student is right and the control wrong on 24/200. The control holds no margin over the student that this sample can settle, so we judge 200 messages too few to call one label source better than the other.
The adapter trained on 9000 rows for 1125 steps, while the control and the student share the 1000 rows and 125 steps, so the difference between the control and the adapter mixes row count and steps with anything else the two runs differ in, and this chapter does not separate them.
Working rule: Compare two adapters only on the same messages and the same rows. Count the messages each one gets right that the other gets wrong, and print both counts with their denominator. Do not set accuracies taken at different thresholds side by side as though they were the same measure.
When the teacher is wrong
A student trained on one label per message learns the labels it is given. Where the teacher is wrong on a row, the student has been shown the wrong team for that row, and it has no way to know otherwise. The teacher's agreement with gold on the training rows, measured above, is the share of the teacher's training labels that are right. The student is measured on test messages and not on its training rows, because its figures on rows it trained on would show what it memorised rather than what it learned.
On this task, the student that learns from the stand-in's labels is not the one to ship, as the figures above show. Chapter 7 quantizes banking77-lora-v1, the human-labelled adapter, and not the student.
Read the copy measure next to accuracy. If the student copies the teacher on most messages and the teacher is wrong on many, the student inherits those errors, and its accuracy against gold falls with the teacher's. If the student disagrees with the teacher on many messages and is still wrong, it is learning something other than the labels, and the first suspect is the number of training rows. That is a rule of thumb from this book's own figures, and this chapter does not test it.
Distilling is worth its cost in some cases. We recommend trying it when gold labels are scarce, when the teacher is strong on your task, and when a sample of gold labels shows that the teacher agrees with them often enough to make the student's errors acceptable. When you already have gold labels for your training rows, train on them, and use a teacher only to check them. A teacher model also changes. Treat a new version of the teacher as a new recording: its labels change, so the student's manifest, its threshold and its measures have to be made again.
In production the failure is quiet. A student that copies its teacher returns a steady stream of teams and a steady hand-off rate, and neither alarms anyone. Check the student against gold labels on a sample of messages that a person has labelled, on a fixed schedule. Chapter 9 describes the same kind of sample for the adapter's hand-offs as a next step, and this book does not build it. Compare the student's answers with those labels, not with the teacher's, because agreement with the teacher says nothing about the team a message belongs to.
The decisions this chapter made are these:
- The teacher labels training rows only, never dev or test rows.
- The teacher is the stand-in, Qwen3-4B-Instruct-2507, under Apache-2.0, and no hosted model is used.
- The student is trained with the same recipe as the human-labelled adapter, and its threshold follows the same rule, with the fallback reported as a fallback.
- The student is compared with a control trained on gold labels for the same rows, on the same messages, and the two share their rows, so a difference between them comes from the labels and not from the rows.
A teacher's labels carry its mistakes into the student, so agreement with gold decides whether distilling is worth it.
Working rule: Train a student on a teacher's labels only when you have too few gold labels for the task and a sample of gold labels shows that the teacher agrees with them well enough. Compare the student with the human-labelled adapter, and with the same recipe on gold labels, on the same messages, before you choose it. When the student loses, ship the adapter and say so in the write-up.
This chapter, in this edition
Edition 2026.11.9Examples passed Nov 20, 20263 of 3 examples in this chapter pass
This book is open to contributions, so anyone can check it or correct it. Found something wrong? Report it with the section's ID, the last part of the link beside each heading, or fix it yourself with a pull request.