더 큰 모델의 답을 작은 모델로 증류하기

읽는 데 약 35분분 걸린다, 6절에 3개 예제이(가) 있음.

2026.11.9판 예제 통과 2026년 11월 20 36개 예제 중 36개 통과 변경 사항

AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식

영어판 2026.11.9판을 번역한 것이다. 원문 읽기

이 장을 마치면 더 큰 교사 모델이 학습 행에 붙인 레이블로 작은 학생 모델을 학습시킬 수 있다. 또한 그 레이블이 정답 레이블과 얼마나 자주 일치하는지 측정할 수 있고, 같은 메시지를 바탕으로 정답 레이블로 학습시킨 학생 모델과 견주어 이 학생 모델을 쓸 만한지 판단할 수 있다.

라우터는 레이블이 붙은 메시지로부터 담당 팀의 이름을 학습한다. 이 장에서는 더 큰 모델이 학습 행의 레이블을 대신 붙이고, 작은 모델이 그 답을 보고 학습할 수 있는지를 묻는다. 정답 레이블은 BANKING77에서 가져온다. 5장에서는 학습 행의 정답 레이블로 헤드를 학습시켰으며, 그때 저장한 어댑터 banking77-lora-v1이 사람이 레이블을 붙인 어댑터다. 이 어댑터의 테스트 결과가 이 장 전체의 기준선이며, 임계값은 5장이 제시한 규칙에 따라 dev에서 고른 뒤 매니페스트에 기록된 값을 그대로 쓴다.

정답 레이블을 만드는 데는 사람의 시간이 든다. 증류는 이를 우회하려는 시도로, 더 큰 모델의 답이 정답 레이블을 대신한다. 여기서 쓰는 더 큰 모델은 대역 모델이다. 이 모델은 이 책의 라우터가 기반으로 삼는 기본 모델인 SmolLM2-135M-Instruct보다 크며, 라우터가 던지는 것과 같은 질문에 답한다.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

What distillation copies

마지막으로 변경된 판: 2026.11.9판

Distillation trains a small model to reproduce a large one. Hinton, Vinyals and Dean's paper, which gave distillation its name, proposed training a small model on the probabilities that a large model gives every class, as soft targets (Hinton et al., 2015, arXiv:1503.02531). Those probabilities carry more information than a single label, the paper argues, and a small model trained this way can often learn from far less data than the large model needs. That is the paper's own claim, from its own experiments, and it is not a rule about every task.

This chapter does not use soft targets. The stand-in returns one team for each message, the team it judges most likely, and the student sees only that team. Kim and Rush used the same simplification for translation, where they replaced the teacher's distribution with its most likely output: "Our simplest approximation is to replace the teacher distribution q with its mode" (Kim and Rush, 2016, arXiv:1606.07947). Their setting is translation, not classification. We borrow the simplification, and we measure it on this task instead of assuming it carries over. Here the teacher's greedy answer plays the part of the mode: it is the one team the teacher returns for each message.

Two things follow from training on one label per message. A student learns the teacher's errors along with its correct answers, because the training loop treats a wrong label as a label, the same as a right one. And the student's errors can differ from the teacher's, which is the reason to measure the student against gold labels as well as against the teacher.

Working rule: Before you train a student on a teacher's answers, measure how often the teacher's answer matches the gold label on rows that you have labelled, and read some of the disagreements. Expect the student to reproduce the teacher's errors, and count them on messages the student has not trained on.

약관이 허용하는 교사 모델

교사 모델은 약관이 그 출력으로 학습하는 것을 허용하는 모델이어야 한다. 이 책에서 대역 모델의 GGUF 빌드는 저장소 unsloth/Qwen3-4B-Instruct-2507-GGUF의 파일 Qwen3-4B-Instruct-2507-Q4_K_M.gguf이며, 교사 모델의 매니페스트에 기록된 리비전을 따른다. 모델은 Qwen이 만든 Qwen3-4B-Instruct-2507이고, 빌드는 unsloth가 만들었다. 우리는 두 곳을 모두 밝힌다. 이 빌드의 모델 카드는 license: apache-2.0를 선언하고 기반 모델로 Qwen/Qwen3-4B-Instruct-2507를 명시한다(고정된 리비전의 모델 카드). 기반 모델의 라이선스는 고정된 커밋에 있으며 Apache License 2.0이다. Apache License 2.0은 두 번째 절에서 "저작물과 그 파생 저작물을 복제하고, 파생 저작물을 작성하고, 공개적으로 전시하고, 공개적으로 실연하고, 재허락하고, 배포할 권리"를 부여한다(Apache License 2.0).

우리는 Apache-2.0이 모델의 출력을 사용하는 데 아무런 제한도 두지 않는 것으로 해석한다. 이 라이선스의 본문은 위의 2절이 밝힌 대로 저작물과 그 파생 저작물을 규율하며, 출력은 언급하지 않는다. 이는 라이선스 본문의 인용이 아니라 우리의 해석이며, 이 모델의 라이선스에만 해당한다. 이 책이 제공하는 어댑터는 Apache-2.0으로 배포되며, 어댑터 옆의 고지문에 라이선스와 출처 표기가 있다.

라이선스는 서로 다르다. Apache-2.0 같은 오픈 웨이트 라이선스는 모델 파일을 규율한다. 호스팅되는 모델은 제공자의 서비스 약관이 규율하며, 이 약관은 서비스나 그 출력을 경쟁 모델의 학습에 쓰는 것을 금지할 수 있다. 약관은 바뀌므로, 학습에 쓰려는 모델의 현재 약관을 사용하는 날에 읽어야 한다. 이 책은 호스팅되는 모델을 교사 모델로 쓰지 않는다.

대역 모델은 오픈 모델이므로 라이선스를 읽을 수 있고, 파일을 해시로 고정할 수 있다. 이 해시는 교사 모델의 매니페스트에 기록된다. 호스팅되는 모델은 제공자가 모델을 바꾸면 답이 달라질 수 있고, 약관이 이러한 사용을 금지할 수도 있다. 이것들은 약관을 읽기 전이라도 파일이 고정된 오픈 교사 모델을 선호할 이유가 된다.

작업 규칙: 학습하는 날에 교사 모델의 라이선스와 이용 약관을 읽는다. 장 안에 교사 모델과 그 라이선스, 리비전을 명시하고, 학생 모델의 매니페스트가 교사 모델의 기록을 가리키게 한다. 약관이 금지하는 모델의 출력으로는 학습하지 않는다. 출력에 제한을 두지 않는 라이선스는 그 라이선스에 대해서만 질문의 답이 되므로, 다른 모델의 답을 짐작하기 전에 그 본문을 읽는다.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Label training rows with the teacher

마지막으로 변경된 판: 2026.11.2판

The teacher labels training rows only. Dev rows are kept for choosing thresholds, and test rows are kept for reporting, as chapter 2 set out. The rows to label are drawn from the training file, outside dev. The labelled rows are the first 1,000 of a seeded random draw of 2,000 such rows, a random subset of them, and the recording sorts them by index before labelling. The labelling run was planned for all 2,000, but it could not finish that many within its time limit, so the book reports the 1,000 rows it labelled.

The draw is seeded. Labels already saved for a chosen row are kept and not labelled again, so a run that stops and continues adds only the rows it has not labelled. We did not test whether a resumed run gives the same labels as one full run.

The stand-in answers each message with the router prompt from chapter 1. It decodes greedily, at temperature zero. In a probe on 40 test messages, two runs gave identical answers, and that probe is the only check we made of repeatability. Its answer is constrained by a grammar to the 77 team names, so each label is a real team.

teacher-agreement.py Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0 통과
"""Measure how often the teacher's label matches gold on the training rows it labelled, and its accuracy on test_200from its recorded answers. The point rests on the agreement line; the example reads recorded answers only."""import osos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import json  # noqa: E402import torch  # noqa: E402torch.set_num_threads(2)import router_lib  # noqa: E402router_lib.setup_runtime()RECORDED = router_lib.HERE / "data" / "recorded"train = router_lib.read_rows("train")teacher_rows = [json.loads(line) for line in (RECORDED / "teacher-labels" / "labels.jsonl").read_text().splitlines()]# row["index"] is the row's position in the training file; field 1 of a row is its gold teamagree = sum(row["label"] == train[row["index"]][1] for row in teacher_rows)n = len(teacher_rows)print(f"training rows labelled by the teacher: {n}")print(f"teacher agrees with gold on training rows: {agree}/{n} ({agree / n:.1%})")print(f"teacher disagrees with gold on training rows: {n - agree}/{n} ({(n - agree) / n:.1%})")test = router_lib.read_rows("test")sample = router_lib.read_sample("test_200")standin = router_lib.standin_answers()# sample holds test row positions; field 1 of a test row is its gold team, field 0 its textgolds = [test[i][1] for i in sample]teacher_preds = [standin[i] for i in sample]correct = sum(p == g for p, g in zip(teacher_preds, golds))print(f"messages on test_200 with a teacher answer: {sum(i in standin for i in sample)}/{len(sample)}")print(f"teacher accuracy on test_200: {correct}/{len(sample)} ({correct / len(sample):.1%})")
출력python teacher-agreement.py
training rows labelled by the teacher: 1000
teacher agrees with gold on training rows: 549/1000 (54.9%)
teacher disagrees with gold on training rows: 451/1000 (45.1%)
messages on test_200 with a teacher answer: 200/200
teacher accuracy on test_200: 109/200 (54.5%)

The teacher's label matches the gold label on 549/1000 (54.9%) of the 1000 training rows it labelled, and it disagrees on 451/1000 (45.1%). A student trained on those disagreeing rows learns the wrong label as the answer to each one. The last two lines of the output come from the stand-in's recorded answers on test_200, the 200 test messages this chapter reports on. There the teacher is right on 109/200 (54.5%) messages.

Read the line teacher agrees with gold on training rows before any student is trained. The line teacher accuracy on test_200 is the teacher's own accuracy on messages no one has used for training, and it is the reference for every copy measure below.

Counts do not show what the disagreements look like. Three training rows, each labelled card_arrival in the gold file, show the pattern. Each message is about a card that has not arrived, described here without its text, since messages are never quoted in this book. The teacher named card_delivery_estimate for two of them and lost_or_stolen_card for the third, so the teacher names a different team from the gold label each time.

Working rule: Measure the teacher's agreement with gold on a labelled sample of rows before you train a student on the labels. Label only rows outside dev and test, and record the teacher's model, its version and its settings in the teacher's recording manifest, and point the student's manifest to it, so each label can be traced.

교사의 레이블로 학생 모델 학습하기

마지막으로 변경된 판: 2026.11.2판

학생 모델은 5장의 레시피를 사용한다. 기반 모델, LoRA 대상, 헤드, 시드, 배치 크기가 모두 같다. 다만 레이블은 정답 파일이 아니라 교사 모델에서 가져온다. 행 수와 스텝 수도 5장과 다르다. 학생 모델은 1000개 행으로 125스텝 동안 학습하고, 사람이 레이블을 붙인 어댑터는 9000개 행으로 1125스텝 동안 학습했다. 일치도 절의 대조 실험은 학생 모델의 행 수와 스텝 수를 그대로 두고 레이블만 바꾼다. 학생 모델의 어댑터는 banking77-distilled-v1이다. 이 어댑터의 매니페스트에는 설정이 기록되어 있고, 교사 모델의 기록인 teacher-labels/manifest.json을 가리킨다. 이 기록에는 교사 모델의 이름과 버전, 설정, 레이블을 붙인 행이 적혀 있으므로, 누구나 각 레이블이 어디에서 왔는지 추적할 수 있다.

레시피는 의도적으로 고정한다. 학생 모델이 레시피와 레이블을 동시에 바꾸면 결과의 차이가 어느 쪽에서 왔는지 독자가 알 수 없다. 레시피를 고정하면 결과가 가리킬 수 있는 차이는 레이블뿐이다.

학생 모델의 임계값은 사람이 레이블을 붙인 어댑터와 같은 규칙으로 dev에서 고른다. 이 규칙은 100분의 1 간격의 격자에서, 넘기지 않은 메시지의 dev 정확도가 0.90 이상이 되는 가장 낮은 임계값을 택한다. 이 목표에 도달하지 못하는 학생 모델을 위해 규칙에는 대체 절차가 있다. 대체 절차는 넘기지 않은 메시지에서 정확도가 가장 높은 임계값을 택하고, 동률이면 가장 낮은 값을 택하며, 모든 메시지를 넘기는 임계값은 제외한다. 출력에는 대체 절차를 사용했다고 표시된다. 이것은 목표에 못 미친 결과를 다루는 이 책의 규칙이므로, 목표를 달성한 학생 모델이 아니라 대체 절차를 쓴 결과로 보고한다. 대체 임계값은 dev에서 그 임계값이 남기는 메시지로 고르는데, 임계값이 높으면 그 메시지가 소수에 그칠 수 있다. 대체 값은 잠정적인 값으로 다루고, 이 장에서 하듯이 test에서 그 값이 치르는 인계 비율과 함께 보고한다.

학생 모델의 임계값은 dev 분할의 일부인 500 of 1003개 메시지로 고른다. dev 메시지 전체를 돌리는 작업은 CPU 코어 두 개의 시간 예산에 맞지 않았기 때문이다. 학생 모델 예제 두 개는 모두 같은 부분을 사용하고, 대조 실험도 자신의 임계값을 고를 때 그 부분을 사용하므로, 두 임계값은 같은 dev 메시지에서 고른 것이다. 5장에서 사람이 레이블을 붙인 어댑터의 임계값은 dev 메시지 1,003개 전체에서 골랐다.

작업 규칙: 학생 모델의 임계값은 사람이 레이블을 붙인 어댑터와 같은 규칙으로 dev에서 고르고, 목표에 도달하는 임계값이 없으면 그 임계값이 대체 값이라고 밝힌다. 학생 모델을 다른 무엇과 비교하기 전에, 그 임계값이 치르는 인계 비율을 정확도 옆에 보고하되, 두 비율의 개수를 함께 밝힌다.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Measure agreement, not only accuracy

마지막으로 변경된 판: 2026.11.9판

Three measures run through this chapter, and each answers its own question. The teacher against gold on the training rows asks whether the teacher is right on the rows it labels, and it is the one above. The student against the teacher on test_200 asks how much of the teacher the student copies. The student against gold on test_200 asks how often the student names the right team, which is the figure a router would be judged on. Each is printed with its denominator: the first is read on the 1,000 training rows, and the other two on the same 200 test messages.

Agreement and accuracy answer different questions. Agreement with the teacher says how much of the teacher the student has learned, whether the teacher is right or wrong. Accuracy against gold says how often the student names the right team. A student can agree with a wrong teacher and be wrong, and it can disagree with a wrong teacher and be right. Printing both lets a reader see which of the two is happening, instead of inferring it from one figure.

student-recorded-adapter.py Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0 통과
"""Score the distilled student banking77-distilled-v1 on test_200 beside the human-labelled adapter, with the student'sthreshold chosen on dev. The point rests on the student's accuracy line and its copy lines; both adapters are loadedand scored live, and nothing is trained."""import osos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import json  # noqa: E402import torch  # noqa: E402torch.set_num_threads(2)import router_lib  # noqa: E402router_lib.setup_runtime()RECORDED = router_lib.HERE / "data" / "recorded"tokenizer, student = router_lib.load_classifier(RECORDED / "distill-student" / "adapter")DEV_N = 500  # the first 500 of the 1,003 dev messages: the run's time budget under two CPUs# reached is False when no threshold meets the target, and the threshold returned is then the fallbackthreshold, reached = router_lib.dev_threshold(student, tokenizer, target=0.90, n=DEV_N)print(f"dev messages used: {DEV_N} of {len(router_lib.read_sample('dev'))}")print(f"dev threshold reaching 0.90 accuracy not handed off: {'yes' if reached else 'no'}")print(f"student threshold chosen on dev: {threshold:.2f}")test = router_lib.read_rows("test")sample = router_lib.read_sample("test_200")# sample holds test row positions; field 1 of a test row is its gold team, field 0 its textgolds = [test[i][1] for i in sample]texts = [test[i][0] for i in sample]student_preds, student_confs = router_lib.classify(student, tokenizer, texts)router_lib.print_copy_report("banking77-distilled-v1", student_preds, student_confs, golds, threshold)# chapter 5 chose the adapter's threshold on dev, and its manifest records the value used herelora = json.loads((RECORDED / "lora-full" / "manifest.json").read_text())_, lora_model = router_lib.load_classifier(RECORDED / "lora-full" / "adapter")lora_preds, lora_confs = router_lib.classify(lora_model, tokenizer, texts)router_lib.print_copy_report("banking77-lora-v1", lora_preds, lora_confs, golds, lora["handoff_threshold"])teacher = router_lib.standin_answers()teacher_preds = [teacher[i] for i in sample]matches = sum(s == t for s, t in zip(student_preds, teacher_preds))# positions within sample where the student's answer is not the gold teamwrong = [k for k in range(len(sample)) if student_preds[k] != golds[k]]wrong_like_teacher = sum(student_preds[k] == teacher_preds[k] for k in wrong)print(f"student answers equal to teacher answers on test_200: {matches}/{len(sample)} ({matches / len(sample):.1%})")print(f"student errors on test_200: {len(wrong)}/{len(sample)} ({len(wrong) / len(sample):.1%})")print(f"student errors equal to the teacher's answer: {wrong_like_teacher}/{len(wrong)}")
출력python student-recorded-adapter.py
dev messages used: 500 of 1003
dev threshold reaching 0.90 accuracy not handed off: no
student threshold chosen on dev: 0.68
banking77-distilled-v1 accuracy on test_200: 69/200 (34.5%)
banking77-distilled-v1 handed off: 150/200 (75.0%)
banking77-distilled-v1 accuracy on messages not handed off: 28/50 (56.0%)
banking77-lora-v1 accuracy on test_200: 172/200 (86.0%)
banking77-lora-v1 handed off: 33/200 (16.5%)
banking77-lora-v1 accuracy on messages not handed off: 156/167 (93.4%)
student answers equal to teacher answers on test_200: 100/200 (50.0%)
student errors on test_200: 131/200 (65.5%)
student errors equal to the teacher's answer: 36/131

The student is well behind the human-labelled adapter, and the gap is not only in hand-offs, as the overall figures below show. On test_200, the student is right on 69/200 (34.5%) messages and hands off 150/200 (75.0%), while the adapter is right on 172/200 (86.0%) messages and hands off 33/200 (16.5%). The student's threshold is the fallback described above. Each run's accuracy on the messages it keeps is at its own threshold, and the output gives both beside this paragraph.

The copy measures come next. The student's answer equals the teacher's on 100/200 (50.0%) messages, and the teacher itself is right on 109/200 (54.5%). A student that copies the teacher closely therefore copies the teacher's wrong answers too. The student makes 131/200 (65.5%) errors on test_200. Of those errors, 36/131 repeat the teacher's answer. The rest differ from it, so a student's mistakes are not only the teacher's mistakes.

The comparison with the adapter alone mixes two things. The adapter learned from the gold labels of the full training split, and the student learned from the teacher's labels on a smaller set of rows. A gap between them could come from the labels, from the number of rows or from both. To separate them, the book records the same recipe on the same rows with gold labels. The control, banking77-gold-1000-v1, is the student's recipe trained on gold labels, and its rows are the rows the teacher labelled.

student-vs-gold-control.py Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0 통과
"""Put the student banking77-distilled-v1 beside its control banking77-gold-1000-v1, which has the same recipe and thesame 1,000 training rows but gold labels, on the same test_200 messages. The point rests on the two counts of messageseach adapter gets right that the other gets wrong, which need no threshold; both adapters are scored live, and nothingis trained."""import jsonimport osos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import torch  # noqa: E402torch.set_num_threads(2)import router_lib  # noqa: E402router_lib.setup_runtime()RECORDED = router_lib.HERE / "data" / "recorded"DEV_N = 500  # the first 500 of the dev messages, as student-recorded-adapter uses: the time budget under two CPUsprint(f"dev messages used: {DEV_N} of {len(router_lib.read_sample('dev'))}")test = router_lib.read_rows("test")sample = router_lib.read_sample("test_200")# sample holds test row positions; field 1 of a test row is its gold team, field 0 its textgolds = [test[i][1] for i in sample]texts = [test[i][0] for i in sample]runs = {}for name, folder in [("banking77-distilled-v1", "distill-student"), ("banking77-gold-1000-v1", "gold-same-rows")]:    tokenizer, model = router_lib.load_classifier(RECORDED / folder / "adapter")    threshold, reached = router_lib.dev_threshold(model, tokenizer, target=0.90, n=DEV_N)    print(f"{name} dev threshold reaching 0.90 accuracy not handed off: {'yes' if reached else 'no'}")    print(f"{name} threshold chosen on dev: {threshold:.2f}")    preds, confs = router_lib.classify(model, tokenizer, texts)    router_lib.print_copy_report(name, preds, confs, golds, threshold)    runs[name] = [p == g for p, g in zip(preds, golds)]# dict order is insertion order: the student first, then the controlstudent, control = runs.values()n = len(golds)print(f"test_200 messages the student gets right and the control gets wrong: {sum(s and not c for s, c in zip(student, control))}/{n}")print(f"test_200 messages the control gets right and the student gets wrong: {sum(c and not s for s, c in zip(student, control))}/{n}")# Recipe sizes from the recorded manifests: the student and its control share rows and steps;# the human-labelled adapter does not. Its manifest names its row count train_rows, not rows, which is why rows_key differs.for name, folder, rows_key in [    ("banking77-distilled-v1", "distill-student", "rows"),    ("banking77-gold-1000-v1", "gold-same-rows", "rows"),    ("banking77-lora-v1", "lora-full", "train_rows"),]:    settings = json.loads((RECORDED / folder / "manifest.json").read_text())["settings"]    print(f"{name} training rows: {settings[rows_key]}")    print(f"{name} training steps: {settings['steps']}")
출력python student-vs-gold-control.py
dev messages used: 500 of 1003
banking77-distilled-v1 dev threshold reaching 0.90 accuracy not handed off: no
banking77-distilled-v1 threshold chosen on dev: 0.68
banking77-distilled-v1 accuracy on test_200: 69/200 (34.5%)
banking77-distilled-v1 handed off: 150/200 (75.0%)
banking77-distilled-v1 accuracy on messages not handed off: 28/50 (56.0%)
banking77-gold-1000-v1 dev threshold reaching 0.90 accuracy not handed off: no
banking77-gold-1000-v1 threshold chosen on dev: 0.97
banking77-gold-1000-v1 accuracy on test_200: 77/200 (38.5%)
banking77-gold-1000-v1 handed off: 197/200 (98.5%)
banking77-gold-1000-v1 accuracy on messages not handed off: 3/3 (100.0%)
test_200 messages the student gets right and the control gets wrong: 24/200
test_200 messages the control gets right and the student gets wrong: 32/200
banking77-distilled-v1 training rows: 1000
banking77-distilled-v1 training steps: 125
banking77-gold-1000-v1 training rows: 1000
banking77-gold-1000-v1 training steps: 125
banking77-lora-v1 training rows: 9000
banking77-lora-v1 training steps: 1125

The control also has no threshold that meets the dev target, so it uses the fallback too, and its threshold is provisional for that reason. It hands off 197/200 (98.5%) of test_200, and it is right on 3/3 (100.0%) of the messages it keeps. Three messages are not a measure. Overall accuracy needs no threshold, because it counts every message whatever the threshold, so the control's 77/200 (38.5%) sits directly beside the student's 69/200 (34.5%).

The fair comparison is on the answers themselves, and it needs no threshold, because it compares the teams. The control is right and the student wrong on 32/200 messages, and the student is right and the control wrong on 24/200. The control holds no margin over the student that this sample can settle, so we judge 200 messages too few to call one label source better than the other.

The adapter trained on 9000 rows for 1125 steps, while the control and the student share the 1000 rows and 125 steps, so the difference between the control and the adapter mixes row count and steps with anything else the two runs differ in, and this chapter does not separate them.

Working rule: Compare two adapters only on the same messages and the same rows. Count the messages each one gets right that the other gets wrong, and print both counts with their denominator. Do not set accuracies taken at different thresholds side by side as though they were the same measure.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

When the teacher is wrong

마지막으로 변경된 판: 2026.11.9판

A student trained on one label per message learns the labels it is given. Where the teacher is wrong on a row, the student has been shown the wrong team for that row, and it has no way to know otherwise. The teacher's agreement with gold on the training rows, measured above, is the share of the teacher's training labels that are right. The student is measured on test messages and not on its training rows, because its figures on rows it trained on would show what it memorised rather than what it learned.

On this task, the student that learns from the stand-in's labels is not the one to ship, as the figures above show. Chapter 7 quantizes banking77-lora-v1, the human-labelled adapter, and not the student.

Read the copy measure next to accuracy. If the student copies the teacher on most messages and the teacher is wrong on many, the student inherits those errors, and its accuracy against gold falls with the teacher's. If the student disagrees with the teacher on many messages and is still wrong, it is learning something other than the labels, and the first suspect is the number of training rows. That is a rule of thumb from this book's own figures, and this chapter does not test it.

Distilling is worth its cost in some cases. We recommend trying it when gold labels are scarce, when the teacher is strong on your task, and when a sample of gold labels shows that the teacher agrees with them often enough to make the student's errors acceptable. When you already have gold labels for your training rows, train on them, and use a teacher only to check them. A teacher model also changes. Treat a new version of the teacher as a new recording: its labels change, so the student's manifest, its threshold and its measures have to be made again.

In production the failure is quiet. A student that copies its teacher returns a steady stream of teams and a steady hand-off rate, and neither alarms anyone. Check the student against gold labels on a sample of messages that a person has labelled, on a fixed schedule. Chapter 9 describes the same kind of sample for the adapter's hand-offs as a next step, and this book does not build it. Compare the student's answers with those labels, not with the teacher's, because agreement with the teacher says nothing about the team a message belongs to.

The decisions this chapter made are these:

  • The teacher labels training rows only, never dev or test rows.
  • The teacher is the stand-in, Qwen3-4B-Instruct-2507, under Apache-2.0, and no hosted model is used.
  • The student is trained with the same recipe as the human-labelled adapter, and its threshold follows the same rule, with the fallback reported as a fallback.
  • The student is compared with a control trained on gold labels for the same rows, on the same messages, and the two share their rows, so a difference between them comes from the labels and not from the rows.

A teacher's labels carry its mistakes into the student, so agreement with gold decides whether distilling is worth it.

Working rule: Train a student on a teacher's labels only when you have too few gold labels for the task and a sample of gold labels shows that the teacher agrees with them well enough. Compare the student with the human-labelled adapter, and with the same recipe on gold labels, on the same messages, before you choose it. When the student loses, ship the adapter and say so in the write-up.

이 판의 이 장

2026.11.9판예제 통과 2026년 11월 20이 장에서 3개 예제개 중 3개가 통과한다

영어판 2026.11.9판을 번역한 것이다.

이 절 인용

형식

각 형식은 이 번역이 따르는 영문 원서를 출처로 표기한다.

모든 형식은 이 판을 명시한다. 링크는 현재 판을 열며, 여기서 절의 제목은 본문이 마지막으로 바뀐 시점을 알려 준다. 접근 날짜와 검색 날짜는 이 자료를 여는 날짜다.

이 페이지의 내용

10장 중 6장: 더 큰 모델의 답을 작은 모델로 증류하기

  1. What distillation copies
  2. 약관이 허용하는 교사 모델
  3. Label training rows with the teacher
  4. 교사의 레이블로 학생 모델 학습하기
  5. Measure agreement, not only accuracy
  6. When the teacher is wrong

맨 위로 돌아가기전체 장