From predicting to following instructions

읽는 데 약 40분분 걸린다, 10절에 9개 예제이(가) 있음.

2026.11.16판 예제 통과 2026년 11월 19 86개 예제 중 86개 통과 변경 사항

AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식

영어판 2026.11.16판을 번역한 것이다. 원문 읽기

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

By the end of this chapter you can explain how a model that predicts the next token becomes one that follows requests, build the two training losses that do it, and measure what that change does and doesn't fix. You'll compare a base model with its instruction-tuned version on the same requests, measure how often each makes up a product detail, check whether its confidence warns you when it does, and find where in a longer context a model stops finding what it was given.

The furniture and home-goods store's product names and descriptions are the text this book models, and every model in the book so far has done one thing: predict the next token of text like its training text. Chapter 10 read SmolLM2-360M, an open model trained that way on general text, and found it scored the store's names far worse, per word, than the store's own small model, and its descriptions worse. People don't want a continuation of their text; they want an answer to their request. Models are taught that in two further steps of training.

Instruction tuning, or supervised fine-tuning, continues training a model on examples of requests paired with good responses, so that predicting the next token of such a pair means answering the request. Wei and colleagues found that tuning on many tasks phrased as instructions made a model better at tasks it had never seen (Wei et al., 2022). Preference tuning then trains a model to prefer the better of two responses to the same request. Ouyang and colleagues had people rank the responses, and trained with reinforcement learning from a reward model learned from those rankings (Ouyang et al., 2022); Direct Preference Optimization, DPO, does preference tuning without a reward model, with a simple loss on the preference pairs themselves (Rafailov et al., 2023).

SmolLM2-360M-Instruct is the base model of chapter 10 after both steps: supervised fine-tuning on public and curated datasets, then DPO on UltraFeedback (model card). UltraFeedback's preference pairs were scored by a large language model, not by people (Cui et al., 2023). This chapter calls SmolLM2-360M the base model and SmolLM2-360M-Instruct the tuned model. The script recordings/instruction_tuning.py ran both once, at pinned revisions, decoding greedily, so each answer is the model's likeliest and the run can be repeated exactly, and saved what they wrote and how likely they found each token it wrote. Every measure of the two models comes from those files.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Build the two training losses by hand

마지막으로 변경된 판: 2026.11.16판

Both steps of tuning are training, with the loop of chapter 8; what changes is the loss. Instruction tuning uses the ordinary next-token loss with one difference: only the response's tokens count. The request is read, as context, but the model isn't trained to predict it, since the point is to answer requests, not to write them.

Lay one product out as a request and a response, its name as the request and its description as the response, the way the store's model already reads it. Then compute that model's loss over every token, and over the response's tokens only:

instruction-loss-by-hand.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""Computes the store's model's loss on one product laid out as a request and a response, over every token and overthe response's tokens only. The prose's point rests on the last printed line, the average loss over the response only."""import numpy as npfrom lib import transformerfrom lib.store import load_products_, vocabulary, manifest = transformer.load()index = {token: i for i, token in enumerate(vocabulary)}# the name is unique in the validation split, so its one row is the productproduct = load_products("validation").query("name == 'double chaise patio furniture cover'").iloc[0]# the text is cut at the model's longest length, so a long description may end earlyids = transformer.product_ids(product["name"], product["description"], index, manifest["settings"]["longest"])sep = ids.index(index[transformer.SEP])           # the name plays the request; the description, the response# the last position predicts nothing, since no token follows it, so its row is droppedprobabilities = transformer.softmax(transformer.forward(ids)[:-1])# position i predicts the token at i + 1, so ids[1:] holds each position's target# each token's loss, given the tokens before itloss = -np.log(probabilities[np.arange(len(ids) - 1), ids[1:]])response = np.arange(1, len(ids)) > sep                             # the description's tokens and its endprint("tokens predicted in the text:", len(loss))print("tokens the instruction loss counts, the response's:", int(response.sum()))print("average loss over every token:", f"{loss.mean():.3f}")print("average loss over the response only:", f"{loss[response].mean():.3f}")
출력python instruction-loss-by-hand.py
tokens predicted in the text: 96
tokens the instruction loss counts, the response's: 90
average loss over every token: 2.267
average loss over the response only: 2.132

Training on the loss over the response alone, across many requests and responses, teaches a model to produce responses. For this product that loss is 2.132, against 2.267 over every token. The instruction loss counts 90 of the 96 tokens the model predicts: the description's tokens and its end.

DPO's loss works on a pair of responses to one request, one preferred and one rejected. DPO compares how much more likely the model being tuned makes each response than the reference model it started from, the model after instruction tuning, and rewards it for raising the preferred one more. With log-probabilities log π of each response under the tuned model and log π_ref under the reference, and a factor β that sets how far the model may move:

margin = β × [(log π(preferred) − log π_ref(preferred)) − (log π(rejected) − log π_ref(rejected))]

loss = −log σ(margin)

with σ the logistic function from chapter 4 (Rafailov et al., 2023, equation 7). Made-up log-probabilities keep the arithmetic in plain view, so this example uses them, with β = 0.1:

dpo-loss-by-hand.py Python 3.12.15 · numpy 2.5.5 통과
"""Computes the DPO margin and loss by hand for two made-up cases, with a fixed beta. The prose's point rests on thetwo loss lines, one for each case."""import numpy as npbeta = 0.1                                       # how far the tuned model may move from the referenceprint("beta:", beta)# Each case: (preferred, rejected) under the tuned model, then (preferred, rejected) under the reference model it# started from. Made-up numbers, to follow the arithmetic.cases = {"tuned model favours the preferred answer more than the reference does": (-12.0, -20.0, -14.0, -18.0),         "tuned model favours it less": (-15.0, -15.0, -14.0, -18.0)}for label, (preferred, rejected, preferred_reference, rejected_reference) in cases.items():    margin = beta * ((preferred - preferred_reference) - (rejected - rejected_reference))    loss = np.log1p(np.exp(-margin))             # -log σ(margin), written so it can't overflow    print(f"{label}, margin:", f"{margin:.2f}")    print(f"{label}, loss:", f"{loss:.3f}")
출력python dpo-loss-by-hand.py
beta: 0.1
tuned model favours the preferred answer more than the reference does, margin: 0.40
tuned model favours the preferred answer more than the reference does, loss: 0.513
tuned model favours it less, margin: -0.40
tuned model favours it less, loss: 0.913

Lowering the loss pushes the preferred response's probability up and the rejected one's down, measured against where the model started. When the tuned model favours the preferred response more than the reference does, the margin is 0.40 and the loss 0.513; when it favours it less, the margin is -0.40 and the loss rises to 0.913.

Working rule: Mask the request out of the loss when you tune a model on requests and responses, and keep the reference model's log-probabilities beside the tuned model's when you train on preferences. Both are small changes to a loop you already have.

기반 모델과 튜닝된 모델에 물어보기

마지막으로 변경된 판: 2026.11.11판

상점에는 설명이 없는 상품들의 설명이 필요하다. 이는 9장이 자체의 작은 모델에 맡긴 작업이다. 기반 모델인 SmolLM2-360M은 요청을 받을 수 없고 이어 쓰기만 할 수 있으므로, 상품 목록의 첫 부분인 Product:, 상품 이름, Description:를 입력으로 받는다. 튜닝된 모델인 SmolLM2-360M-Instruct에는 Write a short product description for:와 상품 이름으로 이루어진 요청을 준다.

두 모델 모두 9장의 상품인 front porch welcome sign wall décor에서 시작한다. 각 모델이 쓴 글을 읽은 다음, 기록에서 설명된 8개 상품 전체를 측정한다. 이 상품들에는 참고할 설명이 하나도 없다.

base-and-tuned.py Python 3.12.15 통과
"""Prints what the base and the tuned model wrote for one product with no description, then measures every such product'stext for materials its name doesn't give and for repeated four-word phrases. The prose's points rest on the last fourprinted lines, the two models' invented-material counts and repeat shares."""from collections import Counterfrom lib import claims, recorded# the recording's saved texts for every product, read back rather than generated againdescribed = recorded.read("instruction-tuning", "descriptions")product = next(row for row in described if row["name"] == "front porch welcome sign wall décor")print("product:", product["name"])print("base model:", " ".join(product["base"]["text"].split()))print("tuned model:", " ".join(product["tuned"]["text"].split()))def repeats(text):    """The share of four-word phrases that repeat an earlier one in the text."""    words = text.lower().split()    # every window of four consecutive words, so a text of n words has n - 3 of them    phrases = [tuple(words[i:i + 4]) for i in range(len(words) - 3)]    # a phrase counts as repeated each time after its first appearance; max keeps a text of fewer than four words from dividing by zero    return sum(count - 1 for count in Counter(phrases).values()) / max(len(phrases), 1)print("products described, none of which has a description to draw on:", len(described))for role in ["base", "tuned"]:    texts = [row[role]["text"] for row in described]    # materials the text names that the product's name doesn't give: a set difference, not the text's whole list    invented = sum(bool(claims.materials(row[role]["text"]) - claims.materials(row["name"])) for row in described)    print(f"{role} model, descriptions that name a material the name doesn't:", f"{invented} of {len(described)}")    print(f"{role} model, repeated four-word phrases:", f"{sum(map(repeats, texts)) / len(texts):.0%}")
출력python base-and-tuned.py
product: front porch welcome sign wall décor
base model: Front porch welcome sign wall décor The front porch is a common feature of many homes. It is a place where people can relax and enjoy the outdoors. It is also a place where people can socialize and meet new friends. The front porch is a great place to hang out with your family
tuned model: "Welcome to our front porch welcome sign wall décor, a space that brings warmth and comfort to your home. This wall is a statement piece that showcases your personality and style. With a mix of rustic and modern elements, you can create a unique and inviting atmosphere that reflects your individuality.
products described, none of which has a description to draw on: 8
base model, descriptions that name a material the name doesn't: 2 of 8
base model, repeated four-word phrases: 14%
tuned model, descriptions that name a material the name doesn't: 1 of 8
tuned model, repeated four-word phrases: 0%

기반 모델은 현관에 대해 쓰고(The front porch is a common feature of many homes), 이어서 휴식과 친구와의 만남에 대해 쓴다. 이 모델은 그럴듯한 방향으로 글을 이어 가지만, 그 방향은 상점이 필요로 하는 방향이 아니다. 튜닝된 모델은 상품 설명의 형태를 갖춘 글을 쓰고(a statement piece that showcases your personality and style), 그러면서 이 표지판을 a space이라고 부르고 This wall이라고도 부른다. 이해하지 못하는 상품에 대해 유창하게 쓰는 것이다. 전체 상품에서 기반 모델의 글은 네 단어 구절의 14%가 반복되고, 튜닝된 모델의 글은 0%가 반복된다. 튜닝의 어느 단계가 반복을 멈추게 했는지는 기록만으로는 알 수 없다.

두 모델 모두 상품에 대해 이름 이상의 것은 알지 못한다. 이름에 없는 재료를 언급한 설명은 기반 모델이 2 of 8개, 튜닝된 모델이 1 of 8개였으며, 이 재료들은 모두 9장의 경우처럼 상점의 데이터가 아니라 모델에서 나온 것이다.

작업 규칙: 요청을 따르게 해야 할 때는 튜닝된 모델을 쓰고, 앞선 글의 문체로 글을 이어 써야 할 때는 기반 모델을 쓴다. 유창함을 지식으로 읽지 않는다. 튜닝된 모델은 같은 사실 부족 상태에서 더 잘 짜인 형태의 설명을 쓸 뿐이다.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Format the request as the model was tuned

마지막으로 변경된 판: 2026.11.16판

The tuned model answered in the previous section because its request came in the layout it was tuned on. A tuned model's chat template is the layout it learned: special tokens that mark where a system message, instructions placed before the user's request, the request itself and the model's reply begin and end. The model learned to answer requests only in that layout, so a request in any other form is only text to continue. Libraries that load the model apply the template from the model's tokenizer files (tokenizer configuration).

The tuned model's full prompt, and its reply beside the one it gives to the same request as plain text, show what it actually read:

chat-template.py Python 3.12.15 통과
"""Prints the full prompt the tuned model read, with its chat template, and its reply with and without the template. Theprose's point rests on the last two printed lines: the same words, with and without the template."""from lib import recordedtemplate = recorded.read("instruction-tuning", "template")# the request is the text between the user marker and its end markerprint("request:", template["with"]["prompt"].split("<|im_start|>user\n")[1].split("<|im_end|>")[0])print("what the tuned model reads, with its chat template:")print(template["with"]["prompt"])print("tuned model, with the template:", " ".join(template["with"]["text"].split()))print("tuned model, without it:", " ".join(template["without"]["text"].split()))
출력python chat-template.py
request: Write a short product description for: front porch welcome sign wall décor
what the tuned model reads, with its chat template:
<|im_start|>system
You are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>
<|im_start|>user
Write a short product description for: front porch welcome sign wall décor<|im_end|>
<|im_start|>assistant

tuned model, with the template: "Welcome to our front porch welcome sign wall décor, a space that brings warmth and comfort to your home. This wall is a statement piece that showcases your personality and style. With a mix of rustic and modern elements, you can create a unique and inviting atmosphere that reflects your individuality.
tuned model, without it: .

Tuning taught the model to follow requests in one layout, not requests in general. With the template, the request sits between <|im_start|>user and <|im_end|>, after a system message the template adds by default, You are a helpful AI assistant named SmolLM, trained by Hugging Face, and before <|im_start|>assistant, where the model's reply begins. The model replies with the description of the previous section. Without the template, given the same words as plain text, the tuned model writes . and stops.

The system message matters too: it's part of every request the model reads, so a different one, such as a description of the store and its tone, can change any reply. That makes it a setting to choose and record, like a decoding setting in chapter 9.

Working rule: Always format requests to a tuned model with its own chat template, applied by the library from the model's tokenizer files, never assembled by hand. Check the template's default system message, since the model reads it on every request, and record the full formatted prompt with every answer you keep.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Measure made-up product details

마지막으로 변경된 판: 2026.11.16판

A model that writes about a product with no facts has to make them up. The fairer test gives it the facts. Both SmolLM2 models are shown a product's name and real description, and asked: What material is this product made of? Answer in a few words, using only the description. The tuned model gets it as a request in its template; the base model gets it as a listing to continue, ending Question:, the question, and Answer:. If the product's data names a material, a good answer names it; if not, a good answer says the description doesn't say.

Checking the answers takes a list of material words, from wood and metal to polyester and leather, in the book's shared module lib.claims. An answer is supported if every material it names is in the product's name or description. An answer is made up if it names a material that the product's name and description don't contain. An answer is declined if it says the product's data doesn't say, and none if it names no material and doesn't decline. The check has limits, and makes judgement calls: it counts engineered wood such as mdf and rubberwood as wood, it counts brushed nickel finish as naming nickel, and it misses a material the list doesn't include. Every answer it classes is recorded for anyone to read. Making up a detail the input doesn't support is one kind of what's called hallucination (Ji et al., 2023), and this test measures exactly that kind.

Class each model's answers for 40 products drawn at random, split by whether the product's data names a material. Where the data names none, will the tuned model make up fewer materials than the base model? Decide before you look.

made-up-materials.py Python 3.12.15 통과
"""Classes both models' answers to the material question, split by whether the product's name or description names amaterial. The prose's point rests on the rows for products whose data names none: made up and declined."""from collections import Counterfrom lib import claims, recordedasked = recorded.read("instruction-tuning", "answers")print("products asked, each with its description given:", len(asked))# a product's data names a material if its description or its name does: the union of the two setsgiven = [bool(claims.materials(row["description"]) | claims.materials(row["name"])) for row in asked]print("products whose name or description names a material:", sum(given))print("products whose name and description name none:", len(given) - sum(given))for role in ["base", "tuned"]:    for label, keep in [("data names a material", True), ("data names none", False)]:        rows = [row for row, named in zip(asked, given) if named == keep]        # each answer is classed against the product's data, by the rules in lib.claims        counts = Counter(claims.classify(row[role]["text"], row["description"], row["name"]) for row in rows)        for kind in ["supported", "made up", "none", "declined"]:            print(f"{role} model, {label}, {kind}:", counts[kind])
출력python made-up-materials.py
products asked, each with its description given: 40
products whose name or description names a material: 28
products whose name and description name none: 12
base model, data names a material, supported: 24
base model, data names a material, made up: 2
base model, data names a material, none: 2
base model, data names a material, declined: 0
base model, data names none, supported: 0
base model, data names none, made up: 7
base model, data names none, none: 5
base model, data names none, declined: 0
tuned model, data names a material, supported: 23
tuned model, data names a material, made up: 3
tuned model, data names a material, none: 2
tuned model, data names a material, declined: 0
tuned model, data names none, supported: 0
tuned model, data names none, made up: 8
tuned model, data names none, none: 4
tuned model, data names none, declined: 0
Answer Base, data names a material Tuned, data names a material Base, data names none Tuned, data names none
Supported 24 23 0 0
Made up 2 3 7 8
None 2 2 5 4
Declined 0 0 0 0

On the 12 products whose name and description name none, the right answer is that the description doesn't say, and neither model gave it once. Both mostly made a material up instead, and the rest of their answers named none without saying why. Of the products, 28 have data that names a material, and on those both models mostly answer from it.

Instruction tuning didn't fix this, and on these products the base model, given a listing to continue, did no worse. The request said to use only the description; when the description had no answer, the models answered anyway, with a material that products like this one often have. Following the form of an instruction isn't the same as knowing when you don't know.

Working rule: Test a model on questions its input can't answer, not only on ones it can, and count made-up answers separately. Check every factual claim a model makes about a product against the product's data, automatically where you can, as lib.claims does for materials.

확신도가 경고가 되는지 확인하기

마지막으로 변경된 판: 2026.11.11판

이전 절에서는 튜닝된 모델이 재료 질문에 내놓은 답을 근거가 있는 답과 지어낸 답으로 분류했다. 지어낸 답에 낮은 확신도가 따라붙었다면, 낮은 확신도로 그런 답을 가려낼 수 있을 것이다. 확신도가 정답을 맞히는 빈도와 일치하는 모델을 **보정되었다(calibrated)**고 한다. Guo와 동료들은 자신들이 연구한 심층 신경망이 흔히 그렇지 않다는 것을 발견했는데, 이 신경망들은 정확도가 정당화하는 수준보다 더 확신했다(Guo et al., 2017). 기록에는 튜닝된 모델이 쓴 각 토큰에 부여한 확률이 종료 토큰까지 포함해 저장되어 있으므로, 각 답의 토큰당 확신도를 기하평균으로 계산할 수 있다.

토큰당 확신도 = exp((log p₁ + … + log pₙ) / n)

이는 평균 로그 확률의 지수이며, 퍼플렉서티가 뒤집어 사용하는 바로 그 평균이다. 따라서 짧은 답과 긴 답을 같은 척도로 측정할 수 있다.

확신도로 두 종류를 구별할 수 있을까? 튜닝된 모델이 근거 있는 답과 지어낸 답에 보인 확신도를 비교하고, 토큰당 0.5라는 임계값이 각각 몇 개를 남기는지 세어 보자.

answer-confidence.py Python 3.12.15 · numpy 2.5.5 통과
"""Compares the tuned model's confidence in its supported and made-up answers to the material question, and counts what athreshold of 0.5 per token keeps. The prose's point rests on the two average confidences and the two threshold counts."""import numpy as npfrom lib import claims, recordedasked = recorded.read("instruction-tuning", "answers")# only supported and made-up answers are compared; declined and none answers are left outby_kind = {"supported": [], "made up": []}for row in asked:    kind = claims.classify(row["tuned"]["text"], row["description"], row["name"])    if kind in by_kind:        # the geometric mean of the token probabilities: exp of their average log-probability        by_kind[kind].append(np.exp(np.mean(np.log(row["tuned"]["token_probabilities"]))))for kind, confidence in by_kind.items():    print(f"tuned model, {kind} answers:", len(confidence))    print(f"tuned model, {kind} answers, average probability per token:", f"{np.mean(confidence):.2f}")# a fixed cut for the comparison, not one fitted to these answersthreshold = 0.5for kind, confidence in by_kind.items():    print(f"tuned model, {kind} answers above {threshold} per token:", f"{sum(c > threshold for c in confidence)} of {len(confidence)}")
출력python answer-confidence.py
tuned model, supported answers: 23
tuned model, supported answers, average probability per token: 0.65
tuned model, made up answers: 11
tuned model, made up answers, average probability per token: 0.45
tuned model, supported answers above 0.5 per token: 21 of 23
tuned model, made up answers above 0.5 per token: 3 of 11

확신도는 걸러 내는 장치가 아니라 신호이다. 가능성의 균형을 기울여 줄 뿐이며, 확신에 찬 오류는 통과시킨다. 지어낸 답의 평균 확신도는 토큰당 0.45로, 근거 있는 답의 0.65보다 낮다. 그러나 토큰당 0.5라는 임계값을 쓰면 근거 있는 답은 21 of 23개를 남기면서도, 지어낸 답이 3 of 11개 그대로 통과한다. 답의 수가 적으므로 이 값들은 의존할 수 있는 비율이 아니라 경향으로 보아야 한다.

작업 규칙: 모델의 확신도만으로 답을 신뢰할지 결정하지 않는다. 자신의 질문에서 확신도가 정답 여부와 어떤 관계인지 측정하고, 임계값을 쓴다면 그 측정값에서 골라 임계값이 통과시키는 답을 세어 본다.

더 긴 컨텍스트에서 잃어버리는 것

마지막으로 변경된 판: 2026.11.11판

소재 질문에서는 모델에 한 번에 제품 하나만 주었다. 그러나 매장의 어시스턴트는 검색 결과, 카테고리 페이지, 비교표처럼 여러 제품의 데이터를 한꺼번에 받고, 그 가운데서 알맞은 제품의 세부 정보를 찾아야 하는 경우가 많다. Liu와 동료들은 모델이 긴 컨텍스트의 처음과 끝에 있는 정보를 중간에 있는 정보보다 더 잘 사용한다는 것을 발견했다(Liu et al., 2024).

질문 대상 제품을 10개의 제품 목록 사이에 첫 번째, 중간, 마지막 중 한 곳에 놓는다. 그런 다음 튜닝한 모델인 SmolLM2-360M-Instruct에 그 제품이 무엇으로 만들어졌는지 묻는다. 데이터에 소재가 적혀 있는 14개 제품에 대해, lib.claims가 확인하는 방식으로 자기 소재만 언급한 답변의 수를 센다.

answer-position.py Python 3.12.15 통과
"""Counts the tuned model's answers that name only the asked product's own materials, with the product first, in the middleor last of a context of listings. The prose's point rests on the three placement lines."""from collections import defaultdictfrom lib import claims, recordedcontext = recorded.read("instruction-tuning", "context")# the other listings in each context; the asked product is one more, as the first printed line showsothers = recorded.read("instruction-tuning", "manifest")["settings"]["context_others"]print("product listings in each context:", others + 1)results = defaultdict(list)for row in context:    # only products whose data names a material: a product with none gives an answer nothing to name    if claims.materials(row["description"]) | claims.materials(row["name"]):        # supported: every material the answer names is in the asked product's own data        results[row["place"]].append(claims.classify(row["tuned"]["text"], row["description"], row["name"]) == "supported")print("products asked about at each place:", len(results["first"]))for place in ["first", "middle", "last"]:    print(f"product placed {place}, answers that name only its own materials:", f"{sum(results[place])} of {len(results[place])}")
출력python answer-position.py
product listings in each context: 10
products asked about at each place: 14
product placed first, answers that name only its own materials: 12 of 14
product placed middle, answers that name only its own materials: 6 of 14
product placed last, answers that name only its own materials: 12 of 14

중간에 놓인 제품이 가장 적게 찾아지며, 이는 Liu와 동료들이 훨씬 큰 모델에서 발견한 양상과 같다. 제품을 첫 번째에 놓으면 12 of 14개의 답변이 자기 소재만 언급하고, 마지막에 놓으면 12 of 14개, 중간에 놓으면 6 of 14개가 그렇다. 개수는 적지만 차이는 크다. 여기서 컨텍스트는 수천 자에 불과해 모델의 한계에 한참 못 미친다. 중간에 있는 세부 정보는 컨텍스트가 가득 차기 훨씬 전부터 사라진다.

작업 규칙: 모델이 가장 필요로 하는 세부 정보는 컨텍스트의 중간이 아니라 처음이나 끝에 두고, 작업이 허용하는 한 관련 없는 텍스트를 적게 주어라. 많은 제품 사이에서 세부 정보를 찾는 모델을 신뢰하기 전에, 더 긴 컨텍스트에서의 검색을 각 위치별로 시험하라.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Adapt a model to your text

마지막으로 변경된 판: 2026.11.16판

Chapter 10 found the general model far worse than the store's small model on the store's names, per word. Instruction tuning changes how a model answers, not what it knows about one store. The way to teach a general model the store's text is fine-tuning: continue training it on that text, the same training loop as chapter 8, starting from the general model's weights instead of random ones.

Full fine-tuning updates every weight, which for a model of hundreds of millions of parameters takes time and memory. LoRA, low-rank adaptation, freezes the model's weights and trains a small correction beside some of them. For a weight matrix that maps k numbers to d numbers, it trains a matrix B of d by r and a matrix A of r by k, and adds their product to the weights: W + BA (Hu et al., 2022). With a small rank r, there's far less to train and store, and Hu and colleagues found it matched full fine-tuning on the tasks they tested.

A LoRA correction would train far fewer weights; count them on SmolLM2-360M, adapting the query and value weights of every layer, at two ranks, against full fine-tuning. The query weights map 960 numbers to 960; the value weights, because keys and values are grouped, map 960 to 320:

lora-parameters.py Python 3.12.15 통과
"""Counts what LoRA would train on SmolLM2-360M, adapting the query and value weights of every layer at two ranks, andwhat full fine-tuning trains. The prose's point rests on the rank 8 lines and the full fine-tuning count."""from lib import open_modelconfig = open_model.config("SmolLM2-360M")width, layers = config["hidden_size"], config["num_hidden_layers"]# keys and values share heads, so their width is the query width scaled by the ratio of key-value heads to headskey_value_width = config["num_key_value_heads"] * width // config["num_attention_heads"]# every parameter of the model: the full fine-tuning count, and the denominator of each sharetotal = open_model.parameters(config)["total"]for rank in [8, 64]:    # LoRA adds two small matrices beside each adapted weight: width x rank, then rank x output width.    added = layers * (rank * (width + width) + rank * (width + key_value_width))       # on the query and value weights    print(f"rank {rank}, parameters trained:", f"{added:,}")    print(f"rank {rank}, share of the model's parameters:", f"{added / total:.2%}")print("parameters trained by full fine-tuning:", f"{total:,}")
출력python lora-parameters.py
rank 8, parameters trained: 819,200
rank 8, share of the model's parameters: 0.23%
rank 64, parameters trained: 6,553,600
rank 64, share of the model's parameters: 1.81%
parameters trained by full fine-tuning: 361,821,120

At rank 8, LoRA trains 819,200 parameters, 0.23% of the model, where full fine-tuning trains all 361,821,120. A rank-8 correction is smaller than the store's own small model, and it can be stored and swapped per task while the base weights stay shared.

This book doesn't run the fine-tuning: our judgement is that even LoRA, over the store's text, would take far longer on a laptop's processor than any recording here. It's the next step for a store that wants a general model to write in its own voice: fine-tune on the catalog, then measure the result against the store's own model, per word, as chapter 10 did. Fine-tuning has risks of its own. A model trained on the catalog learns its gaps along with its text, as the store's small model learned to leave a bulb's description empty in chapter 9. And training on one narrow text can weaken what the model knew before; Biderman and colleagues found LoRA forgets less of it than full fine-tuning (Biderman et al., 2024). Both are measured the same way as everything else in this book: on held-out text, before and after.

Working rule: To teach a general model your text, fine-tune it on that text, with LoRA when memory or storage is tight, and measure the result on held-out text of the same kind. Don't expect instruction tuning or a cleverer prompt to supply knowledge the model never learned.

평가 기준을 돌아보며

마지막으로 변경된 판: 2026.11.12판

이 책은 3장부터 모든 모델을 이름 데이터에서 같은 방식으로 평가했다. 평가 기준은 스토어의 토크나이저가 만드는 토큰당 퍼플렉시티이고, 평가 대상은 스토어의 검증용 이름이다. 테스트용 이름은 따로 남겨 두었다. 검증용 이름을 보고 내린 선택, 예컨대 남겨 둘 에폭이나 혼합 비율이 결과를 실제보다 좋아 보이게 하지 못하도록 하기 위해서다. 이번 최종 비교에서는 앞서 선택 대상이 된 다섯 모델을 4,281 테스트용 이름으로 평가한다. 혼합 비율은 검증 데이터에서 정한 값을 쓴다.

test-names.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""Scores the five models the book chose between on the test names, each name's tokens and its end, and prints eachmodel's test perplexity. The prose's point rests on the five test perplexity lines."""import numpy as npfrom lib import neural, transformerfrom lib.names import name_tokensfrom lib.ngrams import KneserNey, ngrams, padded# The test names, untouched until now: every model scores each name's tokens and its end. The mixing shares are the# ones chosen on the validation names, in chapters 5 and 8.test = name_tokens("test")counted = KneserNey(name_tokens("train"), 4)# padded adds the start marker three times and the end marker, so each name's first tokens have a context and its end is scoredfrom_counts = np.array([counted.probability(c, t) for text in test for c, t in ngrams(padded(text, 4), 4)])from_window = neural.token_probabilities(test)_, vocabulary, _ = transformer.load()index = {token: i for i, token in enumerate(vocabulary)}from_transformer = []for name in test:    # the transformer reads the start marker, the name, then <sep>, which ends a name where the counts end with </s>    ids = [index[transformer.START]] + [index[t] for t in name] + [index[transformer.SEP]]    probabilities = transformer.softmax(transformer.forward(ids)[:-1])    # position i predicts token i + 1, so ids[1:] is each token's target, the end of the name included    from_transformer += list(probabilities[np.arange(len(ids) - 1), ids[1:]])from_transformer = np.array(from_transformer)# pooled over every token of every name, not averaged per nameperplexity = lambda p: f"{np.exp(-np.log(p).mean()):.1f}"print("test names:", f"{len(test):,}")print("test perplexity, the 4-gram baseline:", perplexity(from_counts))print("test perplexity, the window model:", perplexity(from_window))# the mix averages the probabilities, not their logs: a share of 0.2 is 20% of the window modelprint("test perplexity, the window model mixed with the counts, share 0.2:", perplexity(0.2 * from_window + 0.8 * from_counts))print("test perplexity, the store's transformer:", perplexity(from_transformer))print("test perplexity, the store's transformer mixed with the counts, share 0.5:", perplexity((from_transformer + from_counts) / 2))
출력python test-names.py
test names: 4,281
test perplexity, the 4-gram baseline: 22.9
test perplexity, the window model: 35.4
test perplexity, the window model mixed with the counts, share 0.2: 21.6
test perplexity, the store's transformer: 23.7
test perplexity, the store's transformer mixed with the counts, share 0.5: 18.3

검증 데이터에서 내린 선택이 결과를 부풀린 정도는 아주 작았다. 테스트 열의 값이 검증 열의 값에 가깝게 유지된다. 두 점수를 나란히 놓으면 이 책의 이야기를 표 하나로 볼 수 있다.

모델 장 검증용 이름 테스트용 이름
토큰 하나의 횟수, 1-gram 3 687.2
토큰 쌍의 횟수, 2-gram 3 46.9
토큰 네 개의 횟수, 4-gram 기준 모델 3 23.1 22.9
윈도 모델, 작은 신경망 5 36.4 35.4
횟수와 혼합한 윈도 모델 5 21.9 21.6
스토어의 트랜스포머 8 24.0 23.7
횟수와 혼합한 스토어의 트랜스포머 8 18.4 18.3

이 표에는 세 가지 교훈이 담겨 있다. 첫째, 짧고 정형화된 텍스트에서는 횟수 세기만으로도 상당한 성능을 낼 수 있다. 이 책에서 학습된 모델 가운데 이름만으로 4-gram 기준 모델을 이긴 것은 없었고, 두 혼합 모델은 모두 이를 이겼다. 둘째, 학습된 모델은 횟수 기반 모델과 다른 곳에서 틀린다는 점에서 존재 가치가 있다. 그래서 모든 혼합 모델이 구성 요소 두 개를 모두 이겼다. 셋째, 먼 문맥이 중요한 더 긴 텍스트에서는 8장이 설명문에서 측정했듯이 트랜스포머가 단독으로 앞서 나갔다. 크기만으로 결정되는 것은 없었다. 10장의 범용 모델은 수백 배 더 크지만, 스토어의 이름에서 단어당 54,352.8를 기록했고 스토어의 트랜스포머는 199.3를 기록했다.

이 숫자들 중 어느 것도 모델이 유용한지는 알려 주지 않는다. 퍼플렉시티는 모델이 학습 텍스트와 비슷한 텍스트를 얼마나 잘 예측하는지를 측정하며, 이는 자동 완성에 필요한 능력이다. 그러나 설명이 사실인지, 답변이 요청에 맞는지, 구매자가 찾던 것을 발견하는지는 측정하지 않는다. 이 장의 테스트인 지어낸 자료, 확신도, 문맥 속 사실의 위치는 퍼플렉시티를 넘어 작업 자체를 측정하기 위한 첫걸음이다.

작업 규칙: 이 책이 그랬듯이 모델을 비교할 때는 같은 텍스트, 같은 토크나이저, 같은 측정 방식이라는 고정된 평가 기준 하나를 유지한다. 그런 다음 모델을 서비스에 쓰기로 정하기 전에 작업 자체에 맞춰 만든 테스트로 그 작업을 측정한다. 퍼플렉시티가 낮다는 것은 모델을 테스트해 볼 이유이지, 배포할 이유가 아니다.

운영 환경에서 무엇이 잘못되는가

튜닝된 모델은 도움 없이도 사람들이 쓸 수 있는 종류의 모델이며, 그 실패는 성공만큼 유창하게 들린다.

  • 모른다고 말해야 할 때 답을 내놓는다. 기본 SmolLM2 모델과 튜닝된 SmolLM2 모델은 데이터에 재료가 하나도 적혀 있지 않은 대부분의 제품에 대해 재료를 지어냈으며, 어느 쪽도 데이터에 그런 내용이 없다고 말한 적이 없다. 사실을 주장하는 모든 문장은 쇼핑객에게 닿기 전에 제품 데이터와 대조해야 한다.
  • 신뢰도가 오류를 통과시킨다. 지어낸 답은 평균적으로 신뢰도가 더 낮았지만, 좋은 답 대부분을 남기는 임계값을 적용해도 지어낸 답 3 of 11개가 통과했다.
  • 형식은 학습의 일부다. 채팅 템플릿 없이 실행하자 튜닝된 모델은 마침표 하나를 썼다. 모델 파일에 들어 있는 템플릿을 적용하고, 모든 답과 함께 전체 프롬프트를 기록해야 한다.
  • 컨텍스트의 중간 부분은 소실된다. 중요한 세부 내용은 처음이나 끝에 두고, 각 위치에서 검색이 되는지 시험해야 한다.
  • 모델은 여러분의 텍스트를 모른다. 지시 튜닝은 답의 형식을 가르칠 뿐, 상점의 카탈로그를 가르치지 않는다. 카탈로그로 파인튜닝하고, 그 결과를 측정해야 한다.
  • 유창함이 빈틈을 가린다. 튜닝된 모델의 설명은 기본 모델의 설명보다 더 잘 읽히지만 담고 있는 사실은 더 많지 않다. 생성된 텍스트는 읽히는 정도가 아니라 그 안의 주장에 대한 검사로 판단해야 한다.
  • 동작은 모델의 리비전과 템플릿의 기본 메시지에 좌우된다. 녹화본이 하는 것처럼 모델의 리비전을 고정하고, 업데이트가 있을 때마다 이 측정을 다시 실행해야 한다.

이 장에서 결정한 것

  • 이 책은 기반 모델과 튜닝된 모델을 같은 요청으로 비교한다. 고정된 리비전에서 한 번 기록하고 탐욕적으로 디코딩하며, 소재에 대한 lib.claims처럼 누구나 다시 실행할 수 있는 검사로 답변을 측정한다.
  • 지어낸 세부 정보는 입력으로 답할 수 있는 질문뿐 아니라 답할 수 없는 질문에서도 측정한다. 가게의 상품에 대해 두 SmolLM2 모델은 데이터에 소재가 없는 상품의 대부분에 소재를 지어냈고, 한 번도 답을 거절하지 않았다. 그래서 생성된 상품 정보는 항상 그 상품의 데이터와 대조해 확인한다.
  • 모델의 확신은 신호일 뿐 필터가 아니다. 가게 자체의 질문에서 측정한 임계값과 함께 사용할 때만 쓰며, 이 방식이 통과시키는 오류는 센다.
  • 모델에 필요한 세부 정보는 컨텍스트의 처음이나 끝에 둔다. 다른 상품들 사이에 묻어 두어서는 안 된다.
  • 퍼플렉서티는 모델을 비교하는 이 책의 잣대로 남는다. 매번 같은 텍스트, 같은 토크나이저, 같은 이벤트로 비교하며, 각 모델이 선택된 용도는 그 용도를 위해 만든 테스트로 측정한다.
  • 범용 모델에 가게의 텍스트를 가르치는 일은 파인튜닝을 뜻하며, 가게 자체의 모델과 비교해 단어당으로 측정한다. 인스트럭션 튜닝은 모델이 답하는 방식을 바꿀 뿐, 모델이 아는 내용을 바꾸지 않는다. 이 책의 다음 토큰 테스트를 넘어 애플리케이션 안에서 모델을 측정하는 일은 별개의 주제다.

이 판의 이 장

2026.11.16판예제 통과 2026년 11월 19이 장에서 9개 예제개 중 9개가 통과한다

영어판 2026.11.16판을 번역한 것이다.

이 절 인용

형식

각 형식은 이 번역이 따르는 영문 원서를 출처로 표기한다.

모든 형식은 이 판을 명시한다. 링크는 현재 판을 열며, 여기서 절의 제목은 본문이 마지막으로 바뀐 시점을 알려 준다. 접근 날짜와 검색 날짜는 이 자료를 여는 날짜다.

이 페이지의 내용

11장 중 11장: From predicting to following instructions

  1. Build the two training losses by hand
  2. 기반 모델과 튜닝된 모델에 물어보기
  3. Format the request as the model was tuned
  4. Measure made-up product details
  5. 확신도가 경고가 되는지 확인하기
  6. 더 긴 컨텍스트에서 잃어버리는 것
  7. Adapt a model to your text
  8. 평가 기준을 돌아보며
  9. 운영 환경에서 무엇이 잘못되는가
  10. 이 장에서 결정한 것

맨 위로 돌아가기전체 장