Choosing the next token
읽는 데 약 32분분 걸린다, 9절에 7개 예제이(가) 있음.
AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식
영어판 2026.11.16판을 번역한 것이다. 원문 읽기
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
By the end of this chapter you can turn a language model's probabilities into text, and choose how: always the likeliest token, the likeliest whole text by beam search, or a sample shaped by a temperature, a top-k or top-p cut, a repetition penalty and a minimum length. You'll measure what each choice does to the text, and learn which failures no choice of setting can fix.
A language model gives a probability for every possible next token; it doesn't write. To write, something has to choose one token, append it, ask for the next probabilities, and repeat until the model predicts the end marker, </s>. Turning a model's probabilities into text, one chosen token at a time, is called decoding, and it decides much of what the text reads like, with the model unchanged.
The store has a reason to write: of the furniture and home goods in its validation split, 597 of 4,282 products have no description, and so do 14% of its training products. The store's model, the transformer that chapter 8 trained and measured, learned descriptions from the store's own text, so it can draft one from a product's name: start the text with the name and <sep>, as every training text was laid out, and let the model continue. This chapter does that with greedy decoding, beam search and sampling, using the book's shared module lib.decoding, built in NumPy on lib.transformer. Then it measures what each method writes against the store's real descriptions, and fixes two failures that the measurements expose: repetition, and descriptions that come out empty.
가장 가능성이 높은 토큰 고르기
가장 단순한 디코더는 탐욕적(greedy) 디코더다. 이 디코더는 매 단계에서 가장 가능성이 높은 토큰 하나를 고른다. 무작위성이 필요 없으므로 같은 시작에서는 항상 같은 텍스트를 쓰며, 각 단계에는 모델의 순전파 한 번의 비용이 든다. 이렇게 쓰인 텍스트는 토큰마다 모델의 최선의 추측이다.
설명이 없는 상품은 상점이 실제로 필요로 하는 경우이므로, 먼저 그런 상품 하나로 시작한다. front porch welcome sign wall décor를 최대 60 토큰까지 탐욕적으로 디코딩한다. 그다음 두 번째 상품 5 light light bulb를 가져와 그 설명이 비어 있을 확률을 출력한다.
1"""Greedy decoding writes one description of a product that has none, and shows how it repeats itself, and how likely2the model is to leave a bulb's description empty. The prose quotes the greedy description, its repeated phrases, and3the line with the bulb's probability of an empty description."""4from collections import Counter56from lib import decoding, transformer7from lib.store import load_products89validation, training = load_products("validation"), load_products("train")10empty = lambda rows: rows["description"].fillna("").str.strip() == ""11print("validation products:", f"{len(validation):,}")12print("validation products with no description:", f"{empty(validation).sum():,}")13print("training products with no description:", f"{empty(training).mean():.0%}")14bulbs = training[training["name"].str.contains(r"\bbulbs?\b")]15print("training products named as bulbs with no description:", f"{empty(bulbs).mean():.0%}")1617longest = 6018print("longest description, in tokens:", longest)19ids = decoding.generate(decoding.prompt("front porch welcome sign wall décor"), longest=longest) # no rng: each step takes the likeliest token20print("greedy description:", decoding.text(ids))21fourgrams = Counter(tuple(ids[i:i + 4]) for i in range(len(ids) - 3)) # the four tokens from i: a window that slides one token at a time22print("4-token phrases it uses twice or more:", " | ".join(decoding.text(list(g)) for g, n in fourgrams.items() if n > 1))2324end = transformer.load()[1].index(transformer.END) # the index of </s> in the vocabulary25probabilities = transformer.softmax(decoding.next_logits(decoding.prompt("5 light light bulb"))) # the first token after <sep>26print("P(the description is empty), 5 light light bulb:", f"{probabilities[end]:.2f}")
python greedy-description.pyvalidation products: 4,282
validation products with no description: 597
training products with no description: 14%
training products named as bulbs with no description: 24%
longest description, in tokens: 60
greedy description: this welcome sign is a great way to add a touch of rustic charm to your home . it is made of natural wood and features a distressed finish with a distressed finish . this welcome sign is made of wood and has a distressed finish that adds a rustic touch to your home . this sign is made from
4-token phrases it uses twice or more: this welcome sign is | to your home .
P(the description is empty), 5 light light bulb: 0.28
탐욕적으로 만든 설명은 처음에는 상점 자체의 글처럼 읽히다가 곧 반복된다. 그렇게 읽히는 까닭은 실제로 상점의 글을 다시 조합한 것이기 때문이며, this welcome sign is a great way to add a touch of rustic charm to your home가 그 예다. 이 설명은 features a distressed finish with a distressed finish를 쓰고, 뒤에서 has a distressed finish를 다시 쓴다. 또 네 토큰짜리 구절 두 개, this welcome sign is와 to your home .를 각각 두 번 사용하는데, 예제가 이를 나열한다. 탐욕적 디코딩이 반복되는 까닭은 텍스트가 비슷한 지점에 이를 때마다 가능성이 가장 높은 같은 이어쓰기가 선택되기 때문이다. distressed는 간판의 마감으로 모델이 가장 좋아하는 표현이어서, 마감을 쓸 차례가 오면 계속 되돌아온다. Holtzman 등은 대형 모델에서도 같은 현상을 발견했다. 확률을 항상 최대화하는 디코딩은 "밋밋하거나, 일관성이 없거나, 반복 루프에 갇힌" 텍스트로 이어진다(Holtzman et al., 2020).
탐욕적 설명은 간판이 made of natural wood라고도 말하는데, 상품 이름에는 이를 뒷받침하는 내용이 전혀 없다. 전구는 다른 종류의 실패를 보여 준다. 5 light light bulb에 대해 모델은 즉시 끝나는 출력, 곧 빈 설명을 0.28의 확률로 내놓는다. 모델은 빈틈까지 포함해 카탈로그를 있는 그대로 학습했다. 전구라는 이름이 붙은 학습 상품 중 24%가 설명이 없으며, 전체 상품에서는 14%가 그러하다. 빈 설명은 뒤의 절에서 다룬다.
작업 규칙: 매번 같은 출력이 필요하고 텍스트가 짧을 때, 예를 들어 가장 가능성이 높은 다음 단어가 핵심인 검색창 추천에는 탐욕적 디코딩을 쓴다. 구절보다 긴 텍스트에서는 반복이 나타날 것으로 예상하고, 이어지는 절의 방법 중 하나를 고른다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Search for the likeliest text
Greedy decoding picks the likeliest token at each step, which isn't the same as the likeliest text: a slightly less likely first word can lead to much likelier words after it. Beam search looks further, keeping the k likeliest partial texts, the beam, scored by their total log-probability, the sum of the logarithms of a text's token probabilities, where higher means likelier. At each step it extends every text in the beam by its k likeliest next tokens, and keeps the best k of the results. A text that reaches the end marker is finished and stays in the running. The heart of lib.decoding.beam is the line that scores each extension:
candidates += [(score + log_p[token], written + [int(token)]) for token in np.argsort(-log_p, kind="stable")[:width]]
Which description do you expect beam search to rank likeliest for the welcome sign? Run it twice, with a beam of 4. The first run completes the search query outdoor bar by up to 4 tokens, the end of the name included, the way a search box might offer whole phrases; the second writes the welcome sign's description and prints the likeliest text overall and per token, its total log-probability divided by its length.
1"""Beam search with a beam of four, on a search query and on a product's description. The prose quotes the two2completions of outdoor bar, then the likeliest description, its probability and the likeliest per token."""3import numpy as np45from lib import decoding, transformer67_, vocabulary, _ = transformer.load()8width, longest = 4, 49print("beam width:", width)10print("longest completion, in tokens, its end included:", longest)11start = decoding.prompt("outdoor bar")[:-1] # a search query: <s> and the words typed so far12for score, written in decoding.beam(start, width=width, longest=longest, stops=(transformer.SEP,))[:2]: # stop at <sep>, where a name ends; keep the two likeliest13 print("completion of outdoor bar:", decoding.text([t for t in written if vocabulary[t] != transformer.SEP]),14 f"(probability {np.exp(score):.3f})")1516kept = decoding.beam(decoding.prompt("front porch welcome sign wall décor"), width=width) # longest and stops take their defaults: 60 tokens, </s>17score, written = kept[0] # the likeliest text overall18print("likeliest description:", decoding.text(written) or "(empty)")19print("its probability:", f"{np.exp(score):.2f}")20score, written = max(kept, key=lambda candidate: candidate[0] / len(candidate[1])) # the likeliest per token21print("likeliest description per token:", decoding.text(written))
python beam-search.pybeam width: 4
longest completion, in tokens, its end included: 4
completion of outdoor bar: stool (probability 0.365)
completion of outdoor bar: stool cushion (probability 0.089)
likeliest description: (empty)
its probability: 0.15
likeliest description per token: welcome guests to your home with this welcome sign . crafted of iron , this welcome sign is a great way to add a rustic touch to your home .
On the query, beam search gives what a search box should: short, likely phrases, with stool first and then stool cushion. This uses the transformer alone; the mix that chapter 8 chose for autocomplete would need the 4-gram's probability for every candidate at every step, which lib.decoding doesn't compute.
The likeliest description overall is empty. Ending at once has a probability of 0.15, and every longer text multiplies many probabilities below one, so none comes close. Stahlberg and Byrne found the same in translation models: for most sentences the most probable translation was empty, and beam search worked only because it searched badly (Stahlberg and Byrne, 2019). Ranking by probability per token removes that bias, and gives a sensible description. It still invents a material, though: this time the sign is crafted of iron.
Working rule: Use beam search for short completions where the likeliest phrase is the goal, such as search queries, ranked by total probability. For longer text, rank finished texts per token, since total probability favours the shortest, and even then prefer the sampling methods of the next sections, which write text closer to what people write.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Sample, with a temperature
Greedy decoding and beam search use only the top of the distribution. Sampling, which chapter 1 used to generate names, uses all of the distribution: draw the next token at random, in proportion to its probability, so a token with probability 0.2 is chosen one time in five. Before sampling, the distribution can be made sharper or flatter with a temperature, T: divide every logit by T before the softmax. In symbols, with zᵢ the logit of token i, as in chapter 5:
P(token i) = exp(zᵢ / T) / Σⱼ exp(zⱼ / T)
A temperature below 1 sharpens the distribution, giving more of the probability to the likeliest tokens; above 1, it flattens it, giving more to unlikely ones. A temperature near 0 approaches greedy decoding, and a temperature of 1 leaves the model's probabilities as they are. Dividing by T keeps the order of the tokens; it only changes how far apart their probabilities are.
To see what temperature does to one real choice, stop the greedy description early and read the next token's distribution at three temperatures. For each one, the example prints the three likeliest tokens, how many cover 90% of the probability, and how much falls outside temperature 1's 90% cover.
1"""The next token's distribution after the first nine tokens of the greedy description, at three temperatures, and how2many tokens each needs to cover 90% of the probability. The prose quotes the three counts, and the share at 1.5 that3falls outside temperature 1's 90% cover."""4import numpy as np56from lib import decoding, transformer78_, vocabulary, _ = transformer.load()9start = decoding.prompt("front porch welcome sign wall décor")10so_far = decoding.generate(start, longest=9) # the greedy description's first 9 tokens11logits = decoding.next_logits(start + so_far)12print("text so far:", decoding.text(so_far))13print("tokens in the vocabulary:", f"{len(vocabulary):,}")14cover = np.argsort(-transformer.softmax(logits), kind="stable") # at temperature 1, the likeliest first15cover = cover[:int(np.searchsorted(np.cumsum(transformer.softmax(logits)[cover]), 0.9)) + 1] # temperature 1's 90% cover, its last token included16for temperature in [0.5, 1.0, 1.5]:17 probabilities = transformer.softmax(logits / temperature) # divide the logits, then softmax18 top = np.argsort(-probabilities, kind="stable")19 covering = int(np.searchsorted(np.cumsum(probabilities[top]), 0.9)) + 1 # +1: the token that reaches 90% counts20 print(f"temperature {temperature}, top 3:", ", ".join(f'"{vocabulary[i].replace("</w>", "")}" {probabilities[i]:.2f}' for i in top[:3]))21 print(f"temperature {temperature}, tokens needed to cover 90% of the probability:", covering)22 print(f"temperature {temperature}, probability outside temperature 1.0's 90% cover:",23 f"{1 - probabilities[cover].sum():.0%}")
python temperature-by-hand.pytext so far: this welcome sign is a great way to add
tokens in the vocabulary: 4,142
temperature 0.5, top 3: "a" 0.61, "style" 0.20, "some" 0.11
temperature 0.5, tokens needed to cover 90% of the probability: 3
temperature 0.5, probability outside temperature 1.0's 90% cover: 0%
temperature 1.0, top 3: "a" 0.26, "style" 0.15, "some" 0.11
temperature 1.0, tokens needed to cover 90% of the probability: 78
temperature 1.0, probability outside temperature 1.0's 90% cover: 10%
temperature 1.5, top 3: "a" 0.09, "style" 0.06, "some" 0.05
temperature 1.5, tokens needed to cover 90% of the probability: 640
temperature 1.5, probability outside temperature 1.0's 90% cover: 42%
The temperature sets how far the probability spreads. After this welcome sign is a great way to add, at temperature 1 the model needs 78 tokens to cover 90%, and the rest is a long tail, each token in it unlikely. At 0.5, 3 tokens cover 90%. At 1.5, it takes 640 tokens, and 42% of the probability falls outside temperature 1's 90% cover.
| Temperature | Three likeliest tokens | Tokens to cover 90% | Outside T = 1's 90% cover |
|---|---|---|---|
| 0.5 | "a" 0.61, "style" 0.20, "some" 0.11 | 3 | 0% |
| 1 | "a" 0.26, "style" 0.15, "some" 0.11 | 78 | 10% |
| 1.5 | "a" 0.09, "style" 0.06, "some" 0.05 | 640 | 42% |
That tail is the risk in sampling. Most tokens in it are wrong here, and at temperature 1 a sampler draws from it about once in every ten tokens, more often as the temperature rises.
Working rule: Treat temperature as the dial between predictable and varied text. Keep it at or a little below 1 for text that should stay close to what the training text usually says, such as product descriptions. A lower temperature makes text more predictable, not more accurate.
샘플링하고 꼬리를 잘라 내기
온도를 낮추면 꼬리가 줄어들지만 완전히 사라지지는 않는다. 샘플링에 앞서 꼬리를 직접 제거하는 방법이 두 가지 있다. Top-k는 가장 가능성이 높은 토큰 k개만 남기고 그 확률의 합이 1이 되도록 다시 정규화한다(Fan et al., 2018). Top-p는 핵 샘플링(nucleus sampling)이라고도 하며, 확률의 합이 p 이상이 되는 가장 작은 상위 토큰 집합을 남긴다(Holtzman et al., 2020, 식 2). 두 방법 모두 lib.decoding.truncate의 옵션이며, 그 핵심은 세 단계다. 토큰을 정렬하고, 그 순서대로 확률을 더하고, 합이 p에 도달하는 지점을 찾는다.
order = np.argsort(-probabilities, kind="stable")
...
keep = int(np.searchsorted(np.cumsum(probabilities[order]), top_p)) + 1
top-p는 모델의 확신 정도에 맞추어 조정되지만 top-k는 그렇지 않다. 앞 절의 예제에서 모델이 다음 단어를 확신하지 못한 지점에서는 확률의 90%를 채우는 데 토큰 78개가 필요했다. 따라서 top-p 0.9는 그만큼의 토큰을 남기지만 top-k 20은 20개만 남긴다. 모델이 거의 확실하게 예측하는 단어 뒤에서는 top-p는 토큰을 한두 개만 남기고, top-k는 여전히 20개를 남기는데 대부분은 틀린 토큰이다. Holtzman과 동료들은 바로 이 점을 근거로 top-p를 옹호했다. 고정된 k는 "다양한 문맥에 걸쳐 최적이 아니다"라는 것이다.
같은 난수 시드로 네 가지 샘플러가 환영 표지판의 설명을 최대 40개 토큰까지 작성한다. 온도 1에서의 샘플링, 0.7에서의 샘플링, k = 20인 top-k, p = 0.9인 top-p다. 이어서 예제는 온도 1 샘플을 다시 재생하며, 각 단계에서 90% 핵 밖에서 뽑힌 토큰을 나열한다. top-p 0.9라면 이 토큰들을 잘라 냈을 것이다.
1"""Four samplers write the same description from one seed, and the temperature-1 draws are then replayed to mark the2tokens outside each step's 90% nucleus. The last two lines are the ones the prose quotes: how many draws came from3the tail, and which tokens they were."""4import numpy as np56from lib import decoding, transformer78_, vocabulary, _ = transformer.load()9start, longest, seed = decoding.prompt("front porch welcome sign wall décor"), 40, 202610print("longest description, in tokens:", longest)11print("random seed:", seed)12for label, settings in [("sampling, temperature 1.0", {}), ("sampling, temperature 0.7", {"temperature": 0.7}),13 ("top-k 20", {"top_k": 20}), ("top-p 0.9", {"top_p": 0.9})]:14 ids = decoding.generate(start, rng=np.random.default_rng(seed), longest=longest, **settings)15 print(f"{label}:", decoding.text(ids))1617rng, written, tail = np.random.default_rng(seed), [], [] # replay temperature 1.0's draws, step by step18for _ in range(longest):19 probabilities = transformer.softmax(decoding.next_logits(start + written))20 token = int(rng.choice(len(probabilities), p=decoding.truncate(probabilities))) # no cut: draw from all of it21 order = np.argsort(-probabilities, kind="stable")22 before = probabilities[order[:int(np.where(order == token)[0][0])]].sum() # the probability ranked above it23 if before >= 0.9: # top-p 0.9 would have cut this token24 tail.append(vocabulary[token].replace("</w>", ""))25 written.append(token)26 if vocabulary[token] == transformer.END:27 break28print("temperature 1.0, tokens drawn from outside the 90% nucleus:", f"{len(tail)} of {len(written)}")29print("temperature 1.0, those tokens:", ", ".join(tail))
python sample-descriptions.pylongest description, in tokens: 40
random seed: 2026
sampling, temperature 1.0: enhance your home even glywheel design with this welcome welcome sign sign . this piece is ideal for both indoor and inviting . made in the usa , mini spirits decorate .
sampling, temperature 0.7: the sign is a great way to add some extra whimsy to your home and get colorful ! made of high-quality metal , this sign comes with a hanging dustroke hanger to help you gli
top-k 20: if you have a new product with a personalized sign wall décor that will help you entertain . look no further like a home decor , this sign is constructed to make life more wonderful . this welcome sign
top-p 0.9: express your guests with this cast iron sign from the nature of the world . it features a meditating podishade design . the hose and symbol in this metal piece will be
temperature 1.0, tokens drawn from outside the 90% nucleus: 5 of 36
temperature 1.0, those tokens: inviting, in, mini, spir, decorate
꼬리를 자르면 가장 드문 실수는 사라지지만 모든 실수가 사라지는 것은 아니다. 온도 1에서의 샘플링은 갈피를 잡지 못한다. welcome welcome sign sign라고 쓰고, glywheel라는 단어를 지어내고, 표지판이 made in the usa라고 주장한다. 이 가운데 일부만 꼬리에서 나왔다. 꼬리에서 뽑힌 토큰은 5 of 36개였고, inviting, in, mini, spir, decorate. 반복된 단어와 지어낸 단어는 핵 안의 토큰으로 만들어졌으며, top-p도 이 토큰들은 남겼을 것이다. 0.7에서는 글이 더 일관되지만, 탐욕적 디코딩이 나무라고 쓴 곳에서 이번에는 표지판이 made of high-quality metal라고 말한다. top-k와 top-p는 일반 샘플링보다 카탈로그에 가깝게 읽히지만 여전히 지어낸다. top-p의 표지판은 cast iron이며 podishade 디자인이 들어가 있다.
이 장에서 환영 표지판은 여섯 번 재질을 부여받았다. 탐욕적 디코딩, 빔 서치, 이 샘플 세 개, 그리고 아래의 페널티를 적용한 탐욕적 텍스트에서다. 표지판의 이름 어디에도 무엇으로 만들어졌는지는 나와 있지 않다. 모델은 환영 표지판의 설명에 흔히 적히는 내용을 쓰고, 그런 설명에는 대개 재질이 명시된다. 디코딩 방법은 모델이 어떤 그럴듯한 재질을 고르는지를 바꿀 뿐, 어느 방법도 모델이 실제 재질을 고르게 만들 수는 없다.
작업 규칙: 다양한 텍스트가 필요하면 top-p를 0.9 안팎으로, 또는 top-k를 20 안팎으로 두어 꼬리를 자르고, 온도는 1이거나 1보다 약간 낮게 둔다. 결과를 재현해야 할 때는 난수 시드를 고정한다. 어떤 설정으로도 텍스트가 사실이 되리라고 기대해서는 안 된다. 디코딩은 모델이 그럴듯하다고 여기는 것 가운데서 고를 뿐이며, 사실 확인은 별개의 작업이다.
각 설정이 쓴 글을 측정하기
개별 표본은 각 방법의 분위기를 보여 주고, 측정은 각 방법의 득실을 보여 준다. 사람이 글을 읽지 않고도 측정할 수 있는 속성이 몇 가지 있다.
- 반복: 같은 설명 안에서 앞서 나온 구절을 되풀이하는 4토큰 구절의 비율
- 글에 대한 모델 자신의 퍼플렉서티: 모델이 자신이 쓴 글을 얼마나 놀랍다고 여기는지를 모든 토큰에 걸쳐 합산한 값이며, 낮을수록 예측하기 쉬운 글이다
- 설명이 제한 안에서 끝나는지: 실제 설명이 그렇듯 끝나는가
- 상품 이름의 단어를 온전히 하나 사용하는지: 그 상품의 설명이라면 그래야 한다
- 평균 길이: 토큰 수 기준
각 설정은 실제 글에 얼마나 가까운가? 설명이 있는 검증 상품 중에서 20개를 무작위로 뽑아 모든 방법으로 각각 설명을 쓰고, 같은 60토큰 제한에서 잘라 실제 설명과 비교한다.
1"""Seven ways of choosing the next token, measured on the same 20 product descriptions: how often each repeats2itself, how likely the model finds its own text, whether it ends, and whether it stays on the product. The real3descriptions come first, as the bar the others are read against. The prose quotes these lines for each setting."""4import numpy as np56from lib import decoding # the store's model, and the decoding methods this chapter builds7from lib.products import product_texts89longest, sample, seed = 60, 20, 202610# Validation products, which the model never trained on, that have at least one token of description after <sep>11described = [(ids, sep) for ids, sep in product_texts("validation") if sep < len(ids) - 2]12products = [described[i] for i in np.random.default_rng(seed).choice(len(described), sample, replace=False)]13print("products, drawn at random from the validation products with a description:", sample)14print("longest description, in tokens:", longest)151617settings = [("real descriptions", None), ("greedy", {}), ("greedy, repetition penalty 1.3", {"penalty": 1.3}),18 ("sampling, temperature 1.0", {"sample": True}), ("sampling, temperature 0.7", {"sample": True, "temperature": 0.7}),19 ("top-k 20", {"sample": True, "top_k": 20}), ("top-p 0.9", {"sample": True, "top_p": 0.9})]20for label, chosen in settings:21 rows = []22 for number, (ids, sep) in enumerate(products):23 start = ids[:sep + 1] # the start marker, the name and <sep>: the model writes on from here24 if chosen is None:25 written = ids[sep + 1:][:longest] # the description as sold, cut where generating would stop26 else:27 options = {option: value for option, value in chosen.items() if option != "sample"}28 # generate samples when it's given a generator, and is greedy without one. Each product gets its own29 # seed, the same for every sampling setting, so the settings differ only in how they choose.30 rng = np.random.default_rng(seed + number) if chosen.get("sample") else None31 written = decoding.generate(start, rng=rng, longest=longest, **options)32 rows.append(decoding.measures(start, written)) # repetition, each token's log-probability, ended, uses the name, length33 repeated, log_p, ended, uses_name, length = zip(*rows)34 print(f"{label}, repeated 4-token phrases:", f"{np.mean(repeated):.0%}")35 print(f"{label}, the model's perplexity of the text:", f"{np.exp(-np.concatenate(log_p).mean()):.1f}") # over all 20 texts' tokens at once36 print(f"{label}, ended within {longest} tokens:", f"{np.mean(ended):.0%}")37 print(f"{label}, uses a whole word of the name:", f"{np.mean(uses_name):.0%}")38 print(f"{label}, average length in tokens:", f"{np.mean(length):.0f}")
python measure-decoding.pyproducts, drawn at random from the validation products with a description: 20
longest description, in tokens: 60
real descriptions, repeated 4-token phrases: 0%
real descriptions, the model's perplexity of the text: 30.5
real descriptions, ended within 60 tokens: 35%
real descriptions, uses a whole word of the name: 95%
real descriptions, average length in tokens: 53
greedy, repeated 4-token phrases: 11%
greedy, the model's perplexity of the text: 2.7
greedy, ended within 60 tokens: 25%
greedy, uses a whole word of the name: 80%
greedy, average length in tokens: 52
greedy, repetition penalty 1.3, repeated 4-token phrases: 0%
greedy, repetition penalty 1.3, the model's perplexity of the text: 3.9
greedy, repetition penalty 1.3, ended within 60 tokens: 70%
greedy, repetition penalty 1.3, uses a whole word of the name: 80%
greedy, repetition penalty 1.3, average length in tokens: 41
sampling, temperature 1.0, repeated 4-token phrases: 0%
sampling, temperature 1.0, the model's perplexity of the text: 22.6
sampling, temperature 1.0, ended within 60 tokens: 50%
sampling, temperature 1.0, uses a whole word of the name: 50%
sampling, temperature 1.0, average length in tokens: 45
sampling, temperature 0.7, repeated 4-token phrases: 0%
sampling, temperature 0.7, the model's perplexity of the text: 7.2
sampling, temperature 0.7, ended within 60 tokens: 45%
sampling, temperature 0.7, uses a whole word of the name: 65%
sampling, temperature 0.7, average length in tokens: 45
top-k 20, repeated 4-token phrases: 0%
top-k 20, the model's perplexity of the text: 7.6
top-k 20, ended within 60 tokens: 50%
top-k 20, uses a whole word of the name: 80%
top-k 20, average length in tokens: 40
top-p 0.9, repeated 4-token phrases: 0%
top-p 0.9, the model's perplexity of the text: 10.7
top-p 0.9, ended within 60 tokens: 30%
top-p 0.9, uses a whole word of the name: 60%
top-p 0.9, average length in tokens: 47
탐욕 디코딩이 쓴 글은 사람이 쓴 어떤 글보다 훨씬 예측하기 쉬워 퍼플렉서티가 2.7이고, 4토큰 구절의 11%가 반복된다. 실제 설명은 아무것도 반복하지 않으며(4토큰 구절 중 0%), 모델은 이를 상당히 놀랍다고 여겨 퍼플렉서티가 30.5이다. 이 수치는 8장에서 설명에 대해 얻은 값보다 높은데, 표본이 다르고 처음 60토큰만 보기 때문이다. 홀츠먼과 동료들도 대규모로 같은 관찰을 했다. 사람의 글은 모델이 가장 그럴듯하다고 여기는 글이 아니다.
모든 측정에서 실제 설명과 일치하는 설정은 없다. 온도 1의 샘플링은 퍼플렉서티가 22.6로 가장 가깝지만, 상품 이름의 단어를 쓰는 설명은 50%뿐이다. 실제 설명은 95%인 것과 비교하면, 상품에서 벗어나는 것이다. top-p 0.9는 설명이 끝나는 비율이 실제 글과 거의 같아 30% 대 35%이지만, 상품에 머무는 경우는 60%뿐이다. top-k 20은 샘플링 방법 중 상품에 가장 많이 머물러 80%이고, 반복이 없으며 설명의 평균 길이는 40토큰이다. 상품 20개는 작은 표본이므로, 이 값들은 정확한 비율이 아니라 경향으로 읽어야 한다.
이러한 측정이 말해 주지 못하는 것은 설명이 사실인가 하는 점이다. 설명은 유창하고 다채롭고 주제에 맞는데도 틀릴 수 있다. 앞 절의 설명들이 간판의 재질에 관해 그러했다.
작업 규칙: 표본을 읽기 전에, 이와 같은 측정으로 디코딩 설정을 같은 종류의 실제 글과 비교한다. 작업에 어떤 측정이 가장 중요한지 정한다. 상품 설명에서는 상품에 머무르는 것과 반복하지 않는 것이 먼저다. 그런 다음에도 어떤 측정도 잡아내지 못하는 것을 찾기 위해 출력 표본을 읽고, 모든 사실적 주장을 상품 데이터와 대조한다.
반복 억제
탐욕 디코딩의 반복 루프는 샘플링 없이도 해결할 수 있다. Keskar 등은 반복 페널티(repetition penalty)를 제안했다. 이미 생성된 모든 토큰의 로짓을 1보다 큰 계수 θ로 나누어 그 토큰이 선택될 가능성을 낮추는 방법이다(Keskar et al., 2019, 4.1절). 음수 로짓을 나누면 오히려 값이 커지므로, 구현에서는 lib.decoding.penalize이 하듯이 음수 로짓에는 θ를 곱한다. 이렇게 하면 어느 경우에든 해당 토큰의 가능성이 낮아진다. Keskar 등은 계수가 1.2 안팎일 때 반복을 피하는 것과 자연스러운 문장을 쓰는 것 사이에서 균형이 맞는다고 보았다.
탐욕 설명문이 distressed를 두 번째로 쓰려던 지점으로 돌아가 보자. 다음 토큰의 로짓에 페널티를 직접 적용하고, 그 결과를 lib.decoding.penalize의 결과와 대조한 뒤, 적용 전후의 확률을 비교한다. 이어서 페널티를 적용한 채 탐욕적으로 설명문을 작성하여 루프가 사라지는지 확인한다.
1"""The repetition penalty applied by hand to the next token's logits, checked against lib.decoding.penalize, and then2used for a whole greedy description. The two P(distressed) lines are the ones the prose quotes, and the3assert against lib.decoding.penalize is the match it states as fact."""4import numpy as np56from lib import decoding, transformer78_, vocabulary, _ = transformer.load()9start, factor = decoding.prompt("front porch welcome sign wall décor"), 1.310written = decoding.generate(start) # greedy, as in greedy-description11distressed = vocabulary.index("distressed</w>")12position = [i for i, token in enumerate(written) if token == distressed][1] # where greedy writes it a second time13before = written[:position]14print("text so far:", decoding.text(before))15print("penalty factor:", factor)1617logits = decoding.next_logits(start + before)18seen = np.unique(before) # every token the description has used19penalized = logits.copy()20penalized[seen] = np.where(logits[seen] > 0, logits[seen] / factor, logits[seen] * factor) # less likely either way21assert np.allclose(penalized, decoding.penalize(logits, before, factor)) # the hand calculation is the library's22for label, adjusted in [("without a penalty", logits), ("with the penalty", penalized)]:23 probabilities = transformer.softmax(adjusted)24 top = np.argsort(-probabilities, kind="stable")[:3]25 print(f"next token, {label}:", ", ".join(f"{vocabulary[i].replace('</w>', '')} {probabilities[i]:.2f}" for i in top))26 print(f"P(distressed), {label}:", f"{probabilities[distressed]:.2f}")27print("greedy with the penalty:", decoding.text(decoding.generate(start, penalty=factor)))
python repetition-penalty-by-hand.pytext so far: this welcome sign is a great way to add a touch of rustic charm to your home . it is made of natural wood and features a distressed finish with a
penalty factor: 1.3
next token, without a penalty: distressed 0.20, black 0.03, weathered 0.03
P(distressed), without a penalty: 0.20
next token, with the penalty: black 0.04, weathered 0.03, distressed 0.03
P(distressed), with the penalty: 0.03
greedy with the penalty: this welcome sign is a great way to add style and functionality to any room . the worn look of these beautiful signs are made from natural coir , so it can be used indoors or outdoors .
계수가 1.3일 때 distressed의 확률은 0.20에서 0.03로 떨어지고, 그 자리를 다른 토큰이 차지한다. 직접 계산한 값은 라이브러리의 계산과 일치하며, 페널티를 적용한 탐욕 설명문은 루프에 빠지지 않는다. 다만 내용을 지어내는 문제는 여전히 남는다. 이번에는 간판이 야자 껍질에서 얻은 섬유인 natural coir로 만들어졌다고 쓴다.
이 페널티에는 측정 결과에서 확인되는 부작용이 있다. 설명문이 더 일찍 끝난다는 점이다. 페널티는 이미 사용된 모든 토큰에 적용되므로, 설명문에 여러 번 필요한 the나 and 같은 흔한 단어에도 적용된다. 반면 아직 사용되지 않은 종료 토큰에는 적용되지 않는다. 그 결과 종료 토큰이 상대적으로 더 선택되기 쉬워진다. 페널티를 적용한 탐욕 설명문 중 제한 길이 안에서 끝난 비율은 70%로, 페널티가 없을 때의 25%보다 높았다. 평균 길이는 41토큰으로, 페널티가 없을 때의 52토큰보다 짧았다. 반복은 11%에서 0%로 줄었다.
작업 규칙: 탐욕적으로 디코딩하거나 낮은 온도로 디코딩할 때는 1.2에서 1.3 안팎의 반복 페널티를 추가하되, 페널티가 텍스트를 짧게 만들므로 길이에 미치는 영향을 확인한다. 경험적으로, 설명문이 잘려 나온다면 흔한 기능어와 문장부호를 페널티 대상에서 제외한다.
빈 설명 금지하기
탐욕 디코딩 예제에서 모델은 전구에 대해 아무것도 쓰지 않을 준비가 되어 있었고, 빔 서치에서는 표지판에 대한 가장 그럴듯한 설명이 아무것도 아닌 것으로 나타났다. 두 결과는 모두 카탈로그에서 비롯된다. 설명이 없는 제품이 많아서, 모델은 곧바로 끝내는 것이 흔하다고 학습했다. 바로 이런 빈칸을 채우려면 모델이 그것을 재현하지 못하도록 막아야 한다. 간단한 해결책은 최소 길이를 두는 것이다. 설명이 일정 개수의 토큰에 이를 때까지 종료 표지의 로짓을 음의 무한대로 설정하여 종료 표지를 금지한다. lib.decoding.generate은 이를 shortest로 받는다.
탐욕 디코딩이 비워 둔 전구가 시험 사례이다. 최소 길이 없이 한 번, 최소 길이를 10토큰으로 두고 한 번, 모두 두 번 디코딩한다. 출력에는 먼저 설명이 곧바로 끝날 확률이 나온다.
1"""Greedy decoding of the bulb's description, without a minimum length and with one that forbids the end marker until2it has that many tokens. The prose quotes the printed probability of an empty description, and3the description the minimum produces."""4from lib import decoding, transformer56end = transformer.load()[1].index(transformer.END) # the index of </s>, to read its probability7start, shortest = decoding.prompt("5 light light bulb"), 108print("P(the description is empty):", f"{transformer.softmax(decoding.next_logits(start))[end]:.2f}") # the first token is </s>: an empty description9print("greedy description:", decoding.text(decoding.generate(start)) or "(empty)")10print("minimum length, in tokens:", shortest)11print("greedy description, with the minimum:", decoding.text(decoding.generate(start, shortest=shortest))) # shortest: </s> stays forbidden until then
python minimum-length.pyP(the description is empty): 0.28
greedy description: (empty)
minimum length, in tokens: 10
greedy description, with the minimum: this light is a great light for a light , a light , and more light . it is made of metal and has a long , high gloss finish that will make it a perfect accent piece for your home . this light is dimmable , compatible with most standard bulbs , which are not included .
최소 길이는 빈칸을 채우지만 사실을 제공하지는 못한다. 최소 길이가 없으면 탐욕 디코딩은 빈 설명을 택하며, 그 확률은 0.28이다. 최소 길이를 두면 설명을 쓰는데, 이 장의 앞에서 본 두 가지 실패가 그 안에 나타난다. 하나는 a light , a light , and more light처럼 같은 말을 반복하는 것이고, 다른 하나는 이름에 없는 내용, 곧 전구가 made of metal이고 dimmable이며 compatible with most standard bulbs라는 내용을 서술하는 것이다.
작업 규칙: 학습 텍스트에 생성하려는 대상의 빈 예시나 매우 짧은 예시가 포함되어 있다면 최소 길이를 설정한다. 최소 길이는 이 장의 다른 설정을 대체하지 말고 함께 사용하며, 강제로 만들어 낸 텍스트도 다른 텍스트와 똑같이 확인이 필요하다고 예상한다.
운영 환경에서 잘못되는 것들
디코딩의 실패는 대부분 조용히 일어난다.
- 유창한 문장이 지어낸 사실을 말한다. 이 장에서 여섯 개의 설명이 환영 표지판의 재질을 제시했는데, 서로 달랐다. 생성한 상품 문구는 각 사실 주장을 상품 데이터와 대조해 확인하지 않은 채 게시해서는 안 된다. 11장은 모델이 얼마나 자주 지어내는지를 측정한다.
- 학습 텍스트의 빈틈이 출력이 된다. 카탈로그에 빈 설명이 많기 때문에 모델은 전구에 대해 빈 설명을 예측하고, 빔 서치는 그 빈 설명이 가장 그럴듯하다고 판단한다. 최소 길이에 이를 때까지 종료 표지를 금지하라. 그렇지 않으면 모델은 여러분이 메우려는 빈틈을 그대로 재현한다.
- 탐욕적 텍스트는 반복된다. 한 구절보다 긴 텍스트에는 꼬리를 자른 샘플링이나 반복 패널티를 사용하고, 출력 표본에서 반복 정도를 측정하라.
- 샘플링한 텍스트는 기본적으로 재현되지 않는다. 생성한 텍스트마다 난수 시드를 고정하고 기록하라. 그래야 설명 하나에 대한 불만이 들어왔을 때 그 설명을 추적하고 다시 만들어 볼 수 있다.
- 설정이 모델과 함께 변한다. 재학습한 모델은 분포가 다르므로, 이전 모델에 맞춰 조정한 온도나 top-k가 다음 모델에서는 다르게 글을 쓸 수 있다. 재학습할 때마다, 새 모델이 구매자에게 보이는 글을 쓰기 전에 이 장의 측정을 다시 실행하라.
- 토큰마다 순전파가 한 번씩 든다.
lib.decoding는 새 토큰마다 전체 텍스트에 대해 모델을 다시 실행한다. 이 방식은 단순하며 토큰이 60개일 때는 충분하다. 서빙 시스템은 이전 위치의 모든 층의 키와 값을 보관하므로, 새 토큰의 행렬 곱셈은 한 위치만 다룬다. 다만 어텐션은 여전히 이전 모든 위치의 캐시된 키와 값을 읽는다(Pope et al., 2022).
이 장에서 결정한 것
- 자동완성은 가능성이 가장 높은 토큰을 사용한다. 검색창 제안은 8장에서 고른 혼합의 상위 몇 개 토큰을 보여 주며, 총 확률로 순위를 매기는 빔 서치를 쓰면 이를 짧은 구로 확장할 수 있다.
- 상품 설명은 top-k 20 샘플링을 사용한다. 위의 측정 기준에서 반복 없이 상품에서 벗어나지 않은 경우가 가장 많았던 설정이며, 기록해 둔 고정 시드와 최소 길이를 두어 결과가 비어 나오지 않게 한다. 근거가 된 상품이 스무 개뿐이므로, 어떤 설명이든 구매자에게 보이기 전에 더 큰 표본으로 이 선택을 다시 확인한다.
- 빔 서치는 더 긴 텍스트를 토큰당 기준으로 순위를 매긴다. 총 확률은 빈 텍스트에 유리하기 때문이다. 검색어의 짧은 완성문은 총 확률로 순위를 매기는데, 이 경우 구매자가 원하는 것은 가장 가능성이 높은 구이기 때문이다.
- 디코딩 설정은 실제 텍스트에 대한 측정으로 고른다. 무작위로 뽑은 상품 집합에서 반복, 모델 자체의 퍼플렉서티, 설명이 끝맺음되는지, 이름을 사용하는지, 길이를 측정하며, 이는 샘플을 읽기 전에 수행하고 재학습할 때마다 다시 수행한다.
- 생성된 설명은 초안이며 상품 사실의 출처가 아니다. 이름과 카탈로그가 뒷받침하지 못하는 모든 주장은 사람이나 점검 단계가 바로잡는다. 여기서 다룬 모든 방법이, 이름에 재질이 나오지 않는 상품에 재질을 지어냈기 때문이다.
이 판의 이 장
2026.11.16판예제 통과 2026년 11월 19이 장에서 7개 예제개 중 7개가 통과한다
영어판 2026.11.16판을 번역한 것이다.