Choosing the next token
読了まで約34 分分、9 節 節の 7 例 例。
この部分はまだ翻訳されていないため、英語で表示している。
By the end of this chapter you can turn a language model's probabilities into text, and choose how: always the likeliest token, the likeliest whole text by beam search, or a sample shaped by a temperature, a top-k or top-p cut, a repetition penalty and a minimum length. You'll measure what each choice does to the text, and learn which failures no choice of setting can fix.
A language model gives a probability for every possible next token; it doesn't write. To write, something has to choose one token, append it, ask for the next probabilities, and repeat until the model predicts the end marker, </s>. Turning a model's probabilities into text, one chosen token at a time, is called decoding, and it decides much of what the text reads like, with the model unchanged.
The store has a reason to write: of the furniture and home goods in its validation split, 597 of 4,282 products have no description, and so do 14% of its training products. The store's model, the transformer that chapter 8 trained and measured, learned descriptions from the store's own text, so it can draft one from a product's name: start the text with the name and <sep>, as every training text was laid out, and let the model continue. This chapter does that with greedy decoding, beam search and sampling, using the book's shared module lib.decoding, built in NumPy on lib.transformer. Then it measures what each method writes against the store's real descriptions, and fixes two failures that the measurements expose: repetition, and descriptions that come out empty.
最も確からしいトークンを選ぶ
最も単純なデコーダは貪欲法(greedy)である。各ステップで、最も確率の高いトークンを一つだけ選ぶ。乱数を必要としないので、同じ書き出しからは常に同じテキストが得られ、各ステップのコストはモデルの順伝播1回分である。こうして書かれるテキストは、トークンごとに見たモデルの最善の推測である。
説明文のない商品を埋めることが店の必要なので、まずそのような商品から始める。front porch welcome sign wall décorを、最大60トークンまで貪欲にデコードする。次に二つ目の商品として5 light light bulbを取り上げ、その説明文が空である確率を表示する。
1"""Greedy decoding writes one description of a product that has none, and shows how it repeats itself, and how likely2the model is to leave a bulb's description empty. The prose quotes the greedy description, its repeated phrases, and3the line with the bulb's probability of an empty description."""4from collections import Counter56from lib import decoding, transformer7from lib.store import load_products89validation, training = load_products("validation"), load_products("train")10empty = lambda rows: rows["description"].fillna("").str.strip() == ""11print("validation products:", f"{len(validation):,}")12print("validation products with no description:", f"{empty(validation).sum():,}")13print("training products with no description:", f"{empty(training).mean():.0%}")14bulbs = training[training["name"].str.contains(r"\bbulbs?\b")]15print("training products named as bulbs with no description:", f"{empty(bulbs).mean():.0%}")1617longest = 6018print("longest description, in tokens:", longest)19ids = decoding.generate(decoding.prompt("front porch welcome sign wall décor"), longest=longest) # no rng: each step takes the likeliest token20print("greedy description:", decoding.text(ids))21fourgrams = Counter(tuple(ids[i:i + 4]) for i in range(len(ids) - 3)) # the four tokens from i: a window that slides one token at a time22print("4-token phrases it uses twice or more:", " | ".join(decoding.text(list(g)) for g, n in fourgrams.items() if n > 1))2324end = transformer.load()[1].index(transformer.END) # the index of </s> in the vocabulary25probabilities = transformer.softmax(decoding.next_logits(decoding.prompt("5 light light bulb"))) # the first token after <sep>26print("P(the description is empty), 5 light light bulb:", f"{probabilities[end]:.2f}")
python greedy-description.pyvalidation products: 4,282
validation products with no description: 597
training products with no description: 14%
training products named as bulbs with no description: 24%
longest description, in tokens: 60
greedy description: this welcome sign is a great way to add a touch of rustic charm to your home . it is made of natural wood and features a distressed finish with a distressed finish . this welcome sign is made of wood and has a distressed finish that adds a rustic touch to your home . this sign is made from
4-token phrases it uses twice or more: this welcome sign is | to your home .
P(the description is empty), 5 light light bulb: 0.28
貪欲法で得られた説明文は、店自身の文章のように読めるが、やがてループする。そう読めるのは、これが店の文章を組み合わせ直したものだからであり、たとえばthis welcome sign is a great way to add a touch of rustic charm to your homeがそうである。この説明文はfeatures a distressed finish with a distressed finishと書き、後でふたたびhas a distressed finishと書く。さらに、4トークンから成る二つの句、this welcome sign isとto your home .を2回ずつ使っている。これらは例の出力に挙がっている。貪欲法がループするのは、テキストが似た地点に達するたびに、同じ最尤の続きが選ばれるからである。distressedは看板の仕上げについてモデルが最も好む結びであり、仕上げを述べる箇所が来るたびに現れる。Holtzmanらは大規模モデルでも同じ現象を見いだした。常に確率を最大化するデコードは、「退屈で、一貫性がなく、あるいは反復のループに陥る」テキストを導く(Holtzman et al., 2020)。
また、貪欲法の説明文は、この看板がmade of natural woodであると述べているが、商品名のどこにもそのような記述はない。電球は別の失敗を示す。5 light light bulbに対して、モデルは即座の終了、すなわち空の説明文を、確率0.28で出力する。モデルはカタログを、欠落も含めてそのまま学習したのである。電球という名前の学習用商品のうち、説明文のないものは24%であり、全商品での14%に対する割合である。空の説明文への対処は後の節で扱う。
実践ルール: 毎回同じ出力が欲しい場合や、検索ボックスの候補表示のように、最も確からしい次の単語こそが目的となる短いテキストの場合には、貪欲法を使う。句より長いものでは、繰り返しが起こると予想し、以降の節の手法のいずれかを選ぶ。
最も確からしいテキストを探索する
貪欲デコーディングは各ステップで最も確からしいトークンを選ぶが、それは最も確からしいテキストを選ぶこととは同じではない。最初の単語としてやや確率の低いものを選んだ方が、その後に続く単語がはるかに確からしくなることがある。ビームサーチはより先まで見通し、最も確からしい部分テキストをk個保持する。この集合をビームと呼び、各テキストは総対数確率で評価される。総対数確率とは、テキストの各トークンの確率の対数を合計したものであり、値が高いほど確からしい。各ステップでは、ビーム内の各テキストを、次に最も確からしいk個のトークンでそれぞれ延長し、得られた結果のうち上位k個を残す。終端マーカーに達したテキストは完成したものとして扱われ、競争に残り続ける。lib.decoding.beamの中心は、各延長のスコアを計算する次の行である。
candidates += [(score + log_p[token], written + [int(token)]) for token in np.argsort(-log_p, kind="stable")[:width]]
ウェルカムサインの説明として、ビームサーチはどの記述を最も確からしいと順位づけすると予想するだろうか。ビーム幅を4として、2回実行する。1回目は検索クエリoutdoor barを最大4トークン(名前の終端を含む)まで補完する。これは検索ボックスが句全体を提案する場面に相当する。2回目はウェルカムサインの説明を書き、全体として最も確からしいテキストと、トークンあたりの値(総対数確率をその長さで割ったもの)を出力する。
1"""Beam search with a beam of four, on a search query and on a product's description. The prose quotes the two2completions of outdoor bar, then the likeliest description, its probability and the likeliest per token."""3import numpy as np45from lib import decoding, transformer67_, vocabulary, _ = transformer.load()8width, longest = 4, 49print("beam width:", width)10print("longest completion, in tokens, its end included:", longest)11start = decoding.prompt("outdoor bar")[:-1] # a search query: <s> and the words typed so far12for score, written in decoding.beam(start, width=width, longest=longest, stops=(transformer.SEP,))[:2]: # stop at <sep>, where a name ends; keep the two likeliest13 print("completion of outdoor bar:", decoding.text([t for t in written if vocabulary[t] != transformer.SEP]),14 f"(probability {np.exp(score):.3f})")1516kept = decoding.beam(decoding.prompt("front porch welcome sign wall décor"), width=width) # longest and stops take their defaults: 60 tokens, </s>17score, written = kept[0] # the likeliest text overall18print("likeliest description:", decoding.text(written) or "(empty)")19print("its probability:", f"{np.exp(score):.2f}")20score, written = max(kept, key=lambda candidate: candidate[0] / len(candidate[1])) # the likeliest per token21print("likeliest description per token:", decoding.text(written))
python beam-search.pybeam width: 4
longest completion, in tokens, its end included: 4
completion of outdoor bar: stool (probability 0.365)
completion of outdoor bar: stool cushion (probability 0.089)
likeliest description: (empty)
its probability: 0.15
likeliest description per token: welcome guests to your home with this welcome sign . crafted of iron , this welcome sign is a great way to add a rustic touch to your home .
クエリに対しては、ビームサーチは検索ボックスに求められるものを返す。短く確からしい句であり、最初にstool、次にstool cushionが現れる。ここでは Transformer だけを使っている。第8章でオートコンプリート用に選んだ混合方式では、各ステップですべての候補について4-gramの確率が必要になるが、lib.decodingはそれを計算しない。
全体として最も確からしい説明は空である。直ちに終了する確率は0.15であり、それより長いテキストはどれも1未満の確率を多数掛け合わせるため、これに遠く及ばない。StahlbergとByrneは翻訳モデルでも同じ現象を見出した。大半の文において最も確率の高い翻訳は空であり、ビームサーチが機能していたのは、探索が不十分だったからにすぎなかった(Stahlberg and Byrne, 2019)。トークンあたりの確率で順位づけすればこの偏りは取り除かれ、妥当な説明が得られる。それでも素材を作り出してしまう。今回は看板がcrafted of ironであるとされる。
実践ルール: ビームサーチは、検索クエリのように最も確からしい句を得ることが目的である短い補完に用い、総確率で順位づけする。より長いテキストでは、総確率は最も短いものを有利にするため、完成したテキストをトークンあたりで順位づける。ただしその場合でも、人間が書く文章により近いテキストを生成できる次の各節のサンプリング手法を優先する。
温度を指定してサンプリングする
グリーディーデコーディングとビームサーチは、分布の上位だけを使う。第1章で名前を生成するのに用いたサンプリングは、分布の全体を使う。次のトークンを、その確率に比例してランダムに引くのである。したがって、確率0.2のトークンは5回に1回の割合で選ばれる。サンプリングの前に、温度 T によって分布を鋭くすることも平らにすることもできる。方法は、softmax の前にすべてのロジットを T で割ることである。第5章と同様に、zᵢ をトークン i のロジットとすると、次のように書ける。
P(token i) = exp(zᵢ / T) / Σⱼ exp(zⱼ / T)
温度が1より低いと分布は鋭くなり、最も確率の高いトークンに確率がより多く与えられる。1より高いと分布は平らになり、可能性の低いトークンにも確率が多く回る。温度が0に近づくとグリーディーデコーディングに近づき、温度が1ならモデルの確率はそのままである。T で割ってもトークンの順序は変わらず、変わるのは確率の開き具合だけである。
温度が実際の一つの選択に何をもたらすかを見るために、グリーディーデコーディングによる記述を途中で止め、次のトークンの分布を3通りの温度で読む。それぞれの温度について、この例は確率が最も高い3トークン、確率の90%を覆うのに必要なトークン数、そして温度1の90%の範囲の外に出る確率の大きさを出力する。
1"""The next token's distribution after the first nine tokens of the greedy description, at three temperatures, and how2many tokens each needs to cover 90% of the probability. The prose quotes the three counts, and the share at 1.5 that3falls outside temperature 1's 90% cover."""4import numpy as np56from lib import decoding, transformer78_, vocabulary, _ = transformer.load()9start = decoding.prompt("front porch welcome sign wall décor")10so_far = decoding.generate(start, longest=9) # the greedy description's first 9 tokens11logits = decoding.next_logits(start + so_far)12print("text so far:", decoding.text(so_far))13print("tokens in the vocabulary:", f"{len(vocabulary):,}")14cover = np.argsort(-transformer.softmax(logits), kind="stable") # at temperature 1, the likeliest first15cover = cover[:int(np.searchsorted(np.cumsum(transformer.softmax(logits)[cover]), 0.9)) + 1] # temperature 1's 90% cover, its last token included16for temperature in [0.5, 1.0, 1.5]:17 probabilities = transformer.softmax(logits / temperature) # divide the logits, then softmax18 top = np.argsort(-probabilities, kind="stable")19 covering = int(np.searchsorted(np.cumsum(probabilities[top]), 0.9)) + 1 # +1: the token that reaches 90% counts20 print(f"temperature {temperature}, top 3:", ", ".join(f'"{vocabulary[i].replace("</w>", "")}" {probabilities[i]:.2f}' for i in top[:3]))21 print(f"temperature {temperature}, tokens needed to cover 90% of the probability:", covering)22 print(f"temperature {temperature}, probability outside temperature 1.0's 90% cover:",23 f"{1 - probabilities[cover].sum():.0%}")
python temperature-by-hand.pytext so far: this welcome sign is a great way to add
tokens in the vocabulary: 4,142
temperature 0.5, top 3: "a" 0.61, "style" 0.20, "some" 0.11
temperature 0.5, tokens needed to cover 90% of the probability: 3
temperature 0.5, probability outside temperature 1.0's 90% cover: 0%
temperature 1.0, top 3: "a" 0.26, "style" 0.15, "some" 0.11
temperature 1.0, tokens needed to cover 90% of the probability: 78
temperature 1.0, probability outside temperature 1.0's 90% cover: 10%
temperature 1.5, top 3: "a" 0.09, "style" 0.06, "some" 0.05
temperature 1.5, tokens needed to cover 90% of the probability: 640
temperature 1.5, probability outside temperature 1.0's 90% cover: 42%
温度は、確率がどこまで広がるかを決める。this welcome sign is a great way to addの後では、温度1のときモデルが90%を覆うのに必要とするトークンは78個であり、残りは長い裾野で、そこにあるトークンは一つ一つの可能性が低い。0.5では3個で90%を覆う。1.5では640個が必要になり、確率の42%が温度1の90%の範囲の外に出る。
| 温度 | 確率が最も高い3トークン | 90%を覆うトークン数 | T = 1 の90%の範囲の外 |
|---|---|---|---|
| 0.5 | "a" 0.61, "style" 0.20, "some" 0.11 | 3 | 0% |
| 1 | "a" 0.26, "style" 0.15, "some" 0.11 | 78 | 10% |
| 1.5 | "a" 0.09, "style" 0.06, "some" 0.05 | 640 | 42% |
この裾野こそが、サンプリングの危険である。裾野にあるトークンの大半はここでは誤りであり、温度1ではサンプラーは約10トークンに1回の割合でそこから引く。温度が上がるほど、その頻度は増える。
実践ルール: 温度は、予測しやすいテキストと変化に富むテキストとの間のダイヤルとして扱う。商品説明のように、学習テキストが通常述べている内容に近いままであるべきテキストには、温度を1か、1よりやや低い値に保つ。温度を下げるとテキストは予測しやすくなるが、正確になるわけではない。
サンプリングと裾の切り捨て
温度を下げると裾は小さくなるが、完全には消えない。サンプリングの前に裾を直接取り除く方法が二つある。Top-k は確率の高い上位 k 個のトークンだけを残し、それらの確率の和が 1 になるように再正規化する(Fan et al., 2018)。Top-p は核サンプリング(nucleus sampling)とも呼ばれ、確率の合計が少なくとも p に達する、確率の高いトークンの最小の集合を残す(Holtzman et al., 2020、式 2)。どちらも lib.decoding.truncate のオプションであり、その中核は三つの手順からなる。トークンを並べ替え、その順に確率を足し上げ、和が p に達する位置を見つけるのである。
order = np.argsort(-probabilities, kind="stable")
...
keep = int(np.searchsorted(np.cumsum(probabilities[order]), top_p)) + 1
top-k と違い、top-p はモデルの確信度に適応する。前の節の例で、モデルが次の単語に迷っていた箇所では、確率の 90% を覆うのに 78 個のトークンが必要だった。したがって top-p 0.9 はその数だけ残すが、top-k 20 は 20 個しか残さない。モデルがほぼ確実に予測する単語の後では、top-p は 1 個か 2 個のトークンを残すのに対し、top-k は依然として 20 個を残し、その大半は誤りである。Holtzman らはまさにこの点を根拠に top-p を支持し、固定の k は「文脈が変わると最適とは言えない」と論じた。
四つのサンプラーが、それぞれ同じ乱数シードから、ウェルカムサインの説明文を 40 トークンまで書く。温度 1 でのサンプリング、0.7 でのサンプリング、k = 20 の top-k、p = 0.9 の top-p である。続いて、この例は温度 1 のサンプルを再生し、各ステップで 90% の核の外から引かれたトークンを一覧にする。これらは top-p 0.9 であれば切り捨てられていたトークンである。
1"""Four samplers write the same description from one seed, and the temperature-1 draws are then replayed to mark the2tokens outside each step's 90% nucleus. The last two lines are the ones the prose quotes: how many draws came from3the tail, and which tokens they were."""4import numpy as np56from lib import decoding, transformer78_, vocabulary, _ = transformer.load()9start, longest, seed = decoding.prompt("front porch welcome sign wall décor"), 40, 202610print("longest description, in tokens:", longest)11print("random seed:", seed)12for label, settings in [("sampling, temperature 1.0", {}), ("sampling, temperature 0.7", {"temperature": 0.7}),13 ("top-k 20", {"top_k": 20}), ("top-p 0.9", {"top_p": 0.9})]:14 ids = decoding.generate(start, rng=np.random.default_rng(seed), longest=longest, **settings)15 print(f"{label}:", decoding.text(ids))1617rng, written, tail = np.random.default_rng(seed), [], [] # replay temperature 1.0's draws, step by step18for _ in range(longest):19 probabilities = transformer.softmax(decoding.next_logits(start + written))20 token = int(rng.choice(len(probabilities), p=decoding.truncate(probabilities))) # no cut: draw from all of it21 order = np.argsort(-probabilities, kind="stable")22 before = probabilities[order[:int(np.where(order == token)[0][0])]].sum() # the probability ranked above it23 if before >= 0.9: # top-p 0.9 would have cut this token24 tail.append(vocabulary[token].replace("</w>", ""))25 written.append(token)26 if vocabulary[token] == transformer.END:27 break28print("temperature 1.0, tokens drawn from outside the 90% nucleus:", f"{len(tail)} of {len(written)}")29print("temperature 1.0, those tokens:", ", ".join(tail))
python sample-descriptions.pylongest description, in tokens: 40
random seed: 2026
sampling, temperature 1.0: enhance your home even glywheel design with this welcome welcome sign sign . this piece is ideal for both indoor and inviting . made in the usa , mini spirits decorate .
sampling, temperature 0.7: the sign is a great way to add some extra whimsy to your home and get colorful ! made of high-quality metal , this sign comes with a hanging dustroke hanger to help you gli
top-k 20: if you have a new product with a personalized sign wall décor that will help you entertain . look no further like a home decor , this sign is constructed to make life more wonderful . this welcome sign
top-p 0.9: express your guests with this cast iron sign from the nature of the world . it features a meditating podishade design . the hose and symbol in this metal piece will be
temperature 1.0, tokens drawn from outside the 90% nucleus: 5 of 36
temperature 1.0, those tokens: inviting, in, mini, spir, decorate
裾を切り捨てても、取り除けるのはまれな誤りだけであり、すべての誤りではない。温度 1 のサンプリングは迷走する。welcome welcome sign sign と書き、glywheel という語を作り出し、サインが made in the usa だと主張する。このうち裾に由来するのは一部にすぎない。裾から引かれたトークンは 5 of 36 個で、inviting, in, mini, spir, decorate。単語の重複と造語は、top-p が残したはずの核の内側のトークンから作られていた。0.7 では文章のまとまりがよくなるが、貪欲デコーディングが木製と書いたところで、今度はサインが made of high-quality metal だと述べる。top-k と top-p は、単純なサンプリングよりカタログらしい文章になるが、それでも作り話をする。top-p のサインは cast iron で、podishade のデザインだという。
この章を通じて、ウェルカムサインには素材が六回与えられた。貪欲デコーディング、ビームサーチ、これらのサンプル三つ、そして後述のペナルティ付き貪欲法のテキストである。しかし、サインの名前からは何でできているかは分からない。モデルはウェルカムサインの説明が通常述べることを書き、その説明は通常、素材を明記する。デコーディング手法が変えるのは、モデルがどのもっともらしい素材を選ぶかであり、真の素材を選ばせることはどの手法にもできない。
実践ルール: 多様なテキストが必要なときは、top-p を 0.9 前後、または top-k を 20 前後にして裾を切り捨て、温度は 1 か、それよりやや低い値にする。結果を再現する必要があるときは、乱数シードを固定する。どの設定でもテキストが真実になるとは期待しないこと。デコーディングはモデルがもっともらしいと見なすものの中から選ぶだけであり、事実の確認は別の作業である。
各設定が書く文章を測定する
単一のサンプルは各手法の特徴を示すが、トレードオフを示すのは測定である。人がテキストを読まなくても測定できる性質がいくつかある。
- 繰り返し: 同じ説明文の中で、先行する4トークンの句を繰り返している4トークンの句の割合
- モデル自身によるテキストのパープレキシティ: モデルが自分の書いた内容をどれだけ意外だと見なすかを、全トークンにわたってまとめた値である。低いほど予測しやすいことを意味する
- 制限内で説明文が終わるか: 実際の説明文と同様に終わるかどうか
- 商品名の単語を丸ごと1つ使っているか: その商品の説明文であれば使うべきである
- 平均の長さ: トークン数で表す
各設定は実際のテキストにどこまで近づくだろうか。説明文のある検証用商品から 20 件を無作為に選び、それぞれについて各手法で説明文を書かせる。そして、同じ 60 トークンの制限で切り詰めた実際の説明文と比較する。
1"""Seven ways of choosing the next token, measured on the same 20 product descriptions: how often each repeats2itself, how likely the model finds its own text, whether it ends, and whether it stays on the product. The real3descriptions come first, as the bar the others are read against. The prose quotes these lines for each setting."""4import numpy as np56from lib import decoding # the store's model, and the decoding methods this chapter builds7from lib.products import product_texts89longest, sample, seed = 60, 20, 202610# Validation products, which the model never trained on, that have at least one token of description after <sep>11described = [(ids, sep) for ids, sep in product_texts("validation") if sep < len(ids) - 2]12products = [described[i] for i in np.random.default_rng(seed).choice(len(described), sample, replace=False)]13print("products, drawn at random from the validation products with a description:", sample)14print("longest description, in tokens:", longest)151617settings = [("real descriptions", None), ("greedy", {}), ("greedy, repetition penalty 1.3", {"penalty": 1.3}),18 ("sampling, temperature 1.0", {"sample": True}), ("sampling, temperature 0.7", {"sample": True, "temperature": 0.7}),19 ("top-k 20", {"sample": True, "top_k": 20}), ("top-p 0.9", {"sample": True, "top_p": 0.9})]20for label, chosen in settings:21 rows = []22 for number, (ids, sep) in enumerate(products):23 start = ids[:sep + 1] # the start marker, the name and <sep>: the model writes on from here24 if chosen is None:25 written = ids[sep + 1:][:longest] # the description as sold, cut where generating would stop26 else:27 options = {option: value for option, value in chosen.items() if option != "sample"}28 # generate samples when it's given a generator, and is greedy without one. Each product gets its own29 # seed, the same for every sampling setting, so the settings differ only in how they choose.30 rng = np.random.default_rng(seed + number) if chosen.get("sample") else None31 written = decoding.generate(start, rng=rng, longest=longest, **options)32 rows.append(decoding.measures(start, written)) # repetition, each token's log-probability, ended, uses the name, length33 repeated, log_p, ended, uses_name, length = zip(*rows)34 print(f"{label}, repeated 4-token phrases:", f"{np.mean(repeated):.0%}")35 print(f"{label}, the model's perplexity of the text:", f"{np.exp(-np.concatenate(log_p).mean()):.1f}") # over all 20 texts' tokens at once36 print(f"{label}, ended within {longest} tokens:", f"{np.mean(ended):.0%}")37 print(f"{label}, uses a whole word of the name:", f"{np.mean(uses_name):.0%}")38 print(f"{label}, average length in tokens:", f"{np.mean(length):.0f}")
python measure-decoding.pyproducts, drawn at random from the validation products with a description: 20
longest description, in tokens: 60
real descriptions, repeated 4-token phrases: 0%
real descriptions, the model's perplexity of the text: 30.5
real descriptions, ended within 60 tokens: 35%
real descriptions, uses a whole word of the name: 95%
real descriptions, average length in tokens: 53
greedy, repeated 4-token phrases: 11%
greedy, the model's perplexity of the text: 2.7
greedy, ended within 60 tokens: 25%
greedy, uses a whole word of the name: 80%
greedy, average length in tokens: 52
greedy, repetition penalty 1.3, repeated 4-token phrases: 0%
greedy, repetition penalty 1.3, the model's perplexity of the text: 3.9
greedy, repetition penalty 1.3, ended within 60 tokens: 70%
greedy, repetition penalty 1.3, uses a whole word of the name: 80%
greedy, repetition penalty 1.3, average length in tokens: 41
sampling, temperature 1.0, repeated 4-token phrases: 0%
sampling, temperature 1.0, the model's perplexity of the text: 22.6
sampling, temperature 1.0, ended within 60 tokens: 50%
sampling, temperature 1.0, uses a whole word of the name: 50%
sampling, temperature 1.0, average length in tokens: 45
sampling, temperature 0.7, repeated 4-token phrases: 0%
sampling, temperature 0.7, the model's perplexity of the text: 7.2
sampling, temperature 0.7, ended within 60 tokens: 45%
sampling, temperature 0.7, uses a whole word of the name: 65%
sampling, temperature 0.7, average length in tokens: 45
top-k 20, repeated 4-token phrases: 0%
top-k 20, the model's perplexity of the text: 7.6
top-k 20, ended within 60 tokens: 50%
top-k 20, uses a whole word of the name: 80%
top-k 20, average length in tokens: 40
top-p 0.9, repeated 4-token phrases: 0%
top-p 0.9, the model's perplexity of the text: 10.7
top-p 0.9, ended within 60 tokens: 30%
top-p 0.9, uses a whole word of the name: 60%
top-p 0.9, average length in tokens: 47
貪欲法のテキストは、人が書いたどの文章よりもはるかに予測しやすく、パープレキシティは 2.7 である。しかも4トークンの句の 11% が繰り返される。実際の説明文には繰り返しがなく(4トークンの句の 0%)、モデルはそれらをかなり意外だと見なし、パープレキシティは 30.5 である。この値は第8章の説明文の値より高い。今回は別のサンプルであり、先頭の 60 トークンだけを対象にしているためである。Holtzman らは同じ観察を大規模に行っている。人間のテキストは、モデルが最も確からしいと見なすテキストではない。
すべての指標で実際の説明文に一致する設定はない。温度1のサンプリングはパープレキシティでは最も近く 22.6 であるが、商品名の単語を使う説明文は 50% にとどまり、実際の説明文の 95% に及ばない。つまり商品から逸れていく。top-p 0.9 は実際のテキストとほぼ同じ頻度で終わり、30% に対して 35% であるが、商品に沿うのは 60% にすぎない。top-k 20 はサンプリング手法の中で最も商品に沿っており、80% である。繰り返しはなく、説明文は平均 40 トークンである。20件は小さなサンプルなので、これらは正確な率ではなく傾向として読むこと。
これらの指標が語れないのは、説明文が真実かどうかである。説明文は、流暢で、多様で、主題に沿っていても、誤っていることがある。前の節の説明文がそうであり、看板の素材について誤っていた。
実践ルール: サンプルを読む前に、このような指標で、同じ種類の実際のテキストと比べてデコード設定を比較する。その作業にとってどの指標が最も重要かを決める。商品説明であれば、商品に沿うことと繰り返さないことが最優先である。そのうえで、どの指標も捉えないものを見つけるために、出力のサンプルを必ず読み、事実に関する主張はすべて商品のデータと照合する。
繰り返しにペナルティを課す
貪欲デコーディングのループは、サンプリングを使わなくても解消できる。Keskar らは繰り返しペナルティを提案した。すでに生成されたすべてのトークンのロジットを 1 より大きい係数 θ で割り、そのトークンが選ばれにくくなるようにする方法である(Keskar et al., 2019、4.1 節)。負のロジットを割るとかえって値が大きくなるため、実装では負のロジットには θ を掛ける。lib.decoding.penalize もそのようにしており、どちらの場合でもそのトークンは選ばれにくくなる。Keskar らは、係数が 1.2 前後であれば、繰り返しの回避と文章の自然さのバランスが取れることを見いだした。
貪欲な説明文が distressed を再び書こうとしていた箇所に戻る。次のトークンのロジットにペナルティを手計算で適用し、結果を lib.decoding.penalize と照合して、適用前後の確率を比較する。続いて、ループが消えるかどうかを確かめるため、ペナルティを付けて貪欲に文章を生成する。
1"""The repetition penalty applied by hand to the next token's logits, checked against lib.decoding.penalize, and then2used for a whole greedy description. The two P(distressed) lines are the ones the prose quotes, and the3assert against lib.decoding.penalize is the match it states as fact."""4import numpy as np56from lib import decoding, transformer78_, vocabulary, _ = transformer.load()9start, factor = decoding.prompt("front porch welcome sign wall décor"), 1.310written = decoding.generate(start) # greedy, as in greedy-description11distressed = vocabulary.index("distressed</w>")12position = [i for i, token in enumerate(written) if token == distressed][1] # where greedy writes it a second time13before = written[:position]14print("text so far:", decoding.text(before))15print("penalty factor:", factor)1617logits = decoding.next_logits(start + before)18seen = np.unique(before) # every token the description has used19penalized = logits.copy()20penalized[seen] = np.where(logits[seen] > 0, logits[seen] / factor, logits[seen] * factor) # less likely either way21assert np.allclose(penalized, decoding.penalize(logits, before, factor)) # the hand calculation is the library's22for label, adjusted in [("without a penalty", logits), ("with the penalty", penalized)]:23 probabilities = transformer.softmax(adjusted)24 top = np.argsort(-probabilities, kind="stable")[:3]25 print(f"next token, {label}:", ", ".join(f"{vocabulary[i].replace('</w>', '')} {probabilities[i]:.2f}" for i in top))26 print(f"P(distressed), {label}:", f"{probabilities[distressed]:.2f}")27print("greedy with the penalty:", decoding.text(decoding.generate(start, penalty=factor)))
python repetition-penalty-by-hand.pytext so far: this welcome sign is a great way to add a touch of rustic charm to your home . it is made of natural wood and features a distressed finish with a
penalty factor: 1.3
next token, without a penalty: distressed 0.20, black 0.03, weathered 0.03
P(distressed), without a penalty: 0.20
next token, with the penalty: black 0.04, weathered 0.03, distressed 0.03
P(distressed), with the penalty: 0.03
greedy with the penalty: this welcome sign is a great way to add style and functionality to any room . the worn look of these beautiful signs are made from natural coir , so it can be used indoors or outdoors .
係数が 1.3 のとき、distressed の確率は 0.20 から 0.03 に下がり、代わりに別のトークンが選ばれる。手計算の結果はライブラリの結果と一致し、ペナルティを付けた貪欲な説明文はループしない。ただし、作り話をする傾向は残っている。今回は、看板がヤシ殻から取れる繊維である natural coir でできていることになっている。
このペナルティには副作用があり、測定結果にもそれが表れている。説明文が早く終わるのである。ペナルティは、すでに使われたすべてのトークンに課される。説明文の中で何度も必要になる the や and のような一般的な語も対象になるが、まだ使われていない終了マーカーが対象になることはない。そのため、終了は相対的に選ばれやすくなる。実際に、ペナルティを付けた貪欲な説明文のうち、上限内で終わったものは 70% で、ペナルティなしの 25% を上回った。平均の長さは 41 トークンで、ペナルティなしでは 52 トークンだった。繰り返しは 11% から 0% に減った。
実践ルール: 貪欲にデコードするときや低い温度でデコードするときは、1.2 から 1.3 前後の繰り返しペナルティを加え、文章が短くなるため長さへの影響を確認する。経験則として、説明文が途中で切れたようになる場合は、一般的な機能語と句読点をペナルティの対象から外す。
空の説明を禁止する
貪欲法の例では、モデルが電球に対して何も書こうとしないことが分かった。また、ビームサーチでは、看板に対して最も尤もらしい説明が空であることが分かった。どちらの原因もカタログにある。説明のない商品が多いため、モデルはすぐに終了することが一般的だと学習したのである。まさにそうした空白を埋めるには、モデルがそれを再現しないようにしなければならない。簡単な対策は最小長である。説明が一定のトークン数に達するまで、終端マーカーのロジットを負の無限大に設定して、終端マーカーを禁止する。lib.decoding.generateでは、これを shortest として受け取る。
貪欲デコーディングが空のままにした電球を、テストケースとして用いる。最小長なしの場合と、最小長を 10 トークンとした場合の2通りでデコードする。出力は、まず説明が直ちに終わる確率を示す。
1"""Greedy decoding of the bulb's description, without a minimum length and with one that forbids the end marker until2it has that many tokens. The prose quotes the printed probability of an empty description, and3the description the minimum produces."""4from lib import decoding, transformer56end = transformer.load()[1].index(transformer.END) # the index of </s>, to read its probability7start, shortest = decoding.prompt("5 light light bulb"), 108print("P(the description is empty):", f"{transformer.softmax(decoding.next_logits(start))[end]:.2f}") # the first token is </s>: an empty description9print("greedy description:", decoding.text(decoding.generate(start)) or "(empty)")10print("minimum length, in tokens:", shortest)11print("greedy description, with the minimum:", decoding.text(decoding.generate(start, shortest=shortest))) # shortest: </s> stays forbidden until then
python minimum-length.pyP(the description is empty): 0.28
greedy description: (empty)
minimum length, in tokens: 10
greedy description, with the minimum: this light is a great light for a light , a light , and more light . it is made of metal and has a long , high gloss finish that will make it a perfect accent piece for your home . this light is dimmable , compatible with most standard bulbs , which are not included .
最小長は空白を埋めるが、事実を与えることはできない。最小長がなければ、貪欲デコーディングは空の説明を選び、その確率は 0.28 である。最小長を設けると説明が書かれるが、その中にはこの章の前半で見た二つの失敗が現れる。一つは、a light , a light , and more light のように同じ内容を繰り返すことである。もう一つは、名前にない事柄を述べることであり、この電球が made of metal、dimmable、そして compatible with most standard bulbs であると書いている。
実践ルール: 生成対象について、学習テキストに空の例や非常に短い例が含まれている場合は、必ず最小長を設定する。最小長は、この章の他の設定の代わりにではなく、それらと組み合わせて用いる。また、最小長によって強制的に出力させたテキストにも、他のテキストと同様の確認が必要になると考えておく。
本番環境で起こる問題
デコーディングの失敗の多くは目立たない形で起こる。
- 流暢な文章が作り話の事実を述べる。 この章では、6 つの説明文が歓迎看板に素材を与えたが、それらは互いに食い違っていた。生成した商品文は、個々の事実に関する記述を商品データと照合せずに公開してはならない。モデルがどのくらいの頻度で事実を作り出すかは、11 章で測定する。
- 学習テキストの欠落が出力になる。 カタログには空の説明文を持つ電球が多いため、モデルは電球に対して空の説明文を予測し、ビームサーチはその空の説明文を最も確からしいものとして見つける。最小長に達するまで終端マーカーを禁止しなければ、モデルは埋めようとしているその欠落を再現してしまう。
- 貪欲法のテキストは繰り返しに陥る。 句より長い文には、裾を切り落としたサンプリングか繰り返しペナルティを使い、出力の標本で繰り返しを測定する。
- サンプリングしたテキストは、既定では再現できない。 生成したテキストごとに乱数シードを固定して記録する。そうすれば、ある説明文への苦情を追跡して再現できる。
- 設定はモデルとともにずれる。 再学習したモデルは分布が異なるため、前のモデルで調整した temperature や top-k が、次のモデルでは異なる文章を書くことがある。再学習のたびに、新しいモデルが買い物客の目に触れる文章を書く前に、この章の測定をやり直す。
- トークンごとに順伝播が 1 回必要になる。
lib.decodingは、新しいトークンごとにテキスト全体に対してモデルを再実行する。これは単純であり、60 トークンであれば問題ない。サービング用のシステムは、以前の位置における各層のキーと値を保持するため、新しいトークンごとの行列積は 1 つの位置だけを対象とする。ただし、アテンションは、それ以前のすべての位置について、キャッシュされたキーと値を読み取る(Pope et al., 2022)。
この章で決めたこと
- オートコンプリートには最も尤度の高いトークンを用いる。 検索ボックスの候補には、第8章で選んだ混合における上位数個のトークンを表示する。総確率で順位づけするビームサーチを使えば、これらを短いフレーズに延ばせる。
- 商品説明には top-k 20 のサンプリングを用いる。 上記の指標で、繰り返しを起こさずに商品に最も多く話題を留めた設定である。乱数シードは固定して記録し、空の出力にならないよう最小長を設ける。裏づけとなる商品は20点しかないため、この選択は、商品説明が買い物客の目に触れる前に、より大きな標本で再確認する。
- ビームサーチでは、長い文章をトークンあたりで順位づけする。 総確率では空の文章が有利になるためである。検索クエリの短い補完は総確率で順位づけする。この場合、買い物客が求めるのは最も尤度の高いフレーズだからである。
- デコードの設定は、実際の文章に対する測定によって選ぶ。 測定する項目は、繰り返し、モデル自身のパープレキシティ、説明が文として終わるか、商品名を使うか、そして長さである。これらを商品のランダムな集合について、サンプルを読む前に測定し、再学習のたびにもう一度測定する。
- 生成された商品説明は下書きであり、商品に関する事実の出どころにはならない。 商品名とカタログが裏づけられない主張は、人または確認の工程がすべて訂正する。ここで扱ったどの手法も、名前に素材が示されていない商品に対して素材を作り出したためである。
この版における本章
第2026.11.16版2026年11月19日時点で例が通過この章の7 例件のうち7件が合格
英語版の書籍(版2026.11.16)から翻訳した。