Choosing the next token

閱讀約需 38 分鐘 分鐘,9 節 中共有 7 個範例。

版次 2026.11.16 範例於 2026年11 月 19 通過測試 86 個範例中有 86 個通過 變更內容

由 AI 翻譯。經人工以英文驗證。 我們如何製作書籍

譯自英文書,版次 2026.11.16。 閱讀原文

此部分尚未翻譯,因此以英文呈現。

By the end of this chapter you can turn a language model's probabilities into text, and choose how: always the likeliest token, the likeliest whole text by beam search, or a sample shaped by a temperature, a top-k or top-p cut, a repetition penalty and a minimum length. You'll measure what each choice does to the text, and learn which failures no choice of setting can fix.

A language model gives a probability for every possible next token; it doesn't write. To write, something has to choose one token, append it, ask for the next probabilities, and repeat until the model predicts the end marker, </s>. Turning a model's probabilities into text, one chosen token at a time, is called decoding, and it decides much of what the text reads like, with the model unchanged.

The store has a reason to write: of the furniture and home goods in its validation split, 597 of 4,282 products have no description, and so do 14% of its training products. The store's model, the transformer that chapter 8 trained and measured, learned descriptions from the store's own text, so it can draft one from a product's name: start the text with the name and <sep>, as every training text was laid out, and let the model continue. This chapter does that with greedy decoding, beam search and sampling, using the book's shared module lib.decoding, built in NumPy on lib.transformer. Then it measures what each method writes against the store's real descriptions, and fixes two failures that the measurements expose: repetition, and descriptions that come out empty.

選擇機率最高的詞元

最近一次變動於第 2026.11.11 版

最簡單的解碼器是貪婪式(greedy)解碼:每一步都取單一機率最高的詞元。它不需要隨機性,因此相同的起點永遠寫出相同的文字,而且每一步只需要模型做一次前向傳遞。它寫出的文字,逐個詞元來看,就是模型的最佳猜測。

商店需要處理的正是沒有描述的商品,因此先從一個這樣的商品開始:front porch welcome sign wall décor,以貪婪式解碼,最多產生 60 個詞元。接著取第二個商品 5 light light bulb,並印出其描述為空的機率:

greedy-description.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Greedy decoding writes one description of a product that has none, and shows how it repeats itself, and how likelythe model is to leave a bulb's description empty. The prose quotes the greedy description, its repeated phrases, andthe line with the bulb's probability of an empty description."""from collections import Counterfrom lib import decoding, transformerfrom lib.store import load_productsvalidation, training = load_products("validation"), load_products("train")empty = lambda rows: rows["description"].fillna("").str.strip() == ""print("validation products:", f"{len(validation):,}")print("validation products with no description:", f"{empty(validation).sum():,}")print("training products with no description:", f"{empty(training).mean():.0%}")bulbs = training[training["name"].str.contains(r"\bbulbs?\b")]print("training products named as bulbs with no description:", f"{empty(bulbs).mean():.0%}")longest = 60print("longest description, in tokens:", longest)ids = decoding.generate(decoding.prompt("front porch welcome sign wall décor"), longest=longest)   # no rng: each step takes the likeliest tokenprint("greedy description:", decoding.text(ids))fourgrams = Counter(tuple(ids[i:i + 4]) for i in range(len(ids) - 3))   # the four tokens from i: a window that slides one token at a timeprint("4-token phrases it uses twice or more:", " | ".join(decoding.text(list(g)) for g, n in fourgrams.items() if n > 1))end = transformer.load()[1].index(transformer.END)                     # the index of </s> in the vocabularyprobabilities = transformer.softmax(decoding.next_logits(decoding.prompt("5 light light bulb")))   # the first token after <sep>print("P(the description is empty), 5 light light bulb:", f"{probabilities[end]:.2f}")
輸出python greedy-description.py
validation products: 4,282
validation products with no description: 597
training products with no description: 14%
training products named as bulbs with no description: 24%
longest description, in tokens: 60
greedy description: this welcome sign is a great way to add a touch of rustic charm to your home . it is made of natural wood and features a distressed finish with a distressed finish . this welcome sign is made of wood and has a distressed finish that adds a rustic touch to your home . this sign is made from
4-token phrases it uses twice or more: this welcome sign is | to your home .
P(the description is empty), 5 light light bulb: 0.28

貪婪式產生的描述讀起來像商店自己的文字,之後卻開始循環。它之所以讀起來像,是因為它本來就是商店的文字,只是重新組合過,例如 this welcome sign is a great way to add a touch of rustic charm to your home。它寫出 features a distressed finish with a distressed finish,之後又再次寫出 has a distressed finish,並且有兩個四詞元的片語各出現兩次,也就是 this welcome sign is 與 to your home .,範例中列出了這兩個片語。貪婪式解碼之所以循環,是因為文字每次來到相似的位置,勝出的都是同一個機率最高的接續:distressed 是模型最喜愛的告示牌收尾,所以每當需要收尾時,它就會再次出現。Holtzman 等人在大型模型中也發現相同的情形:總是將機率最大化的解碼,會產生「乏味、不連貫,或陷入重複循環」的文字(Holtzman et al., 2020)。

貪婪式產生的描述還說這個告示牌是 made of natural wood,而商品名稱中沒有任何地方這樣說。燈泡則顯示另一種失敗:對於 5 light light bulb,模型給出的是立即結束,也就是空描述,機率為 0.28。模型照原樣學會了這份目錄,缺漏也包含在內。在訓練資料中名稱為燈泡的商品裡,有 24% 沒有描述,而所有商品中的這個比例是 14%。後面的一節會處理空描述的問題。

實用原則: 當你要求每次輸出都相同,而且文字很短時,可以使用貪婪式解碼,例如搜尋框的建議,這類情況的重點就是機率最高的下一個字詞。若文字長於一個片語,應預期它會重複自己,並從後續各節的方法中選擇一種。

搜尋最可能的文字

最近一次變動於第 2026.11.16 版

貪婪解碼在每一步挑選最可能的詞元,但這並不等於最可能的文字:第一個詞稍微不那麼可能,後面卻可能接上可能性高得多的詞。束搜尋(beam search)看得更遠,它保留 k 個最可能的部分文字,稱為束,並以總對數機率為其評分;總對數機率是一段文字各詞元機率之對數的總和,數值越高表示越可能。每一步,它會把束中的每段文字,各以其 k 個最可能的下一個詞元延伸,再從結果中保留最好的 k 個。抵達結束標記的文字即告完成,仍留在比較之列。lib.decoding.beam 的核心是為每個延伸評分的這一行:

            candidates += [(score + log_p[token], written + [int(token)]) for token in np.argsort(-log_p, kind="stable")[:width]]

你預期束搜尋會把哪一則描述排為歡迎看板最可能的描述?請執行兩次,束寬為 4。第一次執行是以最多 4 個詞元(含名稱的結尾)補完搜尋查詢 outdoor bar,就像搜尋框可能提供完整詞組那樣;第二次則撰寫歡迎看板的描述,並印出整體最可能的文字,以及依每個詞元計算的結果,即其總對數機率除以其長度。

在查詢上,束搜尋給出搜尋框應有的結果:簡短而可能的詞組,先是 stool,接著是 stool cushion。這裡只用到 transformer;第 8 章為自動補全選定的混合方式,需要在每一步取得 4-gram 對每個候選詞的機率,而 lib.decoding 並不計算這一項。

整體最可能的描述是空的。立刻結束的機率為 0.15,而每一段更長的文字都要連乘許多小於一的機率,因此沒有任何一段能接近。Stahlberg 與 Byrne 在翻譯模型中也發現同樣的現象:對大多數句子而言,最可能的譯文是空的,束搜尋之所以管用,只是因為它搜尋得不夠徹底(Stahlberg and Byrne, 2019)。改以每個詞元的機率排序可以消除這種偏差,並得到合理的描述。不過它仍會憑空捏造材質:這一次,看板是 crafted of iron。

實用原則: 束搜尋適用於以最可能的詞組為目標的簡短補全,例如搜尋查詢,並以總機率排序。對於較長的文字,應以每個詞元的機率為完成文字排序,因為總機率偏好最短的文字;即使如此,仍宜優先採用後續各節的取樣方法,它們寫出的文字更接近人類的寫作。

以溫度進行取樣

最近一次變動於第 2026.11.16 版

貪婪解碼與束搜尋只用到分布的頂端。取樣(sampling)則用到整個分布,第 1 章生成名字時用的就是它:依各個詞元的機率大小隨機抽出下一個詞元,因此機率為 0.2 的詞元,平均每五次會被選中一次。取樣之前,可以用溫度(temperature)T 讓分布變得更尖銳或更平坦:在 softmax 之前,把每個 logit 都除以 T。以符號表示,令 zᵢ 為詞元 i 的 logit,如第 5 章所述:

P(token i) = exp(zᵢ / T) / Σⱼ exp(zⱼ / T)

溫度低於 1 會使分布變得尖銳,把更多機率給最可能的詞元;高於 1 則使分布變得平坦,把更多機率給不太可能的詞元。溫度趨近 0 時,結果接近貪婪解碼;溫度為 1 時,模型的機率維持原樣。除以 T 不會改變詞元之間的順序,只會改變它們機率之間的差距。

要看出溫度對一次真實選擇有什麼影響,可以提前結束貪婪描述,在三種溫度下讀取下一個詞元的分布。對每種溫度,範例會印出機率最高的三個詞元、涵蓋 90% 機率所需的詞元數,以及落在溫度 1 的 90% 涵蓋範圍之外的機率有多少。

temperature-by-hand.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""The next token's distribution after the first nine tokens of the greedy description, at three temperatures, and howmany tokens each needs to cover 90% of the probability. The prose quotes the three counts, and the share at 1.5 thatfalls outside temperature 1's 90% cover."""import numpy as npfrom lib import decoding, transformer_, vocabulary, _ = transformer.load()start = decoding.prompt("front porch welcome sign wall décor")so_far = decoding.generate(start, longest=9)                   # the greedy description's first 9 tokenslogits = decoding.next_logits(start + so_far)print("text so far:", decoding.text(so_far))print("tokens in the vocabulary:", f"{len(vocabulary):,}")cover = np.argsort(-transformer.softmax(logits), kind="stable")    # at temperature 1, the likeliest firstcover = cover[:int(np.searchsorted(np.cumsum(transformer.softmax(logits)[cover]), 0.9)) + 1]   # temperature 1's 90% cover, its last token includedfor temperature in [0.5, 1.0, 1.5]:    probabilities = transformer.softmax(logits / temperature)     # divide the logits, then softmax    top = np.argsort(-probabilities, kind="stable")    covering = int(np.searchsorted(np.cumsum(probabilities[top]), 0.9)) + 1   # +1: the token that reaches 90% counts    print(f"temperature {temperature}, top 3:", ", ".join(f'"{vocabulary[i].replace("</w>", "")}" {probabilities[i]:.2f}' for i in top[:3]))    print(f"temperature {temperature}, tokens needed to cover 90% of the probability:", covering)    print(f"temperature {temperature}, probability outside temperature 1.0's 90% cover:",          f"{1 - probabilities[cover].sum():.0%}")
輸出python temperature-by-hand.py
text so far: this welcome sign is a great way to add
tokens in the vocabulary: 4,142
temperature 0.5, top 3: "a" 0.61, "style" 0.20, "some" 0.11
temperature 0.5, tokens needed to cover 90% of the probability: 3
temperature 0.5, probability outside temperature 1.0's 90% cover: 0%
temperature 1.0, top 3: "a" 0.26, "style" 0.15, "some" 0.11
temperature 1.0, tokens needed to cover 90% of the probability: 78
temperature 1.0, probability outside temperature 1.0's 90% cover: 10%
temperature 1.5, top 3: "a" 0.09, "style" 0.06, "some" 0.05
temperature 1.5, tokens needed to cover 90% of the probability: 640
temperature 1.5, probability outside temperature 1.0's 90% cover: 42%

溫度決定機率散布的幅度。在 this welcome sign is a great way to add 之後,溫度為 1 時,模型需要 78 個詞元才能涵蓋 90%,其餘則是一條長尾,尾端的每個詞元都不太可能出現。溫度為 0.5 時,只要 3 個詞元就能涵蓋 90%。溫度為 1.5 時,則需要 640 個詞元,而且有 42% 的機率落在溫度 1 的 90% 涵蓋範圍之外。

溫度 機率最高的三個詞元 涵蓋 90% 所需詞元數 落在 T = 1 的 90% 涵蓋範圍之外
0.5 "a" 0.61, "style" 0.20, "some" 0.11 3 0%
1 "a" 0.26, "style" 0.15, "some" 0.11 78 10%
1.5 "a" 0.09, "style" 0.06, "some" 0.05 640 42%

這條長尾正是取樣的風險所在。尾端的詞元在此處大多是錯的,而在溫度 1 時,取樣器大約每十個詞元就會從尾端抽到一次,溫度越高,頻率越高。

**實用原則:**把溫度視為在可預測與多樣之間調整的旋鈕。若文字應貼近訓練文本的一般說法,例如商品描述,就讓溫度維持在 1 或略低於 1。較低的溫度只會讓文字更可預測,並不會讓它更準確。

取樣並截去尾部

最近一次變動於第 2026.11.11 版

降低溫度會縮小尾部,卻無法將它移除。有兩種方法可在取樣之前直接移除尾部。Top-k 只保留機率最高的 k 個 token,並將它們的機率重新正規化,使總和為一(Fan et al., 2018)。Top-p 又稱為核心取樣(nucleus sampling),保留機率最高、且機率總和至少達到 p 的最小 token 集合(Holtzman et al., 2020,公式 2)。兩者都是 lib.decoding.truncate 的選項,其核心有三個步驟:將 token 排序,依該順序累加它們的機率,並找出總和達到 p 的位置:

    order = np.argsort(-probabilities, kind="stable")
    ...
        keep = int(np.searchsorted(np.cumsum(probabilities[order]), top_p)) + 1

Top-p 會隨模型的信心程度調整,top-k 則不會。在上一節的範例中,模型對下一個字沒有把握之處,需要 78 個 token 才能涵蓋 90% 的機率,因此 top-p 0.9 會保留那麼多個 token,而 top-k 20 只會保留 20 個。當模型對某個字的預測近乎確定時,top-p 只保留一、兩個 token,top-k 卻仍保留 20 個,其中大多是錯的。Holtzman 等人正是基於這一點主張使用 top-p:固定的 k 值「在不同的上下文中並非最佳」。

四種取樣器使用相同的隨機種子,為歡迎標誌撰寫描述,長度最多到 40 個 token:溫度 1 的取樣、溫度 0.7 的取樣、k = 20 的 top-k,以及 p = 0.9 的 top-p。接著,範例重現溫度 1 的取樣,並列出每一步中取自 90% 核心之外的 token,這些 token 是 top-p 0.9 會截去的。

sample-descriptions.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Four samplers write the same description from one seed, and the temperature-1 draws are then replayed to mark thetokens outside each step's 90% nucleus. The last two lines are the ones the prose quotes: how many draws came fromthe tail, and which tokens they were."""import numpy as npfrom lib import decoding, transformer_, vocabulary, _ = transformer.load()start, longest, seed = decoding.prompt("front porch welcome sign wall décor"), 40, 2026print("longest description, in tokens:", longest)print("random seed:", seed)for label, settings in [("sampling, temperature 1.0", {}), ("sampling, temperature 0.7", {"temperature": 0.7}),                        ("top-k 20", {"top_k": 20}), ("top-p 0.9", {"top_p": 0.9})]:    ids = decoding.generate(start, rng=np.random.default_rng(seed), longest=longest, **settings)    print(f"{label}:", decoding.text(ids))rng, written, tail = np.random.default_rng(seed), [], []    # replay temperature 1.0's draws, step by stepfor _ in range(longest):    probabilities = transformer.softmax(decoding.next_logits(start + written))    token = int(rng.choice(len(probabilities), p=decoding.truncate(probabilities)))   # no cut: draw from all of it    order = np.argsort(-probabilities, kind="stable")    before = probabilities[order[:int(np.where(order == token)[0][0])]].sum()   # the probability ranked above it    if before >= 0.9:                                        # top-p 0.9 would have cut this token        tail.append(vocabulary[token].replace("</w>", ""))    written.append(token)    if vocabulary[token] == transformer.END:        breakprint("temperature 1.0, tokens drawn from outside the 90% nucleus:", f"{len(tail)} of {len(written)}")print("temperature 1.0, those tokens:", ", ".join(tail))
輸出python sample-descriptions.py
longest description, in tokens: 40
random seed: 2026
sampling, temperature 1.0: enhance your home even glywheel design with this welcome welcome sign sign . this piece is ideal for both indoor and inviting . made in the usa , mini spirits decorate .
sampling, temperature 0.7: the sign is a great way to add some extra whimsy to your home and get colorful ! made of high-quality metal , this sign comes with a hanging dustroke hanger to help you gli
top-k 20: if you have a new product with a personalized sign wall décor that will help you entertain . look no further like a home decor , this sign is constructed to make life more wonderful . this welcome sign
top-p 0.9: express your guests with this cast iron sign from the nature of the world . it features a meditating podishade design . the hose and symbol in this metal piece will be
temperature 1.0, tokens drawn from outside the 90% nucleus: 5 of 36
temperature 1.0, those tokens: inviting, in, mini, spir, decorate

截去尾部只能移除最罕見的錯誤,而非全部錯誤。溫度 1 的取樣會漫無邊際:它寫出 welcome welcome sign sign,自創了一個字 glywheel,並宣稱標誌是 made in the usa。其中只有一部分來自尾部:有 5 of 36 個 token 來自尾部,inviting, in, mini, spir, decorate。重複的字和自創的字,都是由核心之內的 token 組成,而 top-p 會保留這些 token。在 0.7 時,文字較為連貫,但它現在說標誌是 made of high-quality metal,而貪婪解碼說的是木材。Top-k 和 top-p 讀起來比單純取樣更像型錄,但仍會自創內容:top-p 的標誌是 cast iron,帶有 podishade 設計。

在本章中,歡迎標誌已被賦予材質六次,分別來自貪婪解碼、束搜尋、其中三個取樣,以及下方的加上懲罰的貪婪文字,而它的名稱中沒有任何一處說明它由什麼製成。模型寫的是歡迎標誌的描述通常會寫的內容,而這些描述通常會註明材質。解碼方法改變的是模型挑選哪一種合理的材質;沒有任何一種方法能讓它挑出真實的材質。

實用原則: 若要產生多樣的文字,請用 top-p 約 0.9 或 top-k 約 20 截去尾部,並將溫度設在 1 或略低於 1;需要重現結果時,請固定隨機種子。不要期望任何設定能讓文字成為真實的內容:解碼是在模型認為合理的選項中做選擇,而查核事實是另一項工作。

測量各種設定寫出了什麼

最近一次變動於第 2026.11.12 版

單一樣本能呈現各種方法的風格,測量則能呈現其中的取捨。有數項特性不必由人閱讀文字就能測量:

  • **重複率:**同一則描述中,與前面已出現的四詞元片語重複的片語所占比例
  • **模型自身對該文字的困惑度:**模型對自己所寫內容的意外程度,以所有詞元合併計算,數值低表示容易預測
  • 描述是否在限制內結束,如同真實的描述
  • 是否使用了產品名稱中的完整單字,這是針對該產品的描述應有的表現
  • 平均長度,以詞元計

各種設定與真實文字的接近程度如何?從有描述的驗證產品中隨機抽出 20 項,以每種方法為各產品撰寫描述,並與真實描述比較,兩者都在相同的 60 詞元限制處截斷:

measure-decoding.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Seven ways of choosing the next token, measured on the same 20 product descriptions: how often each repeatsitself, how likely the model finds its own text, whether it ends, and whether it stays on the product. The realdescriptions come first, as the bar the others are read against. The prose quotes these lines for each setting."""import numpy as npfrom lib import decoding                    # the store's model, and the decoding methods this chapter buildsfrom lib.products import product_textslongest, sample, seed = 60, 20, 2026# Validation products, which the model never trained on, that have at least one token of description after <sep>described = [(ids, sep) for ids, sep in product_texts("validation") if sep < len(ids) - 2]products = [described[i] for i in np.random.default_rng(seed).choice(len(described), sample, replace=False)]print("products, drawn at random from the validation products with a description:", sample)print("longest description, in tokens:", longest)settings = [("real descriptions", None), ("greedy", {}), ("greedy, repetition penalty 1.3", {"penalty": 1.3}),            ("sampling, temperature 1.0", {"sample": True}), ("sampling, temperature 0.7", {"sample": True, "temperature": 0.7}),            ("top-k 20", {"sample": True, "top_k": 20}), ("top-p 0.9", {"sample": True, "top_p": 0.9})]for label, chosen in settings:    rows = []    for number, (ids, sep) in enumerate(products):        start = ids[:sep + 1]                                 # the start marker, the name and <sep>: the model writes on from here        if chosen is None:            written = ids[sep + 1:][:longest]                 # the description as sold, cut where generating would stop        else:            options = {option: value for option, value in chosen.items() if option != "sample"}            # generate samples when it's given a generator, and is greedy without one. Each product gets its own            # seed, the same for every sampling setting, so the settings differ only in how they choose.            rng = np.random.default_rng(seed + number) if chosen.get("sample") else None            written = decoding.generate(start, rng=rng, longest=longest, **options)        rows.append(decoding.measures(start, written))        # repetition, each token's log-probability, ended, uses the name, length    repeated, log_p, ended, uses_name, length = zip(*rows)    print(f"{label}, repeated 4-token phrases:", f"{np.mean(repeated):.0%}")    print(f"{label}, the model's perplexity of the text:", f"{np.exp(-np.concatenate(log_p).mean()):.1f}")   # over all 20 texts' tokens at once    print(f"{label}, ended within {longest} tokens:", f"{np.mean(ended):.0%}")    print(f"{label}, uses a whole word of the name:", f"{np.mean(uses_name):.0%}")    print(f"{label}, average length in tokens:", f"{np.mean(length):.0f}")
輸出python measure-decoding.py
products, drawn at random from the validation products with a description: 20
longest description, in tokens: 60
real descriptions, repeated 4-token phrases: 0%
real descriptions, the model's perplexity of the text: 30.5
real descriptions, ended within 60 tokens: 35%
real descriptions, uses a whole word of the name: 95%
real descriptions, average length in tokens: 53
greedy, repeated 4-token phrases: 11%
greedy, the model's perplexity of the text: 2.7
greedy, ended within 60 tokens: 25%
greedy, uses a whole word of the name: 80%
greedy, average length in tokens: 52
greedy, repetition penalty 1.3, repeated 4-token phrases: 0%
greedy, repetition penalty 1.3, the model's perplexity of the text: 3.9
greedy, repetition penalty 1.3, ended within 60 tokens: 70%
greedy, repetition penalty 1.3, uses a whole word of the name: 80%
greedy, repetition penalty 1.3, average length in tokens: 41
sampling, temperature 1.0, repeated 4-token phrases: 0%
sampling, temperature 1.0, the model's perplexity of the text: 22.6
sampling, temperature 1.0, ended within 60 tokens: 50%
sampling, temperature 1.0, uses a whole word of the name: 50%
sampling, temperature 1.0, average length in tokens: 45
sampling, temperature 0.7, repeated 4-token phrases: 0%
sampling, temperature 0.7, the model's perplexity of the text: 7.2
sampling, temperature 0.7, ended within 60 tokens: 45%
sampling, temperature 0.7, uses a whole word of the name: 65%
sampling, temperature 0.7, average length in tokens: 45
top-k 20, repeated 4-token phrases: 0%
top-k 20, the model's perplexity of the text: 7.6
top-k 20, ended within 60 tokens: 50%
top-k 20, uses a whole word of the name: 80%
top-k 20, average length in tokens: 40
top-p 0.9, repeated 4-token phrases: 0%
top-p 0.9, the model's perplexity of the text: 10.7
top-p 0.9, ended within 60 tokens: 30%
top-p 0.9, uses a whole word of the name: 60%
top-p 0.9, average length in tokens: 47

貪婪解碼的文字遠比任何人寫的文字更容易預測,困惑度為 2.7,其四詞元片語有 11% 重複。真實描述則毫無重複(四詞元片語重複率為 0%),模型也覺得它們相當出人意料,困惑度為 30.5。這個數字高於第 8 章中描述的數字,原因是這裡的樣本不同,而且只取前 60 個詞元。Holtzman 等人在大規模實驗中也有相同的觀察:人類的文字並不是模型認為機率最高的文字。

沒有任何一種設定能在所有指標上都與真實描述相符。溫度 1 的取樣在困惑度上最接近,為 22.6,但其描述中只有 50% 使用了產品名稱中的單字,真實描述則為 95%,因此它會偏離產品。Top-p 0.9 的結束比例幾乎與真實文字相當,為 30%,真實文字則為 35%,但只有 60% 緊扣產品。Top-k 20 在各種取樣方法中最能緊扣產品,為 80%,沒有重複,描述平均長度為 40 個詞元。二十項產品是很小的樣本,因此這些結果應視為趨勢,而非精確的比率。

這些指標無法說明的是描述是否屬實。描述可以流暢、多樣、切題,卻是錯的;上一節的描述就是如此,關於招牌的材質說得不對。

**實用原則:**在閱讀樣本之前,先以這類指標,對照同類型的真實文字,比較各種解碼設定。決定哪一項指標對這項工作最重要:對產品描述而言,緊扣產品與不重複最優先。接著仍要閱讀一部分輸出樣本,找出指標無法發現的問題,並依產品資料核對每一項事實陳述。

懲罰重複

最近一次變動於第 2026.11.11 版

貪婪解碼的迴圈也可以不靠取樣來修正。Keskar 等人提出了重複懲罰(repetition penalty):把每個已經生成過的 token 的 logit 除以一個大於 1 的係數 θ,使它變得較不可能出現(Keskar et al., 2019,第 4.1 節)。若 logit 為負值,除以 θ 反而會使它變大,因此實作上會改為把負的 logit 乘以 θ,lib.decoding.penalize 就是這麼做的,如此不論正負,該 token 都會變得較不可能出現。Keskar 等人發現,係數約為 1.2 時,能在避免重複與維持文句通順之間取得平衡。

回到貪婪解碼所產生的描述即將第二次寫出 distressed 的地方。請以手算方式,把懲罰套用到下一個 token 的 logit 上,將結果與 lib.decoding.penalize 核對,並比較懲罰前後的機率。接著再以貪婪解碼搭配懲罰來寫一次,看看迴圈是否消失:

repetition-penalty-by-hand.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""The repetition penalty applied by hand to the next token's logits, checked against lib.decoding.penalize, and thenused for a whole greedy description. The two P(distressed) lines are the ones the prose quotes, and theassert against lib.decoding.penalize is the match it states as fact."""import numpy as npfrom lib import decoding, transformer_, vocabulary, _ = transformer.load()start, factor = decoding.prompt("front porch welcome sign wall décor"), 1.3written = decoding.generate(start)                                    # greedy, as in greedy-descriptiondistressed = vocabulary.index("distressed</w>")position = [i for i, token in enumerate(written) if token == distressed][1]   # where greedy writes it a second timebefore = written[:position]print("text so far:", decoding.text(before))print("penalty factor:", factor)logits = decoding.next_logits(start + before)seen = np.unique(before)                                              # every token the description has usedpenalized = logits.copy()penalized[seen] = np.where(logits[seen] > 0, logits[seen] / factor, logits[seen] * factor)   # less likely either wayassert np.allclose(penalized, decoding.penalize(logits, before, factor))   # the hand calculation is the library'sfor label, adjusted in [("without a penalty", logits), ("with the penalty", penalized)]:    probabilities = transformer.softmax(adjusted)    top = np.argsort(-probabilities, kind="stable")[:3]    print(f"next token, {label}:", ", ".join(f"{vocabulary[i].replace('</w>', '')} {probabilities[i]:.2f}" for i in top))    print(f"P(distressed), {label}:", f"{probabilities[distressed]:.2f}")print("greedy with the penalty:", decoding.text(decoding.generate(start, penalty=factor)))
輸出python repetition-penalty-by-hand.py
text so far: this welcome sign is a great way to add a touch of rustic charm to your home . it is made of natural wood and features a distressed finish with a
penalty factor: 1.3
next token, without a penalty: distressed 0.20, black 0.03, weathered 0.03
P(distressed), without a penalty: 0.20
next token, with the penalty: black 0.04, weathered 0.03, distressed 0.03
P(distressed), with the penalty: 0.03
greedy with the penalty: this welcome sign is a great way to add style and functionality to any room . the worn look of these beautiful signs are made from natural coir , so it can be used indoors or outdoors .

在係數為 1.3 時,distressed 的機率從 0.20 降到 0.03,由另一個 token 取而代之。手算的結果與函式庫的計算相符,而且加上懲罰的貪婪描述不再出現迴圈。不過它仍然會捏造內容:這一次,招牌是用 natural coir 製成的,那是一種取自椰子殼的纖維。

這項懲罰有一個副作用,量測結果也顯示了這一點:描述會比較早結束。懲罰會作用在每一個已經用過的 token 上,包括 the 和 and 這類描述中需要反覆使用的常見字詞,卻不會作用在結束標記上,因為結束標記尚未被使用過。於是結束標記的相對機率變高,加上懲罰的貪婪描述中,有 70% 在上限內結束,未加懲罰時則為 25%;平均長度為 41 個 token,未加懲罰時為 52 個。重複率則從 11% 降到 0%。

實用原則: 以貪婪解碼或低溫度解碼時,可加入約 1.2 至 1.3 的重複懲罰,並檢查它對長度的影響,因為它會讓文字變短。依經驗法則,若描述被截得太短,就不要把常見的虛詞和標點符號納入懲罰。

禁止空白描述

最近一次變動於第 2026.11.11 版

貪婪解碼範例發現模型準備好為一顆燈泡什麼都不寫,而束搜尋則發現,對一面標誌來說,最有可能的描述就是什麼都沒有。這兩種情形都來自目錄本身:許多產品沒有描述,因此模型學到立刻結束是常見的做法。要填補的正是這些空缺,就必須阻止模型重現它們。簡單的解法是設定最小長度:在描述達到一定的詞元數之前,將結束標記的 logit 設為負無限大,以禁止它出現。lib.decoding.generate 以 shortest 參數接收這項設定。

貪婪解碼留下空白的那顆燈泡就是測試案例。我們解碼兩次,第一次不設最小長度,第二次設為 10 個詞元;輸出會先給出描述立刻結束的機率:

minimum-length.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Greedy decoding of the bulb's description, without a minimum length and with one that forbids the end marker untilit has that many tokens. The prose quotes the printed probability of an empty description, andthe description the minimum produces."""from lib import decoding, transformerend = transformer.load()[1].index(transformer.END)                    # the index of </s>, to read its probabilitystart, shortest = decoding.prompt("5 light light bulb"), 10print("P(the description is empty):", f"{transformer.softmax(decoding.next_logits(start))[end]:.2f}")   # the first token is </s>: an empty descriptionprint("greedy description:", decoding.text(decoding.generate(start)) or "(empty)")print("minimum length, in tokens:", shortest)print("greedy description, with the minimum:", decoding.text(decoding.generate(start, shortest=shortest)))   # shortest: </s> stays forbidden until then
輸出python minimum-length.py
P(the description is empty): 0.28
greedy description: (empty)
minimum length, in tokens: 10
greedy description, with the minimum: this light is a great light for a light , a light , and more light . it is made of metal and has a long , high gloss finish that will make it a perfect accent piece for your home . this light is dimmable , compatible with most standard bulbs , which are not included .

最小長度能填補空缺,卻無法提供事實。沒有它時,貪婪解碼會選擇空白描述,其機率為 0.28。加上最小長度後,模型確實寫出了描述,而本章稍早的兩種失敗也出現在其中。它重複自己的內容:a light , a light , and more light,而且還陳述了名稱中沒有的事項:這顆燈泡是 made of metal、dimmable,並且 compatible with most standard bulbs。

實用原則: 只要訓練文字中含有空白或極短的同類範例,就應設定最小長度。請將它與本章的其他設定搭配使用,而不是用它取代其他設定,並預期被它強迫產生的文字,與其他任何文字一樣需要檢查。

正式環境中會出什麼問題

最近一次變動於第 2026.11.11 版

解碼的失敗多半悄然無聲:

  • 流暢的文字陳述捏造的事實。 本章有六則描述為迎賓招牌指定了材質,而且彼此矛盾。未經對照商品資料逐一查核每項事實陳述,切勿發布生成的商品文字;第 11 章會量測模型捏造內容的頻率。
  • 訓練文字中的空缺會變成輸出。 模型會為燈泡預測出空白描述,因為目錄中有許多這類項目,而束搜尋(beam search)發現空白描述的可能性最高。請在達到最小長度之前禁止輸出結束標記,否則模型會重現你想要填補的空缺。
  • 貪婪解碼的文字會重複循環。 凡是長於一個片語的文字,都應使用截斷尾部的取樣法,或使用重複懲罰,並在一批輸出樣本上量測重複程度。
  • 取樣產生的文字預設無法重現。 請固定隨機種子,並隨每則生成文字一併記錄,如此一來,針對某則描述的客訴才能被追查並重現。
  • 設定會隨模型而漂移。 重新訓練的模型有不同的分布,因此在上一個模型上調好的溫度或 top-k,用在下一個模型時可能寫出不同的文字。每次重新訓練之後,都要重跑本章的量測,再讓新模型撰寫任何購物者會看到的內容。
  • 每個詞元都要花一次前向傳遞。 lib.decoding 每產生一個新詞元,就會對整段文字重新執行一次模型,這做法簡單,對 60 個詞元來說也夠用。正式服務系統會保留先前各位置在每一層的鍵與值,因此每個新詞元的矩陣乘法只需涵蓋一個位置,不過注意力仍會讀取先前每個位置已快取的鍵與值(Pope et al., 2022)。

本章的決定

  • 自動完成使用可能性最高的詞元。 搜尋框的建議會顯示在第 8 章所選組合中前幾名的詞元,並可用依總機率排序的束搜尋(beam search)將它們延伸為短語。
  • **商品描述使用 top-k 20 取樣,**依上述各項指標,這是最常保留在商品上而不重複的設定;並搭配固定且有紀錄的隨機種子,以及最小長度限制,避免產生空白內容。由於背後只有二十項商品,在任何描述送到消費者眼前之前,這項選擇會先以更大的樣本重新檢查。
  • **束搜尋以每個詞元的平均值來排序較長的文字,**因為總機率會偏向空白文字。搜尋查詢的短補全則依總機率排序,因為消費者要的就是可能性最高的片語。
  • **解碼設定是依真實文字的量測結果來選定:**以一組隨機抽取的商品,量測重複程度、模型本身的困惑度、描述是否有結尾、是否使用商品名稱,以及長度;這些量測在閱讀樣本之前進行,並在每次重新訓練後再做一次。
  • 生成的描述只是草稿,絕不是商品事實的來源。 凡是商品名稱與目錄無法支持的說法,都由人員或檢查步驟更正:這裡的每一種方法,都曾為名稱中沒有提到材質的商品捏造了材質。

本章,於本版次中

版次 2026.11.16範例於 2026年11 月 19 通過測試本章 7 個範例 個範例中有 7 個通過

譯自英文書,版次 2026.11.16。

引用本節

格式

各種格式皆引用英文原書,本譯本即依循該原書。

每種格式都會標示本版。其連結會開啟目前版次,節標題會說明該節內文最後一次變動的時間。存取日期與擷取日期即為你開啟此頁的當日。

本頁內容

第 9 章,共 11 章: Choosing the next token

  1. 選擇機率最高的詞元
  2. 搜尋最可能的文字
  3. 以溫度進行取樣
  4. 取樣並截去尾部
  5. 測量各種設定寫出了什麼
  6. 懲罰重複
  7. 禁止空白描述
  8. 正式環境中會出什麼問題
  9. 本章的決定

返回頂端所有章節