Choosing the next token

阅读约需 37分钟 分钟,9节中含7个示例。

版次 2026.11.16 示例于 2026年11月19日 通过 86个示例中有86个通过 变化内容

AI 翻译。由人工以英文核验。 我们如何制作图书

译自英文版图书,版次 2026.11.16。 阅读原文

这部分尚未翻译,因此以英文显示。

By the end of this chapter you can turn a language model's probabilities into text, and choose how: always the likeliest token, the likeliest whole text by beam search, or a sample shaped by a temperature, a top-k or top-p cut, a repetition penalty and a minimum length. You'll measure what each choice does to the text, and learn which failures no choice of setting can fix.

A language model gives a probability for every possible next token; it doesn't write. To write, something has to choose one token, append it, ask for the next probabilities, and repeat until the model predicts the end marker, </s>. Turning a model's probabilities into text, one chosen token at a time, is called decoding, and it decides much of what the text reads like, with the model unchanged.

The store has a reason to write: of the furniture and home goods in its validation split, 597 of 4,282 products have no description, and so do 14% of its training products. The store's model, the transformer that chapter 8 trained and measured, learned descriptions from the store's own text, so it can draft one from a product's name: start the text with the name and <sep>, as every training text was laid out, and let the model continue. This chapter does that with greedy decoding, beam search and sampling, using the book's shared module lib.decoding, built in NumPy on lib.transformer. Then it measures what each method writes against the store's real descriptions, and fixes two failures that the measurements expose: repetition, and descriptions that come out empty.

选择可能性最大的词元

最近一次修改见于版次 2026.11.11

最简单的解码器是贪心解码:每一步都取可能性最大的那一个词元。它不需要随机性,因此对同一个开头总会写出相同的文本,而且每一步只花费模型的一次前向传播。它写出的文本,逐个词元来看,就是模型的最佳猜测。

没有描述的商品正是商店所需要处理的对象,所以先从一个商品开始:front porch welcome sign wall décor,贪心解码,最多生成 60 个词元。然后取第二个商品 5 light light bulb,输出它的描述为空的概率:

greedy-description.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通过
"""Greedy decoding writes one description of a product that has none, and shows how it repeats itself, and how likelythe model is to leave a bulb's description empty. The prose quotes the greedy description, its repeated phrases, andthe line with the bulb's probability of an empty description."""from collections import Counterfrom lib import decoding, transformerfrom lib.store import load_productsvalidation, training = load_products("validation"), load_products("train")empty = lambda rows: rows["description"].fillna("").str.strip() == ""print("validation products:", f"{len(validation):,}")print("validation products with no description:", f"{empty(validation).sum():,}")print("training products with no description:", f"{empty(training).mean():.0%}")bulbs = training[training["name"].str.contains(r"\bbulbs?\b")]print("training products named as bulbs with no description:", f"{empty(bulbs).mean():.0%}")longest = 60print("longest description, in tokens:", longest)ids = decoding.generate(decoding.prompt("front porch welcome sign wall décor"), longest=longest)   # no rng: each step takes the likeliest tokenprint("greedy description:", decoding.text(ids))fourgrams = Counter(tuple(ids[i:i + 4]) for i in range(len(ids) - 3))   # the four tokens from i: a window that slides one token at a timeprint("4-token phrases it uses twice or more:", " | ".join(decoding.text(list(g)) for g, n in fourgrams.items() if n > 1))end = transformer.load()[1].index(transformer.END)                     # the index of </s> in the vocabularyprobabilities = transformer.softmax(decoding.next_logits(decoding.prompt("5 light light bulb")))   # the first token after <sep>print("P(the description is empty), 5 light light bulb:", f"{probabilities[end]:.2f}")
输出python greedy-description.py
validation products: 4,282
validation products with no description: 597
training products with no description: 14%
training products named as bulbs with no description: 24%
longest description, in tokens: 60
greedy description: this welcome sign is a great way to add a touch of rustic charm to your home . it is made of natural wood and features a distressed finish with a distressed finish . this welcome sign is made of wood and has a distressed finish that adds a rustic touch to your home . this sign is made from
4-token phrases it uses twice or more: this welcome sign is | to your home .
P(the description is empty), 5 light light bulb: 0.28

贪心解码生成的描述起初读起来像商店自己的文本,随后开始循环。它读起来像商店的文本,是因为它本来就是商店的文本,只是经过了重新组合,例如 this welcome sign is a great way to add a touch of rustic charm to your home。它写出 features a distressed finish with a distressed finish,后来又写出 has a distressed finish,并且两个四词元的短语各用了两次,即 this welcome sign is 和 to your home .,示例中列出了它们。贪心解码之所以循环,是因为文本每到相似的位置,胜出的总是同一个可能性最大的后续:distressed 是模型对牌匾收尾方式的首选,所以每当需要写饰面时它就会回来。Holtzman 等人在大型模型中发现了同样的现象:总是最大化概率的解码会产生“平淡、不连贯,或陷入重复循环”的文本(Holtzman et al., 2020)。

贪心解码生成的描述还说这块牌匾是 made of natural wood,而商品名称中没有任何内容说明这一点。灯泡则显示了另一种失败:对于 5 light light bulb,模型给出的是立即结束,也就是空描述,其概率为 0.28。模型学到的是目录的本来面貌,包括其中的空缺。在训练商品中,名称为灯泡的商品有 24% 没有描述,而全部商品中的这一比例为 14%。后面的一节会处理空描述的问题。

**工作准则:**如果希望每次输出都相同,并且文本较短,例如搜索框的建议,而重点就在于可能性最大的下一个词,就使用贪心解码。对于比一个短语更长的内容,要预期它会重复自身,并从后续各节的方法中选择一种。

搜索最可能的文本

最近一次修改见于版次 2026.11.16

贪心解码在每一步选取最可能的词元,但这与最可能的文本并不相同:第一个词的可能性稍低一些,却可能使其后的词可能性高得多。束搜索(beam search)看得更远,它保留 k 个最可能的部分文本,即束,并按总对数概率为其打分;总对数概率是一段文本各词元概率的对数之和,数值越高表示越可能。每一步中,它用每个文本最可能的 k 个下一词元来扩展束中的每个文本,并从结果中保留最好的 k 个。到达结束标记的文本即告完成,并继续参与排名。lib.decoding.beam的核心是为每个扩展打分的那一行:

            candidates += [(score + log_p[token], written + [int(token)]) for token in np.argsort(-log_p, kind="stable")[:width]]

对于欢迎标牌,你预期束搜索会把哪条描述排为最可能?运行两次,束宽为4。第一次运行对搜索查询outdoor bar最多补全4个词元(名称的结尾也计入),就像搜索框可能给出整条短语那样;第二次为欢迎标牌撰写描述,并输出总体上最可能的文本,以及按每个词元计的结果,即其总对数概率除以其长度。

对于查询,束搜索给出了搜索框应当给出的结果:简短而可能的短语,先是stool,然后是stool cushion。这里只使用了 Transformer;第 8 章为自动补全选定的混合方式,需要在每一步为每个候选取得 4 元语法的概率,而lib.decoding并不计算这一概率。

总体上最可能的描述是空文本。立即结束的概率为0.15,而每一段更长的文本都要把许多小于一的概率相乘,因此没有一段能接近它。Stahlberg 和 Byrne 在翻译模型中发现了同样的现象:对大多数句子而言,概率最高的译文是空的,而束搜索之所以有效,只是因为它搜索得不好(Stahlberg and Byrne, 2019)。按每个词元的概率排序可以消除这种偏差,并给出合理的描述。但它仍会编造材质:这一次,标牌是crafted of iron。

工作准则: 对于以最可能的短语为目标的简短补全,例如搜索查询,使用束搜索,并按总概率排序。对于较长的文本,应按每个词元对已完成的文本排序,因为总概率偏向最短的文本;即便如此,也应优先选用后续各节的采样方法,它们写出的文本更接近人类所写。

带温度的采样

最近一次修改见于版次 2026.11.16

贪心解码和束搜索只用到分布的顶部。采样(sampling)则用到整个分布,第 1 章生成名字时用的就是它:按概率的大小随机抽取下一个词元,因此概率为 0.2 的词元每五次被选中一次。采样之前,可以用温度(temperature)T 使分布变得更尖锐或更平坦:在 softmax 之前把每个 logit 都除以 T。用符号表示,设 zᵢ 为词元 i 的 logit,与第 5 章相同:

P(词元 i) = exp(zᵢ / T) / Σⱼ exp(zⱼ / T)

温度低于 1 时,分布变得尖锐,更多的概率归于最可能的词元;高于 1 时,分布变得平坦,更多的概率归于不太可能的词元。温度接近 0 时趋近于贪心解码,温度为 1 时则保持模型给出的概率不变。除以 T 不会改变词元的排序,只会改变它们的概率相距多远。

为了看清温度对一次真实选择的影响,我们提前停下贪心描述,在三种温度下读取下一个词元的分布。对每一种温度,该示例打印出最可能的三个词元、覆盖 90% 概率所需的词元数,以及落在温度 1 的 90% 覆盖范围之外的概率有多少。

temperature-by-hand.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通过
"""The next token's distribution after the first nine tokens of the greedy description, at three temperatures, and howmany tokens each needs to cover 90% of the probability. The prose quotes the three counts, and the share at 1.5 thatfalls outside temperature 1's 90% cover."""import numpy as npfrom lib import decoding, transformer_, vocabulary, _ = transformer.load()start = decoding.prompt("front porch welcome sign wall décor")so_far = decoding.generate(start, longest=9)                   # the greedy description's first 9 tokenslogits = decoding.next_logits(start + so_far)print("text so far:", decoding.text(so_far))print("tokens in the vocabulary:", f"{len(vocabulary):,}")cover = np.argsort(-transformer.softmax(logits), kind="stable")    # at temperature 1, the likeliest firstcover = cover[:int(np.searchsorted(np.cumsum(transformer.softmax(logits)[cover]), 0.9)) + 1]   # temperature 1's 90% cover, its last token includedfor temperature in [0.5, 1.0, 1.5]:    probabilities = transformer.softmax(logits / temperature)     # divide the logits, then softmax    top = np.argsort(-probabilities, kind="stable")    covering = int(np.searchsorted(np.cumsum(probabilities[top]), 0.9)) + 1   # +1: the token that reaches 90% counts    print(f"temperature {temperature}, top 3:", ", ".join(f'"{vocabulary[i].replace("</w>", "")}" {probabilities[i]:.2f}' for i in top[:3]))    print(f"temperature {temperature}, tokens needed to cover 90% of the probability:", covering)    print(f"temperature {temperature}, probability outside temperature 1.0's 90% cover:",          f"{1 - probabilities[cover].sum():.0%}")
输出python temperature-by-hand.py
text so far: this welcome sign is a great way to add
tokens in the vocabulary: 4,142
temperature 0.5, top 3: "a" 0.61, "style" 0.20, "some" 0.11
temperature 0.5, tokens needed to cover 90% of the probability: 3
temperature 0.5, probability outside temperature 1.0's 90% cover: 0%
temperature 1.0, top 3: "a" 0.26, "style" 0.15, "some" 0.11
temperature 1.0, tokens needed to cover 90% of the probability: 78
temperature 1.0, probability outside temperature 1.0's 90% cover: 10%
temperature 1.5, top 3: "a" 0.09, "style" 0.06, "some" 0.05
temperature 1.5, tokens needed to cover 90% of the probability: 640
temperature 1.5, probability outside temperature 1.0's 90% cover: 42%

温度决定了概率铺开的程度。在 this welcome sign is a great way to add 之后,温度为 1 时,模型需要 78 个词元才能覆盖 90%,其余部分是一条长尾,其中每个词元的概率都很小。温度为 0.5 时,3 个词元即可覆盖 90%。温度为 1.5 时,需要 640 个词元,并且有 42% 的概率落在温度 1 的 90% 覆盖范围之外。

温度 最可能的三个词元 覆盖 90% 所需词元数 T = 1 的 90% 覆盖范围之外
0.5 "a" 0.61, "style" 0.20, "some" 0.11 3 0%
1 "a" 0.26, "style" 0.15, "some" 0.11 78 10%
1.5 "a" 0.09, "style" 0.06, "some" 0.05 640 42%

这条长尾正是采样的风险所在。其中大多数词元在这里都是错的,而在温度 1 下,采样器大约每十个词元就会从长尾中抽取一次,温度越高,抽到的次数越多。

**工作准则:**把温度看作在可预测与多样之间调节的旋钮。若文本应当贴近训练文本通常的说法,例如产品描述,就把温度保持在 1 或略低于 1。较低的温度让文本更可预测,而不是更准确。

采样并截去尾部

最近一次修改见于版次 2026.11.11

降低温度会缩小尾部,却不会将其消除。有两种方法可以在采样之前直接去掉尾部。Top-k 只保留概率最高的 k 个词元,并将它们的概率重新归一化,使其和为 1(Fan et al., 2018)。Top-p 也称核采样(nucleus sampling),它保留概率最高的词元中最小的一个集合,使这些词元的概率之和至少达到 p(Holtzman et al., 2020,公式 2)。两者都是 lib.decoding.truncate 的选项,其核心分为三步:对词元排序,按此顺序累加它们的概率,找出累加和达到 p 的位置:

    order = np.argsort(-probabilities, kind="stable")
    ...
        keep = int(np.searchsorted(np.cumsum(probabilities[order]), top_p)) + 1

Top-p 会随模型的把握程度而调整,top-k 则不会。在上一节示例中,模型对下一个词拿不准的那一处,需要 78 个词元才能覆盖 90% 的概率,因此 top-p 0.9 会保留这么多词元,而 top-k 20 只会保留 20 个。在模型几乎可以确定下一个词之后,top-p 只保留一两个词元,top-k 仍然保留 20 个,其中大多数是错的。Holtzman 等人正是基于这一点主张使用 top-p:固定的 k 在不同的上下文中是“次优的”(sub-optimal across varying contexts)。

四种采样器使用同一个随机种子,各自写出欢迎标牌的描述,长度至多 40 个词元:温度为 1 的采样、温度为 0.7 的采样、k = 20 的 top-k,以及 p = 0.9 的 top-p。随后,该示例重放温度为 1 的那次采样,并列出在每一步中从 90% 核之外抽取的词元,这些词元本会被 top-p 0.9 截去。

sample-descriptions.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通过
"""Four samplers write the same description from one seed, and the temperature-1 draws are then replayed to mark thetokens outside each step's 90% nucleus. The last two lines are the ones the prose quotes: how many draws came fromthe tail, and which tokens they were."""import numpy as npfrom lib import decoding, transformer_, vocabulary, _ = transformer.load()start, longest, seed = decoding.prompt("front porch welcome sign wall décor"), 40, 2026print("longest description, in tokens:", longest)print("random seed:", seed)for label, settings in [("sampling, temperature 1.0", {}), ("sampling, temperature 0.7", {"temperature": 0.7}),                        ("top-k 20", {"top_k": 20}), ("top-p 0.9", {"top_p": 0.9})]:    ids = decoding.generate(start, rng=np.random.default_rng(seed), longest=longest, **settings)    print(f"{label}:", decoding.text(ids))rng, written, tail = np.random.default_rng(seed), [], []    # replay temperature 1.0's draws, step by stepfor _ in range(longest):    probabilities = transformer.softmax(decoding.next_logits(start + written))    token = int(rng.choice(len(probabilities), p=decoding.truncate(probabilities)))   # no cut: draw from all of it    order = np.argsort(-probabilities, kind="stable")    before = probabilities[order[:int(np.where(order == token)[0][0])]].sum()   # the probability ranked above it    if before >= 0.9:                                        # top-p 0.9 would have cut this token        tail.append(vocabulary[token].replace("</w>", ""))    written.append(token)    if vocabulary[token] == transformer.END:        breakprint("temperature 1.0, tokens drawn from outside the 90% nucleus:", f"{len(tail)} of {len(written)}")print("temperature 1.0, those tokens:", ", ".join(tail))
输出python sample-descriptions.py
longest description, in tokens: 40
random seed: 2026
sampling, temperature 1.0: enhance your home even glywheel design with this welcome welcome sign sign . this piece is ideal for both indoor and inviting . made in the usa , mini spirits decorate .
sampling, temperature 0.7: the sign is a great way to add some extra whimsy to your home and get colorful ! made of high-quality metal , this sign comes with a hanging dustroke hanger to help you gli
top-k 20: if you have a new product with a personalized sign wall décor that will help you entertain . look no further like a home decor , this sign is constructed to make life more wonderful . this welcome sign
top-p 0.9: express your guests with this cast iron sign from the nature of the world . it features a meditating podishade design . the hose and symbol in this metal piece will be
temperature 1.0, tokens drawn from outside the 90% nucleus: 5 of 36
temperature 1.0, those tokens: inviting, in, mini, spir, decorate

截去尾部只能去掉最罕见的错误,而不是全部错误。温度为 1 的采样会漫无目的地游走:它写出了 welcome welcome sign sign,造出了一个词 glywheel,还称标牌是 made in the usa。其中只有一部分来自尾部:5 of 36 个词元来自尾部,inviting, in, mini, spir, decorate。重复的词和杜撰的词都由核内的词元构成,top-p 本会保留它们。在 0.7 时,文本更连贯,但它现在说标牌是 made of high-quality metal,而贪心解码说的是木头。Top-k 和 top-p 读起来比普通采样更像商品目录,但仍会杜撰:top-p 的标牌是 cast iron,设计为 podishade。

在本章中,欢迎标牌已经六次被赋予材质,分别来自贪心解码、束搜索、其中三次采样,以及下文的带惩罚的贪心文本,而它的名称并没有说明它由什么制成。模型写下的是欢迎标牌的描述通常会写的内容,而这些描述通常会写明材质。解码方法改变的是模型选中哪一种看似合理的材质,没有哪种方法能让它选中真实的那一种。

工作准则: 若要得到多样的文本,可用约 0.9 的 top-p 或约 20 的 top-k 截去尾部,温度取 1 或略低于 1;需要复现结果时,固定随机种子。不要指望任何设置能让文本变得真实:解码只是在模型认为合理的内容中做选择,核查事实是另一项工作。

测量每种设置写出了什么

最近一次修改见于版次 2026.11.12

单个样本能显示每种方法的风格,而测量才能显示其中的取舍。有几项性质无需人来阅读文本就可以测量:

  • **重复率:**同一条描述中,与前文重复的四词元短语所占的比例
  • **模型自身对文本的困惑度:**模型对自己所写内容的意外程度,按所有词元汇总计算,数值低表示可预测
  • 描述是否在限制长度内结束,真实描述通常如此
  • 是否用到了产品名称中的一个完整单词,对该产品的描述理应如此
  • 平均长度,以词元计

每种设置与真实文本的接近程度如何?从有描述的验证集产品中随机抽取 20 个产品,用每种方法为每个产品写一条描述,并与真实描述比较,真实描述同样在 60 个词元处截断:

measure-decoding.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通过
"""Seven ways of choosing the next token, measured on the same 20 product descriptions: how often each repeatsitself, how likely the model finds its own text, whether it ends, and whether it stays on the product. The realdescriptions come first, as the bar the others are read against. The prose quotes these lines for each setting."""import numpy as npfrom lib import decoding                    # the store's model, and the decoding methods this chapter buildsfrom lib.products import product_textslongest, sample, seed = 60, 20, 2026# Validation products, which the model never trained on, that have at least one token of description after <sep>described = [(ids, sep) for ids, sep in product_texts("validation") if sep < len(ids) - 2]products = [described[i] for i in np.random.default_rng(seed).choice(len(described), sample, replace=False)]print("products, drawn at random from the validation products with a description:", sample)print("longest description, in tokens:", longest)settings = [("real descriptions", None), ("greedy", {}), ("greedy, repetition penalty 1.3", {"penalty": 1.3}),            ("sampling, temperature 1.0", {"sample": True}), ("sampling, temperature 0.7", {"sample": True, "temperature": 0.7}),            ("top-k 20", {"sample": True, "top_k": 20}), ("top-p 0.9", {"sample": True, "top_p": 0.9})]for label, chosen in settings:    rows = []    for number, (ids, sep) in enumerate(products):        start = ids[:sep + 1]                                 # the start marker, the name and <sep>: the model writes on from here        if chosen is None:            written = ids[sep + 1:][:longest]                 # the description as sold, cut where generating would stop        else:            options = {option: value for option, value in chosen.items() if option != "sample"}            # generate samples when it's given a generator, and is greedy without one. Each product gets its own            # seed, the same for every sampling setting, so the settings differ only in how they choose.            rng = np.random.default_rng(seed + number) if chosen.get("sample") else None            written = decoding.generate(start, rng=rng, longest=longest, **options)        rows.append(decoding.measures(start, written))        # repetition, each token's log-probability, ended, uses the name, length    repeated, log_p, ended, uses_name, length = zip(*rows)    print(f"{label}, repeated 4-token phrases:", f"{np.mean(repeated):.0%}")    print(f"{label}, the model's perplexity of the text:", f"{np.exp(-np.concatenate(log_p).mean()):.1f}")   # over all 20 texts' tokens at once    print(f"{label}, ended within {longest} tokens:", f"{np.mean(ended):.0%}")    print(f"{label}, uses a whole word of the name:", f"{np.mean(uses_name):.0%}")    print(f"{label}, average length in tokens:", f"{np.mean(length):.0f}")
输出python measure-decoding.py
products, drawn at random from the validation products with a description: 20
longest description, in tokens: 60
real descriptions, repeated 4-token phrases: 0%
real descriptions, the model's perplexity of the text: 30.5
real descriptions, ended within 60 tokens: 35%
real descriptions, uses a whole word of the name: 95%
real descriptions, average length in tokens: 53
greedy, repeated 4-token phrases: 11%
greedy, the model's perplexity of the text: 2.7
greedy, ended within 60 tokens: 25%
greedy, uses a whole word of the name: 80%
greedy, average length in tokens: 52
greedy, repetition penalty 1.3, repeated 4-token phrases: 0%
greedy, repetition penalty 1.3, the model's perplexity of the text: 3.9
greedy, repetition penalty 1.3, ended within 60 tokens: 70%
greedy, repetition penalty 1.3, uses a whole word of the name: 80%
greedy, repetition penalty 1.3, average length in tokens: 41
sampling, temperature 1.0, repeated 4-token phrases: 0%
sampling, temperature 1.0, the model's perplexity of the text: 22.6
sampling, temperature 1.0, ended within 60 tokens: 50%
sampling, temperature 1.0, uses a whole word of the name: 50%
sampling, temperature 1.0, average length in tokens: 45
sampling, temperature 0.7, repeated 4-token phrases: 0%
sampling, temperature 0.7, the model's perplexity of the text: 7.2
sampling, temperature 0.7, ended within 60 tokens: 45%
sampling, temperature 0.7, uses a whole word of the name: 65%
sampling, temperature 0.7, average length in tokens: 45
top-k 20, repeated 4-token phrases: 0%
top-k 20, the model's perplexity of the text: 7.6
top-k 20, ended within 60 tokens: 50%
top-k 20, uses a whole word of the name: 80%
top-k 20, average length in tokens: 40
top-p 0.9, repeated 4-token phrases: 0%
top-p 0.9, the model's perplexity of the text: 10.7
top-p 0.9, ended within 60 tokens: 30%
top-p 0.9, uses a whole word of the name: 60%
top-p 0.9, average length in tokens: 47

贪心文本远比人写的任何文本更容易预测,其困惑度为 2.7,其中 11% 的四词元短语是重复的。真实描述没有任何重复(四词元短语的重复率为 0%),模型觉得它们相当出人意料,困惑度为 30.5。这一数值高于第 8 章中描述的数值,因为这里是另一份样本,而且只取其前 60 个词元。Holtzman 等人在大规模实验中得到了同样的观察:人类的文本并不是模型认为最可能的文本。

没有哪种设置在所有指标上都与真实描述相符。温度为 1 的采样在困惑度上最为接近,为 22.6,但它的描述中只有 50% 用到了产品名称中的单词,而真实描述为 95%,所以它会偏离产品。top-p 0.9 的结束频率几乎与真实文本相当,为 30%,真实文本为 35%,但只有 60% 的描述紧扣产品。在各种采样方法中,top-k 20 最能紧扣产品,为 80%,没有重复,描述平均长 40 个词元。二十个产品是很小的样本,因此应把这些结果当作趋势,而不是精确的比率。

这些指标无法说明的是描述是否属实。一条描述可以流畅、多样、切题却是错的,上一节的描述就是如此,它们在标牌的材质上有误。

**工作准则:**在阅读样本之前,先用这类指标,对照同类的真实文本来比较解码设置。确定对这项任务最重要的指标:对于产品描述,紧扣产品和不重复排在最前。然后仍要阅读一部分输出样本,以发现任何指标都捕捉不到的问题,并对照产品数据核对每一项事实性陈述。

惩罚重复

最近一次修改见于版次 2026.11.11

贪心解码的循环也可以不借助采样来解决。Keskar 等人提出了重复惩罚(repetition penalty):把每个已生成词元的 logit 除以一个大于 1 的因子 θ,使其出现的可能性降低(Keskar et al., 2019,第 4.1 节)。负的 logit 除以 θ 反而会变大,因此实现中对负 logit 乘以 θ,lib.decoding.penalize 就是这样做的,这样无论正负,该词元的可能性都会降低。Keskar 等人发现,因子取 1.2 左右,可以在避免重复与保持行文合理之间取得平衡。

回到贪心描述即将第二次写出 distressed 的位置。先手工把惩罚应用到下一个词元的 logit 上,再用 lib.decoding.penalize 核对结果,并比较惩罚前后的概率。然后带着惩罚用贪心方式写出这段描述,看循环是否消失:

repetition-penalty-by-hand.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通过
"""The repetition penalty applied by hand to the next token's logits, checked against lib.decoding.penalize, and thenused for a whole greedy description. The two P(distressed) lines are the ones the prose quotes, and theassert against lib.decoding.penalize is the match it states as fact."""import numpy as npfrom lib import decoding, transformer_, vocabulary, _ = transformer.load()start, factor = decoding.prompt("front porch welcome sign wall décor"), 1.3written = decoding.generate(start)                                    # greedy, as in greedy-descriptiondistressed = vocabulary.index("distressed</w>")position = [i for i, token in enumerate(written) if token == distressed][1]   # where greedy writes it a second timebefore = written[:position]print("text so far:", decoding.text(before))print("penalty factor:", factor)logits = decoding.next_logits(start + before)seen = np.unique(before)                                              # every token the description has usedpenalized = logits.copy()penalized[seen] = np.where(logits[seen] > 0, logits[seen] / factor, logits[seen] * factor)   # less likely either wayassert np.allclose(penalized, decoding.penalize(logits, before, factor))   # the hand calculation is the library'sfor label, adjusted in [("without a penalty", logits), ("with the penalty", penalized)]:    probabilities = transformer.softmax(adjusted)    top = np.argsort(-probabilities, kind="stable")[:3]    print(f"next token, {label}:", ", ".join(f"{vocabulary[i].replace('</w>', '')} {probabilities[i]:.2f}" for i in top))    print(f"P(distressed), {label}:", f"{probabilities[distressed]:.2f}")print("greedy with the penalty:", decoding.text(decoding.generate(start, penalty=factor)))
输出python repetition-penalty-by-hand.py
text so far: this welcome sign is a great way to add a touch of rustic charm to your home . it is made of natural wood and features a distressed finish with a
penalty factor: 1.3
next token, without a penalty: distressed 0.20, black 0.03, weathered 0.03
P(distressed), without a penalty: 0.20
next token, with the penalty: black 0.04, weathered 0.03, distressed 0.03
P(distressed), with the penalty: 0.03
greedy with the penalty: this welcome sign is a great way to add style and functionality to any room . the worn look of these beautiful signs are made from natural coir , so it can be used indoors or outdoors .

在因子为 1.3 时,distressed 的概率从 0.20 降到 0.03,由另一个词元取而代之。手工计算与库的计算结果一致,带惩罚的贪心描述也不再循环。不过它仍然会编造内容:这一次,招牌是用 natural coir 做成的,那是一种取自椰子壳的纤维。

惩罚还有一个副作用,测量结果显示了这一点:描述结束得更早。惩罚针对每一个已用过的词元,包括描述中需要多次出现的常用词,例如 the 和 and,却从不惩罚尚未用过的结束标记。于是结束标记的相对可能性上升:带惩罚的贪心描述中有 70% 在限制长度内结束,不带惩罚的则为 25%;平均长度为 41 个词元,不带惩罚时为 52 个。其重复率从 11% 降到了 0%。

工作准则: 采用贪心解码或低温度解码时,加上约 1.2 到 1.3 的重复惩罚,并检查它对长度的影响,因为它会使文本变短。经验上,如果描述被截得过短,就不要把常用功能词和标点纳入惩罚。

禁止空描述

最近一次修改见于版次 2026.11.11

贪心解码示例发现,模型准备为一个灯泡什么都不写;而束搜索则发现,发现对于一块标牌,最可能的描述就是空描述。这两种情况都源于商品目录:许多商品没有描述,因此模型学到了立即结束是常见现象。要填补的恰恰是这些空缺,所以必须阻止模型照搬它们。简单的办法是设置最小长度:在描述达到一定数量的词元之前,把结束标记的 logit 设为负无穷,从而禁止它出现。lib.decoding.generate 通过 shortest 接收这一设置。

贪心解码留空的那个灯泡就是测试用例。对它解码两次,第一次不设最小长度,第二次将最小长度设为 10 个词元;输出首先给出其描述立即结束的概率:

minimum-length.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通过
"""Greedy decoding of the bulb's description, without a minimum length and with one that forbids the end marker untilit has that many tokens. The prose quotes the printed probability of an empty description, andthe description the minimum produces."""from lib import decoding, transformerend = transformer.load()[1].index(transformer.END)                    # the index of </s>, to read its probabilitystart, shortest = decoding.prompt("5 light light bulb"), 10print("P(the description is empty):", f"{transformer.softmax(decoding.next_logits(start))[end]:.2f}")   # the first token is </s>: an empty descriptionprint("greedy description:", decoding.text(decoding.generate(start)) or "(empty)")print("minimum length, in tokens:", shortest)print("greedy description, with the minimum:", decoding.text(decoding.generate(start, shortest=shortest)))   # shortest: </s> stays forbidden until then
输出python minimum-length.py
P(the description is empty): 0.28
greedy description: (empty)
minimum length, in tokens: 10
greedy description, with the minimum: this light is a great light for a light , a light , and more light . it is made of metal and has a long , high gloss finish that will make it a perfect accent piece for your home . this light is dimmable , compatible with most standard bulbs , which are not included .

最小长度可以填补空缺,却无法提供事实。不设最小长度时,贪心解码选择空描述,其概率为 0.28。设置最小长度后,它写出了一段描述,而本章前面出现的两种失败都出现在其中。它重复自身,写出 a light , a light , and more light;它还陈述了名称中没有的内容:这个灯泡是 made of metal、dimmable,并且 compatible with most standard bulbs。

工作准则: 只要训练文本中包含所要生成内容的空示例或极短示例,就设置最小长度。要把它与本章的其他设置结合使用,而不是用它取代这些设置,并且要预料到,被它强制生成的文本和其他任何文本一样需要检查。

生产环境中会出什么问题

最近一次修改见于版次 2026.11.11

解码的大多数失败都悄无声息:

  • 流畅的文本会陈述编造的事实。 本章中有六条描述给欢迎标牌指定了材质,而且彼此不一致。在对照商品数据核实每一项事实性陈述之前,切勿发布生成的商品文本;第 11 章会测量模型编造内容的频率。
  • 训练文本中的空缺会变成输出。 模型会为灯泡预测出空描述,因为目录中有许多这样的条目,而束搜索认为空描述的可能性最大。应禁止结束标记在达到最小长度之前出现,否则模型会重现你想要填补的那些空缺。
  • 贪心解码的文本会循环。 对于长于一个短语的文本,应使用截断尾部的采样或重复惩罚,并在一批输出样本上测量重复情况。
  • 采样得到的文本默认不可复现。 应固定随机种子,并随每条生成的文本一同记录,这样针对某条描述的投诉才能被追溯和重现。
  • 设置会随模型漂移。 重新训练后的模型分布不同,因此在上一个模型上调好的温度或 top-k,用在下一个模型上可能写出不同的文本。每次重新训练之后,在新模型写出任何购物者能看到的内容之前,应重新运行本章的各项测量。
  • 每个词元都要付出一次前向计算的代价。 lib.decoding 为每个新词元都在整段文本上重新运行模型,这种做法简单,对于 60 个词元来说也足够好。实际的服务系统会保留每一层在先前位置上的键和值,因此每个新词元的矩阵乘法只覆盖一个位置,不过注意力仍要读取此前每个位置已缓存的键和值(Pope et al., 2022)。

本章的决定

  • 自动补全使用最可能的词元。 搜索框的建议显示第 8 章所选组合中概率最高的几个词元,并可用按总概率排序的束搜索将它们扩展为短语。
  • 商品描述使用 top-k 为 20 的采样。 按上述指标,这一设置最常在不重复的情况下始终围绕商品展开;同时使用固定且有记录的随机种子,并设定最小长度,使描述不会为空。由于背后只有二十件商品,在任何描述送达购物者之前,要先在更大的样本上重新检验这一选择。
  • 束搜索按每个词元对较长文本排序, 因为总概率偏向空文本。搜索查询的简短补全则按总概率排序,其中最可能的短语正是购物者想要的。
  • 解码设置通过对照真实文本的测量来选定: 在一组随机商品上测量重复率、模型自身的困惑度、描述是否有结尾、是否使用商品名称以及描述的长度,这些测量先于阅读样本进行,并在每次重新训练之后再做一次。
  • 生成的描述只是草稿,绝不能作为商品事实的来源。 凡是商品名称和商品目录不能支持的说法,都由人工或检查步骤予以更正:这里的每种方法都曾为名称未给出材质的商品编造了材质。

本章,在本版次中

版次 2026.11.16示例于 2026年11月19日 通过本章 7个示例 个示例中有 7 个通过

译自英文版图书,版次 2026.11.16。

引用本节

格式

每种格式均引用英文原书,本译本以其为准。

每种格式都会注明该版次。其链接会打开当前版次,节标题处会说明该节文本最后一次修改的时间。访问日期和检索日期即为您打开此页的当天。

本页内容

第 9 章,共 11 章: Choosing the next token

  1. 选择可能性最大的词元
  2. 搜索最可能的文本
  3. 带温度的采样
  4. 采样并截去尾部
  5. 测量每种设置写出了什么
  6. 惩罚重复
  7. 禁止空描述
  8. 生产环境中会出什么问题
  9. 本章的决定

返回顶部全部章节