Counting further back
阅读约需 47分钟 分钟,9节中含7个示例。
这部分尚未翻译,因此以英文显示。
By the end of this chapter you can build an n-gram language model that looks several tokens back, smooth it with Kneser-Ney so it handles text it hasn't seen (smoothing gives some probability to what the counts never saw, as chapter 1 showed), measure the baseline that every neural model in this book is measured against, and see why counting, however clever, runs out of road.
Chapter 1 built a model that predicted the next word from the one before it, and two things went wrong. It looked back only one word, so a shopper who typed patio bar got the suggestions for bar. And it gave probability zero to every pair of words it hadn't counted, which made its perplexity on new text infinite. Chapter 2 then gave the book better units to count: the store's tokenizer, which splits every word into pieces it knows, so almost every word is spelled from tokens it has seen. This chapter goes back to counting with those tokens, looks further back, and handles the unseen properly.
Every method here is tried where the store, an online furniture and home-goods shop, needs it: on the product names its search box completes. From this chapter on, every model in the book is scored the same way: by its perplexity per token on the store's validation names, tokenized with the store's tokenizer. Product names are what the search box's autocomplete has to predict, and they're short enough for every example to run in seconds. One fixed text, one tokenizer and one measure make every model in the book comparable with every other on names; later chapters add descriptions, from chapter 8, and per-word scores, in chapter 10, where a model's tokenizer differs.
更长的上下文更贴合训练文本
n 元语法模型(n-gram model)根据前面的 n − 1 个词元来预测每个词元:二元语法(bigram)模型(n = 2)依据前一个词元,三元语法模型(n = 3)依据前两个词元,依此类推。位于被预测词元之前的那些词元称为它的上下文。上下文越多,预测应当越好:在 patio bar 之后,下一个词元比仅在 bar 之后要容易预测得多。计数的方式与第 1 章完全相同,区别在于现在统计的是 n − 1 个词元的上下文及其后跟随的一个词元,并且每段文本前都补上 n − 1 个起始标记,这样连它的第一个词元也有完整的上下文。
在不做平滑的情况下,用统计所用的那些名字来评分,各个阶数显示出模型对已经记住的文本会做什么:
1"""Counts the n-grams of orders 1 to 5 in the training names, and scores each order on those same names with no smoothing.2Read the training perplexity lines, which fall as the order grows, against the seen-once lines, which rise."""3import math4from collections import Counter56from lib.names import name_tokens7from lib.ngrams import ngrams, padded89train = name_tokens("train")10for n in [1, 2, 3, 4, 5]:11 # padded adds n-1 start markers and one end marker, so a name's first token has a full context12 pairs = Counter(pair for text in train for pair in ngrams(padded(text, n), n))13 contexts = Counter()14 for (context, _), count in pairs.items():15 contexts[context] += count16 # the total surprise, in nats: each n-gram's probability is its count over its context's count, the model's own estimate17 surprise = -sum(count * math.log(count / contexts[context]) for (context, _), count in pairs.items())18 print(f"{n}-gram training perplexity: {math.exp(surprise / sum(pairs.values())):.1f}")19 print(f"{n}-grams seen once: {sum(1 for c in pairs.values() if c == 1) / len(pairs):.0%}")
python ngram-memorizes.py1-gram training perplexity: 676.4
1-grams seen once: 8%
2-gram training perplexity: 23.9
2-grams seen once: 60%
3-gram training perplexity: 4.9
3-grams seen once: 76%
4-gram training perplexity: 3.2
4-grams seen once: 82%
5-gram training perplexity: 2.9
5-grams seen once: 85%
随着上下文增长,困惑度迅速下降:没有上下文时为 676.4,有一个词元的上下文时为 23.9,有四个词元的上下文时为 2.9。在四个词元的上下文下,模型在每一步大约只在三个词元之间做选择,因为那么长的 n 元语法大多只出现一次:训练名字中有 85% 的 5 元语法只出现过一次,模型已经记住了每一个之后跟着什么。
这就是陷阱所在。在统计所用的文本上,更长的上下文看起来总是更好,因为只要上下文足够多,每段文本都是独一无二的,模型就能把它背出来。用模型自己的训练文本来评分,衡量的是记忆,这正是第 1 章一开始就留出验证文本的原因。
工作准则: 永远不要根据 n 元语法模型(或任何语言模型)在自身训练文本上的困惑度来评判它:随着模型记住文本,这个值会趋向 1。应当依据在留出文本上的困惑度来选择阶数,本章其余部分就是这样做的。
更长的上下文会缺少证据
记住训练集中的名称,并不能说明模型对新名称的表现。模型只能使用它统计过的上下文,而可能的上下文数量会随长度呈指数增长:词元表有四千个词元时,可能的二元组有一千六百万个,可能的三元组有六百四十亿个。任何训练文本都远远覆盖不了这些组合。因此,上下文越长,新文本用到模型从未见过的上下文的情况就越多。
先确定一个数字:对于 4-gram,验证集名称中有多大比例的 4-gram 从未在训练集名称中出现过?下面给出从 1 阶到 5 阶各阶的比例:
1"""Counts, for each order from 1 to 5, the share of the validation names' n-grams that never occur in the training names.2Compare the lines for 2-grams to 5-grams: that share rises with the order."""3from lib.names import name_tokens4from lib.ngrams import ngrams, padded56train, validation = name_tokens("train"), name_tokens("validation")7print("training tokens:", f"{sum(len(t) for t in train):,}")8print("validation tokens:", f"{sum(len(t) for t in validation):,}")9for n in [1, 2, 3, 4, 5]:10 seen = {pair for text in train for pair in ngrams(padded(text, n), n)} # padded, as in the model, so name starts count too11 held_out = [pair for text in validation for pair in ngrams(padded(text, n), n)]12 unseen = sum(pair not in seen for pair in held_out)13 print(f"{n}-grams in validation never seen in training: {unseen / len(held_out):.1%}")
python unseen-ngrams.pytraining tokens: 337,071
validation tokens: 42,422
1-grams in validation never seen in training: 0.1%
2-grams in validation never seen in training: 12.7%
3-grams in validation never seen in training: 31.1%
4-grams in validation never seen in training: 43.9%
5-grams in validation never seen in training: 53.4%
上下文越长,验证集名称中属于新出现的 n-gram 就越多。未见过的验证词元只占 0.1%,这是商店的分词器在发挥作用。在验证集的二元组中,有 12.7% 未见过,4-gram 中有 43.9%,5-gram 中有 53.4%。当上下文为四个词元时,模型要预测所需的内容,有一半以上是它从未见过的。
增加训练文本的帮助,比你期望的要小。在语言中,少数 n-gram 出现得非常频繁,而大多数 n-gram 很少出现,所以文本每增加一倍,新增的主要是只出现一两次的 n-gram,而这些几乎无助于改善任何一次预测所依据的计数。商店的商品目录以缩小的规模呈现了同样的模式:像 coffee table 这样的常见短语被统计了一千多次,而训练集名称的 4-gram 中有 82% 只被统计了一次。我们的判断(并非测量结果)是:即使目录扩大到十倍,仍会有很大比例的新 4-gram 未被见过,因为新的产品线以及尺寸与材质的新组合会不断出现。
这两个示例合在一起,就是计数方法的核心问题。上下文越长,在见过的情况下预测越准,但被见到的次数却越来越少。本章其余的每一种方法,都是在计数足以支持时使用长上下文,在计数不足时回退到较短的上下文。
工作准则: 在选择 n-gram 的阶数之前,先测量留出集中有多大比例的 n-gram 是训练文本从未包含的。根据经验,当这一比例接近三分之一时(如商店名称中的三元组),该阶数的模型主要取决于它如何处理未见过的情况,因此其平滑方法比阶数更重要。
混合短上下文与长上下文
对于未见过的长上下文,显而易见的补救办法是退回到较短的上下文。插值(interpolation)以平滑的方式实现这种退回:一个词元的概率是三元组、二元组和一元组计数各自给出的结果的加权混合。因此,模型见过的长上下文贡献其尖锐的预测,较短的上下文则补上它没见过的部分。设权重 λ₃、λ₂ 和 λ₁ 之和为 1:
P(token | a b) = λ₃ · P₃(token | a b) + λ₂ · P₂(token | b) + λ₁ · P₁(token)
其中每个 P 都是由计数得到的极大似然估计,与第 1 章的概率相同。Chen 和 Goodman 对平滑方法的研究详细讨论了插值及其变体(Chen and Goodman, 1996)。
手工设定权重能把这一思路讲得很清楚。取一个插值三元组模型,三个阶数的权重分别为 0.6、0.3 和 0.09,剩下的 0.01 平均分给每个词元,这样任何词元的概率都不会为零。当某个上下文从未出现过时,它的权重会转给更低的阶数,因此各概率之和仍为 1:
1"""Builds an interpolated trigram model by hand, with fixed weights, and an add-one bigram model on the same tokens.2The two validation perplexity lines to compare are the interpolated trigram's and the add-one bigram's."""3import math4from collections import Counter56from lib.names import name_tokens7from lib.ngrams import ngrams, padded, perplexity89train, validation = name_tokens("train"), name_tokens("validation")10counts = {k: Counter() for k in (1, 2, 3)} # counts[k][(the k-1 tokens before, token)]11for text in train:12 for context, token in ngrams(padded(text, 3), 3):13 for k in (1, 2, 3):14 # a trigram's context holds two tokens; an order-k model conditions on the last k-1 of them15 counts[k][(context[3 - k:], token)] += 116totals = {k: Counter() for k in (1, 2, 3)}17for k in (1, 2, 3):18 for (context, _), count in counts[k].items():19 totals[k][context] += count20# the tokens the training names contain, plus one slot for a token they never contain21vocabulary = len({token for (_, token) in counts[1]}) + 122weights = {3: 0.6, 2: 0.3, 1: 0.09} # and 0.01 spread evenly over every token232425class Interpolated:26 n = 32728 def probability(self, context, token):29 p, carried = 0.0, 0.030 for k in (3, 2, 1):31 # each order's own context32 weight, total = weights[k] + carried, totals[k][context[3 - k:]]33 if total: # an unseen context passes its weight down34 p, carried = p + weight * counts[k][(context[3 - k:], token)] / total, 0.035 else:36 carried = weight37 # the even 0.01 share, plus any weight passed down from unseen contexts, over the vocabulary38 return p + (0.01 + carried) / vocabulary394041print("interpolated 3-gram validation perplexity:", round(perplexity(Interpolated(), validation), 1))4243# a bigram's context is the one token before the token44bigram_pairs = Counter((context[-1:], token) for text in train for context, token in ngrams(padded(text, 2), 2))45bigram_totals = Counter()46for (context, _), count in bigram_pairs.items():47 bigram_totals[context] += count484950class AddOne:51 n = 25253 def probability(self, context, token):54 # add-one: one added to every pair's count, and one per vocabulary token added to the total55 return (bigram_pairs[(context, token)] + 1) / (bigram_totals[context] + vocabulary)565758print("add-one 2-gram validation perplexity:", round(perplexity(AddOne(), validation), 1))
python interpolate-orders.pyinterpolated 3-gram validation perplexity: 31.4
add-one 2-gram validation perplexity: 218.6
混合多个阶数的效果优于第 1 章的加一平滑。为了公平比较,这里的加一平滑作用于同一批验证词元上的二元组模型,因为在不同词元上算出的困惑度无法相互比较。插值三元组模型的得分为 31.4,加一平滑的得分为 218.6。加一平滑把大部分概率分给了从不出现的词对;混合多个阶数则保留了每个阶数各自掌握的信息。
这种方法有两个弱点。其一,权重是固定的,而且这里的权重是手工选定的,并未经过调优:无论上下文只出现过一次还是出现过一千次,三元组得到的都是同样的 0.6,可是出现过一千次的上下文理应得到更多信任。其二,一元组部分按词元出现的频繁程度分配概率,而如下一节所示,对于退回用的分布,这问错了问题。
**工作准则:**绝不单独使用某一个 n 元组阶数,始终将其与更低的阶数结合,使每个词元都保有一定的概率。优先选择根据各上下文的计数来设定其权重的方法,例如下文的 Kneser-Ney,而不是固定权重。如果确实使用固定权重,应在保留的文本上选择它们,而不是在用来计数的文本上选择。
统计词元所跟随的上下文数
当模型回退到较短的上下文时,它在问一个具体的问题:这个词元出现在一个我们没见过的上下文中的可能性有多大?原始频率回答的是另一个问题。一个词元可以非常常见,却只出现在某一两个特定词元之后,这样的词元对新上下文而言是很差的猜测。更好的度量是词元的延续数(continuation count),即它所跟随的不同词元的个数:一个曾跟随过许多不同词元的词元,很可能也会跟随一个新的词元。
在训练名称中频率最高的 300 个词元里,下面三个词元相对于其出现次数而言,所跟随的不同词元最少,旁边另列两个普通的商品词作对照:
1"""Counts, for each token in the training names, how often it occurs and how many different tokens come right before it.2Look at the three narrowest tokens, which occur often but follow few different tokens, beside table and chair."""3from collections import Counter, defaultdict45from lib.names import name_tokens67occurrences, before = Counter(), defaultdict(set)8for text in name_tokens("train"):9 for previous, token in zip(["<s>"] + text, text): # the start marker stands before each name's first token, so that token has something to follow too10 occurrences[token] += 111 before[token].add(previous) # the different tokens this one follows1213common = [t for t, _ in occurrences.most_common(300)] # the 300 most frequent tokens: the pool the prose ranks14# different tokens followed, per occurrence, lowest first: the three narrowest15narrow = sorted(common, key=lambda t: len(before[t]) / occurrences[t])[:3]16for token in narrow + ["table</w>", "chair</w>"]: # </w> marks a token that ends a word, as the store's tokenizer writes it17 print(f"{token:<14} occurs {occurrences[token]:>5,} times, after {len(before[token]):>4,} different tokens")
python continuation-counts.pytvs</w> occurs 323 times, after 1 different tokens
wide</w> occurs 1,985 times, after 14 different tokens
''</w> occurs 10,335 times, after 85 different tokens
table</w> occurs 3,487 times, after 154 different tokens
chair</w> occurs 2,791 times, after 138 different tokens
词元 tvs 出现了数百次,却只跟随一个其他词元:在该商店的名称中,它总是同一个短语的一部分,即专为电视设计的商品。一个回退到原始频率的模型会认为 tvs 在任何位置都是合理的猜测,而实际上它几乎从来不是。wide 只跟随少数几个词元,多半是数字的英寸标记,而英寸标记本身 '' 多半跟随数字。与此同时,table 和 chair 各自跟随一百多个不同的词元:coffee table、side table、outdoor table。它们的延续数说明了频率所不能说明的事:它们出现在一百多个不同词元之后。
延续数的计算成本很低:对每个词元,保存在它之前出现过的词元集合,再统计该集合的大小。这不需要额外的文本,也不需要调参,只是对同样的计数换一种问法,下一节将在此基础上展开。
这正是 Kneser-Ney 平滑(Kneser and Ney, 1995)背后的思路:当模型回退到较短的上下文时,它使用延续数而不是原始计数,于是在许多上下文中出现的词元获得概率,而只出现在某个固定短语中的词元则得不到。
工作准则: 每当从较长的上下文回退到较短的上下文时,按词元所跟随的不同上下文的个数来加权,而不是按它出现的次数。只有在最高阶,即上下文确实已被见过的地方,才使用原始计数。
这部分尚未翻译,因此以英文显示。
Smooth with Kneser-Ney
Kneser-Ney smoothing combines interpolation of orders with continuation counts. At each order, Kneser-Ney subtracts a fixed discount d from every count it has seen, and gives the probability it took to the next order down, so a context seen often keeps nearly all its own prediction, and a context seen rarely leans on the shorter context, with the discount it frees going to tokens that turn up in many contexts.
The same formula applies at every order. Below the highest, the counts are continuation counts, except for n-grams that start a text, which keep their raw counts, as the KenLM toolkit does. For a context h and a token w, with c(h w) the count of the pair, c(h) the count of the context, and N(h) the number of different tokens seen after h:
P(w | h) = max(c(h w) − d, 0) / c(h) + d · N(h) / c(h) · P(w | shorter h)
The book's shared module lib.ngrams implements interpolated Kneser-Ney as Chen and Goodman describe it (Chen and Goodman, 1999), with a discount of 0.75, a common default. Its lowest order mixes in a uniform share over the vocabulary, so even a token never seen in training keeps a small probability.
Before trusting the module, work out one probability by hand: stool after a name that starts patio bar, order by order. Set it beside the module's answer, and beside the module's probabilities after this context, summed over every token:
1"""Works out the probability of stool after a name that starts patio bar, order by order, with the chapter's formula.2It checks the result against lib.ngrams and against the module's probabilities summed over the vocabulary: the3by hand line should match the lib.ngrams line, and the summed line should be a probability distribution's total."""4from lib import bpe5from lib.names import name_tokens6from lib.ngrams import START, KneserNey78model = KneserNey(name_tokens("train"), 4) # a 4-gram model; below the top order its counts are continuation counts, except for n-grams that start a name9context = tuple([START] + bpe.store_tokenizer().encode("patio bar")) # a name that starts "patio bar"10token, d = "stool</w>", model.d # </w> ends a word; d is the discount, 0.75 by default111213def by_hand(context, k): # the chapter's formula, one order at a time14 if k == 0:15 return 1 / (len(model.vocabulary) + 1) # order 0 is uniform over the vocabulary, with one more slot for an unseen token16 context = context[len(context) - (k - 1):] if k > 1 else () # the last k-1 tokens: the order-k model's own context17 following, total = model.counts[k][context], model.totals[k].get(context, 0)18 if total == 0:19 return by_hand(context, k - 1) # a context never seen gives all its weight to the order below20 kept = max(following[token] - d, 0) / total # the discounted count21 passed_down = d * len(following) / total # what the discount frees for the order below22 print(f"order {k}: kept {kept:.4f}, passed down {passed_down:.4f}")23 return kept + passed_down * by_hand(context, k - 1)242526print("by hand:", f"{by_hand(context, 4):.4f}")27print("lib.ngrams:", f"{model.probability(context, token):.4f}")28total = sum(model.probability(context, t) for t in model.vocabulary)29print("probabilities after the context, summed:", f"{total:.4f}")
python kneser-ney-by-hand.pyorder 4: kept 0.2500, passed down 0.7500
order 3: kept 0.8056, passed down 0.1667
order 2: kept 0.0630, passed down 0.1275
order 1: kept 0.0007, passed down 0.0318
by hand: 0.8621
lib.ngrams: 0.8621
probabilities after the context, summed: 1.0000
Worked by hand, the formula gives 0.8621 for stool, the same as the module's 0.8621. Each order keeps a share and passes the rest down; the shares add up to that answer. Over every token in the vocabulary, the module's probabilities after this context sum to 1.0000, as a probability distribution must.
Kneser-Ney models of every order from 1 to 5, scored on the validation names, show whether longer contexts now pay off:
1"""Builds Kneser-Ney models of orders 1 to 5 on the training names, scores each on the validation names, and estimates a2discount from the 4-gram counts. The lines to read are the validation perplexities, the 3-gram's set against the interpolated trigram's, and the discount line."""3from lib.names import name_tokens4from lib.ngrams import KneserNey, perplexity56train, validation = name_tokens("train"), name_tokens("validation")7for n in [1, 2, 3, 4, 5]:8 model = KneserNey(train, n)9 print(f"{n}-gram validation perplexity: {perplexity(model, validation):.1f}")1011model = KneserNey(train, 4)12# the highest order keeps raw counts, not continuation counts, so these are how often each 4-gram occurs in training13top = [c for following in model.counts[4].values() for c in following.values()]14once, twice = top.count(1), top.count(2)15print("4-grams seen once:", f"{once:,}")16print("4-grams seen twice:", f"{twice:,}")17# Chen and Goodman's estimate of the discount: D = n1 / (n1 + 2 n2), with n1 and n2 the counts seen once and twice18print("discount estimated from them:", f"{once / (once + 2 * twice):.2f}")
python kneser-ney-by-order.py1-gram validation perplexity: 687.2
2-gram validation perplexity: 46.9
3-gram validation perplexity: 24.9
4-gram validation perplexity: 23.1
5-gram validation perplexity: 23.5
4-grams seen once: 164,374
4-grams seen twice: 18,359
discount estimated from them: 0.82
Kneser-Ney beats the hand-weighted interpolated trigram from the interpolation section at the same order, with no weights to choose. Its trigram scores 24.9, against 31.4 for the hand-weighted one.
Longer contexts now help on new text, instead of only on the text the model counted, up to a point: validation perplexity falls from 46.9 for the bigram to 23.1 for the 4-gram, then rises to 23.5 for the 5-gram. The step from trigram to 4-gram is small, and the 5-gram is worse, because by then most contexts are unseen and the extra order adds noise. The discount, too, can be estimated from the counts: Chen and Goodman give D = n₁ / (n₁ + 2 n₂), where n₁ and n₂ are the numbers of n-grams seen once and twice, and on the store's 4-grams it's 0.82, close to the default.
Working rule: Smooth n-gram models with interpolated Kneser-Ney and a discount of 0.75 unless held-out text says otherwise. Raise the order only while validation perplexity keeps falling noticeably, and stop when a step barely helps or hurts, as the step from 4-gram to 5-gram does. A variant, modified Kneser-Ney, uses three discounts instead of one and is the one toolkits such as KenLM build; it would likely lower the baseline a little, and this book keeps the single discount because it's easier to follow by hand.
用三个词元的上下文做自动补全
更长的上下文应当能解决第 1 章中搜索框最糟糕的一条建议:patio bar 得到的建议与 bar 相同。4-gram Kneser-Ney 模型能看到购物者输入的最后三个词元,因此 patio 仍在视野之内。
从 bar 到 patio bar 再到 outdoor bar,建议是否发生了变化?取 4-gram 模型在每个前缀之后最可能的三个下一词元。每个前缀都被视为名称的开头,就像购物者的输入那样,每条建议显示的词元都不带词尾标记:
1"""Asks the 4-gram model for its three likeliest next tokens after bar, patio bar and outdoor bar, each typed as the start2of a name, then counts the training names that contain outdoor bar. The three suggestion lines carry the point, and the last line3counts the training names with outdoor bar in them."""4from lib import bpe5from lib.names import name_tokens6from lib.ngrams import START, KneserNey78model = KneserNey(name_tokens("train"), 4)9tokenizer = bpe.store_tokenizer()10candidates = sorted(model.vocabulary) # sorted first, so tokens with equal probability keep alphabetical order1112for typed in ["bar", "patio bar", "outdoor bar"]:13 # padded with start markers, then the last three tokens: a short prefix still gives a full context14 context = tuple([START] * 3 + tokenizer.encode(typed))[-3:]15 ranked = sorted(candidates, key=lambda t: -model.probability(context, t))[:3] # negated, so the likeliest sort first16 print(f"{typed:<12} -> " + ", ".join(f"{t.replace('</w>', '')} {model.probability(context, t):.2f}" for t in ranked))17# each name as one string of tokens, so the check below matches whole tokens rather than parts of them18names = [" ".join(t) for t in name_tokens("train")]19print("training names containing 'outdoor bar':", sum("outdoor</w> bar</w>" in n for n in names))
python ngram-autocomplete.pybar -> & 0.23, unit 0.13, with 0.11
patio bar -> stool 0.86, & 0.03, with 0.02
outdoor bar -> & 0.23, with 0.19, stool 0.06
training names containing 'outdoor bar': 0
仅在 bar 之后,模型给出的续写有 & 和 with 等,它们属于开头与 bar & counter stools 相似的名称;第 1 章中不考虑单词位置的二元模型,则首先建议 stool。在 patio bar 之后,模型以很高的概率建议 stool:上下文改变了预测结果,这是二元模型做不到的。然而在 outdoor bar 之后,建议又漂回 & 和 with,原因是训练数据中没有任何名称包含 outdoor bar,如示例的最后一行所示,于是模型回退到 bar 之后出现的内容。
诸如 & 和 with 之类的建议确实是店内名称的真实续写,但购物者很少希望它们被列出。搜索框会将它们过滤掉,正如第 1 章对标点所建议的那样,或者改为补全整个短语,而不是单个词元。这属于呈现方式的选择;两种做法下模型的概率是相同的。
outdoor bar 这一行体现了计数方法的局限,虽然只是一个缩影。模型已多次见过 patio bar stool、outdoor patio 和 bar stool,因此具备推测户外吧台通常配有凳子所需的素材。但 outdoor bar stool 的计数是表中独立的一项,没有任何东西把它与 patio bar stool 联系起来。模型需要某种概念,知道 outdoor 与 patio 相似,而计数并不具备这种概念。第 4 章将为店内的词元提供这一概念。
**工作准则:**为建议功能使用模型所能支持的最长上下文,并把训练中罕见的上下文(例如 outdoor bar)与常见上下文分开检查建议:计数模型正是在这类上下文中退回到泛泛的猜测。
这部分尚未翻译,因此以英文显示。
The baseline every later model is measured against
Every chapter from here on builds a model that learns, not one that counts, and each claims to predict the store's text better. A claim like that needs a baseline, a fixed yardstick: one model, one text, one tokenizer and one measure that every later model is compared against. A baseline should be the strongest simple model available, so that beating it means something.
The baseline for this book is the 4-gram Kneser-Ney model of this chapter, scored on the store's validation names with the store's tokenizer. Its validation perplexity is 23.1 per token, from the Kneser-Ney example. The 5-gram model is worse, at 23.5, and holds more counts, so the 4-gram is the baseline: the best of this chapter's counted models on the store's names. A stronger counted baseline exists, modified Kneser-Ney, and the book doesn't use it, for the reason the Kneser-Ney section gives.
The baseline also sets expectations. A perplexity near twenty means the model is, on average, choosing among about twenty likely tokens at each step of a product name. Some steps are nearly certain, such as table after coffee, and some are wide open, such as the first token of a product line's name. A neural model that beats the baseline should do so by predicting the open steps better, because it can relate one context to another, not by memorizing more.
Two things about the baseline will stay fixed for the rest of the book. Every perplexity is per token of the store's tokenizer, on the validation names, so the numbers in every chapter can be set side by side. And the test names stay untouched until the book's final comparison, so nothing in the book is tuned on them.
Working rule: Before building a learned model, build the strongest counted model you can and record its held-out perplexity as the baseline. Report every later model against it, on the same text and with the same tokenizer, and treat a learned model that doesn't beat it as a result to explain, never as a success. Later chapters find exactly that, and earn the learned models their place another way.
生产中会出什么问题
在速度和透明度比最后一点困惑度更重要的场合,例如快速自动补全,N 元语言模型仍然有用。它们会带来以下问题:
- 表会不断变大。 每增加一阶,就多一张计数表,而不同 n 元组的数量几乎与文本同步增长。可以剪除只出现过一次的 n 元组,并在剪枝后度量验证集困惑度,因为一些罕见的 n 元组携带着真实的信息。
- 罕见的上下文只能得到泛泛的答案。 正如
outdoor bar所示,模型很少见到的上下文会退回到较短上下文所给出的建议。除了常见前缀,也要监测罕见前缀的建议。 - 折扣是一种选择。 0.75 的折扣是合理的默认值,而不是定律。如果你的文本与商店的文本差异很大,就在留出文本上对它进行调优。
- 新文本会改变计数。 产品线来来去去,每条建议背后的计数只与上一次重建时一样新。应按计划定期重建,并将近期文本上的验证集困惑度与基线比较,以判断模型何时已经过时。
- 概率之和必须为一。 手写的回退方案常常丢失或重复计算概率,使困惑度看起来比实际更好或更差。请在若干上下文上检查,每个上下文在整个词表上的概率之和是否为一。
本章的决定
- 全书中的每个模型,都通过商店的分词器,按每个词元的困惑度,在商店的验证集名称上评分。 同一份文本、同一个分词器和同一种度量,使各章在名称上的数字可以相互比较。测试集名称留到全书最后的比较中使用,见第 11 章。
- N 元语法模型采用插值 Kneser-Ney 平滑,折扣为 0.75。 在相同阶数下,它优于手工加权的插值,而且无需调节权重。这是因为它按各个上下文的计数比例来信任该上下文,并回退到跟在许多上下文之后的词元。
- 基线是 4 元 Kneser-Ney 模型, 它在验证集名称上的得分为每个词元 23.1。5 元模型得分更差,且保存的计数更多,因此以 4 元模型作为衡量标准。
- 自动补全使用 4 元模型给出的建议,上下文为三个词元。 它能修正
patio bar,但对于outdoor bar这类罕见的上下文,仍会退回到通用的建议,而这正是学习得到的模型必须做得更好的地方。 - 更长的上下文不再有帮助,是因为它们没有出现过,而不是因为它们不重要。 验证集名称的 5 元组中,有一半以上从未在训练集中出现,而模型无法把未见过的上下文与见过的相似上下文联系起来。第 4 章从这个问题入手,为词元赋予向量,以表明哪些词元彼此相似。
本章,在本版次中
版次 2026.11.16示例于 2026年11月19日 通过本章 7个示例 个示例中有 7 个通过
译自英文版图书,版次 2026.11.16。