Counting further back

閱讀約需 43 分鐘 分鐘,9 節 中共有 7 個範例。

版次 2026.11.16 範例於 2026年11 月 19 通過測試 86 個範例中有 86 個通過 變更內容

由 AI 翻譯。經人工以英文驗證。 我們如何製作書籍

譯自英文書,版次 2026.11.16。 閱讀原文

此部分尚未翻譯,因此以英文呈現。

By the end of this chapter you can build an n-gram language model that looks several tokens back, smooth it with Kneser-Ney so it handles text it hasn't seen (smoothing gives some probability to what the counts never saw, as chapter 1 showed), measure the baseline that every neural model in this book is measured against, and see why counting, however clever, runs out of road.

Chapter 1 built a model that predicted the next word from the one before it, and two things went wrong. It looked back only one word, so a shopper who typed patio bar got the suggestions for bar. And it gave probability zero to every pair of words it hadn't counted, which made its perplexity on new text infinite. Chapter 2 then gave the book better units to count: the store's tokenizer, which splits every word into pieces it knows, so almost every word is spelled from tokens it has seen. This chapter goes back to counting with those tokens, looks further back, and handles the unseen properly.

Every method here is tried where the store, an online furniture and home-goods shop, needs it: on the product names its search box completes. From this chapter on, every model in the book is scored the same way: by its perplexity per token on the store's validation names, tokenized with the store's tokenizer. Product names are what the search box's autocomplete has to predict, and they're short enough for every example to run in seconds. One fixed text, one tokenizer and one measure make every model in the book comparable with every other on names; later chapters add descriptions, from chapter 8, and per-word scores, in chapter 10, where a model's tokenizer differs.

較長的上下文更貼合訓練文本

最近一次變動於第 2026.11.16 版

n 元語法模型(n-gram model)以前面 n − 1 個詞元來預測每一個詞元:二元語法(bigram)模型(n = 2)根據一個詞元,三元語法模型(n = 3)根據兩個詞元,依此類推。位於被預測詞元之前的那些詞元稱為它的上下文(context)。上下文越多,預測理應越準:在 patio bar 之後,下一個詞元遠比只在 bar 之後更容易預測。計數的方式與第 1 章完全相同,差別在於現在計數的對象是由 n − 1 個詞元構成的上下文,以及緊接其後的一個詞元;而且每段文本都會補上 n − 1 個起始標記,因此連它的第一個詞元也有完整的上下文。

在不做平滑處理的情況下,以模型當初計數所用的那些名字來評分,各個階數呈現出模型面對已經記住的文本時會有什麼表現:

ngram-memorizes.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Counts the n-grams of orders 1 to 5 in the training names, and scores each order on those same names with no smoothing.Read the training perplexity lines, which fall as the order grows, against the seen-once lines, which rise."""import mathfrom collections import Counterfrom lib.names import name_tokensfrom lib.ngrams import ngrams, paddedtrain = name_tokens("train")for n in [1, 2, 3, 4, 5]:    # padded adds n-1 start markers and one end marker, so a name's first token has a full context    pairs = Counter(pair for text in train for pair in ngrams(padded(text, n), n))    contexts = Counter()    for (context, _), count in pairs.items():        contexts[context] += count    # the total surprise, in nats: each n-gram's probability is its count over its context's count, the model's own estimate    surprise = -sum(count * math.log(count / contexts[context]) for (context, _), count in pairs.items())    print(f"{n}-gram training perplexity: {math.exp(surprise / sum(pairs.values())):.1f}")    print(f"{n}-grams seen once: {sum(1 for c in pairs.values() if c == 1) / len(pairs):.0%}")
輸出python ngram-memorizes.py
1-gram training perplexity: 676.4
1-grams seen once: 8%
2-gram training perplexity: 23.9
2-grams seen once: 60%
3-gram training perplexity: 4.9
3-grams seen once: 76%
4-gram training perplexity: 3.2
4-grams seen once: 82%
5-gram training perplexity: 2.9
5-grams seen once: 85%

隨著上下文變長,困惑度(perplexity)迅速下降:完全沒有上下文時是 676.4,有一個詞元的上下文時是 23.9,有四個詞元時是 2.9。在四個詞元的上下文下,模型在每一步大約只是在三個詞元之間做選擇,因為這麼長的 n 元語法大多只出現過一次:訓練名字中有 85% 的 5 元語法只出現過一次,而模型已經記住了每一個之後接的是什麼。

這就是陷阱所在。在計數所用的文本上,較長的上下文看起來永遠比較好,因為只要上下文夠多,每段文本都是獨一無二的,模型便能把它背誦出來。以模型自己的訓練文本來評分,衡量的是記憶,這正是第 1 章一開始就要留出驗證文本的原因。

實用原則: 切勿以 n 元語法模型(或任何語言模型)在自身訓練文本上的困惑度來評斷它:隨著模型把文本記下來,困惑度會趨近 1。請依據保留文本(held-out text)上的困惑度來選擇階數,本章其餘部分也是這樣做的。

越長的上下文,證據越不足

最近一次變動於第 2026.11.11 版

記住訓練用的名稱,並不能說明新名稱會如何。模型只能使用它計數過的上下文,而可能的上下文數量會隨長度呈指數成長:若有四千個詞元,就有一千六百萬種可能的二元組,以及六百四十億種可能的三元組。沒有任何訓練文本能涵蓋這麼多組合。因此上下文越長,新文本用到模型從未見過之上下文的次數就越多。

先確定一個數字:以 4-gram 而言,驗證用名稱的 4-gram 中,有多少比例從未出現在訓練用名稱裡?以下列出從 1 階到 5 階各階的比例:

unseen-ngrams.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Counts, for each order from 1 to 5, the share of the validation names' n-grams that never occur in the training names.Compare the lines for 2-grams to 5-grams: that share rises with the order."""from lib.names import name_tokensfrom lib.ngrams import ngrams, paddedtrain, validation = name_tokens("train"), name_tokens("validation")print("training tokens:", f"{sum(len(t) for t in train):,}")print("validation tokens:", f"{sum(len(t) for t in validation):,}")for n in [1, 2, 3, 4, 5]:    seen = {pair for text in train for pair in ngrams(padded(text, n), n)}     # padded, as in the model, so name starts count too    held_out = [pair for text in validation for pair in ngrams(padded(text, n), n)]    unseen = sum(pair not in seen for pair in held_out)    print(f"{n}-grams in validation never seen in training: {unseen / len(held_out):.1%}")
輸出python unseen-ngrams.py
training tokens: 337,071
validation tokens: 42,422
1-grams in validation never seen in training: 0.1%
2-grams in validation never seen in training: 12.7%
3-grams in validation never seen in training: 31.1%
4-grams in validation never seen in training: 43.9%
5-grams in validation never seen in training: 53.4%

上下文越長,驗證用名稱中屬於新出現的 n-gram 就越多。只有 0.1% 的驗證詞元未曾見過,這是商店的斷詞器發揮了作用。在驗證用的二元組中,有 12.7% 未曾見過,4-gram 為 43.9%,5-gram 為 53.4%。當上下文有四個詞元時,模型要預測所需的內容,有超過一半是它從未見過的。

增加訓練文本的幫助,比你期望的要小。在語言中,少數 n-gram 出現得非常頻繁,多數則很少出現,所以文本每增加一倍,新增的多半是只出現一兩次的 n-gram,而這對改善任何單一預測背後的計數幾乎沒有幫助。商店的商品目錄以縮小的規模呈現了相同的模式:像 coffee table 這樣的常見詞組被計數超過一千次,而訓練用名稱的 4-gram 中有 82% 只被計數一次。我們的判斷(並非實際測量)是,即使目錄大十倍,仍會有很大比例的新 4-gram 未曾見過,因為新的產品系列,以及尺寸與材質的新組合會不斷出現。

這兩個範例合起來,就是計數方法的核心問題。上下文越長,在見過的情況下預測越準,但被見過的機會卻越來越少。本章其餘的每一種方法,都是在計數足以支持時使用長上下文,在計數不足時退回到較短的上下文。

實用原則: 在選擇 n-gram 的階數之前,先測量保留資料中的 n-gram 有多少比例是訓練文本從未包含的。依經驗法則,當這個比例接近三分之一時(如商店名稱中的三元組),該階數的模型主要取決於它如何處理未見過的情況,因此其平滑方法比階數本身更重要。

混合短語境與長語境

最近一次變動於第 2026.11.16 版

對於未見過的長語境,最直接的補救方式是退而求助較短的語境。內插(interpolation)以平滑的方式退讓:一個詞元的機率是三元、二元與一元計數所給出預測的加權混合,因此模型見過的長語境能貢獻其精確的預測,較短的語境則補足它未曾見過的部分。設權重 λ₃、λ₂ 與 λ₁ 總和為一:

P(token | a b) = λ₃ · P₃(token | a b) + λ₂ · P₂(token | b) + λ₁ · P₁(token)

其中每個 P 都是依計數得到的最大概似估計,如同第 1 章的機率。Chen 與 Goodman 對平滑方法的研究詳細討論了內插及其變體(Chen and Goodman, 1996)。

以手動設定的權重可以讓這個概念更清楚。取一個內插三元模型,三種階數的權重分別為 0.6、0.3 與 0.09,最後的 0.01 平均分給每個詞元,使任何詞元的機率都不為零。當某個語境從未出現過,它的權重會移交給較低的階數,因此機率總和仍為一:

interpolate-orders.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Builds an interpolated trigram model by hand, with fixed weights, and an add-one bigram model on the same tokens.The two validation perplexity lines to compare are the interpolated trigram's and the add-one bigram's."""import mathfrom collections import Counterfrom lib.names import name_tokensfrom lib.ngrams import ngrams, padded, perplexitytrain, validation = name_tokens("train"), name_tokens("validation")counts = {k: Counter() for k in (1, 2, 3)}             # counts[k][(the k-1 tokens before, token)]for text in train:    for context, token in ngrams(padded(text, 3), 3):        for k in (1, 2, 3):            # a trigram's context holds two tokens; an order-k model conditions on the last k-1 of them            counts[k][(context[3 - k:], token)] += 1totals = {k: Counter() for k in (1, 2, 3)}for k in (1, 2, 3):    for (context, _), count in counts[k].items():        totals[k][context] += count# the tokens the training names contain, plus one slot for a token they never containvocabulary = len({token for (_, token) in counts[1]}) + 1weights = {3: 0.6, 2: 0.3, 1: 0.09}                   # and 0.01 spread evenly over every tokenclass Interpolated:    n = 3    def probability(self, context, token):        p, carried = 0.0, 0.0        for k in (3, 2, 1):            # each order's own context            weight, total = weights[k] + carried, totals[k][context[3 - k:]]            if total:                                    # an unseen context passes its weight down                p, carried = p + weight * counts[k][(context[3 - k:], token)] / total, 0.0            else:                carried = weight        # the even 0.01 share, plus any weight passed down from unseen contexts, over the vocabulary        return p + (0.01 + carried) / vocabularyprint("interpolated 3-gram validation perplexity:", round(perplexity(Interpolated(), validation), 1))# a bigram's context is the one token before the tokenbigram_pairs = Counter((context[-1:], token) for text in train for context, token in ngrams(padded(text, 2), 2))bigram_totals = Counter()for (context, _), count in bigram_pairs.items():    bigram_totals[context] += countclass AddOne:    n = 2    def probability(self, context, token):        # add-one: one added to every pair's count, and one per vocabulary token added to the total        return (bigram_pairs[(context, token)] + 1) / (bigram_totals[context] + vocabulary)print("add-one 2-gram validation perplexity:", round(perplexity(AddOne(), validation), 1))
輸出python interpolate-orders.py
interpolated 3-gram validation perplexity: 31.4
add-one 2-gram validation perplexity: 218.6

混合不同階數的效果勝過第 1 章的加一平滑。為了公平比較,這裡的加一平滑是在同一批驗證詞元上以二元模型執行,因為不同詞元上的困惑度無法互相比較。內插三元模型的得分為 31.4,加一平滑的得分為 218.6。加一平滑把大部分機率分給了從未出現的詞對;混合階數則保留了每個階數所掌握的資訊。

它有兩個弱點。其一,權重是固定的,而且這些權重是手動挑選、並未調校:無論語境只出現一次或一千次,三元模型都拿到同樣的 0.6,然而出現一千次的語境理應得到更多信任。其二,一元的部分依詞元出現的頻率比例給予機率,而如下一節所示,這對於作為退讓的來源而言,問的是錯誤的問題。

實用原則: 切勿單獨使用單一階數的 n 元模型;一律將它與較低的階數結合,使每個詞元都保有一些機率。比起固定權重,應優先採用依計數決定各語境權重的方法,例如下文的 Kneser-Ney。若確實使用固定權重,應在保留的文本上選定,而非在用來計數的文本上選定。

計算詞元所接續的上下文數量

最近一次變動於第 2026.11.16 版

當模型退回到較短的上下文時,它問的是一個特定的問題:這個詞元在我們沒見過的上下文中出現的可能性有多大?原始頻率回答的是另一個問題。一個詞元可以非常常見,卻只出現在一兩個特定詞元之後,這樣的詞元對新上下文而言是很差的猜測。較好的衡量方式是詞元的接續計數(continuation count),也就是它所接續的不同詞元的數量:一個曾經接在許多不同詞元之後的詞元,很可能也會接在一個新的詞元之後。

在訓練名稱中最常出現的 300 個詞元裡,以下三個詞元就其出現次數而言,所接在其後的不同詞元最少;表中並列了兩個一般的商品用詞作為對照:

continuation-counts.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Counts, for each token in the training names, how often it occurs and how many different tokens come right before it.Look at the three narrowest tokens, which occur often but follow few different tokens, beside table and chair."""from collections import Counter, defaultdictfrom lib.names import name_tokensoccurrences, before = Counter(), defaultdict(set)for text in name_tokens("train"):    for previous, token in zip(["<s>"] + text, text):      # the start marker stands before each name's first token, so that token has something to follow too        occurrences[token] += 1        before[token].add(previous)                 # the different tokens this one followscommon = [t for t, _ in occurrences.most_common(300)]     # the 300 most frequent tokens: the pool the prose ranks# different tokens followed, per occurrence, lowest first: the three narrowestnarrow = sorted(common, key=lambda t: len(before[t]) / occurrences[t])[:3]for token in narrow + ["table</w>", "chair</w>"]:       # </w> marks a token that ends a word, as the store's tokenizer writes it    print(f"{token:<14} occurs {occurrences[token]:>5,} times, after {len(before[token]):>4,} different tokens")
輸出python continuation-counts.py
tvs</w>        occurs   323 times, after    1 different tokens
wide</w>       occurs 1,985 times, after   14 different tokens
''</w>         occurs 10,335 times, after   85 different tokens
table</w>      occurs 3,487 times, after  154 different tokens
chair</w>      occurs 2,791 times, after  138 different tokens

詞元 tvs 出現了數百次,卻只接在另一個詞元之後:在這間商店的名稱中,它永遠是同一個片語的一部分,即專為電視設計的商品。若模型退回到以原始頻率判斷,會認為 tvs 在任何地方都是合理的猜測,但實際上幾乎從來不是。wide 只接在少數幾個詞元之後,多半是數字的英寸符號,而英寸符號本身 '' 則多半接在數字之後。相對地,table 與 chair 各自接在一百多個不同的詞元之後:coffee table、side table、outdoor table。它們的接續計數說明了頻率所沒有說明的事:它們出現在一百多個不同詞元之後。

接續計數的計算成本很低:對每個詞元,記錄在它之前出現過的詞元集合,然後計算集合的大小。這不需要額外的文字,也不需要調參,只是對同樣的計數提出不同的問題,而下一節就建立在這些計數之上。

這正是 Kneser-Ney 平滑法(Kneser and Ney, 1995)背後的洞見:當模型退回到較短的上下文時,使用接續計數而不是原始計數,讓出現在許多上下文中的詞元獲得機率,而只出現在單一固定片語中的詞元則不會。

實用原則: 每當從長上下文退回到短上下文時,依詞元所接續的不同上下文數量來加權,而不是依其出現頻率。只有在最高階、也就是上下文確實出現過的情況下,才使用原始計數。

此部分尚未翻譯,因此以英文呈現。

Smooth with Kneser-Ney

最近一次變動於第 2026.11.16 版

Kneser-Ney smoothing combines interpolation of orders with continuation counts. At each order, Kneser-Ney subtracts a fixed discount d from every count it has seen, and gives the probability it took to the next order down, so a context seen often keeps nearly all its own prediction, and a context seen rarely leans on the shorter context, with the discount it frees going to tokens that turn up in many contexts.

The same formula applies at every order. Below the highest, the counts are continuation counts, except for n-grams that start a text, which keep their raw counts, as the KenLM toolkit does. For a context h and a token w, with c(h w) the count of the pair, c(h) the count of the context, and N(h) the number of different tokens seen after h:

P(w | h) = max(c(h w) − d, 0) / c(h) + d · N(h) / c(h) · P(w | shorter h)

The book's shared module lib.ngrams implements interpolated Kneser-Ney as Chen and Goodman describe it (Chen and Goodman, 1999), with a discount of 0.75, a common default. Its lowest order mixes in a uniform share over the vocabulary, so even a token never seen in training keeps a small probability.

Before trusting the module, work out one probability by hand: stool after a name that starts patio bar, order by order. Set it beside the module's answer, and beside the module's probabilities after this context, summed over every token:

kneser-ney-by-hand.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Works out the probability of stool after a name that starts patio bar, order by order, with the chapter's formula.It checks the result against lib.ngrams and against the module's probabilities summed over the vocabulary: theby hand line should match the lib.ngrams line, and the summed line should be a probability distribution's total."""from lib import bpefrom lib.names import name_tokensfrom lib.ngrams import START, KneserNeymodel = KneserNey(name_tokens("train"), 4)     # a 4-gram model; below the top order its counts are continuation counts, except for n-grams that start a namecontext = tuple([START] + bpe.store_tokenizer().encode("patio bar"))     # a name that starts "patio bar"token, d = "stool</w>", model.d                # </w> ends a word; d is the discount, 0.75 by defaultdef by_hand(context, k):                         # the chapter's formula, one order at a time    if k == 0:        return 1 / (len(model.vocabulary) + 1)   # order 0 is uniform over the vocabulary, with one more slot for an unseen token    context = context[len(context) - (k - 1):] if k > 1 else ()     # the last k-1 tokens: the order-k model's own context    following, total = model.counts[k][context], model.totals[k].get(context, 0)    if total == 0:        return by_hand(context, k - 1)           # a context never seen gives all its weight to the order below    kept = max(following[token] - d, 0) / total                  # the discounted count    passed_down = d * len(following) / total                     # what the discount frees for the order below    print(f"order {k}: kept {kept:.4f}, passed down {passed_down:.4f}")    return kept + passed_down * by_hand(context, k - 1)print("by hand:", f"{by_hand(context, 4):.4f}")print("lib.ngrams:", f"{model.probability(context, token):.4f}")total = sum(model.probability(context, t) for t in model.vocabulary)print("probabilities after the context, summed:", f"{total:.4f}")
輸出python kneser-ney-by-hand.py
order 4: kept 0.2500, passed down 0.7500
order 3: kept 0.8056, passed down 0.1667
order 2: kept 0.0630, passed down 0.1275
order 1: kept 0.0007, passed down 0.0318
by hand: 0.8621
lib.ngrams: 0.8621
probabilities after the context, summed: 1.0000

Worked by hand, the formula gives 0.8621 for stool, the same as the module's 0.8621. Each order keeps a share and passes the rest down; the shares add up to that answer. Over every token in the vocabulary, the module's probabilities after this context sum to 1.0000, as a probability distribution must.

Kneser-Ney models of every order from 1 to 5, scored on the validation names, show whether longer contexts now pay off:

kneser-ney-by-order.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Builds Kneser-Ney models of orders 1 to 5 on the training names, scores each on the validation names, and estimates adiscount from the 4-gram counts. The lines to read are the validation perplexities, the 3-gram's set against the interpolated trigram's, and the discount line."""from lib.names import name_tokensfrom lib.ngrams import KneserNey, perplexitytrain, validation = name_tokens("train"), name_tokens("validation")for n in [1, 2, 3, 4, 5]:    model = KneserNey(train, n)    print(f"{n}-gram validation perplexity: {perplexity(model, validation):.1f}")model = KneserNey(train, 4)# the highest order keeps raw counts, not continuation counts, so these are how often each 4-gram occurs in trainingtop = [c for following in model.counts[4].values() for c in following.values()]once, twice = top.count(1), top.count(2)print("4-grams seen once:", f"{once:,}")print("4-grams seen twice:", f"{twice:,}")# Chen and Goodman's estimate of the discount: D = n1 / (n1 + 2 n2), with n1 and n2 the counts seen once and twiceprint("discount estimated from them:", f"{once / (once + 2 * twice):.2f}")
輸出python kneser-ney-by-order.py
1-gram validation perplexity: 687.2
2-gram validation perplexity: 46.9
3-gram validation perplexity: 24.9
4-gram validation perplexity: 23.1
5-gram validation perplexity: 23.5
4-grams seen once: 164,374
4-grams seen twice: 18,359
discount estimated from them: 0.82

Kneser-Ney beats the hand-weighted interpolated trigram from the interpolation section at the same order, with no weights to choose. Its trigram scores 24.9, against 31.4 for the hand-weighted one.

Longer contexts now help on new text, instead of only on the text the model counted, up to a point: validation perplexity falls from 46.9 for the bigram to 23.1 for the 4-gram, then rises to 23.5 for the 5-gram. The step from trigram to 4-gram is small, and the 5-gram is worse, because by then most contexts are unseen and the extra order adds noise. The discount, too, can be estimated from the counts: Chen and Goodman give D = n₁ / (n₁ + 2 n₂), where n₁ and n₂ are the numbers of n-grams seen once and twice, and on the store's 4-grams it's 0.82, close to the default.

Working rule: Smooth n-gram models with interpolated Kneser-Ney and a discount of 0.75 unless held-out text says otherwise. Raise the order only while validation perplexity keeps falling noticeably, and stop when a step barely helps or hurts, as the step from 4-gram to 5-gram does. A variant, modified Kneser-Ney, uses three discounts instead of one and is the one toolkits such as KenLM build; it would likely lower the baseline a little, and this book keeps the single discount because it's easier to follow by hand.

以三個詞元的上下文進行自動完成

最近一次變動於第 2026.11.11 版

較長的上下文應能改善第 1 章中搜尋框最糟的建議:當時 patio bar 得到的建議與 bar 相同。4-gram Kneser-Ney 模型能看到購物者輸入的最後三個詞元,因此 patio 仍在其視野內。

從 bar 到 patio bar 再到 outdoor bar,建議是否有所改變?取 4-gram 模型在每個前綴之後最可能的三個下一個詞元。每個前綴都被視為名稱的開頭,就如同購物者的輸入方式,而每個建議顯示的詞元都不含字尾標記:

ngram-autocomplete.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Asks the 4-gram model for its three likeliest next tokens after bar, patio bar and outdoor bar, each typed as the startof a name, then counts the training names that contain outdoor bar. The three suggestion lines carry the point, and the last linecounts the training names with outdoor bar in them."""from lib import bpefrom lib.names import name_tokensfrom lib.ngrams import START, KneserNeymodel = KneserNey(name_tokens("train"), 4)tokenizer = bpe.store_tokenizer()candidates = sorted(model.vocabulary)       # sorted first, so tokens with equal probability keep alphabetical orderfor typed in ["bar", "patio bar", "outdoor bar"]:    # padded with start markers, then the last three tokens: a short prefix still gives a full context    context = tuple([START] * 3 + tokenizer.encode(typed))[-3:]    ranked = sorted(candidates, key=lambda t: -model.probability(context, t))[:3]     # negated, so the likeliest sort first    print(f"{typed:<12} -> " + ", ".join(f"{t.replace('</w>', '')} {model.probability(context, t):.2f}" for t in ranked))# each name as one string of tokens, so the check below matches whole tokens rather than parts of themnames = [" ".join(t) for t in name_tokens("train")]print("training names containing 'outdoor bar':", sum("outdoor</w> bar</w>" in n for n in names))
輸出python ngram-autocomplete.py
bar          -> & 0.23, unit 0.13, with 0.11
patio bar    -> stool 0.86, & 0.03, with 0.02
outdoor bar  -> & 0.23, with 0.19, stool 0.06
training names containing 'outdoor bar': 0

單獨在 bar 之後,模型建議的接續詞包括 & 與 with,它們屬於開頭類似 bar & counter stools 的名稱;第 1 章的 bigram 不考慮該字出現的位置,最先建議的是 stool。在 patio bar 之後,模型以高機率建議 stool:上下文改變了預測結果,這是 bigram 模型做不到的。然而在 outdoor bar 之後,建議又漂回 & 與 with,原因是訓練資料中沒有任何名稱包含 outdoor bar(如範例最後一行所示),模型便退回到 bar 之後會出現的內容。

像 & 與 with 這類建議確實是商店名稱的真實接續,但購物者很少希望看到它們被提供出來。搜尋框會將它們過濾掉,如同第 1 章對標點符號的建議,或者改為完成整個詞組,而非單一詞元。這屬於呈現方式的選擇;兩種做法下模型的機率相同。

outdoor bar 這一行是計數方法極限的縮影。模型已多次見過 patio bar stool、outdoor patio 與 bar stool,因此具備推測戶外吧檯大概會搭配高腳椅的素材。但 outdoor bar stool 的計數在表中是獨立的一筆項目,沒有任何東西將它與 patio bar stool 連結起來。模型需要某種概念,知道 outdoor 與 patio 是相近的,而計數並不具備這種概念。第 4 章會為商店的詞元提供這種概念。

**實用原則:**建議時請使用模型所能支援的最長上下文,並將訓練中罕見的上下文(例如 outdoor bar)與常見上下文分開檢查:計數式模型正是在這種情況下退回到泛用的猜測。

所有後續模型的比較基準

最近一次變動於第 2026.11.16 版

從這裡開始的每一章,建立的都是會學習的模型,而不是只會計數的模型,而且每個模型都宣稱能更準確地預測商店的文字。這樣的宣稱需要一個基準(baseline),也就是一把固定的尺:同一個模型、同一份文字、同一種分詞器與同一個衡量指標,所有後續模型都與之比較。基準應當是手邊最強的簡單模型,如此超越它才有意義。

本書的基準是本章的 4-gram Kneser-Ney 模型,以商店的驗證名稱和商店的分詞器計分。它的驗證困惑度為每個詞元 23.1,數值出自 Kneser-Ney 範例。5-gram 模型的表現較差,為 23.5,而且保存了更多計數,所以 4-gram 是基準:它是本章各個計數模型中,在商店名稱上表現最好的。更強的計數基準確實存在,即修正版 Kneser-Ney,但本書不採用,理由見 Kneser-Ney 節。

基準也設定了預期。困惑度接近二十,表示模型在商品名稱的每一步,平均是在約二十個可能的詞元之間做選擇。有些步驟幾乎可以確定,例如 table 之後接 coffee;有些步驟則完全開放,例如某個產品線名稱的第一個詞元。若神經模型超越了基準,應當是因為它能把一種上下文與另一種上下文相互關聯,因而更準確地預測那些開放的步驟,而不是因為記住了更多內容。

基準有兩點在本書其餘部分都保持不變。其一,每一個困惑度都是以商店分詞器的詞元為單位、在驗證名稱上計算的,因此各章的數字可以並排比較。其二,測試名稱在本書最後的比較之前完全不動,所以書中沒有任何東西是依據它們調整的。

**實用原則:**在建立學習式模型之前,先建立你所能做到最強的計數模型,並把它在保留資料上的困惑度記錄為基準。之後的每個模型都要在相同的文字、相同的分詞器上與它比較;學習式模型若沒有勝過基準,應視為有待解釋的結果,絕不能當作成功。後續各章正好發現了這種情形,並以另一種方式為學習式模型贏得其地位。

正式環境中會出什麼問題

在速度與透明度比最後一點困惑度(perplexity)更重要的場合,例如快速自動完成,n-gram 模型仍然有用。它們會帶來下列問題:

  • 表格會變大。 每增加一個階數,就多一張計數表,而不同 n-gram 的數量幾乎與文字量同步成長。可以剪除只出現過一次的 n-gram,並在剪除後量測驗證集的困惑度,因為有些罕見的 n-gram 帶有實際的資訊。
  • 罕見的上下文只得到泛用的答案。 如 outdoor bar 所示,模型很少見過的上下文,會退回到較短上下文所建議的結果。除了常見的前綴之外,也要監看罕見前綴的建議。
  • 折扣值是一種選擇。 0.75 的折扣值是合理的預設值,而不是定律。如果你的文字與商店的文字差異很大,請以保留的驗證文字(held-out text)來調整它。
  • 新文字會改變計數。 產品線來來去去,每項建議背後的計數,只和最近一次重建時一樣新。請依排程重建,並將近期文字上的驗證困惑度與基準值比較,以判斷模型何時已經過時。
  • 機率總和必須為一。 手寫的退回(backoff)機制常常遺失或重複計算機率,使困惑度看起來比實際更好或更差。請在幾個上下文上檢查,每個上下文在整個詞彙表上的機率總和是否為一。

本章的決定

  • 本書中的每個模型,都以店家驗證用名稱上的每詞元困惑度(perplexity per token)評分,並透過店家的斷詞器(tokenizer)計算。 同一份文字、同一個斷詞器、同一種度量,使每一章在名稱上的數字都能互相比較;測試用名稱則保留到本書最後的比較,見第 11 章。
  • N-gram 模型採用內插式 Kneser-Ney 平滑法,折扣值為 0.75。 在相同階數下,它勝過手動加權的內插法,且不必調整任何權重,因為它依各個上下文的計數比例來信任該上下文,並回退到跟在許多上下文之後的詞元。
  • 基準線是 4-gram Kneser-Ney 模型, 在驗證用名稱上為每詞元 23.1。5-gram 模型的分數較差,且儲存的計數更多,因此以 4-gram 作為衡量標準。
  • 自動完成使用 4-gram 模型的建議,上下文為三個詞元。 它能修正 patio bar,但遇到罕見的上下文,例如 outdoor bar,仍會退回泛用的建議,這正是學習式模型必須做得更好的地方。
  • 較長的上下文不再有幫助,是因為它們未曾出現過,而不是因為它們不重要。 驗證用名稱的 5-gram 中,有超過一半從未出現在訓練資料裡,而模型無法把未見過的上下文與相似的已見上下文聯繫起來。第 4 章從這個問題著手,做法是賦予詞元向量,以表示哪些詞元彼此相似。

本章,於本版次中

版次 2026.11.16範例於 2026年11 月 19 通過測試本章 7 個範例 個範例中有 7 個通過

譯自英文書,版次 2026.11.16。

引用本節

格式

各種格式皆引用英文原書,本譯本即依循該原書。

每種格式都會標示本版。其連結會開啟目前版次,節標題會說明該節內文最後一次變動的時間。存取日期與擷取日期即為你開啟此頁的當日。

本頁內容

第 3 章,共 11 章: Counting further back

  1. 較長的上下文更貼合訓練文本
  2. 越長的上下文,證據越不足
  3. 混合短語境與長語境
  4. 計算詞元所接續的上下文數量
  5. Smooth with Kneser-Ney
  6. 以三個詞元的上下文進行自動完成
  7. 所有後續模型的比較基準
  8. 正式環境中會出什麼問題
  9. 本章的決定

返回頂端所有章節