神經語言模型

閱讀約需 43 分鐘 分鐘,11 節 中共有 10 個範例。

版次 2026.11.16 範例於 2026年11 月 19 通過測試 86 個範例中有 86 個通過 變更內容

由 AI 翻譯。經人工以英文驗證。 我們如何製作書籍

譯自英文書,版次 2026.11.16。 閱讀原文

讀完本章後,你可以建立一個神經語言模型,根據前面一個視窗內的詞元來預測下一個詞元,手動計算它的梯度並加以驗證,用 PyTorch 訓練它,並與計數式基準模型做公平的比較。你也會看到,學得的模型並不會因為是學得的就勝出:這個模型單獨使用時輸不過計數法,兩者混合時才有幫助。

第 3 章建立了本書的基準模型:一個 4-gram Kneser-Ney 模型,它以計數的方式,根據前面三個詞元來預測商店產品名稱中的每一個詞元。它的限制在於無法把一個上下文與另一個上下文關聯起來:在一家販售家具與家居用品的商店的名稱中,它有 patio bar 的計數,卻沒有 outdoor bar 的計數,兩者之間也沒有任何連結。第 4 章說明了向量可以表達詞元之間的關係,因為從商店的文字中學得向量時,patio 與 outdoor 的向量會很接近。本章把向量放進模型裡。神經語言模型(neural language model)會把前面的詞元讀成向量,用學得的運算將它們組合起來,並為詞彙表中的每個詞元產生一個機率。它的向量與運算是一起學得的,方法是訓練它預測下一個詞元。這個設計由 Bengio 等人提出(Bengio et al., 2003)。

文字由商店提供,測試方式則與第 3 章以來每個模型所面對的相同:透過商店的分詞器,計算驗證集名稱上每個詞元的困惑度。數值越低越好,基準模型的得分為 23.1。

從嵌入視窗預測下一個詞元

最近一次變動於第 2026.11.16 版

本章的模型會查看前面詞元所構成的固定視窗(window),此處為四個詞元,與 5-gram 所計數的上下文相同。每個詞元都會到嵌入表(embedding table)中查出對應的向量;嵌入表是一個矩陣,詞彙表中的每個詞元各佔一列數字,如第 4 章所述,差別在於這張表是為了預測下一個詞元而學得的,而不是由計數算出的。這四列向量會並排接成一個長向量 x。隱藏層(hidden layer)根據其輸入 x 計算 h = tanh(xW + b):先把 x 乘上權重矩陣 W,加上偏置向量 b,再把每個結果通過 tanh。tanh 是一個把任何數字壓縮到 −1 至 1 範圍內的函數,使這一層能以單靠加總做不到的方式組合其輸入。接著,輸出層(output layer)根據隱藏層的輸出 h 計算 z = hU + c,為詞彙表中的每個詞元各給一個分數,稱為 logits。

最後一步是把分數轉換成機率。Softmax 對每個 logit 取指數,再除以總和,因此結果皆為正數且總和為一,分數越高,機率越高:

P(token i) = exp(zᵢ) / Σⱼ exp(zⱼ)

以下用訓練好的權重走過一次預測,看模型如何把四個詞元變成機率:預測以 patio bar 開頭的名稱之後的下一個詞元。範例中的每一行都對應上述其中一個公式,所印出的形狀則顯示每個步驟做了什麼:

window-model-by-hand.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""One prediction of the trained window model, computed by hand from its saved weights: the embeddings, the tanh layer,the output scores and the softmax. The prose's point rests on the shapes, the parameter counts and the three likeliesttokens after `patio bar`."""import numpy as npfrom lib import bpe, neuralfrom lib.ngrams import STARTweights, vocabulary, manifest = neural.load()index, width = {t: i for i, t in enumerate(vocabulary)}, manifest["settings"]["context"]   # token -> row; the window's width, from the saved settingscontext = ([START] * width + bpe.store_tokenizer().encode("patio bar"))[-width:]   # pad with start markers, keep the last width tokensids = np.array([[index[t] for t in context]])vectors = weights["embedding"][ids]                                    # one 64-number vector per tokenjoined = vectors.reshape(1, -1)                                        # -1 lets numpy work out the length: 4 vectors of 64 make 256hidden = np.tanh(joined @ weights["hidden_w"] + weights["hidden_b"])   # one layer, squashed into -1 to 1logits = hidden @ weights["output_w"] + weights["output_b"]            # one score for every tokenprobabilities = np.exp(logits - logits.max())                          # subtract the largest score first: same probabilities, and exp can't overflowprobabilities /= probabilities.sum()                                   # softmax: scores into probabilitiesprint("context:", " ".join(context))for name, array in [("embeddings", vectors), ("joined", joined), ("hidden", hidden), ("logits", logits)]:    print(f"{name} shape:", array.shape)print("vocabulary size:", f"{len(vocabulary):,}")print("parameters:", f"{sum(w.size for w in weights.values()):,}")for part in ["embedding", "hidden_w", "output_w"]:    print(f"parameters in {part}:", f"{weights[part].size:,}")for i in np.argsort(-probabilities[0], kind="stable")[:3]:             # the three likeliest; a stable sort keeps ties in vocabulary order    print(f"P({vocabulary[i]} | patio bar): {probabilities[0, i]:.2f}")
輸出python window-model-by-hand.py
context: <s> <s> patio</w> bar</w>
embeddings shape: (1, 4, 64)
joined shape: (1, 256)
hidden shape: (1, 256)
logits shape: (1, 4140)
vocabulary size: 4,140
parameters: 1,394,732
parameters in embedding: 264,960
parameters in hidden_w: 65,536
parameters in output_w: 1,059,840
P(stool</w> | patio bar): 0.63
P(with</w> | patio bar): 0.23
P(&</w> | patio bar): 0.05

由於這個名稱目前只有兩個詞元,視窗以起始標記補齊。四個詞元各自變成一個含 64 個數字的向量;接起來成為一個含 256 個數字的向量;隱藏層再將它轉換為另外 256 個數字;輸出層則為模型所知的 4,140 個詞元各給一個分數,這些詞元即商店分詞器的詞彙表,再加上起始與結束標記。視窗模型共有 1,394,732 個參數,也就是它所學習的數字。其中約四分之三位於輸出層的權重 1,059,840,因為輸出層需要為每個詞元準備一欄,其餘大部分則在嵌入表中。它的預測相當合理:stool 的機率為 0.63,其次是 with 與 &,這些都能接續 patio bar with stools 之類的名稱。

**實用原則:**對於你所訓練的任何模型,至少要親手實作一次前向傳遞,並如本範例一般印出每個中間陣列的形狀:形狀錯誤是神經網路程式碼中常見的錯誤,而形狀正好說明了模型實際在計算什麼。詞彙表很大時,預期輸出層會佔去大部分的參數。

以反向傳播計算梯度

最近一次變動於第 2026.11.16 版

權重隨機的模型預測得很差;訓練就是調整權重,使預測變好。要衡量模型預測得有多差,使用損失(loss):L = −log P(下一個詞元),也就是模型賦予實際接續出現之詞元的對數機率取負值,這就是第 1 章中被平均後得到困惑度(perplexity)的那個驚訝程度。訓練的目標是降低平均損失,損失越低,困惑度也越低。要降低損失,每個權重都需要一個梯度(gradient):若該權重稍有改變,損失會隨之改變多少,記作 ∂L/∂w。

反向傳播(backpropagation)一次算出所有梯度,做法是把微積分的連鎖律沿著前向傳遞的相反方向逐步套用:從損失到 logits,從 logits 經過輸出層到隱藏層,穿過 tanh,再回到用過的嵌入向量(Rumelhart et al., 1986)。其中有兩個步驟值得牢記。損失對 logits 的梯度,是 softmax 機率在真實詞元處減去 1。而通過 tanh 的梯度要乘上 1 − h²,在 h 接近 ±1 之處這個值很小。本書共用的模組 lib.tinynet 以一個迷你版的商店模型,逐行完整寫出整個過程,該模型有 5 個詞元、2 個數值的嵌入向量、長度為 2 的視窗,以及 3 個隱藏單元。

反向傳播的程式碼常會出現不易察覺的錯誤,因此要用一個較慢卻不會出錯的方法來檢查:把某個權重往上、往下各微調一個極小的量,測量損失如何變動,再與反向傳播的結果比較。商店的模型需要超過一百萬次微調,所以我們對迷你模型的每個參數都做檢查:

gradients-by-hand.py Python 3.12.15 · numpy 2.5.5 通過
"""Backpropagation's gradients for the tiny model, checked against nudging every one of its parameters up and down.The assert is the check the prose relies on: the two agree to six decimal places, for every parameter."""import numpy as npfrom lib.tinynet import backprop, forward, parametersp = parameters()                                  # the tiny model: 5 tokens, a window of 2, 3 hidden units; seeded, so the same every runcontext, target = np.array([1, 3]), 4             # the window: token ids 1 and 3; the token that came next is id 4worst, step = 0.0, 1e-6                           # the nudge: small enough that the loss is nearly a straight line over itgradients = backprop(p, context, target)for name, value in p.items():                     # nudge each parameter both ways, and see how the loss moves    for i in np.ndindex(value.shape):        value[i] += step        up = forward(p, context, target)[3]       # forward returns (joined, hidden, logits, loss): index 3 is the loss        value[i] -= 2 * step                      # from up to down is two steps, so the estimate is centred on the value        down = forward(p, context, target)[3]        value[i] += step                          # back where it started        worst = max(worst, abs((up - down) / (2 * step) - gradients[name][i]))agrees = worst < 1e-6                             # the largest disagreement over all parameters, against six decimal placesassert agreesprint("loss:", round(float(forward(p, context, target)[3]), 4))print("parameters checked:", sum(v.size for v in p.values()))print("backprop agrees with nudging to 6 decimal places:", "yes" if agrees else "no")
輸出python gradients-by-hand.py
loss: 1.5293
parameters checked: 45
backprop agrees with nudging to 6 decimal places: yes

反向傳播得到的全部 45 個梯度,與微調估計值在小數點後六位皆相符,範例中以斷言加以驗證。微調法每個參數需要兩次前向傳遞,所以沒有人用它來訓練,但在迷你模型上,它是確認反向傳播正確的基準。

實用原則: 手寫的梯度,先在迷你模型上與微調估計值比對,再放心使用。若使用的是會自動計算梯度的函式庫,如 PyTorch 一節所示,則改為把函式庫的前向傳遞與自己寫的前向傳遞比對。

執行梯度下降步驟

最近一次變動於第 2026.11.16 版

有了梯度,訓練就是一個迴圈。梯度下降(gradient descent)會讓每個權重沿著其梯度的反方向走一小步,即 w ← w − η · ∂L/∂w,其中 η 稱為步長(step size),也稱為學習率(learning rate),決定每一步走多遠。每一步都會讓損失在其所計算的範例上略為變小;在許多範例上反覆進行後,權重便會穩定在模型能準確預測的位置。

以 0.5 的步長執行二十步梯度下降,使用 lib.tinynet 的梯度,在三筆範例上訓練上一節中商店的小型模型,可以看到損失逐步下降:

gradient-descent-by-hand.py Python 3.12.15 · numpy 2.5.5 通過
"""Twenty steps of gradient descent on three examples of the tiny model, printing the average loss before training andafter one, five and twenty steps. The prose's point rests on the average loss at those four points."""import numpy as npfrom lib.tinynet import backprop, forward, parametersp = parameters()                                             # seeded, so every run starts from the same weightsexamples = [(np.array([1, 3]), 4), (np.array([0, 1]), 3), (np.array([3, 4]), 2)]     # three (window, next token) pairs: each window is two token ids, then the id of the token after itrate = 0.5                                                   # the step sizedef average_loss():    return np.mean([forward(p, c, t)[3] for c, t in examples])print("average loss before:", f"{average_loss():.3f}")for step in range(1, 21):                                    # twenty steps, each one a pass over all three examples    for c, t in examples:        for name, gradient in backprop(p, c, t).items():            p[name] -= rate * gradient / len(examples)       # w <- w - rate * dL/dw, with each gradient averaged over the examples    if step in (1, 5, 20):        print(f"average loss after {step} steps:", f"{average_loss():.3f}")
輸出python gradient-descent-by-hand.py
average loss before: 1.568
average loss after 1 steps: 1.346
average loss after 5 steps: 0.646
average loss after 20 steps: 0.109

在範例印出的每一步,平均損失都在下降,從訓練前的 1.568 降到二十步後的 0.109。只有三筆範例可學,這個小型模型可以把損失壓到接近零,這其實是死記;商店的模型則是從數十萬筆範例中學習,同樣的迴圈必須找出在這些範例中普遍成立的規律。

實際訓練在這個迴圈上有兩點不同。實際訓練的每一步是在一個批次(batch)的範例上計算,一次幾百筆,而不是全部範例,因此每一步的成本很低,各步之間也會略有差異。此外,它採用改良過的更新方式,而不是固定步長,下一節會說明。

**實用原則:**開始訓練任何新模型時,先用少數幾筆範例跑幾步,確認損失確實下降:如果在三筆範例上損失都降不下來,代表程式有錯,而不是問題困難。步長可以多試幾個值來選擇:太小,損失下降得很慢;太大,損失會上下跳動甚至增加。

以 PyTorch 訓練

最近一次變動於第 2026.11.16 版

若每個模型都要手寫反向傳播,既耗時又容易出錯。PyTorch 是廣受使用的神經網路訓練函式庫,它能根據前向傳遞自動計算梯度,並且快速執行運算。本書用它來訓練模型,程式在 recordings/neural_lm.py 中,它建立的視窗模型與第一節相同:一張嵌入表、一層 tanh 層與一層輸出層,並以資料庫中的訓練名稱進行訓練。這個腳本的核心,是上一節的迴圈,以每批 512 筆預測來執行:

loss = loss_of(model(train_x[pick]), train_y[pick])   # forward pass and average loss on one batch
optimizer.zero_grad()                                  # clear the previous batch's gradients
loss.backward()                                        # backpropagation: a gradient for every weight
optimizer.step()                                       # move every weight against its gradient

訓練腳本使用的最佳化器是 Adam,它是梯度下降法的改良版(Kingma and Ba, 2015)。它為每個權重保存梯度的滾動平均,以及梯度平方的滾動平均,並以前者除以後者的平方根作為步幅。梯度一貫較大的權重,所得步幅與梯度較小的權重大致相同,因此同一個步長設定就適用於所有權重。

訓練仰賴機器的浮點數運算,這部分在第 4 章中說明,所以訓練只執行一次,本書的範例則讀取它儲存的權重,如同讀取所有此類結果一樣。這些範例從不匯入 PyTorch:lib.neural 以 NumPy 根據那些權重重新計算模型的前向傳遞。這樣做要成立,前提是 NumPy 版本算出的結果與 PyTorch 所算出的相同,因此訓練腳本儲存了 PyTorch 自己針對 64 個驗證情境所算出的 logits,下一個範例便拿來比對:

same-as-pytorch.py Python 3.12.15 · numpy 2.5.5 通過
"""The trained model's logits, recomputed in NumPy from its saved weights and compared with the logits PyTorch computedfor the same saved validation contexts. The prose's point rests on the agreement, which the last line prints."""import numpy as npfrom lib import neuralweights, _, manifest = neural.load()check = np.load("data/recorded/neural-lm/check.npz")                 # saved by the training script, so this runs without PyTorch: contexts, and PyTorch's logits for themours = neural.forward(weights, check["contexts"])print("PyTorch version:", manifest["torch"])print("contexts compared:", len(check["contexts"]))print("logits compared:", f"{ours.size:,}")                          # every logit: one per context and per token in the vocabularyprint("same as PyTorch to 4 decimal places:", bool(np.abs(ours - check["logits"]).max() < 1e-4))   # 1e-4: four decimal places
輸出python same-as-pytorch.py
PyTorch version: 2.14.1+cpu
contexts compared: 64
logits compared: 264,960
same as PyTorch to 4 decimal places: True

NumPy 的前向傳遞重現了 PyTorch 的 264,960 個 logits,精確到小數點後四位,此模型是以 PyTorch 2.14.1+cpu 訓練而成。本章所報告的每個關於已訓練模型的數字,都來自那段 NumPy 程式碼,唯獨訓練曲線例外,那是腳本在訓練時記錄下來的。

**實用原則:**以能為你計算梯度的函式庫來訓練,同時自行保留一段簡短而獨立的前向傳遞來檢查它的輸出,如同這裡的 lib.neural 所做的。每次儲存權重時,都要記錄函式庫對少數輸入的輸出,使檢查能在任何地方、在訓練結束很久之後執行。

此部分尚未翻譯,因此以英文呈現。

Stop when validation loss turns up

最近一次變動於第 2026.11.16 版

A model with 1.4 million parameters, trained on a few hundred thousand tokens, can memorize its training text, as chapter 3's 5-gram did. Training perplexity keeps falling as it does. Validation perplexity falls at first, while the model learns patterns that hold for new names, then rises, once it starts fitting quirks of the training names. A validation perplexity that turns up while training perplexity keeps falling is overfitting, and the simplest defence is early stopping: measure validation loss after every epoch, one full pass over the training text, and keep the weights from the best epoch.

The training script recorded both perplexities after each of its twelve epochs. Read them for where validation stops improving:

training-curve.py Python 3.12.15 通過
"""The training script's record of training and validation perplexity after each epoch, and the epoch whose weights werekept. The prose's point rests on the validation column, and on the two validation figures printed after the table."""import jsonimport mathcurves = json.load(open("data/recorded/neural-lm/curves.json", encoding="utf-8"))manifest = json.load(open("data/recorded/neural-lm/manifest.json", encoding="utf-8"))print(f"{'epoch':>5}  {'training perplexity':>19}  {'validation perplexity':>21}")for c in curves:                                             # losses are averages of log probabilities: exp turns them into perplexities    print(f"{c['epoch']:>5}  {math.exp(c['train_loss']):>19.1f}  {math.exp(c['validation_loss']):>21.1f}")best = min(curves, key=lambda c: c["validation_loss"])       # the epoch with the lowest validation loss, which the kept epoch should matchprint("epoch kept:", manifest["kept_epoch"])print("validation perplexity at the kept epoch:", f"{math.exp(best['validation_loss']):.1f}")print("validation perplexity at the last epoch:", f"{math.exp(curves[-1]['validation_loss']):.1f}")   # curves[-1]: the final epoch's record, whatever was kept
輸出python training-curve.py
epoch  training perplexity  validation perplexity
    1                109.3                   51.4
    2                 37.6                   39.9
    3                 27.3                   37.3
    4                 22.4                   36.6
    5                 19.3                   36.4
    6                 17.2                   36.9
    7                 15.6                   37.8
    8                 14.4                   38.6
    9                 13.5                   39.7
   10                 12.7                   40.9
   11                 12.0                   42.1
   12                 11.5                   43.7
epoch kept: 5
validation perplexity at the kept epoch: 36.4
validation perplexity at the last epoch: 43.7

In epoch 1, training perplexity is higher than validation, because the training figure averages over the whole epoch, including its first batches, when the model was still untrained, while validation is measured at the epoch's end. Validation perplexity bottoms out at 36.4 in epoch 5 and climbs from there, to 43.7 by the last epoch, while training perplexity falls at every epoch after the first. The training script kept the weights from the best epoch, and every result in the rest of this chapter uses them.

The gap between the two curves is the model memorizing. It grows every epoch, because the training names repeat their product lines and phrasing, and this model has no regularization, which is anything in training that discourages memorizing, so early stopping is its only defence.

Working rule: Always record validation loss per epoch and keep the best epoch's weights, never the last. When the curves part early, as they do here after a few epochs, the model has more capacity than its training text supports: regularize it, use a smaller model, or get more text of the same kind.

與基準模型比較

最近一次變動於第 2026.11.11 版

真正的檢驗,是本書每個模型都要面對的那一項:在驗證名稱上計算每個詞元的困惑度,並與 4-gram 基準模型比較。在這些名稱上,哪一項更重要:關聯相似的上下文,還是精確的計數?神經模型能關聯相似的上下文,4-gram 做不到;而 4-gram 對它見過的每個上下文都有精確的計數。

neural-vs-baseline.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""The window model and the 4-gram baseline, each scored by perplexity per token on the same validation names. Theprose's point rests on the two printed perplexities."""from lib import neuralfrom lib.names import name_tokensfrom lib.ngrams import KneserNey, perplexitytrain, validation = name_tokens("train"), name_tokens("validation")     # both models are scored on the same tokens of the same namesprint("window model validation perplexity:", f"{neural.perplexity(validation):.1f}")      # the window model, from its saved weightsprint("4-gram baseline validation perplexity:", f"{perplexity(KneserNey(train, 4), validation):.1f}")   # a 4-gram: each token is predicted from the three before it
輸出python neural-vs-baseline.py
window model validation perplexity: 36.4
4-gram baseline validation perplexity: 23.1

以計數為基礎的模型大幅勝出:視窗模型的驗證困惑度是 36.4,基準模型則是 23.1。

為了找出差距出現在哪裡,我們依三個詞元的上下文是否出現在訓練名稱中,將驗證詞元分成兩部分,並分別為兩個模型評分:

seen-and-unseen-contexts.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Splits the validation tokens by whether their three-token context occurs in the training names, and scores both modelson each part. The prose's point rests on the four perplexities printed for the two parts."""import mathimport numpy as npfrom lib import neuralfrom lib.names import name_tokensfrom lib.ngrams import KneserNey, ngrams, paddedtrain, validation = name_tokens("train"), name_tokens("validation")counted = KneserNey(train, 4)pairs = [(c, t) for text in validation for c, t in ngrams(padded(text, 4), 4)]   # padding gives each name's first tokens a contextseen = np.array([c in counted.totals[4] for c, _ in pairs])     # was this 3-token context in the training names?from_counts = np.array([counted.probability(c, t) for c, t in pairs])from_network = neural.token_probabilities(validation)           # in the same order as pairs, so each entry is for the same tokenprint("validation tokens with a context seen in training:", f"{seen.mean():.0%}")for part, where in [("seen", seen), ("unseen", ~seen)]:                  # ~seen: the complement, the contexts the training names lack    for model, p in [("4-gram", from_counts), ("network", from_network)]:        print(f"{model} perplexity, {part} contexts: {math.exp(-np.log(p[where]).mean()):.1f}")   # exp of the average surprise, over this part
輸出python seen-and-unseen-contexts.py
validation tokens with a context seen in training: 70%
4-gram perplexity, seen contexts: 17.9
network perplexity, seen contexts: 26.1
4-gram perplexity, unseen contexts: 41.2
network perplexity, unseen contexts: 78.5

網路在兩部分都比較差,而且在未見過的上下文上差距更大,而在這類上下文中,它關聯上下文的能力原本應該最有幫助。在這部分,4-gram 的分數是 41.2,網路是 78.5。在見過的上下文上(占驗證詞元的 70%),4-gram 的分數是 17.9,網路是 26.1。

因此,這個網路並非只在計數精確之處落敗;它也沒有學會良好的泛化。就這個模型與這份文本而言,有兩個特徵最可能是原因。第一,網路沒有正則化,在數個訓練週期後就發生過度擬合,如訓練曲線所示。第二,商店的名稱簡短且有固定格式:三個詞元的上下文就涵蓋了名稱的大部分內容,而這正是 Kneser-Ney 計數最強的情境。Bengio 等人在更大、更多樣的文本上得到相反的結果:在 Brown 語料庫(約 800,000 個訓練詞)上,他們最好的網路單獨使用時測試困惑度為 268,與三元語法模型混合後為 252,而最好的 n-gram 模型為 312(Bengio et al., 2003,第 4.2 節與表 1)。學習得來的模型並不會因為是學習得來就比較好;必須加以測量。

實用原則: 在宣稱學習得來的模型有幫助之前,務必在相同的文本與詞元上,將它與強大的計數基準模型比較,並依見過與未見過的上下文拆分比較,看出各模型在哪裡勝出。不要因為一個小型資料集,就斷定神經模型行不通,或是斷定它們總是行得通。

將神經網路與計數模型混合

最近一次變動於第 2026.11.11 版

一個平均表現較差的模型,只要它的錯誤與較佳模型的錯誤不同,仍然可能有用。當兩個模型犯的錯誤不同時,混合它們的機率可以勝過兩者,Bengio 等人就是以此方式處理他們的網路與經過平滑的三元語法(trigram)模型(Bengio et al., 2003,第 4 節)。混合的做法是對每個詞元取兩個模型機率的加權平均,與第 3 章的內插相同。

混合後能否勝過兩端的模型?我們在驗證用的名字上計分,讓網路在混合中所占的比重分別為 0、0.2、0.3、0.5 和 1,其中 0 表示只用計數模型,1 表示只用網路:

mix-neural-and-counts.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Mixes the network's and the 4-gram's probability of every validation token, at five shares for the network, and printsthe validation perplexity of each mix. The prose's point rests on the printed perplexity at share 0.2, and at the two ends."""import mathimport numpy as npfrom lib import neuralfrom lib.names import name_tokensfrom lib.ngrams import KneserNey, ngrams, paddedtrain, validation = name_tokens("train"), name_tokens("validation")from_network = neural.token_probabilities(validation)           # the network's probability of each actual next token, in ordercounted = KneserNey(train, 4)# the 4-gram's probability for the same tokens, in the same order: each name padded with three start markers firstfrom_counts = np.array([counted.probability(c, t) for text in validation for c, t in ngrams(padded(text, 4), 4)])for share in [0.0, 0.2, 0.3, 0.5, 1.0]:                # the network's share of the mix; the counts get the rest    mixed = share * from_network + (1 - share) * from_counts    print(f"validation perplexity, network share {share:.1f}: {math.exp(-np.log(mixed).mean()):.1f}")
輸出python mix-neural-and-counts.py
validation perplexity, network share 0.0: 23.1
validation perplexity, network share 0.2: 21.9
validation perplexity, network share 0.3: 22.1
validation perplexity, network share 0.5: 22.9
validation perplexity, network share 1.0: 36.4

混合後的模型勝過兩個單獨的模型。當網路的比重為 0.2 時,驗證困惑度(perplexity)為 21.9,優於基準模型的 23.1,也遠優於單獨使用網路的 36.4。網路的平均表現較差,但在計數模型給某個詞元極低機率的地方,網路所占的比重使該機率不致過小,因而降低了平均驚訝程度。

所保留的訓練週期(epoch)與比重都是依據驗證用的名字挑選的,因此這項改善略微偏於樂觀;本書最後會在測試用的名字上進行比較來檢驗。此外,混合之所以有幫助,只是因為兩個模型的錯誤不同:如果它們犯的是同樣的錯誤,將兩者平均不會有任何增益。

實用原則: 當學習而得的模型與計數模型表現相近時,先嘗試混合兩者的機率,再決定選用哪一個,並在保留的文本上選定比重。若其中一個模型明顯較強,比重應保持較小,並在依賴這項改善之前,先以另外的測試集確認。

更多文字只有在種類相同時才有幫助

最近一次變動於第 2026.11.11 版

如果神經網路因為學習的文字太少而過度擬合,最直接的解法就是增加文字。商店有很多現成的文字:商品描述。訓練腳本可以用名稱和描述一起訓練同一個模型,本書也記錄了這個版本,並以相同的驗證名稱評分。

以下在驗證名稱上比較這兩個已記錄的模型,一個只用名稱訓練,另一個用名稱與描述訓練:

more-text-of-another-kind.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Scores the window model trained on names alone against the one trained on names and descriptions, counts the trainingtokens of each kind of text, and prints the second model's validation perplexity after each of its epochs. The prose'spoint rests on the two validation perplexities, and on the training token counts printed beside them."""import jsonimport mathfrom lib import neuralfrom lib.names import name_tokensvalidation = name_tokens("validation")for name in ["neural-lm", "neural-lm-all-text"]:               # the recorded runs: names alone, then names and descriptions    manifest = json.load(open(f"data/recorded/{name}/manifest.json", encoding="utf-8"))    print(f"learned from {manifest['settings']['text']}:", f"{neural.perplexity(validation, name):.1f}")from lib import bpefrom lib.store import load_productstokenizer = bpe.store_tokenizer()names_tokens = sum(len(t) for t in name_tokens("train"))      # counted in the store's tokens, the ones the model readsdescription_tokens = sum(len(tokenizer.encode(d)) for d in load_products("train")["description"])   # the training split only, as for namesprint("training tokens, names:", f"{names_tokens:,}")print("training tokens, names and descriptions:", f"{names_tokens + description_tokens:,}")curve = json.load(open("data/recorded/neural-lm-all-text/curves.json", encoding="utf-8"))print("all-text model, validation perplexity by epoch:", ", ".join(f"{math.exp(c['validation_loss']):.1f}" for c in curve))   # one per epoch, in order
輸出python more-text-of-another-kind.py
learned from names: 36.4
learned from all: 59.0
training tokens, names: 337,071
training tokens, names and descriptions: 2,744,320
all-text model, validation perplexity by epoch: 73.6, 64.0, 60.8, 59.0

加入描述之後,模型在名稱上的表現反而變差。訓練名稱包含 337,071 個詞元,名稱加描述則包含 2,744,320 個,約為前者的八倍。只從名稱學習的模型得分為 36.4,從名稱與描述學習的模型得分為 59.0。第二個模型訓練了四個 epoch,停止時驗證困惑度仍在下降,但每個 epoch 的降幅愈來愈小:73.6, 64.0, 60.8, 59.0。我們的判斷是,即使再增加 epoch,也無法接近只用名稱訓練的模型。

描述是另一種類型的文字。它們以完整句子寫成,用到 your、this 和 perfect 這類名稱裡很少出現的詞,而主要從描述學習的模型,會把名稱當成描述來預測。新增的文字讓模型學到了與任務不符的分布。模型學的是它拿到的文字,而困惑度衡量的是它預測這類文字的能力:不同種類的資料可能讓它變差,這類資料需要加權,或與目標資料分開處理。

實用原則: 在加入訓練文字之前,先確認它與你要預測的文字相似;兩種都訓練一次,並在目標文字上衡量,而不是假設愈多愈好。當額外文字差異很大時,只用目標文字訓練,或是安排每種文字的版面,讓模型能分辨各種類型,如第 8 章在商品名稱與描述之間加入分隔符號的做法。

此部分尚未翻譯,因此以英文呈現。

Suggest from similar contexts

最近一次變動於第 2026.11.11 版

The network lost on average, including on contexts the training names lack, but averages hide single cases. Chapter 3's autocomplete showed the 4-gram's worst case, outdoor bar, a context no training name contains, where it fell back to generic continuations.

Next, the network's three likeliest tokens after five name prefixes, and a few embeddings, tell whether outdoor bar gets the stool suggestion that patio bar does:

neural-autocomplete.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""The network's three likeliest next tokens after five typed prefixes, its probability of `stool` after `outdoor bar`,and the cosine similarity of `patio` with three other tokens in its embedding table. The prose's point rests on thesuggestions after each prefix, and on the three cosines printed last."""import numpy as npfrom lib import bpe, neuralfrom lib.ngrams import STARTweights, vocabulary, manifest = neural.load()index, width = {t: i for i, t in enumerate(vocabulary)}, manifest["settings"]["context"]tokenizer = bpe.store_tokenizer()for typed in ["bar", "patio bar", "outdoor bar", "kitchen bar", "wine bar"]:    # each typed prefix is read as the start of a name    context = ([START] * width + tokenizer.encode(typed))[-width:]               # pad with start markers, keep the last width tokens    logits = neural.forward(weights, np.array([[index[t] for t in context]]))[0]   # forward takes a batch: one row, so take row 0    probabilities = np.exp(neural.log_softmax(logits[None])[0])                 # log_softmax wants a batch too: add an axis, then take it off    best = np.argsort(-probabilities, kind="stable")[:3]                        # the three likeliest; a stable sort keeps ties in vocabulary order    # the end-of-word marker </w> is shown as nothing, so the tokens read as the words they came from    print(f"{typed:<12} -> " + ", ".join(f"{vocabulary[i].replace('</w>', '')} {probabilities[i]:.2f}" for i in best))outdoor_bar = ([START] * width + tokenizer.encode("outdoor bar"))[-width:]      # the same padding as abovep_stool = np.exp(neural.log_softmax(neural.forward(weights, np.array([[index[t] for t in outdoor_bar]])))[0, index["stool</w>"]])print("P(stool | outdoor bar):", f"{p_stool:.2f}")table = weights["embedding"] / np.linalg.norm(weights["embedding"], axis=1, keepdims=True)   # unit-length rows, so a dot product is a cosinefor a, b in [("patio</w>", "outdoor</w>"), ("patio</w>", "kitchen</w>"), ("patio</w>", "wine</w>")]:    print(f"cosine({a.replace('</w>', '')}, {b.replace('</w>', '')}) in the model's embeddings:", f"{float(table[index[a]] @ table[index[b]]):.2f}")
輸出python neural-autocomplete.py
bar          -> with 0.22, cabinet 0.14, cart 0.13
patio bar    -> stool 0.63, with 0.23, & 0.05
outdoor bar  -> stool 0.40, with 0.13, & 0.12
kitchen bar  -> & 0.53, with 0.17, cart 0.09
wine bar     -> with 0.23, cabinet 0.16, </s> 0.14
P(stool | outdoor bar): 0.40
cosine(patio, outdoor) in the model's embeddings: 0.16
cosine(patio, kitchen) in the model's embeddings: 0.07
cosine(patio, wine) in the model's embeddings: 0.10

After outdoor bar, the network suggests stool first, with probability 0.40, where the 4-gram put & and with ahead of it. The control prefixes show that stool isn't automatic after bar: after bar alone, kitchen bar and wine bar, it isn't among the network's three suggestions. And in the network's own embedding table, patio is a little closer to outdoor, with a cosine similarity of 0.16, than to kitchen or wine. That's consistent with the network having carried something from patio bar over to outdoor bar, though the similarity is weak, and one prefix proves little.

This is one case, not an average: the seen and unseen split showed the network worse on unseen contexts overall. It shows what a learned model can do that counting can't, and that this one does it only sometimes.

Working rule: Look at a learned model's predictions on contexts the training text lacks, with control prefixes beside them, but judge it by measurements over many contexts, not by examples. For autocomplete in production, use the mix, and test it on rare prefixes as well as common ones.

正式環境中會出什麼問題

最近一次變動於第 2026.11.11 版

神經語言模型在帶來新優勢的同時,也帶來新的失敗模式:

  • 過度擬合不會有聲響。 訓練損失看起來總是很好。如果沒有驗證曲線,訓練過久的模型,上線後的表現會比訓練較少的模型更差。請保留最佳的 epoch,並留意訓練與驗證之間的差距。
  • 保存你要上線的權重。 在另一種處理器上重新訓練,得到的權重會略有不同,因此要保留產生這些權重的程式碼與設定,以及函式庫對少數輸入所給出的輸出,就像這裡的訓練腳本所做的一樣。
  • 輸出層佔據主要成本。 每一步都要為詞彙表中的每個詞元評分,這佔了大部分的運算量與大部分的參數。詞彙表變大時,最先增加成本的就是這一層。
  • 不相符的訓練文字會造成損害。 加入與目標不同的文字,例如拿商品描述來訓練商品名稱的模型,會使預測變差,而訓練損失看起來卻沒有問題。每次更動訓練資料之後,都要在目標文字上重新衡量。
  • 平均值會掩蓋罕見的情境。 模型整體分數可能很高,卻仍會在不常見的前綴上失敗,而購物者尋找新東西時,正好會輸入這類前綴。請分別為見過與未見過的情境評分,如同本章所做的,並特別留意未見過的情境。
  • 混合模型在服務時需要兩個模型。 把神經網路與計數混合,會使需要載入並保持最新的內容加倍。每次重新訓練網路時,都要重新建立計數,讓兩者描述的是同一份文字。

本章的決定

  • 本書的神經網路模型一律在 PyTorch 中訓練一次,存放於 recordings/, 範例再依儲存的權重以 NumPy 重新計算。NumPy 的前向傳遞能將 PyTorch 的 logits 重現到小數點後四位,因此範例不需要 PyTorch。
  • 模型保留驗證表現最佳那個 epoch 的權重。 視窗模型的驗證困惑度在第 5 個 epoch 後最低,之後每個 epoch 都上升,訓練困惑度卻持續下降:模型是在死記訓練用的名稱。
  • 在商店的名稱上,單獨使用時視窗模型輸給 4-gram 基準模型,無論是已見過或未見過的上下文都是如此;混合後則勝過基準模型。 網路佔比 0.2 時得到 21.9,是本書目前為止表現最好的模型。
  • 學習式模型與基準模型的比較,要分別在已見過與未見過的上下文上進行, 而不只看平均值,因為這兩部分可能呈現不同的結果。在此例中,網路在兩者上都落後,而在未見過的上下文上落後得更多。
  • 訓練文本必須與目標文本相近。 加入商品描述後,模型在名稱上的表現變差,儘管名稱加描述的詞元數量約為單獨名稱的八倍,因此每個模型的訓練文本,都要依其將被評分的文本來挑選。
  • 名稱很短,所以從第 8 章起,本書也會在商店的商品描述上衡量模型。

本章,於本版次中

版次 2026.11.16範例於 2026年11 月 19 通過測試本章 10 個範例 個範例中有 10 個通過

譯自英文書,版次 2026.11.16。

引用本節

格式

各種格式皆引用英文原書,本譯本即依循該原書。

每種格式都會標示本版。其連結會開啟目前版次,節標題會說明該節內文最後一次變動的時間。存取日期與擷取日期即為你開啟此頁的當日。

本頁內容

第 5 章,共 11 章: 神經語言模型

  1. 從嵌入視窗預測下一個詞元
  2. 以反向傳播計算梯度
  3. 執行梯度下降步驟
  4. 以 PyTorch 訓練
  5. Stop when validation loss turns up
  6. 與基準模型比較
  7. 將神經網路與計數模型混合
  8. 更多文字只有在種類相同時才有幫助
  9. Suggest from similar contexts
  10. 正式環境中會出什麼問題
  11. 本章的決定

返回頂端所有章節