訓練小型模型

閱讀約需 52 分鐘 分鐘,10 節 中共有 7 個範例。

版次 2026.11.16 範例於 2026年11 月 19 通過測試 86 個範例中有 86 個通過 變更內容

由 AI 翻譯。經人工以英文驗證。 我們如何製作書籍

譯自英文書,版次 2026.11.16。 閱讀原文

讀完本章後,你能從頭到尾訓練一個小型 transformer,並評斷其結果。你會把文字轉成批次,用 AdamW 更新權重,設定步長的排程,閱讀訓練曲線,以計數基準線公平地衡量模型,並找出在這個規模下,更大的寬度、更多的文字與更多的訓練各能帶來什麼,也會找出模型在哪些地方勝過計數,又在哪些地方仍需要搭配計數。

第 7 章組裝了商店的模型,並用它訓練好的權重檢查每個部分。本章要問的是這些權重從何而來:一次訓練執行,記錄在 recordings/transformer.py 中。那次執行讓模型學會預測商店型錄中的每個下一個 token,也就是商店在線上販售的家具與家居用品的名稱和描述。接著,本章針對模型的兩項用途進行衡量:在搜尋框中建議產品名稱的下一個字詞,以及預測描述,描述的文字夠長,遠處的上下文也會產生影響。

如同第 5 章,訓練只在 PyTorch 中執行了一次。範例讀取的是它儲存的內容:權重、曲線與設定,這些都記錄在該次執行的資訊清單中。若想自行訓練,請在獨立的環境中安裝 recordings/requirements.txt,這會一併安裝 PyTorch 的 CPU 版本,然後在本書的資料夾中執行 python recordings/transformer.py:它會把相同的檔案寫入 examples/data/recorded/transformer/,並可在筆記型電腦的處理器上執行。

將商品轉為批次

最近一次變動於第 2026.11.16 版

訓練的第一步,是把商店的商品目錄轉成模型可以學習的數字。每項商品成為一段文字:起始標記、名稱、分隔符號 <sep>、描述與結束標記,並由商店的分詞器進行分詞。超過模型長度上限的文字會被截斷。每個訓練步驟一次取用一個批次的文字,使運算能以整個矩陣進行;長度不同的文字則以填充符號補到該批次中最長者的長度。

每段文字會讓模型在每個位置各做一次預測。輸入是一段文字去掉最後一個符元後的符元,目標則是去掉第一個符元後的符元,因此位置 i 的訓練目標是預測符元 i + 1。由於有因果遮罩,一次前向傳遞就能同時完成所有這些預測。損失只對非填充的目標取平均:填充會耗費運算,卻學不到任何東西。

填充是需要留意的成本,因此請依訓練腳本的做法建立洗牌後一個訓練週期的批次,並計算每個批次中有多少比例是填充:

training-batches.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Builds one shuffled epoch of batches the way the training script does, and measures the padding in them.The prose's point rests on the share of targets that are padding over that epoch."""import mathimport numpy as npfrom lib import transformerfrom lib.products import product_texts_, vocabulary, manifest = transformer.load()settings = manifest["settings"]pad, end = vocabulary.index(transformer.PAD), vocabulary.index(transformer.END)texts = [ids for ids, _ in product_texts("train")]size = settings["batch"]order = np.random.default_rng(settings["seed"]).permutation(len(texts))   # one shuffled epoch, from the run's seed, so it repeatspadding, targets_seen, shares = 0, 0, []for start in range(0, len(texts), size):    batch = [texts[i] for i in order[start:start + size]]                      # this step's texts: the next `size` in the shuffled order    longest = max(len(ids) for ids in batch)    padded = np.array([ids + [pad] * (longest - len(ids)) for ids in batch])   # short texts filled out with <pad>    inputs, targets = padded[:, :-1], padded[:, 1:]                            # each position predicts the next token    padding, targets_seen = padding + (targets == pad).sum(), targets_seen + targets.size    shares.append((targets == pad).mean())    if start == 0:        first_shape = (inputs.shape, targets.shape)print("training texts:", f"{len(texts):,}")print("tokens in them:", f"{sum(len(ids) for ids in texts):,}")print(f"texts cut at the length limit, {settings['longest']} tokens:", f"{np.mean([ids[-1] != end for ids in texts]):.1%}")   # a cut text lost its end markerprint("texts per batch:", size)print("one batch's inputs and targets:", *first_shape)print("share of targets that are padding, over one shuffled epoch:", f"{padding / targets_seen:.0%}")print("least and most padding in a batch:", f"{min(shares):.0%} and {max(shares):.0%}")print("steps per epoch:", f"{math.ceil(len(texts) / size):,}")   # the last, partly filled batch is a step tooprint("epochs in the recorded run:", settings["epochs"])print("steps in the recorded run:", f"{manifest['steps']:,}")
輸出python training-batches.py
training texts: 34,252
tokens in them: 2,843,222
texts cut at the length limit, 160 tokens: 1.0%
texts per batch: 32
one batch's inputs and targets: (32, 151) (32, 151)
share of targets that are padding, over one shuffled epoch: 46%
least and most padding in a batch: 30% and 59%
steps per epoch: 1,071
epochs in the recorded run: 16
steps in the recorded run: 17,136

訓練切分包含 34,252 段文字與 2,843,222 個符元。這個數字包含每段文字的起始、分隔與結束標記,這些標記並未計入第 4 章的統計所得的 2,744,320 個符元;也不含被長度上限截去的部分;被截斷的文字只佔 1.0%。

每個批次包含 32 段文字,因此走過一遍全部訓練文字,即一個訓練週期,需要 1,071 個步驟;而記錄下來的那次訓練共進行了 17,136 個步驟,涵蓋 16 個訓練週期。

在一個經過洗牌的訓練週期中,有 46% 的目標是填充,視批次而定,介於 30% and 59% 之間,這是因為一段很長的描述會決定與它同批的所有較短文字的寬度。每個批次都是一個矩形:每段文字一列,每個位置一欄。

這種規模的填充是浪費:每次前向傳遞中,近半數的運算是在產生之後會被損失忽略的預測。它不會改變模型學到的內容,因為填充從不計入損失,卻使每個步驟的成本幾乎加倍。Krell 等人發現,常見資料集中最多有一半的符元是填充,並提出打包(packing)的做法:在每一列中將數段文字首尾相接,同時避免注意力跨越文字之間(Krell et al., 2021)。將長度相近的文字放進同一批次,也能減少填充。

**實用原則:**以輸入與位移後的目標來建立批次,將填充從損失中遮罩掉,並在每個訓練週期以固定的隨機種子對文字洗牌,使整次訓練可以重現。請量測批次中填充所佔的比例;若比例偏高,先打包文字或依長度分組,再考慮購買更大的機器。

使用 AdamW 與梯度裁剪進行更新

最近一次變動於第 2026.11.16 版

批次備妥之後,每一步都是第 5 章的迴圈:前向傳遞、損失、梯度、更新。recordings/transformer.py中的下列幾行就是完整的訓練步驟,每個批次執行一次:

        ids = pad([train[i] for i in order[start:start + SETTINGS["batch"]]])
        logits = model(ids[:, :-1])
        loss = torch.nn.functional.cross_entropy(logits.transpose(1, 2), ids[:, 1:], ignore_index=index[PAD])
        optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()
        scheduler.step()

前三行即前一節的內容:經過填充的批次、去掉最後一個詞元的輸入,以及針對位移後的目標所計算、並忽略填充位置的損失。另有兩行與第 5 章不同。

優化器是 AdamW,也就是第 5 章的 Adam 再加上與梯度分開施行的權重衰減(weight decay)。權重衰減在每一步將各個權重減去其數值乘以步長再乘以衰減係數的量,此處衰減係數為 0.1(見 PyTorch 的 AdamW)。Loshchilov 與 Hutter 指出,以這種方式施行衰減,與 Adam 搭配的效果優於把衰減項加進損失中(Loshchilov and Hutter, 2019)。衰減是一種正則化,而第 5 章的視窗模型缺少這一項:它把每個權重拉向零,因此只有在訓練文本持續推動時,權重才會維持在較大的值。

訓練腳本也會對梯度進行裁剪:當所有梯度的合併長度(把全部梯度視為一個長向量來計算)超過 1.0 時,就把它們一併縮小到該長度(見 PyTorch 的 clip_grad_norm_)。如此一來,單一異常批次不會把權重推離太遠。Pascanu 等人正是為此提出裁剪:罕見而極大的梯度,否則會抵消訓練已學到的大部分成果(Pascanu et al., 2013)。

**實用原則:**使用 AdamW 並搭配適度的權重衰減,同時依梯度的合併長度進行裁剪。讓訓練步驟短到可以整段讀完,使其中每個選擇都清楚可見:損失忽略什麼、優化器衰減什麼,以及裁剪的對象是什麼。

此部分尚未翻譯,因此以英文呈現。

Warm up, then decay the step size

最近一次變動於第 2026.11.16 版

The last line of the loop, scheduler.step(), changes the step size after every batch, following a schedule, which sets the step size for each step of the run. For the first 200 steps, the step size rises in a straight line from nearly zero to its peak, 0.003: a warm-up, as the transformer paper's schedule had (Vaswani et al., 2017, section 5.3). In Adam's first steps, its running average of squared gradients rests on only a few batches, so its step sizes vary widely; Liu and colleagues showed that warm-up works by reducing that variance (Liu et al., 2020). After the warm-up, the step size falls along half a cosine curve, to a tenth of the peak by the last step (Loshchilov and Hutter, 2017). Large steps early cover ground; small steps late let the weights settle into a low point instead of bouncing around it. In symbols, after the warm-up:

rate = peak × (0.1 + 0.9 × ½ (1 + cos(π × progress)))

where progress runs from 0 at the end of the warm-up to 1 at the last step.

Whether the schedule matches the run's is checkable: compute it by hand from the recorded settings, and set it against the step sizes the training script saved every tenth step:

learning-rate-schedule.py Python 3.12.15 · numpy 2.5.5 通過
"""Computes the warm-up and cosine schedule by hand from the recorded settings, and checks it against the step sizesthe training script saved. The prose relies on the check: the assert fails the run if the two ever differ."""import jsonimport mathfrom lib import transformer_, _, manifest = transformer.load()settings, total = manifest["settings"], manifest["steps"]recorded_rates = json.loads((transformer.RECORDED / "curves.json").read_text())["rates"]   # every 10th step's, after the scheduler's stepdef rate(step):    """Warm up linearly to the peak, then follow half a cosine down to a tenth of it."""    if step < settings["warmup_steps"]:        return settings["peak_rate"] * (step + 1) / settings["warmup_steps"]   # steps count from 0, so step 0 is already a share of the peak    progress = (step - settings["warmup_steps"]) / (total - settings["warmup_steps"])   # 0 at the warm-up's end, 1 at the last step    return settings["peak_rate"] * (0.1 + 0.9 * 0.5 * (1 + math.cos(math.pi * progress)))print("warm-up steps:", settings["warmup_steps"])print("peak step size:", settings["peak_rate"])print("weight decay:", settings["weight_decay"])for step in [0, settings["warmup_steps"] // 2, settings["warmup_steps"], total // 2, total - 1]:    print(f"step size at step {step:,}:", f"{rate(step):.2e}")gap = max(abs(recorded - rate(10 * i + 1)) for i, recorded in enumerate(recorded_rates))   # recorded_rates[i] was saved after step 10 * i, so it is the size used at step 10 * i + 1assert gap < 1e-9                   # every saved step size matches the hand calculation, to a billionthprint("same as the recorded step sizes:", "yes")
輸出python learning-rate-schedule.py
warm-up steps: 200
peak step size: 0.003
weight decay: 0.1
step size at step 0: 1.50e-05
step size at step 100: 1.51e-03
step size at step 200: 3.00e-03
step size at step 8,568: 1.68e-03
step size at step 17,135: 3.00e-04
same as the recorded step sizes: yes

The hand calculation matches every saved step size. The step size starts at 1.50e-05, peaks at 3.00e-03 at the warm-up's end, and is 3.00e-04 at the last step, a tenth of that peak.

The decay matters. The comparison run trained for four epochs at the same width, with the same seed and the same batches, so its schedule finished its decay at step 4,284, and at that step it scored 27.6 on the validation names. The recorded run was still near its peak step size at that step, and scored 32.6. Same data, same steps: the difference is the decay.

Working rule: Warm the step size up over the first few hundred steps, then decay it, and match the schedule's length to the run's. Save the step size as training runs, as the training script does, and check it against the schedule you meant: a schedule bug looks like a training problem, and is far easier to find this way.

解讀訓練曲線

最近一次變動於第 2026.11.11 版

在批次、更新與排程都設定好之後,這家商店的模型訓練了 16 個 epoch,腳本在每個 epoch 結束後,分別在驗證集上量測它對名稱與描述的表現。它保留的是名稱與描述驗證損失相加後最低的那個 epoch 的權重,做法與第 5 章保留最佳 epoch 相同。

以每個 token 的困惑度(perplexity)來看,記錄下來的曲線逐 epoch 如下:

transformer-training-curve.py Python 3.12.15 · numpy 2.5.5 通過
"""Prints the recorded run's perplexity for each epoch, then the epoch kept and its validation perplexities.The prose's point rests on the two validation columns and the epoch kept."""import jsonimport mathfrom lib import transformer_, _, manifest = transformer.load()curves = json.loads((transformer.RECORDED / "curves.json").read_text())["epochs"]print("perplexity per token; training mixes names and descriptions, as the training text does")print("epoch  training  validation names  validation descriptions")for curve in curves:    print(f"{curve['epoch']:>5}  {math.exp(curve['train_loss']):>8.1f}  {math.exp(curve['validation_name_loss']):>16.1f}"          f"  {math.exp(curve['validation_description_loss']):>23.1f}")kept = next(curve for curve in curves if curve["epoch"] == manifest["kept_epoch"])   # the epoch whose weights were keptprint("epoch kept, the lowest name and description losses added together:", manifest["kept_epoch"])print("validation perplexity on names, kept epoch:", f"{math.exp(kept['validation_name_loss']):.1f}")print("validation perplexity on descriptions, kept epoch:", f"{math.exp(kept['validation_description_loss']):.1f}")print("validation perplexity on names, epoch 4:", f"{math.exp(curves[3]['validation_name_loss']):.1f}")   # the list starts at epoch 1, so index 3 is epoch 4
輸出python transformer-training-curve.py
perplexity per token; training mixes names and descriptions, as the training text does
epoch  training  validation names  validation descriptions
    1     131.4              50.5                     50.4
    2      41.4              38.8                     36.2
    3      32.8              34.5                     31.5
    4      29.0              32.6                     29.3
    5      26.6              30.9                     27.8
    6      24.8              30.2                     26.4
    7      23.3              28.9                     25.0
    8      21.8              28.1                     23.9
    9      20.5              27.1                     22.9
   10      19.2              26.5                     22.2
   11      17.9              25.8                     21.6
   12      16.8              25.1                     20.8
   13      15.8              24.6                     20.4
   14      14.9              24.2                     20.0
   15      14.2              24.0                     19.9
   16      13.8              24.1                     19.9
epoch kept, the lowest name and description losses added together: 15
validation perplexity on names, kept epoch: 24.0
validation perplexity on descriptions, kept epoch: 19.9
validation perplexity on names, epoch 4: 32.6

驗證困惑度一路下降,到第 15 個 epoch,也就是被保留的那個 epoch 之後趨於平穩:最後一個 epoch 並沒有更好。在保留的 epoch,名稱的驗證困惑度是 24.0,描述則是 19.9。如同第 5 章所述,第 1 個 epoch 的訓練困惑度高於驗證困惑度,因為它是對該 epoch 最初幾個批次取平均,而那時模型什麼都還不知道。訓練欄位和訓練文本一樣,混合了名稱與描述,因此它呈現的是趨勢,不能拿來與任一個驗證欄位的數字並列比較。

第 5 章的視窗模型在第 5 個 epoch 之後就不再進步;這個模型則持續進步了長得多的時間。兩次執行在幾個方面不同,都可能解釋這個差異。這個模型從名稱與描述學習,共 2,843,222 個 token,而視窗模型只從名稱學習,有 337,071 個;它的步長會衰減;它有權重衰減;而且注意力機制在每個位置都套用相同的權重。這次執行並未區分這些原因。它確實顯示的是訓練與驗證之間的差距,這個差距每個 epoch 都在擴大:到最後,模型預測訓練文本的表現遠勝於新文本,表示它開始記憶了。

描述的分數比名稱低,但這並不表示模型更擅長處理描述。每種文本都有自己的可預測性:描述會重複其範本與名稱中的詞句,而名稱則是用寥寥數語概括一項新產品。困惑度是用來比較不同模型在同一份文本上的表現,絕不能用來比較不同文本。

實用原則: 對你在意的每一種文本,逐 epoch 記錄驗證損失,並保留最佳 epoch 的權重。要把訓練與驗證之間的差距視為警訊,而不只看訓練損失。絕不要跨不同種類的文本比較困惑度。

與名稱的基準模型比較

最近一次變動於第 2026.11.11 版

訓練過程讀完之後,就可以衡量結果。對名稱而言,最強的計數模型是以訓練名稱訓練的 4-gram 基準模型。它在驗證名稱上的得分為 23.1,這是本書的衡量標準:自第 3 章以來,每個模型都以這些名稱評分。同樣的 4-gram 若以完整文本(名稱與描述)訓練,在這些名稱上的得分為 27.7,原因是描述稀釋了它對名稱走向的計數。

比較時必須計算相同的預測。4-gram 預測每個名稱的各個詞元,接著預測其結束標記;transformer 預測每個名稱的各個詞元,接著預測 <sep>,兩者作用相同,因為它標示名稱在何處結束。名稱之後不需要接任何內容,所以 transformer 只讀取名稱本身。兩個模型評分的是同樣的 46,704 個預測。

先猜一猜:單靠 transformer,能否擊敗第 5 章的視窗模型未能擊敗的計數基準模型?然後比照第 5 章的做法,為兩個模型以及兩者的混合評分:

transformer-on-names.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Scores every validation name, its end included, with the 4-gram baseline and with the transformer, then mixes themat several shares. The prose's point rests on the share 0.5 line, the lowest of the five."""import numpy as npfrom lib import transformerfrom lib.names import name_tokensfrom lib.ngrams import KneserNey, ngrams, padded_, vocabulary, _ = transformer.load()index = {token: i for i, token in enumerate(vocabulary)}   # each token's id in the vocabulary, which is its column in the model's outputcounted = KneserNey(name_tokens("train"), 4)                     # chapter 3's baseline, trained on namesfrom_counts, from_transformer = [], []for name in name_tokens("validation"):    ids = [index[transformer.START]] + [index[token] for token in name] + [index[transformer.SEP]]   # the name as the model reads it    probabilities = transformer.softmax(transformer.forward(ids)[:-1])   # the last position predicts past <sep>, which a name never needs    from_transformer += list(probabilities[np.arange(len(ids) - 1), ids[1:]])   # each position's probability for the token after it    from_counts += [counted.probability(context, token) for context, token in ngrams(padded(name, 4), 4)]   # then </s>from_counts, from_transformer = np.array(from_counts), np.array(from_transformer)print("validation tokens scored, each name's end included:", f"{len(from_counts):,}")for share in [0.0, 0.3, 0.5, 0.7, 1.0]:    mixed = (1 - share) * from_counts + share * from_transformer    print(f"validation perplexity on names, transformer share {share:.1f}:", f"{np.exp(-np.log(mixed).mean()):.1f}")
輸出python transformer-on-names.py
validation tokens scored, each name's end included: 46,704
validation perplexity on names, transformer share 0.0: 23.1
validation perplexity on names, transformer share 0.3: 18.7
validation perplexity on names, transformer share 0.5: 18.4
validation perplexity on names, transformer share 0.7: 18.8
validation perplexity on names, transformer share 1.0: 24.0

以各占一半的比例混合時,transformer 與計數模型在名稱上的得分為 18.4,是本書所有模型中在名稱上表現最好的。單獨使用 transformer 時得分為 24.0,略遜於基準模型的 23.1。

單獨的 transformer 遠勝於第 5 章的視窗模型,後者的得分為 36.4,而混合模型也勝過第 5 章最佳的混合結果 21.9。

因此在名稱上,單靠計數仍然難以被超越,原因如第 5 章所述:名稱短而有固定格式,三個詞元的上下文就涵蓋了名稱所含的大部分資訊。但 transformer 出錯的地方與計數模型不同,混合後兩者的長處都能受益。比例是依驗證名稱選定的,因此這項增益略微偏樂觀;測試名稱則保持不動,留待本書最後的比較。

實用原則: 以你能建立的最強基準模型,在完全相同的預測上與新模型比較,包括各自如何標示文本的結尾。當學習式模型與計數模型的表現接近時,就把兩者混合,並在保留的文本上選定比例。

此部分尚未翻譯,因此以英文呈現。

Compare on descriptions

最近一次變動於第 2026.11.11 版

The names comparison favours counting, because names are short. Descriptions run many times longer, and chapter 6 showed the model copying from the name many tokens back, so this is where its longer reach should pay. A fair baseline here is a 4-gram trained on the same texts the transformer learned from, the name, <sep> and description joined, and scored on the same tokens: each description's tokens and its end, with the same text before them.

Put the longer reach to the test on descriptions: fit the 4-gram on every training text, then score it, the transformer and an even mix on the descriptions of the first 500 validation products, leaving out texts cut at the length limit and products with no description. It also splits the tokens, as chapter 5 did, by whether their three-token context occurs in the training texts:

transformer-on-descriptions.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Scores the descriptions of the first 500 validation products with a 4-gram trained on whole texts, with the transformer,and with their even mix, then splits the tokens by whether their context was seen in training. The prose's point rests on the4-gram baseline and transformer lines, the half-and-half line and the unseen-context lines."""import numpy as npfrom lib import transformerfrom lib.names import name_tokensfrom lib.ngrams import KneserNey, ngrams, paddedfrom lib.products import product_texts_, vocabulary, _ = transformer.load()end = vocabulary.index(transformer.END)as_tokens = lambda ids: [vocabulary[i] for i in ids[1:] if i != end]          # padded() adds its own <s> and </s>counted = KneserNey([as_tokens(ids) for ids, _ in product_texts("train")], 4)   # whole texts: name, <sep>, descriptionfrom_counts, from_transformer, seen, products = [], [], [], 0for ids, sep in product_texts("validation", 500):    if ids[-1] != end or sep == len(ids) - 2:        continue                                          # cut at the length limit, or no description    pairs = list(ngrams(padded(as_tokens(ids), 4), 4))[sep:]    # from <sep> on: the description's tokens, then </s>    from_counts += [counted.probability(context, token) for context, token in pairs]    seen += [context in counted.totals[4] for context, _ in pairs]   # was this 3-token context in the training texts?    probabilities = transformer.softmax(transformer.forward(ids)[sep:-1])   # from <sep> on: each position predicts the next token    from_transformer += list(probabilities[np.arange(len(probabilities)), ids[sep + 1:]])   # the description starts after <sep>, at sep + 1    products += 1from_counts, from_transformer, seen = np.array(from_counts), np.array(from_transformer), np.array(seen)perplexity = lambda p: f"{np.exp(-np.log(p).mean()):.1f}"print("products scored:", products)print("description tokens scored:", f"{len(from_counts):,}")mixed = (from_counts + from_transformer) / 2for label, p in [("4-gram baseline", from_counts), ("transformer", from_transformer), ("half and half", mixed)]:    print(f"{label}, validation perplexity on descriptions:", perplexity(p))print("description tokens whose 3-token context is in the training texts:", f"{seen.mean():.0%}")for part, where in [("seen", seen), ("unseen", ~seen)]:    print(f"4-gram, {part} contexts:", perplexity(from_counts[where]))    print(f"transformer, {part} contexts:", perplexity(from_transformer[where]))names = [list(ngrams(padded(name + [transformer.SEP], 4), 4))[:-1] for name in name_tokens("validation")]   # to <sep>: the last pair predicts </s>, which a name stops short ofon_names = [counted.probability(context, token) for pairs in names for context, token in pairs]print("this 4-gram, trained on whole texts, validation perplexity on names:", perplexity(np.array(on_names)))
輸出python transformer-on-descriptions.py
products scored: 446
description tokens scored: 35,651
4-gram baseline, validation perplexity on descriptions: 22.5
transformer, validation perplexity on descriptions: 20.1
half and half, validation perplexity on descriptions: 14.7
description tokens whose 3-token context is in the training texts: 74%
4-gram, seen contexts: 16.3
transformer, seen contexts: 16.6
4-gram, unseen contexts: 57.6
transformer, unseen contexts: 35.0
this 4-gram, trained on whole texts, validation perplexity on names: 27.7

On descriptions, the transformer beats counting alone: it scores 20.1 on the 35,651 description tokens of 446 products, against the 4-gram's 22.5. It is the first learned model in this book to beat a counted model of the same text on its own.

The split shows where they differ. 74% of the tokens have a context the training texts contain, and on those the two models tie: 16.3 for the 4-gram, 16.6 for the transformer. On the rest, the transformer scores 35.0 where the 4-gram scores 57.6. Descriptions are formulaic enough that counts do as well as the transformer wherever they have seen the context; where they haven't, the transformer reads further back and relates what it sees to what it learned, and the counts can only back off.

That's also why the mix works so well. Half and half, the two score 14.7, far below either alone: on any one token, at least one of the two is often confident and right, and the even mix keeps half of that probability, enough to avoid the worst surprises of each.

Working rule: Measure a model on the kind of text it will predict, against a baseline that sees the same text, and expect the ranking to change from one kind of text to another. Split the comparison by seen and unseen contexts to see where each model wins. Try a mix before concluding that either model is enough: the mix gained more than any change to either model that this chapter measured.

寬度、資料與訓練步數能換來什麼

最近一次變動於第 2026.11.11 版

已記錄的模型只是眾多選擇中的一種:寬度為 128 個數值,使用全部訓練文字訓練 16 個 epoch。針對規模大得多的模型所做的研究發現,模型愈大、看過的文字愈多、訓練愈久,損失就愈平穩地下降,而且較大的模型能從看到的每個 token 學到更多(Kaplan et al., 2020)。在運算預算固定的情況下,Hoffmann 等人發現模型大小與訓練文字量應該一起增加(Hoffmann et al., 2022)。Muennighoff 等人則發現,將同一份文字重複最多約四個 epoch,效果幾乎與新文字相當,之後每多重複一次,增益就愈小(Muennighoff et al., 2023)。這些結論在這個玩具規模下是否成立,要靠測量來確認,而不是預先假設。

因此,訓練腳本除了主要的一次訓練之外,還記錄了七次成本較低的訓練,每次只改變一件事:

  • 三種寬度,即 64、128 與 256,各訓練四個 epoch,每個注意力頭都維持 32 個數值寬;
  • 隨機抽取的四分之一與二分之一訓練商品,各訓練四個 epoch;
  • 同樣的四分之一與二分之一資料,以更頻繁的重複,訓練到與全部資料四個 epoch 相同的步數。

哪一種改變換來的效果最大?請並排閱讀這八次訓練,內容包括各自的大小、步數,以及在最佳 epoch 與最後一個 epoch 的困惑度:

width-data-and-steps.py Python 3.12.15 · numpy 2.5.5 通過
"""Reads eight recorded runs and prints each one's parameters, steps, best epoch, and validation perplexities at its bestand last epochs. The prose's point rests on the perplexity-on-names lines."""import jsonimport mathfrom lib import transformerruns = [("width 64, 4 epochs", "transformer-width-64"), ("width 128, 4 epochs", "transformer-width-128"),        ("width 256, 4 epochs", "transformer-width-256"), ("width 128, 16 epochs", "transformer"),        ("a quarter of the data, 4 epochs", "transformer-data-25"), ("half the data, 4 epochs", "transformer-data-50"),        ("a quarter of the data, 16 epochs", "transformer-data-25-same-steps"),        ("half the data, 8 epochs", "transformer-data-50-same-steps")]for label, folder in runs:    run = transformer.RECORDED.parent / folder   # every run sits in the folder beside the main run's    manifest = json.loads((run / "manifest.json").read_text())    curves = json.loads((run / "curves.json").read_text())["epochs"]    best = min(curves, key=lambda curve: curve["validation_name_loss"] + curve["validation_description_loss"])   # the epoch the script kept    print(f"{label}, parameters:", f"{manifest['parameters']:,}")    print(f"{label}, steps:", f"{manifest['steps']:,}")    print(f"{label}, best epoch:", best["epoch"])    print(f"{label}, perplexity on names:", f"{math.exp(best['validation_name_loss']):.1f}")    print(f"{label}, perplexity on descriptions:", f"{math.exp(best['validation_description_loss']):.1f}")    print(f"{label}, perplexity on names, last epoch:", f"{math.exp(curves[-1]['validation_name_loss']):.1f}")   # the last epoch trained, not the one kept    print(f"{label}, perplexity on descriptions, last epoch:", f"{math.exp(curves[-1]['validation_description_loss']):.1f}")
輸出python width-data-and-steps.py
width 64, 4 epochs, parameters: 478,656
width 64, 4 epochs, steps: 4,284
width 64, 4 epochs, best epoch: 4
width 64, 4 epochs, perplexity on names: 36.9
width 64, 4 epochs, perplexity on descriptions: 35.9
width 64, 4 epochs, perplexity on names, last epoch: 36.9
width 64, 4 epochs, perplexity on descriptions, last epoch: 35.9
width 128, 4 epochs, parameters: 1,334,144
width 128, 4 epochs, steps: 4,284
width 128, 4 epochs, best epoch: 4
width 128, 4 epochs, perplexity on names: 27.6
width 128, 4 epochs, perplexity on descriptions: 24.7
width 128, 4 epochs, perplexity on names, last epoch: 27.6
width 128, 4 epochs, perplexity on descriptions, last epoch: 24.7
width 256, 4 epochs, parameters: 4,273,920
width 256, 4 epochs, steps: 4,284
width 256, 4 epochs, best epoch: 4
width 256, 4 epochs, perplexity on names: 23.5
width 256, 4 epochs, perplexity on descriptions: 19.5
width 256, 4 epochs, perplexity on names, last epoch: 23.5
width 256, 4 epochs, perplexity on descriptions, last epoch: 19.5
width 128, 16 epochs, parameters: 1,334,144
width 128, 16 epochs, steps: 17,136
width 128, 16 epochs, best epoch: 15
width 128, 16 epochs, perplexity on names: 24.0
width 128, 16 epochs, perplexity on descriptions: 19.9
width 128, 16 epochs, perplexity on names, last epoch: 24.1
width 128, 16 epochs, perplexity on descriptions, last epoch: 19.9
a quarter of the data, 4 epochs, parameters: 1,334,144
a quarter of the data, 4 epochs, steps: 1,072
a quarter of the data, 4 epochs, best epoch: 4
a quarter of the data, 4 epochs, perplexity on names: 51.9
a quarter of the data, 4 epochs, perplexity on descriptions: 54.1
a quarter of the data, 4 epochs, perplexity on names, last epoch: 51.9
a quarter of the data, 4 epochs, perplexity on descriptions, last epoch: 54.1
half the data, 4 epochs, parameters: 1,334,144
half the data, 4 epochs, steps: 2,144
half the data, 4 epochs, best epoch: 4
half the data, 4 epochs, perplexity on names: 35.2
half the data, 4 epochs, perplexity on descriptions: 34.8
half the data, 4 epochs, perplexity on names, last epoch: 35.2
half the data, 4 epochs, perplexity on descriptions, last epoch: 34.8
a quarter of the data, 16 epochs, parameters: 1,334,144
a quarter of the data, 16 epochs, steps: 4,288
a quarter of the data, 16 epochs, best epoch: 11
a quarter of the data, 16 epochs, perplexity on names: 40.0
a quarter of the data, 16 epochs, perplexity on descriptions: 39.6
a quarter of the data, 16 epochs, perplexity on names, last epoch: 44.1
a quarter of the data, 16 epochs, perplexity on descriptions, last epoch: 45.9
half the data, 8 epochs, parameters: 1,334,144
half the data, 8 epochs, steps: 4,288
half the data, 8 epochs, best epoch: 8
half the data, 8 epochs, perplexity on names: 30.4
half the data, 8 epochs, perplexity on descriptions: 28.6
half the data, 8 epochs, perplexity on names, last epoch: 30.4
half the data, 8 epochs, perplexity on descriptions, last epoch: 28.6

寬度的幫助很大,但需要付出代價。在步數相同時,名稱的困惑度從寬度 64 的 36.9,降到寬度 128 的 27.6,再降到寬度 256 的 23.5。在名稱上,訓練四個 epoch 的最寬模型,甚至勝過訓練十六個 epoch 的已記錄模型,後者為 24.0。但寬度 256 有 4,273,920 個參數,寬度 128 則為 1,334,144 個,每一步的成本也大約高出相應的倍數。

新文字的幫助不會增加每一步的成本。在步數相同時,隨機四分之一的商品在最佳時於名稱上得到 40.0,二分之一得到 30.4,全部則得到 27.6。文字量從一半加倍到全部,所得增益小於寬度從 128 加倍到 256,但不需要額外運算;而從四分之一加倍到一半的第一次加倍,增益比另外兩者都大。

重複在一段時間內有幫助,之後反而有害。四分之一的文字看過四次,得到 51.9;看得更多次時,成績會進步到第 11 個 epoch,之後轉差,十六輪後結束時為 44.1:模型把太常看到的內容背了下來。在相同步數下,重複的文字始終追不上新文字。

更多步數的幫助最小。已記錄模型的十六個 epoch 比四個 epoch 好,在名稱上是 24.0 對 27.6,但步數是後者的四倍。

這些訓練也顯示了本書規模的限制。商店的訓練文字固定為其目錄所含的內容,也就是 2,843,222 個 token。第 7 章曾拿 SmolLM2-360M 的設計與商店的模型相比,它以 4 兆個 token 訓練(模型卡),是商店訓練文字量的一百萬倍以上,這讓一個寬 960 個數值、深 32 層的模型能持續從新文字學習,而不是背誦舊文字。

**實用原則:**在把手上的文字重複超過幾次之前,先取得更多與你要預測的內容同類的文字;這樣做每一步都不會增加成本。若運算資源有餘,先嘗試以較少的 epoch 訓練較寬的模型,再考慮把窄模型訓練得更久,並且兩種都要測量。每次訓練只改變一件事,並在重要之處固定步數,否則比較結果會同時衡量兩件事。

此部分尚未翻譯,因此以英文呈現。

最近一次變動於第 2026.11.11 版

Averages say how good a model is; the search box shows what that means to a shopper. Chapter 3's 4-gram suggested & and with after outdoor bar, a context no training name contains, and chapter 5's window model suggested stool. A few training descriptions do contain outdoor bar, and the transformer learned from them as well as from the names, while the 4-gram learned from names alone.

A shopper would see the transformer's three likeliest next tokens, and those of the half-and-half mix that the names comparison favoured, after the same five prefixes, each treated as the start of a name. A suggestion of <sep> means the model expects the name to end there:

transformer-autocomplete.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 通過
"""Asks the transformer and its half-and-half mix with the 4-gram for their three likeliest next tokens after five prefixes,treated as the start of a name, and prints the probability of stool after outdoor bar. The prose's point rests on thestool lines and the control prefixes beside them."""import numpy as npfrom lib import bpe, transformerfrom lib.names import name_tokensfrom lib.ngrams import KneserNey_, vocabulary, _ = transformer.load()index = {token: i for i, token in enumerate(vocabulary)}counted = KneserNey(name_tokens("train"), 4)tokenizer = bpe.store_tokenizer()as_counted = [{transformer.SEP: "</s>"}.get(token, token) for token in vocabulary]   # <sep> ends a name, as </s> doesshow = lambda token: token.replace("</w>", "")for prefix in ["bar", "patio bar", "outdoor bar", "kitchen bar", "wine bar"]:    tokens = tokenizer.encode(prefix)    ids = [index[transformer.START]] + [index[token] for token in tokens]    from_transformer = transformer.softmax(transformer.forward(ids)[-1])    context = tuple(["<s>"] * 3 + tokens)[-3:]   # the 4-gram looks at the last three tokens, padded with start markers for a short prefix    from_counts = np.array([counted.probability(context, token) for token in as_counted])    mixed = (from_transformer + from_counts) / 2    for label, p in [("transformer", from_transformer), ("half and half", mixed)]:        top = np.argsort(-p, kind="stable")[:3]   # the three likeliest; a stable sort keeps tied tokens in vocabulary order        print(f"{prefix:<12} {label:<13} -> " + ", ".join(f"{show(vocabulary[i])} {p[i]:.2f}" for i in top))    if prefix == "outdoor bar":        stool = index["stool</w>"]   # </w> marks the end of a word in the tokenizer, so this is the whole word stool        print("P(stool | outdoor bar), transformer:", f"{from_transformer[stool]:.2f}")        print("P(stool | outdoor bar), 4-gram:", f"{from_counts[stool]:.2f}")        print("P(stool | outdoor bar), half and half:", f"{mixed[stool]:.2f}")
輸出python transformer-autocomplete.py
bar          transformer   -> with 0.17, & 0.14, and 0.09
bar          half and half -> & 0.19, with 0.14, unit 0.07
patio bar    transformer   -> stool 0.51, table 0.25, set 0.02
patio bar    half and half -> stool 0.69, table 0.13, & 0.01
outdoor bar  transformer   -> stool 0.50, with 0.05, table 0.04
outdoor bar  half and half -> stool 0.28, & 0.12, with 0.12
P(stool | outdoor bar), transformer: 0.50
P(stool | outdoor bar), 4-gram: 0.06
P(stool | outdoor bar), half and half: 0.28
kitchen bar  transformer   -> room 0.14, with 0.08, cabinet 0.08
kitchen bar  half and half -> with 0.14, & 0.12, room 0.07
wine bar     transformer   -> with 0.43, <sep> 0.22, cabinet 0.08
wine bar     half and half -> with 0.30, <sep> 0.21, cabinet 0.13

After outdoor bar, the transformer gives stool 0.50, where the 4-gram gives it 0.06; the mix, at 0.28, still puts it first. The control prefixes show the models aren't suggesting stool after every bar: after kitchen bar and wine bar they suggest other continuations, and after wine bar both expect, with good reason, that the name may already be complete. After patio bar, a context the names contain often, the mix is more confident in stool than the transformer alone.

One prefix is an example, not a measure. The measure is the names comparison, and there the mix won. For the search box, that's the model to serve.

Working rule: Serve the mix for autocomplete, and test it on rare prefixes as well as common ones, with control prefixes beside each. Treat a single striking suggestion as an illustration of a measured gain, never as the evidence for it.

正式環境中會出什麼問題

訓練出問題時,過程中往往看不出異狀:

  • 填充(padding)浪費運算。 這裡每一步的預測中,將近一半是填充。在購買更多運算資源之前,先將文本打包,或依長度分組。
  • 排程的錯誤看起來像模型的問題。 依經驗法則,暖機階段永遠不結束,或衰減開始得太早,都會使損失下降得太慢;上方的衰減比較顯示了排程能使結果改變多少。請儲存步長,並對照你原本打算使用的排程加以檢查。
  • 以驗證資料做出的選擇會讓結果看起來比實際更好。 保留的訓練週期與混合比例都是在驗證集上選出的,因此其數字略微偏於樂觀。請保留一份測試集,直到最後都不要動用。
  • 重複的文本會被記住。 小型資料集被看過許多次後,訓練損失看起來良好,但在新文本上表現不佳。請計算訓練看過每段文本的次數,並且寧可取得新文本,也不要增加訓練週期。
  • 只用一種文本來衡量一個模型。 Transformer 在名稱上排名第二,次於計數方法,在描述上則排名第一。在選擇要部署哪個模型之前,請在每個模型將要服務的各種文本上分別評分。
  • 結果取決於機器。 PyTorch 不保證不同版本或平台之間的結果完全相同(可重現性)。請記錄權重、曲線與確切的腳本,就像 recordings/transformer.py 以腳本的雜湊值所做的那樣,並將它們放在一起保存。

此部分尚未翻譯,因此以英文呈現。

What this chapter decided

  • The store's model is trained in one recorded run: batches of 32 product texts, AdamW with weight decay 0.1, gradients clipped, a warm-up of 200 steps to a peak step size of 0.003, cosine decay to a tenth, and the weights kept from epoch 15 of 16.
  • Every comparison counts the same predictions. On names, both models predict each name's tokens and its end, against chapter 3's names-only baseline, the stronger one for names. On descriptions, a 4-gram trained on the same texts scores the same tokens, with the same text before them.
  • On the validation names, the transformer alone roughly ties the 4-gram baseline, and mixed half and half with it scores 18.4, the best on names in this book so far. The mix is the model the book uses for autocomplete, replacing chapter 5's mix of the window model and the counts.
  • From this chapter on, models are also measured on the validation descriptions. There the transformer beats the baseline alone, 20.1 against 22.5, by winning where the counts haven't seen the context, and the mix scores 14.7.
  • At this scale, new text is the cheapest gain, width the largest at a price, and repetition helps only for a few passes. The recorded model stays at width 128, because every example from chapter 6 on reads it and the examples have to run in seconds on a laptop; a wider one would score better.

本章,於本版次中

版次 2026.11.16範例於 2026年11 月 19 通過測試本章 7 個範例 個範例中有 7 個通過

譯自英文書,版次 2026.11.16。

引用本節

格式

各種格式皆引用英文原書,本譯本即依循該原書。

每種格式都會標示本版。其連結會開啟目前版次,節標題會說明該節內文最後一次變動的時間。存取日期與擷取日期即為你開啟此頁的當日。

本頁內容

第 8 章,共 11 章: 訓練小型模型

  1. 將商品轉為批次
  2. 使用 AdamW 與梯度裁剪進行更新
  3. Warm up, then decay the step size
  4. 解讀訓練曲線
  5. 與名稱的基準模型比較
  6. Compare on descriptions
  7. 寬度、資料與訓練步數能換來什麼
  8. Suggest the next word in the search box
  9. 正式環境中會出什麼問題
  10. What this chapter decided

返回頂端所有章節