训练一个小模型
阅读约需 45分钟 分钟,10节中含7个示例。
学完本章,你可以从头到尾训练一个小型 transformer,并对结果做出判断。你将把文本切成批次,用 AdamW 更新权重,设定步长的调度方式,阅读训练曲线,拿计数基线对模型做公平的度量,弄清在这个规模下,更大的宽度、更多的文本和更长的训练各能带来什么,并找出模型在哪些地方胜过计数,又在哪些地方仍需要计数相伴。
第 7 章组装了商店的模型,并用它训练好的权重检查了每一个部分。本章要问的是:这些权重从何而来?答案是一次训练运行,记录在 recordings/transformer.py 中。那次运行教会模型预测商店目录中的每一个下一个词元,目录里是商店在线销售的家具和家居用品的名称与描述。随后,本章在模型要完成的两项任务上对它进行度量:在搜索框中建议商品名称的下一个词,以及预测描述,因为描述的文本足够长,远处的上下文会起作用。
与第 5 章一样,训练只在 PyTorch 中运行了一次。示例读取的是它保存下来的内容:权重、曲线和设置,这些都记录在该次运行的清单中。如果想自己训练,请在单独的环境中安装 recordings/requirements.txt,这会装上 PyTorch 的 CPU 版本,然后在本书所在的文件夹中运行 python recordings/transformer.py:它会把同样的文件写入 examples/data/recorded/transformer/,并且在笔记本电脑的处理器上就能运行。
将商品转为批次
训练的第一步,是把商店的商品目录转成模型可以学习的数字。每件商品成为一条文本:起始标记、名称、分隔符 <sep>、描述和结束标记,由商店的分词器切分为词元。超过模型长度上限的文本会被截断。每个训练步一次取一个批次的文本,使运算可以在整块矩阵上进行;长度不同的文本,则用填充词元补齐到该批次中最长文本的长度。
随后,每条文本在每个位置上都让模型做出一个预测。输入是文本去掉最后一个词元后的部分,目标是文本去掉第一个词元后的部分,因此位置 i 被训练去预测第 i + 1 个词元。由于有因果掩码,一次前向传播就能同时完成所有这些预测。损失只在非填充的目标上取平均:填充会消耗计算,却不教给模型任何东西。
填充是需要留意的开销,因此按训练脚本的做法构建一个轮次中经过打乱的批次,并统计每个批次中有多少是填充:
1"""Builds one shuffled epoch of batches the way the training script does, and measures the padding in them.2The prose's point rests on the share of targets that are padding over that epoch."""3import math45import numpy as np67from lib import transformer8from lib.products import product_texts910_, vocabulary, manifest = transformer.load()11settings = manifest["settings"]12pad, end = vocabulary.index(transformer.PAD), vocabulary.index(transformer.END)13texts = [ids for ids, _ in product_texts("train")]14size = settings["batch"]1516order = np.random.default_rng(settings["seed"]).permutation(len(texts)) # one shuffled epoch, from the run's seed, so it repeats17padding, targets_seen, shares = 0, 0, []18for start in range(0, len(texts), size):19 batch = [texts[i] for i in order[start:start + size]] # this step's texts: the next `size` in the shuffled order20 longest = max(len(ids) for ids in batch)21 padded = np.array([ids + [pad] * (longest - len(ids)) for ids in batch]) # short texts filled out with <pad>22 inputs, targets = padded[:, :-1], padded[:, 1:] # each position predicts the next token23 padding, targets_seen = padding + (targets == pad).sum(), targets_seen + targets.size24 shares.append((targets == pad).mean())25 if start == 0:26 first_shape = (inputs.shape, targets.shape)2728print("training texts:", f"{len(texts):,}")29print("tokens in them:", f"{sum(len(ids) for ids in texts):,}")30print(f"texts cut at the length limit, {settings['longest']} tokens:", f"{np.mean([ids[-1] != end for ids in texts]):.1%}") # a cut text lost its end marker31print("texts per batch:", size)32print("one batch's inputs and targets:", *first_shape)33print("share of targets that are padding, over one shuffled epoch:", f"{padding / targets_seen:.0%}")34print("least and most padding in a batch:", f"{min(shares):.0%} and {max(shares):.0%}")35print("steps per epoch:", f"{math.ceil(len(texts) / size):,}") # the last, partly filled batch is a step too36print("epochs in the recorded run:", settings["epochs"])37print("steps in the recorded run:", f"{manifest['steps']:,}")
python training-batches.pytraining texts: 34,252
tokens in them: 2,843,222
texts cut at the length limit, 160 tokens: 1.0%
texts per batch: 32
one batch's inputs and targets: (32, 151) (32, 151)
share of targets that are padding, over one shuffled epoch: 46%
least and most padding in a batch: 30% and 59%
steps per epoch: 1,071
epochs in the recorded run: 16
steps in the recorded run: 17,136
训练集包含 34,252 条文本、2,843,222 个词元。这个数目包含每条文本的起始、分隔和结束标记,而第 4 章的统计所得的 2,744,320 个词元不含这些标记;同时,这个数目不含被长度上限截去的部分,被截断的文本只有 1.0%。
每个批次包含 32 条文本,因此把训练文本完整过一遍,即一个轮次,需要 1,071 步;记录下来的那次运行共 17,136 步,历时 16 个轮次。
在一个经过打乱的轮次中,有 46% 的目标是填充,各批次之间的比例在 30% and 59% 之间,因为一条很长的描述就决定了与它同批的所有较短文本的宽度。每个批次是一个矩形:每条文本一行,每个位置一列。
这种规模的填充是浪费:每次前向传播中近一半的计算,得出的是随后被损失忽略的预测。它不改变模型学到的内容,因为填充从不计入损失,但会使每一步的开销几乎翻倍。Krell 等人发现,常见数据集中填充可占全部词元的一半之多,并提出了打包:把若干条文本首尾相接放进同一行,同时使注意力不跨越文本之间的边界(Krell et al., 2021)。把长度相近的文本放进同一批次,同样可以减少填充。
工作准则: 以输入和错位一位的目标来构建批次,把填充排除在损失之外,并在每个轮次用固定的随机种子打乱文本,使运行可以复现。测量批次中填充所占的比例;如果比例很大,先打包文本或按长度分组,再考虑购买更大的机器。
使用 AdamW 与梯度裁剪进行更新
批次备好之后,每一步都是第 5 章中的循环:前向传播、损失、梯度、更新。recordings/transformer.py 中的下面这几行就是全部内容,每个批次运行一次:
ids = pad([train[i] for i in order[start:start + SETTINGS["batch"]]])
logits = model(ids[:, :-1])
loss = torch.nn.functional.cross_entropy(logits.transpose(1, 2), ids[:, 1:], ignore_index=index[PAD])
optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
optimizer.step()
scheduler.step()
前三行即上一节的内容:一个经过填充的批次,去掉最后一个词元的输入,以及在移位后的目标上计算、并忽略填充的损失。有两行与第 5 章不同。
优化器是 AdamW,即第 5 章中的 Adam,并将权重衰减与梯度分开施加。权重衰减在每一步中把每个权重减去其自身的值乘以步长再乘以衰减系数,这里的衰减系数为 0.1(PyTorch 的 AdamW)。Loshchilov 和 Hutter 证明,与把衰减加到损失中相比,以这种方式施加衰减在配合 Adam 时效果更好(Loshchilov and Hutter, 2019)。衰减是一种正则化,而第 5 章的窗口模型缺少这一点:它把每个权重拉向零,因此只有训练文本持续推动某个权重时,它才会保持较大。
训练脚本还会对梯度进行裁剪:当梯度的总长度(把所有梯度视作一个长向量来度量)超过 1.0 时,把它们统一缩小到该长度(PyTorch 的 clip_grad_norm_)。这样,一个异常的批次就不会把权重甩得很远。Pascanu 等人正是为此提出了裁剪:罕见而极大的梯度,否则会抵消训练已学到的大部分内容(Pascanu et al., 2013)。
**工作准则:**使用权重衰减适中的 AdamW,并按梯度的合长度进行裁剪。让训练步骤短到可以完整读完,使其中的每个选择都清晰可见:损失忽略什么、优化器衰减什么、裁剪的又是什么。
先预热,再衰减步长
循环的最后一行 scheduler.step() 会在每个批次之后改变步长,所依据的是一个调度(schedule),它为运行中的每一步设定步长。在最初的 200 步中,步长从接近零开始沿直线上升到峰值 0.003:这是一次预热(warm-up),Transformer 论文的调度即是如此(Vaswani et al., 2017,第 5.3 节)。在 Adam 的最初几步,其平方梯度的滑动平均只依赖于少数几个批次,因此它的步长波动很大;Liu 等人表明,预热的作用在于降低这种方差(Liu et al., 2020)。预热结束后,步长沿半条余弦曲线下降,到最后一步降至峰值的十分之一(Loshchilov and Hutter, 2017)。早期的大步长用来快速推进;后期的小步长让权重沉降到低点,而不是在低点周围来回跳动。用公式表示,预热之后:
rate = peak × (0.1 + 0.9 × ½ (1 + cos(π × progress)))
其中 progress 从预热结束时的 0 增长到最后一步时的 1。
调度是否与这次运行相符,是可以检验的:根据记录下来的设置手工计算一遍,再与训练脚本每隔十步保存的步长对照:
1"""Computes the warm-up and cosine schedule by hand from the recorded settings, and checks it against the step sizes2the training script saved. The prose relies on the check: the assert fails the run if the two ever differ."""3import json4import math56from lib import transformer78_, _, manifest = transformer.load()9settings, total = manifest["settings"], manifest["steps"]10recorded_rates = json.loads((transformer.RECORDED / "curves.json").read_text())["rates"] # every 10th step's, after the scheduler's step111213def rate(step):14 """Warm up linearly to the peak, then follow half a cosine down to a tenth of it."""15 if step < settings["warmup_steps"]:16 return settings["peak_rate"] * (step + 1) / settings["warmup_steps"] # steps count from 0, so step 0 is already a share of the peak17 progress = (step - settings["warmup_steps"]) / (total - settings["warmup_steps"]) # 0 at the warm-up's end, 1 at the last step18 return settings["peak_rate"] * (0.1 + 0.9 * 0.5 * (1 + math.cos(math.pi * progress)))192021print("warm-up steps:", settings["warmup_steps"])22print("peak step size:", settings["peak_rate"])23print("weight decay:", settings["weight_decay"])24for step in [0, settings["warmup_steps"] // 2, settings["warmup_steps"], total // 2, total - 1]:25 print(f"step size at step {step:,}:", f"{rate(step):.2e}")26gap = max(abs(recorded - rate(10 * i + 1)) for i, recorded in enumerate(recorded_rates)) # recorded_rates[i] was saved after step 10 * i, so it is the size used at step 10 * i + 127assert gap < 1e-9 # every saved step size matches the hand calculation, to a billionth28print("same as the recorded step sizes:", "yes")
python learning-rate-schedule.pywarm-up steps: 200
peak step size: 0.003
weight decay: 0.1
step size at step 0: 1.50e-05
step size at step 100: 1.51e-03
step size at step 200: 3.00e-03
step size at step 8,568: 1.68e-03
step size at step 17,135: 3.00e-04
same as the recorded step sizes: yes
手工计算的结果与保存的每一个步长都吻合。步长起始于 1.50e-05,在预热结束时达到峰值 3.00e-03,到最后一步为 3.00e-04,恰为该峰值的十分之一。
衰减很重要。对照运行以相同的宽度、相同的种子和相同的批次训练了四个轮次,因此它的调度在第 4,284 步完成衰减,在这一步,它在验证集名字上的困惑度为 27.6。而记录下来的那次运行在这一步仍接近其峰值步长,验证困惑度为 32.6。数据相同,步数相同:差别就在衰减。
工作准则: 在最初的几百步里把步长预热上去,然后再衰减,并使调度的长度与运行的长度相匹配。像训练脚本那样,在训练过程中保存步长,并与你原本打算使用的调度对照检查:调度方面的错误看上去像是训练问题,用这种方式要容易发现得多。
读训练曲线
在批次、更新和学习率计划都设定之后,商店的模型训练了 16 个轮次,脚本在每个轮次之后,分别在名称和描述上用验证集对它做了度量。它保留的是名称与描述验证损失相加后最低的那个轮次的权重,做法与第 5 章保留最佳轮次相同。
按每个词元的困惑度来读,记录下来的曲线逐轮次如下:
1"""Prints the recorded run's perplexity for each epoch, then the epoch kept and its validation perplexities.2The prose's point rests on the two validation columns and the epoch kept."""3import json4import math56from lib import transformer78_, _, manifest = transformer.load()9curves = json.loads((transformer.RECORDED / "curves.json").read_text())["epochs"]1011print("perplexity per token; training mixes names and descriptions, as the training text does")12print("epoch training validation names validation descriptions")13for curve in curves:14 print(f"{curve['epoch']:>5} {math.exp(curve['train_loss']):>8.1f} {math.exp(curve['validation_name_loss']):>16.1f}"15 f" {math.exp(curve['validation_description_loss']):>23.1f}")16kept = next(curve for curve in curves if curve["epoch"] == manifest["kept_epoch"]) # the epoch whose weights were kept17print("epoch kept, the lowest name and description losses added together:", manifest["kept_epoch"])18print("validation perplexity on names, kept epoch:", f"{math.exp(kept['validation_name_loss']):.1f}")19print("validation perplexity on descriptions, kept epoch:", f"{math.exp(kept['validation_description_loss']):.1f}")20print("validation perplexity on names, epoch 4:", f"{math.exp(curves[3]['validation_name_loss']):.1f}") # the list starts at epoch 1, so index 3 is epoch 4
python transformer-training-curve.pyperplexity per token; training mixes names and descriptions, as the training text does
epoch training validation names validation descriptions
1 131.4 50.5 50.4
2 41.4 38.8 36.2
3 32.8 34.5 31.5
4 29.0 32.6 29.3
5 26.6 30.9 27.8
6 24.8 30.2 26.4
7 23.3 28.9 25.0
8 21.8 28.1 23.9
9 20.5 27.1 22.9
10 19.2 26.5 22.2
11 17.9 25.8 21.6
12 16.8 25.1 20.8
13 15.8 24.6 20.4
14 14.9 24.2 20.0
15 14.2 24.0 19.9
16 13.8 24.1 19.9
epoch kept, the lowest name and description losses added together: 15
validation perplexity on names, kept epoch: 24.0
validation perplexity on descriptions, kept epoch: 19.9
validation perplexity on names, epoch 4: 32.6
验证困惑度一直下降到第 15 轮,也就是保留的那个轮次,然后趋于平稳:最后一个轮次并没有更好。在保留的轮次上,名称的验证困惑度是 24.0,描述的是 19.9。与第 5 章一样,第 1 轮的训练困惑度高于验证困惑度,因为它是对该轮次最初几个批次求的平均,而那时模型还一无所知。训练列像训练文本一样把名称和描述混在一起,所以它显示的是趋势,而不是可以与任一验证列并列比较的数值。
第 5 章的窗口模型在第 5 轮之后就不再改进;这个模型则在长得多的时间里持续改进。两次运行在好几个方面不同,都可能解释这一点。这个模型从名称和描述中学习,共 2,843,222 个词元,而窗口模型只从名称中学习,有 337,071 个词元;它的步长会衰减(学习率逐步降低);它使用权重衰减;注意力在每个位置上应用相同的权重。这次运行并不能把这些原因区分开。它确实能显示的是训练与验证之间的差距,这一差距每个轮次都在扩大:到最后,模型对训练文本的预测远好于对新文本的预测,所以它已开始记忆。
描述的得分比名称低,但这并不意味着模型更擅长描述。每一种文本都有自己的可预测性:描述会重复其模板和名称中的短语,而名称则用寥寥数词概括一个新产品。困惑度用来比较不同模型在同一文本上的表现,绝不能用来相互比较不同的文本。
工作准则: 对你关心的每一种文本,分别记录每个轮次的验证损失,并保留最佳轮次的权重。要把训练与验证之间的差距当作警示来读,而不能只看训练损失。绝不要跨不同种类的文本比较困惑度。
与基线在人名上的比较
训练读完之后,就可以衡量结果了。在人名上,最强的计数模型是用训练集人名训练的 4-gram 基线。它在验证集人名上的得分为 23.1,这是本书的衡量标尺:自第 3 章以来,每个模型都在这些人名上打分。同样的 4-gram 若用完整文本(人名和描述)训练,在这些人名上的得分为 27.7,原因是描述稀释了它对人名构成方式的计数。
比较必须针对相同的预测。4-gram 预测每个人名的各个词元,然后预测其结束标记;Transformer 预测每个人名的各个词元,然后预测 <sep>,后者起的作用相同,因为它标出了人名在哪里结束。人名之后不需要任何内容,所以 Transformer 只读取人名本身。两个模型对同样的 46,704 个预测打分。
先来猜一猜:仅靠 Transformer,能否击败第 5 章的窗口模型未能击败的计数基线?然后像第 5 章那样,为这两个模型以及二者的混合模型打分:
1"""Scores every validation name, its end included, with the 4-gram baseline and with the transformer, then mixes them2at several shares. The prose's point rests on the share 0.5 line, the lowest of the five."""3import numpy as np45from lib import transformer6from lib.names import name_tokens7from lib.ngrams import KneserNey, ngrams, padded89_, vocabulary, _ = transformer.load()10index = {token: i for i, token in enumerate(vocabulary)} # each token's id in the vocabulary, which is its column in the model's output11counted = KneserNey(name_tokens("train"), 4) # chapter 3's baseline, trained on names12from_counts, from_transformer = [], []13for name in name_tokens("validation"):14 ids = [index[transformer.START]] + [index[token] for token in name] + [index[transformer.SEP]] # the name as the model reads it15 probabilities = transformer.softmax(transformer.forward(ids)[:-1]) # the last position predicts past <sep>, which a name never needs16 from_transformer += list(probabilities[np.arange(len(ids) - 1), ids[1:]]) # each position's probability for the token after it17 from_counts += [counted.probability(context, token) for context, token in ngrams(padded(name, 4), 4)] # then </s>18from_counts, from_transformer = np.array(from_counts), np.array(from_transformer)1920print("validation tokens scored, each name's end included:", f"{len(from_counts):,}")21for share in [0.0, 0.3, 0.5, 0.7, 1.0]:22 mixed = (1 - share) * from_counts + share * from_transformer23 print(f"validation perplexity on names, transformer share {share:.1f}:", f"{np.exp(-np.log(mixed).mean()):.1f}")
python transformer-on-names.pyvalidation tokens scored, each name's end included: 46,704
validation perplexity on names, transformer share 0.0: 23.1
validation perplexity on names, transformer share 0.3: 18.7
validation perplexity on names, transformer share 0.5: 18.4
validation perplexity on names, transformer share 0.7: 18.8
validation perplexity on names, transformer share 1.0: 24.0
Transformer 与计数各占一半混合时,在人名上的得分为 18.4,是本书中所有模型在人名上的最佳成绩。Transformer 单独使用时得分为 24.0,略差于基线的 23.1。
单独的 Transformer 远好于第 5 章的窗口模型(其得分为 36.4),而混合模型也胜过第 5 章的最佳混合结果 21.9。
因此在人名上,单靠计数仍然很难被超越,原因正如第 5 章所述:人名短小且有固定格式,三个词元的上下文已经涵盖了人名所含的大部分信息。但 Transformer 出错的地方与计数不同,混合模型从两者中都获得了增益。混合比例是在验证集人名上选定的,因此这一增益略显乐观;测试集人名保持原封不动,留作本书最后的比较。
**工作准则:**用你能构建的最强基线与新模型比较,并且要针对完全相同的预测,包括各自如何标记文本的结束。当学习得到的模型与计数模型表现接近时,把它们混合起来,并在留出的文本上选择混合比例。
这部分尚未翻译,因此以英文显示。
Compare on descriptions
The names comparison favours counting, because names are short. Descriptions run many times longer, and chapter 6 showed the model copying from the name many tokens back, so this is where its longer reach should pay. A fair baseline here is a 4-gram trained on the same texts the transformer learned from, the name, <sep> and description joined, and scored on the same tokens: each description's tokens and its end, with the same text before them.
Put the longer reach to the test on descriptions: fit the 4-gram on every training text, then score it, the transformer and an even mix on the descriptions of the first 500 validation products, leaving out texts cut at the length limit and products with no description. It also splits the tokens, as chapter 5 did, by whether their three-token context occurs in the training texts:
1"""Scores the descriptions of the first 500 validation products with a 4-gram trained on whole texts, with the transformer,2and with their even mix, then splits the tokens by whether their context was seen in training. The prose's point rests on the34-gram baseline and transformer lines, the half-and-half line and the unseen-context lines."""4import numpy as np56from lib import transformer7from lib.names import name_tokens8from lib.ngrams import KneserNey, ngrams, padded9from lib.products import product_texts1011_, vocabulary, _ = transformer.load()12end = vocabulary.index(transformer.END)13as_tokens = lambda ids: [vocabulary[i] for i in ids[1:] if i != end] # padded() adds its own <s> and </s>14counted = KneserNey([as_tokens(ids) for ids, _ in product_texts("train")], 4) # whole texts: name, <sep>, description1516from_counts, from_transformer, seen, products = [], [], [], 017for ids, sep in product_texts("validation", 500):18 if ids[-1] != end or sep == len(ids) - 2:19 continue # cut at the length limit, or no description20 pairs = list(ngrams(padded(as_tokens(ids), 4), 4))[sep:] # from <sep> on: the description's tokens, then </s>21 from_counts += [counted.probability(context, token) for context, token in pairs]22 seen += [context in counted.totals[4] for context, _ in pairs] # was this 3-token context in the training texts?23 probabilities = transformer.softmax(transformer.forward(ids)[sep:-1]) # from <sep> on: each position predicts the next token24 from_transformer += list(probabilities[np.arange(len(probabilities)), ids[sep + 1:]]) # the description starts after <sep>, at sep + 125 products += 126from_counts, from_transformer, seen = np.array(from_counts), np.array(from_transformer), np.array(seen)27perplexity = lambda p: f"{np.exp(-np.log(p).mean()):.1f}"2829print("products scored:", products)30print("description tokens scored:", f"{len(from_counts):,}")31mixed = (from_counts + from_transformer) / 232for label, p in [("4-gram baseline", from_counts), ("transformer", from_transformer), ("half and half", mixed)]:33 print(f"{label}, validation perplexity on descriptions:", perplexity(p))34print("description tokens whose 3-token context is in the training texts:", f"{seen.mean():.0%}")35for part, where in [("seen", seen), ("unseen", ~seen)]:36 print(f"4-gram, {part} contexts:", perplexity(from_counts[where]))37 print(f"transformer, {part} contexts:", perplexity(from_transformer[where]))3839names = [list(ngrams(padded(name + [transformer.SEP], 4), 4))[:-1] for name in name_tokens("validation")] # to <sep>: the last pair predicts </s>, which a name stops short of40on_names = [counted.probability(context, token) for pairs in names for context, token in pairs]41print("this 4-gram, trained on whole texts, validation perplexity on names:", perplexity(np.array(on_names)))
python transformer-on-descriptions.pyproducts scored: 446
description tokens scored: 35,651
4-gram baseline, validation perplexity on descriptions: 22.5
transformer, validation perplexity on descriptions: 20.1
half and half, validation perplexity on descriptions: 14.7
description tokens whose 3-token context is in the training texts: 74%
4-gram, seen contexts: 16.3
transformer, seen contexts: 16.6
4-gram, unseen contexts: 57.6
transformer, unseen contexts: 35.0
this 4-gram, trained on whole texts, validation perplexity on names: 27.7
On descriptions, the transformer beats counting alone: it scores 20.1 on the 35,651 description tokens of 446 products, against the 4-gram's 22.5. It is the first learned model in this book to beat a counted model of the same text on its own.
The split shows where they differ. 74% of the tokens have a context the training texts contain, and on those the two models tie: 16.3 for the 4-gram, 16.6 for the transformer. On the rest, the transformer scores 35.0 where the 4-gram scores 57.6. Descriptions are formulaic enough that counts do as well as the transformer wherever they have seen the context; where they haven't, the transformer reads further back and relates what it sees to what it learned, and the counts can only back off.
That's also why the mix works so well. Half and half, the two score 14.7, far below either alone: on any one token, at least one of the two is often confident and right, and the even mix keeps half of that probability, enough to avoid the worst surprises of each.
Working rule: Measure a model on the kind of text it will predict, against a baseline that sees the same text, and expect the ranking to change from one kind of text to another. Split the comparison by seen and unseen contexts to see where each model wins. Try a mix before concluding that either model is enough: the mix gained more than any change to either model that this chapter measured.
宽度、数据和步数能换来什么
已记录的模型只是众多选择中的一种:宽度为 128 个数,在全部训练文本上训练 16 个 epoch。对大得多的模型所做的研究发现,随着模型变大、见到的文本变多、训练时间变长,损失会平滑下降,而且更大的模型从每个词元中学到的东西更多(Kaplan et al., 2020)。在计算预算固定的情况下,Hoffmann 等人发现,模型规模与训练文本应当一同增长(Hoffmann et al., 2022)。Muennighoff 等人则发现,把同一份文本重复最多约四个 epoch,效果几乎与新文本相当,此后每多重复一次,收益就更小(Muennighoff et al., 2023)。这些结论在这个玩具规模上是否成立,需要测量,而不能假定。
因此,训练脚本在主运行之外还记录了七次成本更低的运行,每次只改变一个因素:
- 三种宽度,64、128 和 256,各训练四个 epoch,每个头都保持 32 个数宽;
- 训练商品的随机四分之一和随机一半,各训练四个 epoch;
- 同样的四分之一和一半,通过更频繁地重复,训练到与全部数据四个 epoch 相同的步数。
哪一种改变收益最大?把八次运行并排来看,每次运行的规模、步数,以及最佳 epoch 和最后一个 epoch 的困惑度如下:
1"""Reads eight recorded runs and prints each one's parameters, steps, best epoch, and validation perplexities at its best2and last epochs. The prose's point rests on the perplexity-on-names lines."""3import json4import math56from lib import transformer78runs = [("width 64, 4 epochs", "transformer-width-64"), ("width 128, 4 epochs", "transformer-width-128"),9 ("width 256, 4 epochs", "transformer-width-256"), ("width 128, 16 epochs", "transformer"),10 ("a quarter of the data, 4 epochs", "transformer-data-25"), ("half the data, 4 epochs", "transformer-data-50"),11 ("a quarter of the data, 16 epochs", "transformer-data-25-same-steps"),12 ("half the data, 8 epochs", "transformer-data-50-same-steps")]1314for label, folder in runs:15 run = transformer.RECORDED.parent / folder # every run sits in the folder beside the main run's16 manifest = json.loads((run / "manifest.json").read_text())17 curves = json.loads((run / "curves.json").read_text())["epochs"]18 best = min(curves, key=lambda curve: curve["validation_name_loss"] + curve["validation_description_loss"]) # the epoch the script kept19 print(f"{label}, parameters:", f"{manifest['parameters']:,}")20 print(f"{label}, steps:", f"{manifest['steps']:,}")21 print(f"{label}, best epoch:", best["epoch"])22 print(f"{label}, perplexity on names:", f"{math.exp(best['validation_name_loss']):.1f}")23 print(f"{label}, perplexity on descriptions:", f"{math.exp(best['validation_description_loss']):.1f}")24 print(f"{label}, perplexity on names, last epoch:", f"{math.exp(curves[-1]['validation_name_loss']):.1f}") # the last epoch trained, not the one kept25 print(f"{label}, perplexity on descriptions, last epoch:", f"{math.exp(curves[-1]['validation_description_loss']):.1f}")
python width-data-and-steps.pywidth 64, 4 epochs, parameters: 478,656
width 64, 4 epochs, steps: 4,284
width 64, 4 epochs, best epoch: 4
width 64, 4 epochs, perplexity on names: 36.9
width 64, 4 epochs, perplexity on descriptions: 35.9
width 64, 4 epochs, perplexity on names, last epoch: 36.9
width 64, 4 epochs, perplexity on descriptions, last epoch: 35.9
width 128, 4 epochs, parameters: 1,334,144
width 128, 4 epochs, steps: 4,284
width 128, 4 epochs, best epoch: 4
width 128, 4 epochs, perplexity on names: 27.6
width 128, 4 epochs, perplexity on descriptions: 24.7
width 128, 4 epochs, perplexity on names, last epoch: 27.6
width 128, 4 epochs, perplexity on descriptions, last epoch: 24.7
width 256, 4 epochs, parameters: 4,273,920
width 256, 4 epochs, steps: 4,284
width 256, 4 epochs, best epoch: 4
width 256, 4 epochs, perplexity on names: 23.5
width 256, 4 epochs, perplexity on descriptions: 19.5
width 256, 4 epochs, perplexity on names, last epoch: 23.5
width 256, 4 epochs, perplexity on descriptions, last epoch: 19.5
width 128, 16 epochs, parameters: 1,334,144
width 128, 16 epochs, steps: 17,136
width 128, 16 epochs, best epoch: 15
width 128, 16 epochs, perplexity on names: 24.0
width 128, 16 epochs, perplexity on descriptions: 19.9
width 128, 16 epochs, perplexity on names, last epoch: 24.1
width 128, 16 epochs, perplexity on descriptions, last epoch: 19.9
a quarter of the data, 4 epochs, parameters: 1,334,144
a quarter of the data, 4 epochs, steps: 1,072
a quarter of the data, 4 epochs, best epoch: 4
a quarter of the data, 4 epochs, perplexity on names: 51.9
a quarter of the data, 4 epochs, perplexity on descriptions: 54.1
a quarter of the data, 4 epochs, perplexity on names, last epoch: 51.9
a quarter of the data, 4 epochs, perplexity on descriptions, last epoch: 54.1
half the data, 4 epochs, parameters: 1,334,144
half the data, 4 epochs, steps: 2,144
half the data, 4 epochs, best epoch: 4
half the data, 4 epochs, perplexity on names: 35.2
half the data, 4 epochs, perplexity on descriptions: 34.8
half the data, 4 epochs, perplexity on names, last epoch: 35.2
half the data, 4 epochs, perplexity on descriptions, last epoch: 34.8
a quarter of the data, 16 epochs, parameters: 1,334,144
a quarter of the data, 16 epochs, steps: 4,288
a quarter of the data, 16 epochs, best epoch: 11
a quarter of the data, 16 epochs, perplexity on names: 40.0
a quarter of the data, 16 epochs, perplexity on descriptions: 39.6
a quarter of the data, 16 epochs, perplexity on names, last epoch: 44.1
a quarter of the data, 16 epochs, perplexity on descriptions, last epoch: 45.9
half the data, 8 epochs, parameters: 1,334,144
half the data, 8 epochs, steps: 4,288
half the data, 8 epochs, best epoch: 8
half the data, 8 epochs, perplexity on names: 30.4
half the data, 8 epochs, perplexity on descriptions: 28.6
half the data, 8 epochs, perplexity on names, last epoch: 30.4
half the data, 8 epochs, perplexity on descriptions, last epoch: 28.6
宽度带来很大的帮助,但要付出代价。步数相同时,名称上的困惑度从宽度 64 时的 36.9,降到宽度 128 时的 27.6,再降到宽度 256 时的 23.5。在名称上,训练四个 epoch 的最宽模型,甚至胜过训练十六个 epoch 的已记录模型,后者为 24.0。但宽度 256 有 4,273,920 个参数,宽度 128 则为 1,334,144,每一步的开销也大致按此比例增加。
新文本不增加每一步的开销,却同样有帮助。步数相同时,商品的随机四分之一在名称上最好成绩为 40.0,一半为 30.4,全部为 27.6。文本从一半加倍到全部,收益小于宽度从 128 加倍到 256,但不需要额外计算;而第一次加倍,即从四分之一到一半,收益比这两者都大。
重复在一段时间内有帮助,之后反而有害。四分之一的文本看四遍,得分为 51.9;看得更多时,成绩一直改善到第 11 个 epoch,随后变差,十六遍之后以 44.1 结束:模型把见得过多的内容记住了。在步数相同的情况下,重复的文本始终没有赶上新文本。
更多步数帮助最小。已记录模型的十六个 epoch 比四个 epoch 有所改进,在名称上为 24.0,而后者为 27.6,代价是四倍的步数。
这些运行也显示了本书规模的局限。商店的训练文本固定为其目录所含的内容,即 2,843,222 个词元。SmolLM2-360M 的设计在第 7 章中与商店的模型做过比较,它在 4 万亿个词元上训练(模型卡),是前者的一百万倍以上,这使得一个宽 960 个数、深 32 层的模型能够不断从新文本中学习,而不是记住旧文本。
**工作准则:**在把已有文本重复几次以上之前,先获取更多你要预测的那类文本;这样做每一步都不增加开销。若计算资源有余,先尝试用更少的 epoch 训练更宽的模型,再考虑把窄模型训练更久,并对两者都做测量。每次运行只改变一个因素,并在关键之处保持步数不变,否则比较会同时测量两件事。
这部分尚未翻译,因此以英文显示。
Suggest the next word in the search box
Averages say how good a model is; the search box shows what that means to a shopper. Chapter 3's 4-gram suggested & and with after outdoor bar, a context no training name contains, and chapter 5's window model suggested stool. A few training descriptions do contain outdoor bar, and the transformer learned from them as well as from the names, while the 4-gram learned from names alone.
A shopper would see the transformer's three likeliest next tokens, and those of the half-and-half mix that the names comparison favoured, after the same five prefixes, each treated as the start of a name. A suggestion of <sep> means the model expects the name to end there:
1"""Asks the transformer and its half-and-half mix with the 4-gram for their three likeliest next tokens after five prefixes,2treated as the start of a name, and prints the probability of stool after outdoor bar. The prose's point rests on the3stool lines and the control prefixes beside them."""4import numpy as np56from lib import bpe, transformer7from lib.names import name_tokens8from lib.ngrams import KneserNey910_, vocabulary, _ = transformer.load()11index = {token: i for i, token in enumerate(vocabulary)}12counted = KneserNey(name_tokens("train"), 4)13tokenizer = bpe.store_tokenizer()14as_counted = [{transformer.SEP: "</s>"}.get(token, token) for token in vocabulary] # <sep> ends a name, as </s> does15show = lambda token: token.replace("</w>", "")1617for prefix in ["bar", "patio bar", "outdoor bar", "kitchen bar", "wine bar"]:18 tokens = tokenizer.encode(prefix)19 ids = [index[transformer.START]] + [index[token] for token in tokens]20 from_transformer = transformer.softmax(transformer.forward(ids)[-1])21 context = tuple(["<s>"] * 3 + tokens)[-3:] # the 4-gram looks at the last three tokens, padded with start markers for a short prefix22 from_counts = np.array([counted.probability(context, token) for token in as_counted])23 mixed = (from_transformer + from_counts) / 224 for label, p in [("transformer", from_transformer), ("half and half", mixed)]:25 top = np.argsort(-p, kind="stable")[:3] # the three likeliest; a stable sort keeps tied tokens in vocabulary order26 print(f"{prefix:<12} {label:<13} -> " + ", ".join(f"{show(vocabulary[i])} {p[i]:.2f}" for i in top))27 if prefix == "outdoor bar":28 stool = index["stool</w>"] # </w> marks the end of a word in the tokenizer, so this is the whole word stool29 print("P(stool | outdoor bar), transformer:", f"{from_transformer[stool]:.2f}")30 print("P(stool | outdoor bar), 4-gram:", f"{from_counts[stool]:.2f}")31 print("P(stool | outdoor bar), half and half:", f"{mixed[stool]:.2f}")
python transformer-autocomplete.pybar transformer -> with 0.17, & 0.14, and 0.09
bar half and half -> & 0.19, with 0.14, unit 0.07
patio bar transformer -> stool 0.51, table 0.25, set 0.02
patio bar half and half -> stool 0.69, table 0.13, & 0.01
outdoor bar transformer -> stool 0.50, with 0.05, table 0.04
outdoor bar half and half -> stool 0.28, & 0.12, with 0.12
P(stool | outdoor bar), transformer: 0.50
P(stool | outdoor bar), 4-gram: 0.06
P(stool | outdoor bar), half and half: 0.28
kitchen bar transformer -> room 0.14, with 0.08, cabinet 0.08
kitchen bar half and half -> with 0.14, & 0.12, room 0.07
wine bar transformer -> with 0.43, <sep> 0.22, cabinet 0.08
wine bar half and half -> with 0.30, <sep> 0.21, cabinet 0.13
After outdoor bar, the transformer gives stool 0.50, where the 4-gram gives it 0.06; the mix, at 0.28, still puts it first. The control prefixes show the models aren't suggesting stool after every bar: after kitchen bar and wine bar they suggest other continuations, and after wine bar both expect, with good reason, that the name may already be complete. After patio bar, a context the names contain often, the mix is more confident in stool than the transformer alone.
One prefix is an example, not a measure. The measure is the names comparison, and there the mix won. For the search box, that's the model to serve.
Working rule: Serve the mix for autocomplete, and test it on rare prefixes as well as common ones, with control prefixes beside each. Treat a single striking suggestion as an illustration of a measured gain, never as the evidence for it.
生产环境中会出什么问题
训练出问题时,过程中往往看不出异常:
- 填充浪费算力。 这里每一步的预测中,近一半是填充。在购买更多算力之前,先把文本打包,或按长度分组。
- 调度方式的错误看起来像模型的问题。 根据经验,预热一直不结束,或衰减开始得过早,都会表现为损失下降过慢;上文的衰减对比显示了调度方式能在多大程度上改变结果。请保存步长,并对照你本来想要的调度方式检查它。
- 依据验证数据做出的选择会使结果显得更好看。 保留哪个轮次和混合比例,都是在验证集上选定的,因此验证集上的数字略微偏乐观。请保留一份测试集,直到最后都不要动用。
- 重复出现的文本会被记住。 小数据集被反复看到许多次,训练损失看起来正常,而在新文本上表现很差。请统计训练中每条文本被看到的次数,并且宁可选用新文本,也不要增加轮次。
- 一个模型只在一类文本上得到衡量。 在名称上,Transformer 排在计数方法之后,位居第二;在描述上则排第一。在选择部署哪个模型之前,请在它将要服务的每一类文本上为每个模型打分。
- 结果依赖于机器。 PyTorch 不保证在不同版本或平台之间得到相同的结果(可复现性)。请记录权重、曲线和确切的脚本,就像
recordings/transformer.py通过脚本的哈希值所做的那样,并把它们保存在一起。
本章的决定
- **本店的模型在一次有记录的训练中完成:**每批 32 条商品文本,使用权重衰减为 0.1 的 AdamW,对梯度进行裁剪,先用 200 步预热,升至 0.003 的峰值步长,再按余弦曲线衰减到峰值的十分之一,并保留第 15 轮的权重,该次训练共 16 轮。
- **每一项比较所统计的预测都相同。**在名称上,两个模型都预测每个名称的各个词元及其结尾,对照的是第 3 章中仅用名称训练的基线,这是对名称而言更强的基线。在描述上,用相同文本训练的 4-gram 模型对相同的词元打分,其前文也相同。
- **在验证集的名称上,单独的 Transformer 与 4-gram 基线大致持平,而与基线各占一半混合后得分为 18.4,**这是本书迄今在名称上的最好成绩。这个混合模型就是本书用于自动补全的模型,取代了第 5 章中由窗口模型与计数混合而成的模型。
- **从本章起,模型也在验证集的描述上进行度量。**在描述上,Transformer 单独就胜过基线,为 20.1,对 22.5,原因是它在计数模型没有见过的上下文中占优;混合模型的得分为 14.7。
- **在这一规模下,新增文本是代价最低的收益,增加宽度的收益最大但要付出代价,重复训练只在少数几遍内有帮助。**记录在案的模型保持宽度 128,因为从第 6 章起的每个示例都要读取它,而这些示例必须能在笔记本电脑上数秒内运行;更宽的模型会得分更好。
本章,在本版次中
版次 2026.11.16示例于 2026年11月19日 通过本章 7个示例 个示例中有 7 个通过
译自英文版图书,版次 2026.11.16。