神经语言模型
阅读约需 46分钟 分钟,11节中含10个示例。
读完本章,你将能够构建一个神经语言模型,根据前面若干词元构成的窗口预测下一个词元;手工推导并检验它的梯度;用 PyTorch 训练它;并在公平的条件下与计数基线进行比较。你还会看到,学习得来的模型并不会因为是学出来的就占优:这个模型单独使用时不敌计数方法,而在两者混合时则有帮助。
第 3 章建立了本书的基线,即 4-gram Kneser-Ney 模型,它通过计数,由前面三个词元预测商店商品名称中的每个词元。它的局限在于无法把一个上下文与另一个上下文联系起来:在一家出售家具和家居用品的商店的名称中,它有 patio bar 的计数,却没有 outdoor bar 的计数,二者之间没有任何联系。第 4 章表明向量可以把词元联系起来,因为从商店文本中学得的 patio 和 outdoor 具有相近的向量。本章把向量放进模型之中。神经语言模型把前面的词元读作向量,用学习得到的运算将它们组合起来,并为词表中的每个词元给出一个概率。它通过训练来预测下一个词元,从而同时学习向量和运算。这一设计由 Bengio 等人提出(Bengio et al., 2003)。
文本由商店提供,检验方式则与第 3 章以来每个模型所面对的相同:经由商店的分词器,在验证集名称上计算每个词元的困惑度。困惑度越低越好,基线的得分为 23.1。
从嵌入窗口进行预测
本章的模型查看前面若干词元组成的固定窗口,这里取四个,与 5-gram 所统计的上下文相同。每个词元都要到嵌入表中查找,嵌入表是一个矩阵,词表中每个词元对应一行数字,与第 4 章中的做法一样,不同之处在于,这张表是为预测下一个词元而学习得到的,而不是由计数算出的。四行向量并排拼接成一个长向量 x。隐藏层由其输入 x 计算 h = tanh(xW + b):它先用权重矩阵 W 乘以 x,再加上偏置向量 b,然后把每个结果送入 tanh。tanh 是一个把任意数值压缩到 −1 至 1 范围内的函数,因此该层能够以单纯求和做不到的方式组合各个输入。随后,输出层由隐藏层的输出 h 计算 z = hU + c,为词表中的每个词元给出一个分数,这些分数称为 logits。
最后一步把分数转换为概率。Softmax 对每个 logit 取指数,再除以所有指数之和,因此结果均为正数且总和为一,分数越高,概率越大:
P(token i) = exp(zᵢ) / Σⱼ exp(zⱼ)
用训练好的权重逐步演算一次预测,可以看出模型如何把四个词元变成概率:这里预测的是以 patio bar 开头的名称之后的下一个词元。该示例的每一行对应上面的一个公式,其输出的形状则显示了每一步所做的事情:
1"""One prediction of the trained window model, computed by hand from its saved weights: the embeddings, the tanh layer,2the output scores and the softmax. The prose's point rests on the shapes, the parameter counts and the three likeliest3tokens after `patio bar`."""4import numpy as np56from lib import bpe, neural7from lib.ngrams import START89weights, vocabulary, manifest = neural.load()10index, width = {t: i for i, t in enumerate(vocabulary)}, manifest["settings"]["context"] # token -> row; the window's width, from the saved settings11context = ([START] * width + bpe.store_tokenizer().encode("patio bar"))[-width:] # pad with start markers, keep the last width tokens12ids = np.array([[index[t] for t in context]])1314vectors = weights["embedding"][ids] # one 64-number vector per token15joined = vectors.reshape(1, -1) # -1 lets numpy work out the length: 4 vectors of 64 make 25616hidden = np.tanh(joined @ weights["hidden_w"] + weights["hidden_b"]) # one layer, squashed into -1 to 117logits = hidden @ weights["output_w"] + weights["output_b"] # one score for every token18probabilities = np.exp(logits - logits.max()) # subtract the largest score first: same probabilities, and exp can't overflow19probabilities /= probabilities.sum() # softmax: scores into probabilities2021print("context:", " ".join(context))22for name, array in [("embeddings", vectors), ("joined", joined), ("hidden", hidden), ("logits", logits)]:23 print(f"{name} shape:", array.shape)24print("vocabulary size:", f"{len(vocabulary):,}")25print("parameters:", f"{sum(w.size for w in weights.values()):,}")26for part in ["embedding", "hidden_w", "output_w"]:27 print(f"parameters in {part}:", f"{weights[part].size:,}")28for i in np.argsort(-probabilities[0], kind="stable")[:3]: # the three likeliest; a stable sort keeps ties in vocabulary order29 print(f"P({vocabulary[i]} | patio bar): {probabilities[0, i]:.2f}")
python window-model-by-hand.pycontext: <s> <s> patio</w> bar</w>
embeddings shape: (1, 4, 64)
joined shape: (1, 256)
hidden shape: (1, 256)
logits shape: (1, 4140)
vocabulary size: 4,140
parameters: 1,394,732
parameters in embedding: 264,960
parameters in hidden_w: 65,536
parameters in output_w: 1,059,840
P(stool</w> | patio bar): 0.63
P(with</w> | patio bar): 0.23
P(&</w> | patio bar): 0.05
由于这个名称目前只有两个词元,窗口用起始标记补齐。四个词元各自变成一个含 64 个数的向量;拼接后得到一个含 256 个数的向量;隐藏层再把它变换为另外 256 个数;输出层则为模型所认识的 4,140 个词元各给出一个分数,这些词元包括存储分词器的词表以及起始和结束标记。该窗口模型有 1,394,732 个参数,即它所学习的数值。其中约四分之三位于输出层的权重 1,059,840 中,因为输出层需要为每个词元准备一列,其余的大部分在嵌入表中。它的预测是合理的:stool 的概率为 0.63,其次是 with 和 &,它们可以接在诸如 patio bar with stools 这样的名称之后。
工作准则: 对于自己训练的任何模型,至少手工搭建一次前向传播,并像本示例那样打印每个中间数组的形状:形状错误是神经网络代码中常见的缺陷,而形状恰好说明了模型在计算什么。词表较大时,应预料到输出层占据了大部分参数。
用反向传播计算梯度
权重随机的模型预测很差;训练就是调整权重,使预测变好。要衡量模型预测得有多差,使用损失:L = −log P(下一个词元),即模型赋予实际出现的下一个词元的对数概率的相反数,也就是第 1 章中被平均为困惑度的那种意外程度。训练会降低平均损失,而损失越低,困惑度越低。要降低损失,每个权重都需要一个梯度:当该权重发生微小变化时,损失会改变多少,记作 ∂L/∂w。
反向传播一次算出全部梯度,做法是把微积分的链式法则沿前向传播的相反方向逐步应用:从损失到 logits,从 logits 经输出层到隐藏层,经过 tanh,再回到被用到的嵌入(Rumelhart et al., 1986)。其中有两步值得牢记。损失关于 logits 的梯度,等于 softmax 概率在真实词元处减去 1。而经过 tanh 的梯度要乘以 1 − h²,在 h 接近 ±1 的地方这个值很小。本书的共享模块 lib.tinynet 针对商店模型的一个微型版本,逐行写出了整个过程,该微型版本有 5 个词元,嵌入为 2 个数,窗口为 2,隐藏单元为 3 个。
反向传播的代码常常出现不易察觉的错误,所以要用一种更慢但不会出错的方法来核对:把某一个权重向上和向下各微调一个很小的量,测量损失如何变化,再作比较。商店的模型需要一百多万次微调,因此对微型模型的每个参数都进行核对:
1"""Backpropagation's gradients for the tiny model, checked against nudging every one of its parameters up and down.2The assert is the check the prose relies on: the two agree to six decimal places, for every parameter."""3import numpy as np45from lib.tinynet import backprop, forward, parameters67p = parameters() # the tiny model: 5 tokens, a window of 2, 3 hidden units; seeded, so the same every run8context, target = np.array([1, 3]), 4 # the window: token ids 1 and 3; the token that came next is id 4910worst, step = 0.0, 1e-6 # the nudge: small enough that the loss is nearly a straight line over it11gradients = backprop(p, context, target)12for name, value in p.items(): # nudge each parameter both ways, and see how the loss moves13 for i in np.ndindex(value.shape):14 value[i] += step15 up = forward(p, context, target)[3] # forward returns (joined, hidden, logits, loss): index 3 is the loss16 value[i] -= 2 * step # from up to down is two steps, so the estimate is centred on the value17 down = forward(p, context, target)[3]18 value[i] += step # back where it started19 worst = max(worst, abs((up - down) / (2 * step) - gradients[name][i]))20agrees = worst < 1e-6 # the largest disagreement over all parameters, against six decimal places21assert agrees22print("loss:", round(float(forward(p, context, target)[3]), 4))23print("parameters checked:", sum(v.size for v in p.values()))24print("backprop agrees with nudging to 6 decimal places:", "yes" if agrees else "no")
python gradients-by-hand.pyloss: 1.5293
parameters checked: 45
backprop agrees with nudging to 6 decimal places: yes
反向传播得到的全部 45 个梯度,与微调估计值在小数点后六位上一致,示例中对此做了断言。微调方法每个参数需要两次前向传播,所以没有人用它来训练,但在微型模型上,它是确认反向传播正确的基准。
工作准则: 在信任手写梯度之前,先在微型模型上用微调估计值核对。如果使用的是替你计算梯度的库,如 PyTorch 一节中所做的那样,则改为用你自己的前向传播来核对库的前向传播。
执行梯度下降步骤
有了梯度,训练就成了一个循环。梯度下降(gradient descent)让每个权重沿着其梯度的反方向移动一小步,即 w ← w − η · ∂L/∂w,其中 η 称为步长(step size),也叫学习率(learning rate),决定每一步走多远。每一步都会让损失在计算它所用的那些样本上略微减小;在大量样本上反复进行,权重便会稳定在模型预测良好的位置。
使用lib.tinynet给出的梯度,以 0.5 的步长对上一节中商店的微型模型在三个样本上执行二十步梯度下降,可以看到损失不断下降:
1"""Twenty steps of gradient descent on three examples of the tiny model, printing the average loss before training and2after one, five and twenty steps. The prose's point rests on the average loss at those four points."""3import numpy as np45from lib.tinynet import backprop, forward, parameters67p = parameters() # seeded, so every run starts from the same weights8examples = [(np.array([1, 3]), 4), (np.array([0, 1]), 3), (np.array([3, 4]), 2)] # three (window, next token) pairs: each window is two token ids, then the id of the token after it9rate = 0.5 # the step size101112def average_loss():13 return np.mean([forward(p, c, t)[3] for c, t in examples])141516print("average loss before:", f"{average_loss():.3f}")17for step in range(1, 21): # twenty steps, each one a pass over all three examples18 for c, t in examples:19 for name, gradient in backprop(p, c, t).items():20 p[name] -= rate * gradient / len(examples) # w <- w - rate * dL/dw, with each gradient averaged over the examples21 if step in (1, 5, 20):22 print(f"average loss after {step} steps:", f"{average_loss():.3f}")
python gradient-descent-by-hand.pyaverage loss before: 1.568
average loss after 1 steps: 1.346
average loss after 5 steps: 0.646
average loss after 20 steps: 0.109
在该示例打印的每一步中,平均损失都在下降,从训练前的1.568降到二十步之后的0.109。只有三个样本可学时,微型模型可以把损失压到接近于零,这就是记忆;商店的模型要从数十万个样本中学习,同样的循环必须找出在这些样本中普遍成立的规律。
真实的训练在这个循环上有两处不同。真实训练的每一步是在一个批次(batch)的样本上计算的,每次几百个,而不是全部样本,因此每一步的开销很小,各步之间也会略有波动。此外,它使用经过改进的更新方式,而不是固定步长,下一节将对此加以说明。
工作准则: 开始任何新模型时,先在少量样本上运行几步,检查损失是否下降:如果在三个样本上损失都不下降,说明有缺陷,而不是问题本身困难。步长靠多试几个来选定:太小,损失下降缓慢;太大,损失会来回跳动甚至增大。
这部分尚未翻译,因此以英文显示。
Train with PyTorch
Writing backpropagation by hand for every model would be slow and error-prone. PyTorch, a widely used library for training neural networks, computes gradients automatically from the forward pass, and runs the arithmetic fast. The book trains its model with it, in recordings/neural_lm.py, which builds the same window model as the first section: an embedding table, one tanh layer and an output layer, trained on the store's training names. The heart of the script is the loop of the previous section, on batches of 512 predictions:
loss = loss_of(model(train_x[pick]), train_y[pick]) # forward pass and average loss on one batch
optimizer.zero_grad() # clear the previous batch's gradients
loss.backward() # backpropagation: a gradient for every weight
optimizer.step() # move every weight against its gradient
The training script's optimizer is Adam, a refinement of gradient descent (Kingma and Ba, 2015). It keeps, for every weight, a running average of its gradients and a running average of their squares, and steps by the first divided by the square root of the second. A weight whose gradients are consistently large gets steps of about the same size as one whose gradients are small, so one step size works for all of them.
Training depends on the machine's floating-point arithmetic, which chapter 4 explains, so it runs once and the book's examples read the weights it saved, as they read every such result. The examples never import PyTorch: lib.neural recomputes the model's forward pass in NumPy, from those weights. That only works if the NumPy version computes what PyTorch computed, so the training script saved PyTorch's own logits for 64 validation contexts, and the next example compares:
1"""The trained model's logits, recomputed in NumPy from its saved weights and compared with the logits PyTorch computed2for the same saved validation contexts. The prose's point rests on the agreement, which the last line prints."""3import numpy as np45from lib import neural67weights, _, manifest = neural.load()8check = np.load("data/recorded/neural-lm/check.npz") # saved by the training script, so this runs without PyTorch: contexts, and PyTorch's logits for them910ours = neural.forward(weights, check["contexts"])11print("PyTorch version:", manifest["torch"])12print("contexts compared:", len(check["contexts"]))13print("logits compared:", f"{ours.size:,}") # every logit: one per context and per token in the vocabulary14print("same as PyTorch to 4 decimal places:", bool(np.abs(ours - check["logits"]).max() < 1e-4)) # 1e-4: four decimal places
python same-as-pytorch.pyPyTorch version: 2.14.1+cpu
contexts compared: 64
logits compared: 264,960
same as PyTorch to 4 decimal places: True
The NumPy forward pass reproduces all 264,960 of PyTorch's logits to four decimal places, for the model trained with PyTorch 2.14.1+cpu. Every number this chapter reports about the trained model comes from that NumPy code, except the training curve, which the script recorded as it trained.
Working rule: Train with a library that computes gradients for you, and keep a short, independent forward pass of your own to check its outputs, as lib.neural does here. Record the library's outputs for a few inputs whenever you save weights, so the check can run anywhere, long after training.
这部分尚未翻译,因此以英文显示。
Stop when validation loss turns up
A model with 1.4 million parameters, trained on a few hundred thousand tokens, can memorize its training text, as chapter 3's 5-gram did. Training perplexity keeps falling as it does. Validation perplexity falls at first, while the model learns patterns that hold for new names, then rises, once it starts fitting quirks of the training names. A validation perplexity that turns up while training perplexity keeps falling is overfitting, and the simplest defence is early stopping: measure validation loss after every epoch, one full pass over the training text, and keep the weights from the best epoch.
The training script recorded both perplexities after each of its twelve epochs. Read them for where validation stops improving:
1"""The training script's record of training and validation perplexity after each epoch, and the epoch whose weights were2kept. The prose's point rests on the validation column, and on the two validation figures printed after the table."""3import json4import math56curves = json.load(open("data/recorded/neural-lm/curves.json", encoding="utf-8"))7manifest = json.load(open("data/recorded/neural-lm/manifest.json", encoding="utf-8"))89print(f"{'epoch':>5} {'training perplexity':>19} {'validation perplexity':>21}")10for c in curves: # losses are averages of log probabilities: exp turns them into perplexities11 print(f"{c['epoch']:>5} {math.exp(c['train_loss']):>19.1f} {math.exp(c['validation_loss']):>21.1f}")12best = min(curves, key=lambda c: c["validation_loss"]) # the epoch with the lowest validation loss, which the kept epoch should match13print("epoch kept:", manifest["kept_epoch"])14print("validation perplexity at the kept epoch:", f"{math.exp(best['validation_loss']):.1f}")15print("validation perplexity at the last epoch:", f"{math.exp(curves[-1]['validation_loss']):.1f}") # curves[-1]: the final epoch's record, whatever was kept
python training-curve.pyepoch training perplexity validation perplexity
1 109.3 51.4
2 37.6 39.9
3 27.3 37.3
4 22.4 36.6
5 19.3 36.4
6 17.2 36.9
7 15.6 37.8
8 14.4 38.6
9 13.5 39.7
10 12.7 40.9
11 12.0 42.1
12 11.5 43.7
epoch kept: 5
validation perplexity at the kept epoch: 36.4
validation perplexity at the last epoch: 43.7
In epoch 1, training perplexity is higher than validation, because the training figure averages over the whole epoch, including its first batches, when the model was still untrained, while validation is measured at the epoch's end. Validation perplexity bottoms out at 36.4 in epoch 5 and climbs from there, to 43.7 by the last epoch, while training perplexity falls at every epoch after the first. The training script kept the weights from the best epoch, and every result in the rest of this chapter uses them.
The gap between the two curves is the model memorizing. It grows every epoch, because the training names repeat their product lines and phrasing, and this model has no regularization, which is anything in training that discourages memorizing, so early stopping is its only defence.
Working rule: Always record validation loss per epoch and keep the best epoch's weights, never the last. When the curves part early, as they do here after a few epochs, the model has more capacity than its training text supports: regularize it, use a smaller model, or get more text of the same kind.
与基线比较
真正的检验与本书中每个模型所面对的相同:在验证名称上按词元计算困惑度,并与 4-gram 基线比较。在这些名称上,哪一点更重要:关联相似的上下文,还是精确的计数?神经模型能够关联相似的上下文,而 4-gram 做不到;4-gram 则对它见过的每个上下文都有精确的计数。
1"""The window model and the 4-gram baseline, each scored by perplexity per token on the same validation names. The2prose's point rests on the two printed perplexities."""3from lib import neural4from lib.names import name_tokens5from lib.ngrams import KneserNey, perplexity67train, validation = name_tokens("train"), name_tokens("validation") # both models are scored on the same tokens of the same names8print("window model validation perplexity:", f"{neural.perplexity(validation):.1f}") # the window model, from its saved weights9print("4-gram baseline validation perplexity:", f"{perplexity(KneserNey(train, 4), validation):.1f}") # a 4-gram: each token is predicted from the three before it
python neural-vs-baseline.pywindow model validation perplexity: 36.4
4-gram baseline validation perplexity: 23.1
基于计数的模型胜出,而且差距很大:窗口模型的验证困惑度为 36.4,基线为 23.1。
为了看清差距出在哪里,我们按三个词元的上下文是否出现在训练名称中来拆分验证词元,并分别在两部分上为两个模型打分:
1"""Splits the validation tokens by whether their three-token context occurs in the training names, and scores both models2on each part. The prose's point rests on the four perplexities printed for the two parts."""3import math45import numpy as np67from lib import neural8from lib.names import name_tokens9from lib.ngrams import KneserNey, ngrams, padded1011train, validation = name_tokens("train"), name_tokens("validation")12counted = KneserNey(train, 4)13pairs = [(c, t) for text in validation for c, t in ngrams(padded(text, 4), 4)] # padding gives each name's first tokens a context14seen = np.array([c in counted.totals[4] for c, _ in pairs]) # was this 3-token context in the training names?15from_counts = np.array([counted.probability(c, t) for c, t in pairs])16from_network = neural.token_probabilities(validation) # in the same order as pairs, so each entry is for the same token1718print("validation tokens with a context seen in training:", f"{seen.mean():.0%}")19for part, where in [("seen", seen), ("unseen", ~seen)]: # ~seen: the complement, the contexts the training names lack20 for model, p in [("4-gram", from_counts), ("network", from_network)]:21 print(f"{model} perplexity, {part} contexts: {math.exp(-np.log(p[where]).mean()):.1f}") # exp of the average surprise, over this part
python seen-and-unseen-contexts.pyvalidation tokens with a context seen in training: 70%
4-gram perplexity, seen contexts: 17.9
network perplexity, seen contexts: 26.1
4-gram perplexity, unseen contexts: 41.2
network perplexity, unseen contexts: 78.5
网络在两部分上都更差,在未见过的上下文上差距更大,而这恰恰是它关联上下文的能力本应最有帮助的地方。在未见过的上下文上,4-gram 的得分为 41.2,网络为 78.5。在已见过的上下文上(它们占验证词元的 70%),4-gram 的得分为 17.9,网络为 26.1。
因此,这个网络并不只是在计数精确的地方落败;它也没有学会良好的泛化。这个模型和这份文本的两个特点最有可能是原因。其一,网络没有正则化,几个轮次之后就会过拟合,正如训练曲线所显示的那样。其二,商店的名称短小且有固定格式:三个词元的上下文就涵盖了一个名称的大部分内容,而这正是 Kneser-Ney 计数最为强势的场景。Bengio 等人在更大、更多样的文本上得到了相反的结果:在 Brown 语料库上(约 800,000 个训练词),他们最好的网络单独使用时测试困惑度为 268,与三元模型混合后为 252,而最好的 n-gram 模型为 312(Bengio et al., 2003,第 4.2 节和表 1)。学习得到的模型并不会因为是学出来的就更好;必须加以度量。
工作准则: 在声称学习得到的模型有帮助之前,务必在相同的文本和词元上,将它与强大的基于计数的基线相比较,并按已见和未见上下文拆分比较,以看清每个模型在哪里取胜。不要根据一个小数据集就断定神经模型不起作用,或者断定它们总是有效。
将网络与计数模型混合
一个平均表现较差的模型,只要它的错误与较好模型的错误不同,仍然可能有用。当两个模型犯的错误不同时,将二者的概率混合起来,效果可以胜过其中任何一个。Bengio 等人正是这样处理他们的网络和平滑三元模型的(Bengio et al., 2003,第 4 节)。这种混合是对两个模型给出的每个词元概率做加权平均,与第 3 章的插值相同。
混合后的效果是否胜过两端?取网络在混合中所占的份额为 0、0.2、0.3、0.5 和 1,对验证集中的名字打分,其中 0 表示只用计数模型,1 表示只用网络:
1"""Mixes the network's and the 4-gram's probability of every validation token, at five shares for the network, and prints2the validation perplexity of each mix. The prose's point rests on the printed perplexity at share 0.2, and at the two ends."""3import math45import numpy as np67from lib import neural8from lib.names import name_tokens9from lib.ngrams import KneserNey, ngrams, padded1011train, validation = name_tokens("train"), name_tokens("validation")12from_network = neural.token_probabilities(validation) # the network's probability of each actual next token, in order13counted = KneserNey(train, 4)14# the 4-gram's probability for the same tokens, in the same order: each name padded with three start markers first15from_counts = np.array([counted.probability(c, t) for text in validation for c, t in ngrams(padded(text, 4), 4)])1617for share in [0.0, 0.2, 0.3, 0.5, 1.0]: # the network's share of the mix; the counts get the rest18 mixed = share * from_network + (1 - share) * from_counts19 print(f"validation perplexity, network share {share:.1f}: {math.exp(-np.log(mixed).mean()):.1f}")
python mix-neural-and-counts.pyvalidation perplexity, network share 0.0: 23.1
validation perplexity, network share 0.2: 21.9
validation perplexity, network share 0.3: 22.1
validation perplexity, network share 0.5: 22.9
validation perplexity, network share 1.0: 36.4
混合模型胜过了两个单独的模型。网络份额为 0.2 时,验证集困惑度为 21.9,优于基线的 23.1,也远优于单独使用网络时的 36.4。网络的平均表现较差,但在计数模型给某个词元的概率极小时,网络的份额使该概率不至于过小,从而降低了平均意外程度。
所保留的训练轮次和份额都是通过观察验证集上的名字选定的,因此这里的提升略显乐观;本书最终在测试集名字上所做的比较会对此加以检验。此外,混合之所以有帮助,只是因为两个模型的错误不同:如果它们犯的错误相同,将二者平均不会带来任何收益。
工作准则: 当学习得到的模型与计数模型表现接近时,先尝试混合二者的概率,再决定选用哪一个,并在留出文本上选择份额。当其中一个模型明显更强时,应让份额保持较小;在依赖这一提升之前,应在单独的测试集上确认它。
更多文本只有在类型相同时才有帮助
如果网络过拟合是因为学习的文本太少,最直接的办法就是增加文本。商店有现成的来源:商品描述。训练脚本可以用名称和描述一起训练同一个模型,本书也记录了这个版本,并在同样的验证名称上对它评分。
在验证名称上比较这两个已记录的模型,一个用名称训练,另一个用名称和描述训练:
1"""Scores the window model trained on names alone against the one trained on names and descriptions, counts the training2tokens of each kind of text, and prints the second model's validation perplexity after each of its epochs. The prose's3point rests on the two validation perplexities, and on the training token counts printed beside them."""4import json5import math67from lib import neural8from lib.names import name_tokens910validation = name_tokens("validation")11for name in ["neural-lm", "neural-lm-all-text"]: # the recorded runs: names alone, then names and descriptions12 manifest = json.load(open(f"data/recorded/{name}/manifest.json", encoding="utf-8"))13 print(f"learned from {manifest['settings']['text']}:", f"{neural.perplexity(validation, name):.1f}")1415from lib import bpe16from lib.store import load_products1718tokenizer = bpe.store_tokenizer()19names_tokens = sum(len(t) for t in name_tokens("train")) # counted in the store's tokens, the ones the model reads20description_tokens = sum(len(tokenizer.encode(d)) for d in load_products("train")["description"]) # the training split only, as for names21print("training tokens, names:", f"{names_tokens:,}")22print("training tokens, names and descriptions:", f"{names_tokens + description_tokens:,}")23curve = json.load(open("data/recorded/neural-lm-all-text/curves.json", encoding="utf-8"))24print("all-text model, validation perplexity by epoch:", ", ".join(f"{math.exp(c['validation_loss']):.1f}" for c in curve)) # one per epoch, in order
python more-text-of-another-kind.pylearned from names: 36.4
learned from all: 59.0
training tokens, names: 337,071
training tokens, names and descriptions: 2,744,320
all-text model, validation perplexity by epoch: 73.6, 64.0, 60.8, 59.0
加入描述之后,模型对名称的预测反而变差了。训练名称包含 337,071 个词元,名称和描述合在一起包含 2,744,320 个,约为前者的八倍。只从名称学习的模型得分为 36.4,从名称和描述一起学习的模型得分为 59.0。第二个模型训练了四个轮次,停止时验证困惑度仍在下降,但每个轮次的降幅越来越小:73.6, 64.0, 60.8, 59.0。我们的判断是,即使增加轮次,也无法使它接近只用名称训练的模型。
描述是另一种类型的文本。它们以完整的句子写成,其中有 your、this 和 perfect 这类名称里很少出现的词,主要从描述中学习的模型会把名称当作描述来预测。新增的文本让模型学到了与任务不符的分布。模型学到的是给它的文本,困惑度衡量的是它预测此类文本的能力:另一种类型的数据可能使它变差,因此需要加权,或者与目标文本分开处理。
**工作准则:**添加训练文本之前,先检查它是否与要预测的文本相似;两者都训练一遍,并在目标文本上度量,而不要假定越多越好。当额外的文本差异很大时,只用目标文本训练,或者安排好每种文本的版式,使模型能够区分它们的类型,如第 8 章在商品名称与描述之间使用分隔符。
这部分尚未翻译,因此以英文显示。
Suggest from similar contexts
The network lost on average, including on contexts the training names lack, but averages hide single cases. Chapter 3's autocomplete showed the 4-gram's worst case, outdoor bar, a context no training name contains, where it fell back to generic continuations.
Next, the network's three likeliest tokens after five name prefixes, and a few embeddings, tell whether outdoor bar gets the stool suggestion that patio bar does:
1"""The network's three likeliest next tokens after five typed prefixes, its probability of `stool` after `outdoor bar`,2and the cosine similarity of `patio` with three other tokens in its embedding table. The prose's point rests on the3suggestions after each prefix, and on the three cosines printed last."""4import numpy as np56from lib import bpe, neural7from lib.ngrams import START89weights, vocabulary, manifest = neural.load()10index, width = {t: i for i, t in enumerate(vocabulary)}, manifest["settings"]["context"]11tokenizer = bpe.store_tokenizer()1213for typed in ["bar", "patio bar", "outdoor bar", "kitchen bar", "wine bar"]: # each typed prefix is read as the start of a name14 context = ([START] * width + tokenizer.encode(typed))[-width:] # pad with start markers, keep the last width tokens15 logits = neural.forward(weights, np.array([[index[t] for t in context]]))[0] # forward takes a batch: one row, so take row 016 probabilities = np.exp(neural.log_softmax(logits[None])[0]) # log_softmax wants a batch too: add an axis, then take it off17 best = np.argsort(-probabilities, kind="stable")[:3] # the three likeliest; a stable sort keeps ties in vocabulary order18 # the end-of-word marker </w> is shown as nothing, so the tokens read as the words they came from19 print(f"{typed:<12} -> " + ", ".join(f"{vocabulary[i].replace('</w>', '')} {probabilities[i]:.2f}" for i in best))2021outdoor_bar = ([START] * width + tokenizer.encode("outdoor bar"))[-width:] # the same padding as above22p_stool = np.exp(neural.log_softmax(neural.forward(weights, np.array([[index[t] for t in outdoor_bar]])))[0, index["stool</w>"]])23print("P(stool | outdoor bar):", f"{p_stool:.2f}")24table = weights["embedding"] / np.linalg.norm(weights["embedding"], axis=1, keepdims=True) # unit-length rows, so a dot product is a cosine25for a, b in [("patio</w>", "outdoor</w>"), ("patio</w>", "kitchen</w>"), ("patio</w>", "wine</w>")]:26 print(f"cosine({a.replace('</w>', '')}, {b.replace('</w>', '')}) in the model's embeddings:", f"{float(table[index[a]] @ table[index[b]]):.2f}")
python neural-autocomplete.pybar -> with 0.22, cabinet 0.14, cart 0.13
patio bar -> stool 0.63, with 0.23, & 0.05
outdoor bar -> stool 0.40, with 0.13, & 0.12
kitchen bar -> & 0.53, with 0.17, cart 0.09
wine bar -> with 0.23, cabinet 0.16, </s> 0.14
P(stool | outdoor bar): 0.40
cosine(patio, outdoor) in the model's embeddings: 0.16
cosine(patio, kitchen) in the model's embeddings: 0.07
cosine(patio, wine) in the model's embeddings: 0.10
After outdoor bar, the network suggests stool first, with probability 0.40, where the 4-gram put & and with ahead of it. The control prefixes show that stool isn't automatic after bar: after bar alone, kitchen bar and wine bar, it isn't among the network's three suggestions. And in the network's own embedding table, patio is a little closer to outdoor, with a cosine similarity of 0.16, than to kitchen or wine. That's consistent with the network having carried something from patio bar over to outdoor bar, though the similarity is weak, and one prefix proves little.
This is one case, not an average: the seen and unseen split showed the network worse on unseen contexts overall. It shows what a learned model can do that counting can't, and that this one does it only sometimes.
Working rule: Look at a learned model's predictions on contexts the training text lacks, with control prefixes beside them, but judge it by measurements over many contexts, not by examples. For autocomplete in production, use the mix, and test it on rare prefixes as well as common ones.
生产中会出什么问题
神经语言模型在带来新优势的同时,也带来了新的故障模式:
- 过拟合是无声的。 训练损失看起来总是很好。没有验证曲线,训练过久的模型上线后的表现,会比训练较少的模型更差。请保留最佳的轮次,并留意训练与验证之间的差距。
- 保存你要上线的权重。 在另一种处理器上重新训练,得到的权重会略有不同,因此要保留生成这些权重的代码和设置,以及库对少量输入给出的输出,本章的训练脚本就是这样做的。
- 输出层占主导。 在每一步为词表中的每个词元打分,占去了大部分计算量和大部分参数。词表变大,首先在这里增加成本。
- 不匹配的训练文本有害。 加入与目标不同的文本,例如用描述文本来预测名称(如用描述文本训练名称模型),会使预测变差,而训练损失看起来却没有问题。每次更改训练数据之后,都要在目标文本上做评测。
- 平均值会掩盖罕见的上下文。 模型总体得分可能很高,却仍然在一些不常见的前缀上失败,购物者寻找新东西时正会输入这类前缀。要像本章那样,把见过和没见过的上下文分开评分,并关注没见过的那部分。
- 混合需要在服务时同时具备两个模型。 把神经网络与计数混合,会使需要加载和保持更新的内容翻倍。每当重新训练网络,就要重新构建计数,使两者描述的是同一份文本。
本章的决定
- 本书的神经网络模型只在 PyTorch 中训练一次,结果保存在
recordings/中, 各个示例再根据保存的权重用 NumPy 重新计算。NumPy 前向传播得到的 logits 与 PyTorch 的结果在小数点后四位上一致,因此示例无需安装 PyTorch。 - 模型保留验证集上表现最好的那一轮 epoch 的权重。 窗口模型的验证困惑度在第 5 个 epoch 之后最低,此后每个 epoch 都在上升,而训练困惑度则持续下降:模型是在记忆训练用的名称。
- 在商店的名称上,窗口模型单独使用时不如 4-gram 基线,无论是已见上下文还是未见上下文,都是如此;两者混合后则优于基线。 当网络占比为 0.2 时,结果为 21.9,是本书迄今为止最好的模型。
- 学习得到的模型要在已见上下文和未见上下文上分别与基线比较, 而不只是比较平均值,因为这两部分可能呈现不同的结果。在这里,网络在两部分上都输给了基线,在未见上下文上输得更多。
- 训练文本必须与目标文本相似。 加入商品描述之后,模型在名称上的表现反而变差了,尽管名称与描述合在一起的词元数约为单独名称的 8 倍。因此,每个模型的训练文本都要根据它将要被评分的文本来选择。
- 名称很短,所以从第 8 章起,本书还会在商店的商品描述上评估模型。
本章,在本版次中
版次 2026.11.16示例于 2026年11月19日 通过本章 10个示例 个示例中有 10 个通过
译自英文版图书,版次 2026.11.16。