신경 언어 모델

읽는 데 약 39분분 걸린다, 11절에 10개 예제이(가) 있음.

2026.11.16판 예제 통과 2026년 11월 19 86개 예제 중 86개 통과 변경 사항

AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식

영어판 2026.11.16판을 번역한 것이다. 원문 읽기

이 장을 마치면 앞선 토큰들의 창(window)으로 다음 토큰을 예측하는 신경 언어 모델을 만들고, 그 기울기를 손으로 계산해 검증하며, PyTorch로 학습시키고, 횟수 세기로 만든 기준 모델과 공정하게 비교하여 측정할 수 있다. 또한 학습된 모델이 학습되었다는 이유만으로 이기지는 않는다는 점도 보게 된다. 이 모델은 단독으로는 횟수 세기에 지지만, 둘을 섞으면 도움이 된다.

3장에서는 이 책의 기준 모델을 만들었다. 이 모델은 4-gram Kneser-Ney 모델로, 매장 상품명의 각 토큰을 그 앞의 세 토큰으로부터 횟수를 세어 예측한다. 이 모델의 한계는 한 문맥을 다른 문맥과 연관 짓지 못한다는 점이다. 가구와 생활용품을 파는 매장의 상품명에서 patio bar에 대한 횟수는 있지만 outdoor bar에 대한 횟수는 없으며, 둘을 이어 주는 것은 아무것도 없다. 4장은 벡터가 토큰들을 연관 지을 수 있음을 보였다. patio와 outdoor는 매장의 텍스트로 학습하면 가까운 벡터를 얻기 때문이다. 이 장은 벡터를 모델 안에 넣는다. 신경 언어 모델(neural language model)은 이전 토큰들을 벡터로 읽고, 학습된 산술로 이를 결합하여, 어휘의 모든 토큰에 대한 확률을 만들어 낸다. 이 모델은 다음 토큰을 예측하도록 학습되면서 벡터와 산술을 함께 학습한다. 이 설계는 Bengio와 동료들이 소개한 것이다(Bengio et al., 2003).

텍스트는 매장이 제공하며, 시험은 3장 이후 모든 모델이 맞닥뜨리는 것과 같다. 매장의 토크나이저를 거친 검증용 상품명에서 토큰당 퍼플렉서티를 측정하는 것이다. 값이 낮을수록 좋으며, 기준 모델의 점수는 23.1이다.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Predict from a window of embeddings

마지막으로 변경된 판: 2026.11.16판

This chapter's model looks at a fixed window of the previous tokens, four here, the same context a 5-gram counts. Each token is looked up in an embedding table, a matrix with one row of numbers per token in the vocabulary, as in chapter 4, except that this table is learned for predicting the next token rather than computed from counts. The four rows are joined side by side into one long vector, x. A hidden layer computes h = tanh(xW + b) from its input x: it multiplies x by a matrix of weights W, adds a vector of biases b, and passes each result through tanh, a function that squashes any number into the range from −1 to 1, so the layer can combine its inputs in ways sums alone can't. An output layer then computes z = hU + c from the hidden layer's output h, one score per token in the vocabulary, called the logits.

The last step turns scores into probabilities. Softmax exponentiates every logit and divides by the sum, so the results are positive and sum to one, and a higher score means a higher probability:

P(token i) = exp(zᵢ) / Σⱼ exp(zⱼ)

Worked through with the trained weights, one prediction shows how the model turns four tokens into probabilities: the next token after a name starting patio bar. Each line of the example is one of the formulas above, and its printed shapes show what each step does:

window-model-by-hand.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""One prediction of the trained window model, computed by hand from its saved weights: the embeddings, the tanh layer,the output scores and the softmax. The prose's point rests on the shapes, the parameter counts and the three likeliesttokens after `patio bar`."""import numpy as npfrom lib import bpe, neuralfrom lib.ngrams import STARTweights, vocabulary, manifest = neural.load()index, width = {t: i for i, t in enumerate(vocabulary)}, manifest["settings"]["context"]   # token -> row; the window's width, from the saved settingscontext = ([START] * width + bpe.store_tokenizer().encode("patio bar"))[-width:]   # pad with start markers, keep the last width tokensids = np.array([[index[t] for t in context]])vectors = weights["embedding"][ids]                                    # one 64-number vector per tokenjoined = vectors.reshape(1, -1)                                        # -1 lets numpy work out the length: 4 vectors of 64 make 256hidden = np.tanh(joined @ weights["hidden_w"] + weights["hidden_b"])   # one layer, squashed into -1 to 1logits = hidden @ weights["output_w"] + weights["output_b"]            # one score for every tokenprobabilities = np.exp(logits - logits.max())                          # subtract the largest score first: same probabilities, and exp can't overflowprobabilities /= probabilities.sum()                                   # softmax: scores into probabilitiesprint("context:", " ".join(context))for name, array in [("embeddings", vectors), ("joined", joined), ("hidden", hidden), ("logits", logits)]:    print(f"{name} shape:", array.shape)print("vocabulary size:", f"{len(vocabulary):,}")print("parameters:", f"{sum(w.size for w in weights.values()):,}")for part in ["embedding", "hidden_w", "output_w"]:    print(f"parameters in {part}:", f"{weights[part].size:,}")for i in np.argsort(-probabilities[0], kind="stable")[:3]:             # the three likeliest; a stable sort keeps ties in vocabulary order    print(f"P({vocabulary[i]} | patio bar): {probabilities[0, i]:.2f}")
출력python window-model-by-hand.py
context: <s> <s> patio</w> bar</w>
embeddings shape: (1, 4, 64)
joined shape: (1, 256)
hidden shape: (1, 256)
logits shape: (1, 4140)
vocabulary size: 4,140
parameters: 1,394,732
parameters in embedding: 264,960
parameters in hidden_w: 65,536
parameters in output_w: 1,059,840
P(stool</w> | patio bar): 0.63
P(with</w> | patio bar): 0.23
P(&</w> | patio bar): 0.05

The window is padded with start markers, because the name has only two tokens so far. Each of the four tokens becomes a vector of 64 numbers; joined, they make one vector of 256; the hidden layer turns that into 256 more; and the output layer gives one score for each of the 4,140 tokens the model knows, the store tokenizer's vocabulary plus the start and end markers. The window model has 1,394,732 parameters, the numbers it learns. About three quarters of them are in the output layer's weights, 1,059,840, because it needs a column for every token, and most of the rest are in the embedding table. Its prediction is sensible: stool with probability 0.63, then with and &, which continue names such as patio bar with stools.

Working rule: Build the forward pass of any model you train by hand at least once, and print the shape of every intermediate array, as this example does: a wrong shape is a common bug in neural code, and the shapes say exactly what the model computes. Expect the output layer to hold most of the parameters whenever the vocabulary is large.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Compute gradients by backpropagation

마지막으로 변경된 판: 2026.11.16판

A model with random weights predicts badly; training adjusts the weights to predict well. To measure how badly a model predicts, use the loss: L = −log P(next token), the negative log probability the model gave to the token that actually came next, the same surprise chapter 1 averaged into perplexity. Training lowers the average loss, and lower loss means lower perplexity. To lower the loss, each weight needs a gradient: how much the loss would change if that weight changed a little, written ∂L/∂w.

Backpropagation computes all the gradients at once, by applying the chain rule of calculus backwards through the forward pass: from the loss to the logits, from the logits through the output layer to the hidden layer, through tanh, and back to the embeddings that were used (Rumelhart et al., 1986). Two of its steps are worth knowing by heart. The gradient of the loss with respect to the logits is the softmax probabilities minus 1 at the true token. And the gradient through tanh is multiplied by 1 − h², which is small where h is near ±1. The book's shared module lib.tinynet writes the whole of it out, line by line, for a tiny version of the store's model, with 5 tokens, embeddings of 2 numbers, a window of 2 and 3 hidden units.

Backpropagation code often goes subtly wrong, so check it against a slower method that can't: nudge one weight up and down by a tiny amount, measure how the loss moves, and compare. The store's model would take more than a million nudges, so check every parameter of the tiny one:

gradients-by-hand.py Python 3.12.15 · numpy 2.5.5 통과
"""Backpropagation's gradients for the tiny model, checked against nudging every one of its parameters up and down.The assert is the check the prose relies on: the two agree to six decimal places, for every parameter."""import numpy as npfrom lib.tinynet import backprop, forward, parametersp = parameters()                                  # the tiny model: 5 tokens, a window of 2, 3 hidden units; seeded, so the same every runcontext, target = np.array([1, 3]), 4             # the window: token ids 1 and 3; the token that came next is id 4worst, step = 0.0, 1e-6                           # the nudge: small enough that the loss is nearly a straight line over itgradients = backprop(p, context, target)for name, value in p.items():                     # nudge each parameter both ways, and see how the loss moves    for i in np.ndindex(value.shape):        value[i] += step        up = forward(p, context, target)[3]       # forward returns (joined, hidden, logits, loss): index 3 is the loss        value[i] -= 2 * step                      # from up to down is two steps, so the estimate is centred on the value        down = forward(p, context, target)[3]        value[i] += step                          # back where it started        worst = max(worst, abs((up - down) / (2 * step) - gradients[name][i]))agrees = worst < 1e-6                             # the largest disagreement over all parameters, against six decimal placesassert agreesprint("loss:", round(float(forward(p, context, target)[3]), 4))print("parameters checked:", sum(v.size for v in p.values()))print("backprop agrees with nudging to 6 decimal places:", "yes" if agrees else "no")
출력python gradients-by-hand.py
loss: 1.5293
parameters checked: 45
backprop agrees with nudging to 6 decimal places: yes

All 45 gradients from backpropagation agree with the nudged estimates to six decimal places, which the example asserts. The nudging method costs two forward passes per parameter, which is why no one trains with it, but on a tiny model it's the reference that tells you backpropagation is right.

Working rule: Check hand-written gradients against nudged estimates on a tiny model before trusting them. When you use a library that computes gradients for you, as the PyTorch section does, check the library's forward pass against your own instead.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Take gradient descent steps

마지막으로 변경된 판: 2026.11.16판

With gradients in hand, training is a loop. Gradient descent moves every weight a small step against its gradient, w ← w − η · ∂L/∂w, where η, the step size, also called the learning rate, sets how far each step goes. Each step makes the loss a little smaller for the examples it was computed on, and repeated over many examples, the weights settle where the model predicts well.

Twenty steps of gradient descent at a step size of 0.5, using lib.tinynet's gradients, on three examples for the store's tiny model from the previous section, show the loss falling:

gradient-descent-by-hand.py Python 3.12.15 · numpy 2.5.5 통과
"""Twenty steps of gradient descent on three examples of the tiny model, printing the average loss before training andafter one, five and twenty steps. The prose's point rests on the average loss at those four points."""import numpy as npfrom lib.tinynet import backprop, forward, parametersp = parameters()                                             # seeded, so every run starts from the same weightsexamples = [(np.array([1, 3]), 4), (np.array([0, 1]), 3), (np.array([3, 4]), 2)]     # three (window, next token) pairs: each window is two token ids, then the id of the token after itrate = 0.5                                                   # the step sizedef average_loss():    return np.mean([forward(p, c, t)[3] for c, t in examples])print("average loss before:", f"{average_loss():.3f}")for step in range(1, 21):                                    # twenty steps, each one a pass over all three examples    for c, t in examples:        for name, gradient in backprop(p, c, t).items():            p[name] -= rate * gradient / len(examples)       # w <- w - rate * dL/dw, with each gradient averaged over the examples    if step in (1, 5, 20):        print(f"average loss after {step} steps:", f"{average_loss():.3f}")
출력python gradient-descent-by-hand.py
average loss before: 1.568
average loss after 1 steps: 1.346
average loss after 5 steps: 0.646
average loss after 20 steps: 0.109

The average loss falls at each step the example prints, from 1.568 before training to 0.109 after twenty. With only three examples to learn, the tiny model can drive its loss close to zero, which is memorizing; the store's model learns from hundreds of thousands of examples, where the same loop has to find patterns that hold across them.

Real training changes two things about this loop. Each step of real training is computed on a batch of examples, a few hundred at a time, instead of all of them, so each step is cheap and the steps vary a little. And it uses a refined update instead of a fixed step size, as the next section describes.

Working rule: Start any new model by running a few steps on a handful of examples and checking that the loss falls: a loss that won't fall on three examples means a bug, not a hard problem. Choose the step size by trying a few: too small and the loss crawls, too large and it jumps around or grows.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Train with PyTorch

마지막으로 변경된 판: 2026.11.16판

Writing backpropagation by hand for every model would be slow and error-prone. PyTorch, a widely used library for training neural networks, computes gradients automatically from the forward pass, and runs the arithmetic fast. The book trains its model with it, in recordings/neural_lm.py, which builds the same window model as the first section: an embedding table, one tanh layer and an output layer, trained on the store's training names. The heart of the script is the loop of the previous section, on batches of 512 predictions:

loss = loss_of(model(train_x[pick]), train_y[pick])   # forward pass and average loss on one batch
optimizer.zero_grad()                                  # clear the previous batch's gradients
loss.backward()                                        # backpropagation: a gradient for every weight
optimizer.step()                                       # move every weight against its gradient

The training script's optimizer is Adam, a refinement of gradient descent (Kingma and Ba, 2015). It keeps, for every weight, a running average of its gradients and a running average of their squares, and steps by the first divided by the square root of the second. A weight whose gradients are consistently large gets steps of about the same size as one whose gradients are small, so one step size works for all of them.

Training depends on the machine's floating-point arithmetic, which chapter 4 explains, so it runs once and the book's examples read the weights it saved, as they read every such result. The examples never import PyTorch: lib.neural recomputes the model's forward pass in NumPy, from those weights. That only works if the NumPy version computes what PyTorch computed, so the training script saved PyTorch's own logits for 64 validation contexts, and the next example compares:

same-as-pytorch.py Python 3.12.15 · numpy 2.5.5 통과
"""The trained model's logits, recomputed in NumPy from its saved weights and compared with the logits PyTorch computedfor the same saved validation contexts. The prose's point rests on the agreement, which the last line prints."""import numpy as npfrom lib import neuralweights, _, manifest = neural.load()check = np.load("data/recorded/neural-lm/check.npz")                 # saved by the training script, so this runs without PyTorch: contexts, and PyTorch's logits for themours = neural.forward(weights, check["contexts"])print("PyTorch version:", manifest["torch"])print("contexts compared:", len(check["contexts"]))print("logits compared:", f"{ours.size:,}")                          # every logit: one per context and per token in the vocabularyprint("same as PyTorch to 4 decimal places:", bool(np.abs(ours - check["logits"]).max() < 1e-4))   # 1e-4: four decimal places
출력python same-as-pytorch.py
PyTorch version: 2.14.1+cpu
contexts compared: 64
logits compared: 264,960
same as PyTorch to 4 decimal places: True

The NumPy forward pass reproduces all 264,960 of PyTorch's logits to four decimal places, for the model trained with PyTorch 2.14.1+cpu. Every number this chapter reports about the trained model comes from that NumPy code, except the training curve, which the script recorded as it trained.

Working rule: Train with a library that computes gradients for you, and keep a short, independent forward pass of your own to check its outputs, as lib.neural does here. Record the library's outputs for a few inputs whenever you save weights, so the check can run anywhere, long after training.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Stop when validation loss turns up

마지막으로 변경된 판: 2026.11.16판

A model with 1.4 million parameters, trained on a few hundred thousand tokens, can memorize its training text, as chapter 3's 5-gram did. Training perplexity keeps falling as it does. Validation perplexity falls at first, while the model learns patterns that hold for new names, then rises, once it starts fitting quirks of the training names. A validation perplexity that turns up while training perplexity keeps falling is overfitting, and the simplest defence is early stopping: measure validation loss after every epoch, one full pass over the training text, and keep the weights from the best epoch.

The training script recorded both perplexities after each of its twelve epochs. Read them for where validation stops improving:

training-curve.py Python 3.12.15 통과
"""The training script's record of training and validation perplexity after each epoch, and the epoch whose weights werekept. The prose's point rests on the validation column, and on the two validation figures printed after the table."""import jsonimport mathcurves = json.load(open("data/recorded/neural-lm/curves.json", encoding="utf-8"))manifest = json.load(open("data/recorded/neural-lm/manifest.json", encoding="utf-8"))print(f"{'epoch':>5}  {'training perplexity':>19}  {'validation perplexity':>21}")for c in curves:                                             # losses are averages of log probabilities: exp turns them into perplexities    print(f"{c['epoch']:>5}  {math.exp(c['train_loss']):>19.1f}  {math.exp(c['validation_loss']):>21.1f}")best = min(curves, key=lambda c: c["validation_loss"])       # the epoch with the lowest validation loss, which the kept epoch should matchprint("epoch kept:", manifest["kept_epoch"])print("validation perplexity at the kept epoch:", f"{math.exp(best['validation_loss']):.1f}")print("validation perplexity at the last epoch:", f"{math.exp(curves[-1]['validation_loss']):.1f}")   # curves[-1]: the final epoch's record, whatever was kept
출력python training-curve.py
epoch  training perplexity  validation perplexity
    1                109.3                   51.4
    2                 37.6                   39.9
    3                 27.3                   37.3
    4                 22.4                   36.6
    5                 19.3                   36.4
    6                 17.2                   36.9
    7                 15.6                   37.8
    8                 14.4                   38.6
    9                 13.5                   39.7
   10                 12.7                   40.9
   11                 12.0                   42.1
   12                 11.5                   43.7
epoch kept: 5
validation perplexity at the kept epoch: 36.4
validation perplexity at the last epoch: 43.7

In epoch 1, training perplexity is higher than validation, because the training figure averages over the whole epoch, including its first batches, when the model was still untrained, while validation is measured at the epoch's end. Validation perplexity bottoms out at 36.4 in epoch 5 and climbs from there, to 43.7 by the last epoch, while training perplexity falls at every epoch after the first. The training script kept the weights from the best epoch, and every result in the rest of this chapter uses them.

The gap between the two curves is the model memorizing. It grows every epoch, because the training names repeat their product lines and phrasing, and this model has no regularization, which is anything in training that discourages memorizing, so early stopping is its only defence.

Working rule: Always record validation loss per epoch and keep the best epoch's weights, never the last. When the curves part early, as they do here after a few epochs, the model has more capacity than its training text supports: regularize it, use a smaller model, or get more text of the same kind.

기준 모델과 비교

마지막으로 변경된 판: 2026.11.11판

진짜 시험은 이 책의 모든 모델이 치르는 것이다. 검증용 이름에서 토큰당 퍼플렉시티를 4-gram 기준 모델과 비교한다. 이 이름들에서는 비슷한 문맥을 서로 연관 짓는 것과 정확한 횟수를 세는 것 중 어느 쪽이 더 중요한가? 신경망 모델은 4-gram이 할 수 없는 일, 곧 비슷한 문맥을 서로 연관 짓는 일을 할 수 있다. 반면 4-gram은 본 적 있는 모든 문맥에 대해 정확한 횟수를 가지고 있다.

neural-vs-baseline.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""The window model and the 4-gram baseline, each scored by perplexity per token on the same validation names. Theprose's point rests on the two printed perplexities."""from lib import neuralfrom lib.names import name_tokensfrom lib.ngrams import KneserNey, perplexitytrain, validation = name_tokens("train"), name_tokens("validation")     # both models are scored on the same tokens of the same namesprint("window model validation perplexity:", f"{neural.perplexity(validation):.1f}")      # the window model, from its saved weightsprint("4-gram baseline validation perplexity:", f"{perplexity(KneserNey(train, 4), validation):.1f}")   # a 4-gram: each token is predicted from the three before it
출력python neural-vs-baseline.py
window model validation perplexity: 36.4
4-gram baseline validation perplexity: 23.1

횟수를 세는 모델이 큰 차이로 이긴다. 윈도 모델의 검증 퍼플렉시티는 36.4이고, 기준 모델은 23.1이다.

어디에서 차이가 나는지 보기 위해, 검증 토큰을 세 토큰짜리 문맥이 학습용 이름에 나타나는지에 따라 나누고 두 모델을 각 부분에서 평가한다.

seen-and-unseen-contexts.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""Splits the validation tokens by whether their three-token context occurs in the training names, and scores both modelson each part. The prose's point rests on the four perplexities printed for the two parts."""import mathimport numpy as npfrom lib import neuralfrom lib.names import name_tokensfrom lib.ngrams import KneserNey, ngrams, paddedtrain, validation = name_tokens("train"), name_tokens("validation")counted = KneserNey(train, 4)pairs = [(c, t) for text in validation for c, t in ngrams(padded(text, 4), 4)]   # padding gives each name's first tokens a contextseen = np.array([c in counted.totals[4] for c, _ in pairs])     # was this 3-token context in the training names?from_counts = np.array([counted.probability(c, t) for c, t in pairs])from_network = neural.token_probabilities(validation)           # in the same order as pairs, so each entry is for the same tokenprint("validation tokens with a context seen in training:", f"{seen.mean():.0%}")for part, where in [("seen", seen), ("unseen", ~seen)]:                  # ~seen: the complement, the contexts the training names lack    for model, p in [("4-gram", from_counts), ("network", from_network)]:        print(f"{model} perplexity, {part} contexts: {math.exp(-np.log(p[where]).mean()):.1f}")   # exp of the average surprise, over this part
출력python seen-and-unseen-contexts.py
validation tokens with a context seen in training: 70%
4-gram perplexity, seen contexts: 17.9
network perplexity, seen contexts: 26.1
4-gram perplexity, unseen contexts: 41.2
network perplexity, unseen contexts: 78.5

신경망은 두 부분 모두에서 더 나쁘며, 문맥을 연관 짓는 능력이 가장 도움이 되어야 할 본 적 없는 문맥에서 격차가 더 크다. 그 부분에서 4-gram은 41.2, 신경망은 78.5를 기록한다. 검증 토큰의 70%를 차지하는 본 적 있는 문맥에서는 4-gram이 17.9, 신경망이 26.1를 기록한다.

따라서 이 신경망은 횟수가 정확한 곳에서만 지는 것이 아니다. 일반화도 잘 배우지 못했다. 이 모델과 이 텍스트의 두 가지 특징이 가장 그럴듯한 이유이다. 첫째, 학습 곡선에서 보았듯이 이 신경망에는 정규화가 없어서 몇 에폭 뒤에 과적합된다. 둘째, 가게의 이름은 짧고 정형화되어 있다. 세 토큰의 문맥이 이름이 담은 내용의 대부분을 덮는데, 이는 Kneser-Ney의 횟수가 가장 강한 환경이다. Bengio와 동료들은 더 크고 다양한 텍스트에서 반대의 결과를 얻었다. 학습 단어가 약 800,000개인 Brown 코퍼스에서 그들의 최고 신경망 단독의 테스트 퍼플렉시티는 268이었고, 트라이그램 모델과 혼합하면 252였으며, 최고의 n-gram 모델은 312였다(Bengio et al., 2003, 4.2절과 표 1). 학습된 모델이라고 해서 저절로 더 나은 것은 아니다. 직접 측정해 보아야 한다.

작업 규칙: 학습된 모델이 도움이 된다고 주장하기 전에 항상 같은 텍스트와 같은 토큰에서 강력한 횟수 기반 기준 모델과 비교하고, 비교를 본 적 있는 문맥과 본 적 없는 문맥으로 나누어 각 모델이 어디에서 이기는지 확인한다. 작은 데이터셋 하나만 보고 신경망 모델이 통하지 않는다거나 언제나 통한다고 결론짓지 않는다.

신경망과 빈도 모델 섞기

마지막으로 변경된 판: 2026.11.11판

평균적으로 더 나쁜 모델이라도 그 실수가 더 나은 모델의 실수와 다르다면 여전히 쓸모가 있다. 두 모델이 서로 다른 실수를 할 때는 두 확률을 섞은 결과가 두 모델보다 나을 수 있다. Bengio와 동료들은 자신들의 신경망과 평활화한 트라이그램 모델로 이를 보였다(Bengio et al., 2003, 4절). 이 혼합은 3장의 보간에서처럼 각 토큰에 대한 두 모델의 확률을 가중 평균한 것이다.

혼합이 양쪽 모델을 모두 앞설까? 혼합에서 신경망의 비중을 0, 0.2, 0.3, 0.5, 1로 바꿔 가며 검증용 이름을 채점해 보자. 0은 빈도 모델만 쓰는 경우이고 1은 신경망만 쓰는 경우이다.

mix-neural-and-counts.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""Mixes the network's and the 4-gram's probability of every validation token, at five shares for the network, and printsthe validation perplexity of each mix. The prose's point rests on the printed perplexity at share 0.2, and at the two ends."""import mathimport numpy as npfrom lib import neuralfrom lib.names import name_tokensfrom lib.ngrams import KneserNey, ngrams, paddedtrain, validation = name_tokens("train"), name_tokens("validation")from_network = neural.token_probabilities(validation)           # the network's probability of each actual next token, in ordercounted = KneserNey(train, 4)# the 4-gram's probability for the same tokens, in the same order: each name padded with three start markers firstfrom_counts = np.array([counted.probability(c, t) for text in validation for c, t in ngrams(padded(text, 4), 4)])for share in [0.0, 0.2, 0.3, 0.5, 1.0]:                # the network's share of the mix; the counts get the rest    mixed = share * from_network + (1 - share) * from_counts    print(f"validation perplexity, network share {share:.1f}: {math.exp(-np.log(mixed).mean()):.1f}")
출력python mix-neural-and-counts.py
validation perplexity, network share 0.0: 23.1
validation perplexity, network share 0.2: 21.9
validation perplexity, network share 0.3: 22.1
validation perplexity, network share 0.5: 22.9
validation perplexity, network share 1.0: 36.4

혼합은 두 모델을 모두 앞선다. 신경망의 비중이 0.2일 때 검증 퍼플렉시티는 21.9로, 기준 모델의 23.1보다 좋고 신경망 단독의 36.4보다는 훨씬 좋다. 신경망은 평균적으로는 더 나쁘지만, 빈도 모델이 어떤 토큰에 매우 작은 확률을 주는 곳에서는 신경망의 비중 덕분에 그 확률이 극단적으로 작아지지 않고, 그 결과 평균 놀라움이 낮아진다.

선택한 에폭과 비중은 모두 검증용 이름을 보고 정했으므로 이 이득은 다소 낙관적이다. 이 책의 최종 비교가 테스트용 이름으로 이를 확인할 것이다. 또한 혼합이 도움이 되는 것은 두 모델의 오류가 서로 다르기 때문이다. 두 모델이 같은 실수를 한다면 평균을 내도 얻는 것이 없다.

작업 규칙: 학습한 모델과 빈도 기반 모델의 성능이 비슷하면 하나를 고르기 전에 두 확률을 섞어 보고, 비중은 따로 떼어 둔 텍스트에서 정한다. 한 모델이 훨씬 강할 때는 비중을 작게 유지하고, 혼합에 의존하기 전에 별도의 테스트 세트에서 이득을 확인한다.

더 많은 텍스트는 같은 종류일 때만 도움이 된다

마지막으로 변경된 판: 2026.11.11판

신경망이 너무 적은 텍스트에서 학습해 과적합된다면, 가장 먼저 떠오르는 해결책은 텍스트를 더 늘리는 것이다. 상점에는 상품 설명이라는 텍스트가 충분히 있다. 학습 스크립트는 같은 모델을 이름과 설명을 함께 사용해 학습시킬 수 있으며, 이 책은 그 버전도 같은 검증용 이름으로 평가해 기록해 두었다.

기록된 두 모델을 검증용 이름에서 비교해 보자. 하나는 이름만으로 학습했고, 다른 하나는 이름과 설명으로 학습했다.

more-text-of-another-kind.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""Scores the window model trained on names alone against the one trained on names and descriptions, counts the trainingtokens of each kind of text, and prints the second model's validation perplexity after each of its epochs. The prose'spoint rests on the two validation perplexities, and on the training token counts printed beside them."""import jsonimport mathfrom lib import neuralfrom lib.names import name_tokensvalidation = name_tokens("validation")for name in ["neural-lm", "neural-lm-all-text"]:               # the recorded runs: names alone, then names and descriptions    manifest = json.load(open(f"data/recorded/{name}/manifest.json", encoding="utf-8"))    print(f"learned from {manifest['settings']['text']}:", f"{neural.perplexity(validation, name):.1f}")from lib import bpefrom lib.store import load_productstokenizer = bpe.store_tokenizer()names_tokens = sum(len(t) for t in name_tokens("train"))      # counted in the store's tokens, the ones the model readsdescription_tokens = sum(len(tokenizer.encode(d)) for d in load_products("train")["description"])   # the training split only, as for namesprint("training tokens, names:", f"{names_tokens:,}")print("training tokens, names and descriptions:", f"{names_tokens + description_tokens:,}")curve = json.load(open("data/recorded/neural-lm-all-text/curves.json", encoding="utf-8"))print("all-text model, validation perplexity by epoch:", ", ".join(f"{math.exp(c['validation_loss']):.1f}" for c in curve))   # one per epoch, in order
출력python more-text-of-another-kind.py
learned from names: 36.4
learned from all: 59.0
training tokens, names: 337,071
training tokens, names and descriptions: 2,744,320
all-text model, validation perplexity by epoch: 73.6, 64.0, 60.8, 59.0

설명을 추가하면 모델은 이름에 대해 오히려 나빠진다. 학습용 이름은 337,071개의 토큰을 담고 있고, 이름과 설명을 합치면 2,744,320개로 약 여덟 배가 된다. 이름만으로 학습한 모델의 점수는 36.4이고, 이름과 설명으로 학습한 모델의 점수는 59.0이다. 두 번째 모델은 네 에폭 동안 학습했으며, 멈추었을 때 검증 퍼플렉시티는 에폭마다 더 작은 폭으로 계속 내려가는 중이었다. 그 추이는 다음과 같다. 73.6, 64.0, 60.8, 59.0 우리의 판단으로는 에폭을 더 늘려도 이름만으로 학습한 모델에 근접하지 못한다.

설명은 다른 종류의 텍스트이다. 설명은 완전한 문장으로 쓰이며, 이름에는 좀처럼 나오지 않는 your, this, perfect 같은 단어를 포함한다. 설명에서 주로 학습한 모델은 이름을 설명처럼 예측한다. 추가한 텍스트가 모델에 과제에 맞지 않는 분포를 가르친 것이다. 모델은 주어진 텍스트를 학습하고, 퍼플렉시티는 그와 같은 텍스트를 얼마나 잘 예측하는지를 측정한다. 다른 종류의 데이터는 모델을 더 나쁘게 만들 수 있으므로 가중치를 조정하거나 따로 분리해 두어야 한다.

작업 규칙: 학습 텍스트를 추가하기 전에, 그것이 예측하려는 텍스트와 닮았는지 확인한다. 많을수록 좋다고 가정하지 말고, 둘을 함께 학습시킨 뒤 대상 텍스트에서 측정한다. 추가 텍스트가 많이 다르면 대상 텍스트만으로 학습하거나, 모델이 텍스트의 종류를 구분할 수 있도록 각 텍스트를 배치한다. 8장에서 상품의 이름과 설명 사이에 구분자를 두는 것이 그 예이다.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Suggest from similar contexts

마지막으로 변경된 판: 2026.11.11판

The network lost on average, including on contexts the training names lack, but averages hide single cases. Chapter 3's autocomplete showed the 4-gram's worst case, outdoor bar, a context no training name contains, where it fell back to generic continuations.

Next, the network's three likeliest tokens after five name prefixes, and a few embeddings, tell whether outdoor bar gets the stool suggestion that patio bar does:

neural-autocomplete.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""The network's three likeliest next tokens after five typed prefixes, its probability of `stool` after `outdoor bar`,and the cosine similarity of `patio` with three other tokens in its embedding table. The prose's point rests on thesuggestions after each prefix, and on the three cosines printed last."""import numpy as npfrom lib import bpe, neuralfrom lib.ngrams import STARTweights, vocabulary, manifest = neural.load()index, width = {t: i for i, t in enumerate(vocabulary)}, manifest["settings"]["context"]tokenizer = bpe.store_tokenizer()for typed in ["bar", "patio bar", "outdoor bar", "kitchen bar", "wine bar"]:    # each typed prefix is read as the start of a name    context = ([START] * width + tokenizer.encode(typed))[-width:]               # pad with start markers, keep the last width tokens    logits = neural.forward(weights, np.array([[index[t] for t in context]]))[0]   # forward takes a batch: one row, so take row 0    probabilities = np.exp(neural.log_softmax(logits[None])[0])                 # log_softmax wants a batch too: add an axis, then take it off    best = np.argsort(-probabilities, kind="stable")[:3]                        # the three likeliest; a stable sort keeps ties in vocabulary order    # the end-of-word marker </w> is shown as nothing, so the tokens read as the words they came from    print(f"{typed:<12} -> " + ", ".join(f"{vocabulary[i].replace('</w>', '')} {probabilities[i]:.2f}" for i in best))outdoor_bar = ([START] * width + tokenizer.encode("outdoor bar"))[-width:]      # the same padding as abovep_stool = np.exp(neural.log_softmax(neural.forward(weights, np.array([[index[t] for t in outdoor_bar]])))[0, index["stool</w>"]])print("P(stool | outdoor bar):", f"{p_stool:.2f}")table = weights["embedding"] / np.linalg.norm(weights["embedding"], axis=1, keepdims=True)   # unit-length rows, so a dot product is a cosinefor a, b in [("patio</w>", "outdoor</w>"), ("patio</w>", "kitchen</w>"), ("patio</w>", "wine</w>")]:    print(f"cosine({a.replace('</w>', '')}, {b.replace('</w>', '')}) in the model's embeddings:", f"{float(table[index[a]] @ table[index[b]]):.2f}")
출력python neural-autocomplete.py
bar          -> with 0.22, cabinet 0.14, cart 0.13
patio bar    -> stool 0.63, with 0.23, & 0.05
outdoor bar  -> stool 0.40, with 0.13, & 0.12
kitchen bar  -> & 0.53, with 0.17, cart 0.09
wine bar     -> with 0.23, cabinet 0.16, </s> 0.14
P(stool | outdoor bar): 0.40
cosine(patio, outdoor) in the model's embeddings: 0.16
cosine(patio, kitchen) in the model's embeddings: 0.07
cosine(patio, wine) in the model's embeddings: 0.10

After outdoor bar, the network suggests stool first, with probability 0.40, where the 4-gram put & and with ahead of it. The control prefixes show that stool isn't automatic after bar: after bar alone, kitchen bar and wine bar, it isn't among the network's three suggestions. And in the network's own embedding table, patio is a little closer to outdoor, with a cosine similarity of 0.16, than to kitchen or wine. That's consistent with the network having carried something from patio bar over to outdoor bar, though the similarity is weak, and one prefix proves little.

This is one case, not an average: the seen and unseen split showed the network worse on unseen contexts overall. It shows what a learned model can do that counting can't, and that this one does it only sometimes.

Working rule: Look at a learned model's predictions on contexts the training text lacks, with control prefixes beside them, but judge it by measurements over many contexts, not by examples. For autocomplete in production, use the mix, and test it on rare prefixes as well as common ones.

운영 환경에서 잘못되는 것들

마지막으로 변경된 판: 2026.11.11판

신경망 언어 모델은 새로운 강점과 함께 새로운 실패 양상을 가져온다.

  • 과적합은 눈에 띄지 않게 일어난다. 학습 손실은 언제나 좋아 보인다. 검증 곡선이 없으면 너무 오래 학습한 모델이 덜 학습한 모델보다 나쁜 상태로 배포된다. 가장 좋은 에폭을 보관하고, 학습 손실과 검증 손실의 간격을 지켜본다.
  • 배포하는 가중치를 저장한다. 다른 프로세서에서 다시 학습하면 가중치가 조금 달라진다. 따라서 가중치를 만든 코드와 설정, 그리고 몇 가지 입력에 대한 라이브러리의 출력을 이곳의 학습 스크립트처럼 함께 보관한다.
  • 출력층이 비용의 대부분을 차지한다. 모든 단계에서 어휘의 모든 토큰에 점수를 매기는 일이 계산량과 매개변수의 대부분을 차지한다. 어휘가 커지면 가장 먼저 여기에서 비용이 든다.
  • 목표와 맞지 않는 학습 텍스트는 해가 된다. 이름을 다루는데 설명문을 추가한 경우처럼 목표 텍스트와 다른 텍스트를 더하면, 학습 손실은 괜찮아 보여도 예측은 나빠진다. 학습 데이터를 바꿀 때마다 목표 텍스트로 성능을 측정한다.
  • 평균은 드문 문맥을 가린다. 모델이 전체적으로는 높은 점수를 받으면서도, 구매자가 새로운 것을 찾을 때 입력하는 보기 드문 접두사에서는 실패할 수 있다. 이 장에서 했듯이 본 문맥과 보지 못한 문맥을 따로 채점하고, 보지 못한 문맥을 주시한다.
  • 혼합하려면 서빙 시점에 두 모델이 모두 필요하다. 신경망과 빈도 집계를 섞으면 적재하고 최신 상태로 유지해야 할 대상이 두 배가 된다. 신경망을 다시 학습할 때마다 빈도 집계도 다시 만들어, 둘이 같은 텍스트를 나타내게 한다.

이 장에서 결정한 것

  • 이 책의 신경망 모델은 PyTorch로 recordings/에서 한 번만 학습하며, 예제는 저장된 가중치로 NumPy에서 모델을 다시 계산한다. NumPy 순전파는 PyTorch의 로짓을 소수점 넷째 자리까지 재현하므로, 예제에는 PyTorch가 필요하지 않다.
  • 모델은 검증 성능이 가장 좋았던 에포크의 가중치를 유지한다. 윈도 모델의 검증 퍼플렉시티는 에포크 5 이후에 가장 낮았고, 그 뒤로는 매 에포크 올랐다. 반면 학습 퍼플렉시티는 계속 떨어졌다. 모델이 학습용 이름을 외우고 있었던 것이다.
  • 상점의 이름에서 윈도 모델은 단독으로는 4-그램 기준 모델에 지며, 본 문맥과 보지 못한 문맥 모두에서 그렇다. 둘을 섞으면 기준 모델을 이긴다. 신경망 비중 0.2에서 21.9이 나오며, 이는 지금까지 이 책의 모델 중 가장 좋은 값이다.
  • 학습된 모델은 평균만이 아니라 본 문맥과 보지 못한 문맥에서 따로 기준 모델과 비교한다. 두 부분이 서로 다른 이야기를 할 수 있기 때문이다. 여기서는 신경망이 두 경우 모두 졌고, 보지 못한 문맥에서 더 크게 졌다.
  • 학습 텍스트는 대상 텍스트와 닮아야 한다. 설명을 추가하자 모델은 이름에서 더 나빠졌다. 이름과 설명을 합친 토큰 수가 이름만의 약 여덟 배였는데도 그랬다. 그러므로 모든 모델의 학습 텍스트는 그 모델이 평가받을 텍스트에 맞추어 고른다.
  • 이름은 짧으므로 8장부터는 상점의 설명에 대해서도 모델을 측정한다.

이 판의 이 장

2026.11.16판예제 통과 2026년 11월 19이 장에서 10개 예제개 중 10개가 통과한다

영어판 2026.11.16판을 번역한 것이다.

이 절 인용

형식

각 형식은 이 번역이 따르는 영문 원서를 출처로 표기한다.

모든 형식은 이 판을 명시한다. 링크는 현재 판을 열며, 여기서 절의 제목은 본문이 마지막으로 바뀐 시점을 알려 준다. 접근 날짜와 검색 날짜는 이 자료를 여는 날짜다.

이 페이지의 내용

11장 중 5장: 신경 언어 모델

  1. Predict from a window of embeddings
  2. Compute gradients by backpropagation
  3. Take gradient descent steps
  4. Train with PyTorch
  5. Stop when validation loss turns up
  6. 기준 모델과 비교
  7. 신경망과 빈도 모델 섞기
  8. 더 많은 텍스트는 같은 종류일 때만 도움이 된다
  9. Suggest from similar contexts
  10. 운영 환경에서 잘못되는 것들
  11. 이 장에서 결정한 것

맨 위로 돌아가기전체 장