Counting further back

읽는 데 약 39분분 걸린다, 9절에 7개 예제이(가) 있음.

2026.11.16판 예제 통과 2026년 11월 19 86개 예제 중 86개 통과 변경 사항

AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식

영어판 2026.11.16판을 번역한 것이다. 원문 읽기

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

By the end of this chapter you can build an n-gram language model that looks several tokens back, smooth it with Kneser-Ney so it handles text it hasn't seen (smoothing gives some probability to what the counts never saw, as chapter 1 showed), measure the baseline that every neural model in this book is measured against, and see why counting, however clever, runs out of road.

Chapter 1 built a model that predicted the next word from the one before it, and two things went wrong. It looked back only one word, so a shopper who typed patio bar got the suggestions for bar. And it gave probability zero to every pair of words it hadn't counted, which made its perplexity on new text infinite. Chapter 2 then gave the book better units to count: the store's tokenizer, which splits every word into pieces it knows, so almost every word is spelled from tokens it has seen. This chapter goes back to counting with those tokens, looks further back, and handles the unseen properly.

Every method here is tried where the store, an online furniture and home-goods shop, needs it: on the product names its search box completes. From this chapter on, every model in the book is scored the same way: by its perplexity per token on the store's validation names, tokenized with the store's tokenizer. Product names are what the search box's autocomplete has to predict, and they're short enough for every example to run in seconds. One fixed text, one tokenizer and one measure make every model in the book comparable with every other on names; later chapters add descriptions, from chapter 8, and per-word scores, in chapter 10, where a model's tokenizer differs.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Longer contexts fit the training text better

마지막으로 변경된 판: 2026.11.16판

An n-gram model predicts each token from the n − 1 tokens before it: a bigram model (n = 2) from one, a trigram model (n = 3) from two, and so on. The tokens before the one being predicted are its context. More context should mean better predictions: after patio bar, the next token is far more predictable than after bar alone. Counting works exactly as in chapter 1, except that the counts are now of a context of n − 1 tokens followed by a token, and each text is padded with n − 1 start markers, so even its first token has a full context.

Scored on the names they were counted from, with no smoothing, the orders show what a model does with text it has already memorized:

ngram-memorizes.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""Counts the n-grams of orders 1 to 5 in the training names, and scores each order on those same names with no smoothing.Read the training perplexity lines, which fall as the order grows, against the seen-once lines, which rise."""import mathfrom collections import Counterfrom lib.names import name_tokensfrom lib.ngrams import ngrams, paddedtrain = name_tokens("train")for n in [1, 2, 3, 4, 5]:    # padded adds n-1 start markers and one end marker, so a name's first token has a full context    pairs = Counter(pair for text in train for pair in ngrams(padded(text, n), n))    contexts = Counter()    for (context, _), count in pairs.items():        contexts[context] += count    # the total surprise, in nats: each n-gram's probability is its count over its context's count, the model's own estimate    surprise = -sum(count * math.log(count / contexts[context]) for (context, _), count in pairs.items())    print(f"{n}-gram training perplexity: {math.exp(surprise / sum(pairs.values())):.1f}")    print(f"{n}-grams seen once: {sum(1 for c in pairs.values() if c == 1) / len(pairs):.0%}")
출력python ngram-memorizes.py
1-gram training perplexity: 676.4
1-grams seen once: 8%
2-gram training perplexity: 23.9
2-grams seen once: 60%
3-gram training perplexity: 4.9
3-grams seen once: 76%
4-gram training perplexity: 3.2
4-grams seen once: 82%
5-gram training perplexity: 2.9
5-grams seen once: 85%

Perplexity falls fast as the context grows: 676.4 with no context at all, 23.9 with one token, and 2.9 with four. At four tokens of context the model is choosing among about three tokens at each step, because most n-grams that long occur only once: 85% of the 5-grams in the training names are seen a single time, and the model has memorized what followed each.

That's the trap. On the text it counted, a longer context always looks better, because with enough context every text is unique and the model can recite it. Scoring a model on its own training text measures memory, which is why chapter 1 set aside validation text in the first place.

Working rule: Never judge an n-gram model, or any language model, by its perplexity on its own training text: it falls toward 1 as the model memorizes. Choose the order by perplexity on held-out text, as the rest of this chapter does.

더 긴 문맥은 근거가 부족해진다

마지막으로 변경된 판: 2026.11.11판

학습 데이터의 이름을 외웠다고 해서 새로운 이름에 대해 무언가를 알게 되는 것은 아니다. 모델은 세어 본 문맥만 사용할 수 있는데, 가능한 문맥의 수는 길이에 따라 지수적으로 늘어난다. 토큰이 사천 개이면 가능한 쌍은 천육백만 개이고 가능한 세 토큰 조합은 육백사십억 개이다. 어떤 학습 텍스트도 이를 모두 포괄하지 못한다. 따라서 문맥이 길수록 새 텍스트가 모델이 한 번도 본 적 없는 문맥을 사용하는 경우가 더 잦아진다.

먼저 수치를 확인해 보자. 4-gram의 경우 검증 이름의 4-gram 가운데 학습 이름에 한 번도 나타나지 않는 비율은 얼마인가? 1부터 5까지 모든 차수의 비율은 다음과 같다.

unseen-ngrams.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""Counts, for each order from 1 to 5, the share of the validation names' n-grams that never occur in the training names.Compare the lines for 2-grams to 5-grams: that share rises with the order."""from lib.names import name_tokensfrom lib.ngrams import ngrams, paddedtrain, validation = name_tokens("train"), name_tokens("validation")print("training tokens:", f"{sum(len(t) for t in train):,}")print("validation tokens:", f"{sum(len(t) for t in validation):,}")for n in [1, 2, 3, 4, 5]:    seen = {pair for text in train for pair in ngrams(padded(text, n), n)}     # padded, as in the model, so name starts count too    held_out = [pair for text in validation for pair in ngrams(padded(text, n), n)]    unseen = sum(pair not in seen for pair in held_out)    print(f"{n}-grams in validation never seen in training: {unseen / len(held_out):.1%}")
출력python unseen-ngrams.py
training tokens: 337,071
validation tokens: 42,422
1-grams in validation never seen in training: 0.1%
2-grams in validation never seen in training: 12.7%
3-grams in validation never seen in training: 31.1%
4-grams in validation never seen in training: 43.9%
5-grams in validation never seen in training: 53.4%

문맥이 길수록 검증 이름의 n-gram 가운데 새로운 것이 더 많아진다. 검증 토큰 중 처음 보는 토큰은 0.1%에 불과하며, 이는 상점의 토크나이저가 제 역할을 하고 있다는 뜻이다. 검증 바이그램 중에서는 12.7%가 처음 보는 것이고, 4-gram에서는 43.9%, 5-gram에서는 53.4%가 그렇다. 문맥이 토큰 네 개일 때 모델은 예측에 필요한 것의 절반이 넘는 부분을 한 번도 본 적이 없다.

학습 텍스트를 늘려도 기대만큼 도움이 되지 않는다. 언어에서는 소수의 n-gram이 매우 자주 나타나고 대부분은 드물게 나타난다. 그래서 텍스트를 두 배로 늘릴 때마다 추가되는 것은 대개 한두 번만 나타나는 n-gram이며, 이는 어떤 하나의 예측을 뒷받침하는 횟수를 거의 개선하지 못한다. 상점의 카탈로그에서도 같은 양상을 작은 규모로 볼 수 있다. coffee table 같은 흔한 구절은 천 번이 넘게 세어지지만, 학습 이름의 4-gram 중 82%는 한 번만 세어진다. 측정한 결과가 아니라 우리의 판단이지만, 카탈로그가 열 배 크더라도 새로운 4-gram이 상당 비율 처음 보는 것으로 남을 것이다. 새로운 제품군과 크기 및 재질의 새로운 조합이 계속 등장하기 때문이다.

두 예제를 함께 보면 카운팅의 핵심 문제가 드러난다. 더 긴 문맥은 본 적이 있을 때 예측을 더 잘하지만, 본 적이 있는 경우는 점점 줄어든다. 이 장의 나머지 방법은 모두 카운트가 뒷받침할 때는 긴 문맥을 사용하고, 그렇지 않을 때는 더 짧은 문맥으로 물러나는 방식이다.

작업 규칙: n-gram 차수를 정하기 전에, 학습 텍스트에 한 번도 나오지 않는 보류된 n-gram의 비율을 측정하라. 경험칙으로, 그 비율이 상점 이름의 트라이그램처럼 삼분의 일에 가까워지면 그 차수의 모델은 처음 보는 것을 어떻게 다루느냐에 주로 좌우되므로, 차수보다 스무딩이 더 중요하다.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Mix short and long contexts

마지막으로 변경된 판: 2026.11.16판

The obvious fix for unseen long contexts is to fall back on shorter ones. Interpolation falls back smoothly: the probability of a token is a weighted mix of what the trigram, bigram and unigram counts say, so a long context the model has seen contributes its sharp prediction, and the shorter ones fill in what it hasn't. With weights λ₃, λ₂ and λ₁ that sum to one:

P(token | a b) = λ₃ · P₃(token | a b) + λ₂ · P₂(token | b) + λ₁ · P₁(token)

where each P is a maximum-likelihood estimate from counts, as in chapter 1's probabilities. Chen and Goodman's study of smoothing methods covers interpolation and its variants in detail (Chen and Goodman, 1996).

Hand-set weights make the idea plain. Take an interpolated trigram with 0.6, 0.3 and 0.09 for the three orders, and the last 0.01 spread evenly over every token, so no token gets probability zero. When a context was never seen, its weight passes to the orders below, so the probabilities still sum to one:

interpolate-orders.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""Builds an interpolated trigram model by hand, with fixed weights, and an add-one bigram model on the same tokens.The two validation perplexity lines to compare are the interpolated trigram's and the add-one bigram's."""import mathfrom collections import Counterfrom lib.names import name_tokensfrom lib.ngrams import ngrams, padded, perplexitytrain, validation = name_tokens("train"), name_tokens("validation")counts = {k: Counter() for k in (1, 2, 3)}             # counts[k][(the k-1 tokens before, token)]for text in train:    for context, token in ngrams(padded(text, 3), 3):        for k in (1, 2, 3):            # a trigram's context holds two tokens; an order-k model conditions on the last k-1 of them            counts[k][(context[3 - k:], token)] += 1totals = {k: Counter() for k in (1, 2, 3)}for k in (1, 2, 3):    for (context, _), count in counts[k].items():        totals[k][context] += count# the tokens the training names contain, plus one slot for a token they never containvocabulary = len({token for (_, token) in counts[1]}) + 1weights = {3: 0.6, 2: 0.3, 1: 0.09}                   # and 0.01 spread evenly over every tokenclass Interpolated:    n = 3    def probability(self, context, token):        p, carried = 0.0, 0.0        for k in (3, 2, 1):            # each order's own context            weight, total = weights[k] + carried, totals[k][context[3 - k:]]            if total:                                    # an unseen context passes its weight down                p, carried = p + weight * counts[k][(context[3 - k:], token)] / total, 0.0            else:                carried = weight        # the even 0.01 share, plus any weight passed down from unseen contexts, over the vocabulary        return p + (0.01 + carried) / vocabularyprint("interpolated 3-gram validation perplexity:", round(perplexity(Interpolated(), validation), 1))# a bigram's context is the one token before the tokenbigram_pairs = Counter((context[-1:], token) for text in train for context, token in ngrams(padded(text, 2), 2))bigram_totals = Counter()for (context, _), count in bigram_pairs.items():    bigram_totals[context] += countclass AddOne:    n = 2    def probability(self, context, token):        # add-one: one added to every pair's count, and one per vocabulary token added to the total        return (bigram_pairs[(context, token)] + 1) / (bigram_totals[context] + vocabulary)print("add-one 2-gram validation perplexity:", round(perplexity(AddOne(), validation), 1))
출력python interpolate-orders.py
interpolated 3-gram validation perplexity: 31.4
add-one 2-gram validation perplexity: 218.6

Mixing orders beats chapter 1's add-one smoothing. For a fair comparison, add-one here runs on a bigram model of the same validation tokens, since perplexities over different tokens can't be compared. The interpolated trigram scores 31.4, and add-one scores 218.6. Add-one spreads most of its probability over pairs that never happen; mixing orders keeps what each order knows.

It has two weaknesses. The weights are fixed, and these were picked by hand, not tuned: the same 0.6 goes to the trigram whether its context was seen once or a thousand times, though a context seen a thousand times deserves more trust. And the unigram part gives each token probability in proportion to how often it occurs, which, as the next section shows, is the wrong question for a fallback.

Working rule: Never use a single n-gram order alone; always combine it with lower orders, so every token keeps some probability. Prefer methods that set each context's weight from its counts, such as Kneser-Ney below, over fixed weights. If you do use fixed weights, choose them on held-out text, not the text you counted.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Count the contexts a token follows

마지막으로 변경된 판: 2026.11.16판

When a model backs off to a shorter context, it's asking a specific question: how likely is this token to show up in a context we haven't seen? Raw frequency answers a different one. A token can be very common and still occur only after one or two particular tokens, and such a token is a poor guess for a new context. The better measure is a token's continuation count, the number of different tokens it follows: a token that has followed many different tokens is likely to follow a new one too.

Among the 300 most frequent tokens in the training names, these three come after the fewest different tokens for how often they occur, set beside two ordinary product words:

continuation-counts.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""Counts, for each token in the training names, how often it occurs and how many different tokens come right before it.Look at the three narrowest tokens, which occur often but follow few different tokens, beside table and chair."""from collections import Counter, defaultdictfrom lib.names import name_tokensoccurrences, before = Counter(), defaultdict(set)for text in name_tokens("train"):    for previous, token in zip(["<s>"] + text, text):      # the start marker stands before each name's first token, so that token has something to follow too        occurrences[token] += 1        before[token].add(previous)                 # the different tokens this one followscommon = [t for t, _ in occurrences.most_common(300)]     # the 300 most frequent tokens: the pool the prose ranks# different tokens followed, per occurrence, lowest first: the three narrowestnarrow = sorted(common, key=lambda t: len(before[t]) / occurrences[t])[:3]for token in narrow + ["table</w>", "chair</w>"]:       # </w> marks a token that ends a word, as the store's tokenizer writes it    print(f"{token:<14} occurs {occurrences[token]:>5,} times, after {len(before[token]):>4,} different tokens")
출력python continuation-counts.py
tvs</w>        occurs   323 times, after    1 different tokens
wide</w>       occurs 1,985 times, after   14 different tokens
''</w>         occurs 10,335 times, after   85 different tokens
table</w>      occurs 3,487 times, after  154 different tokens
chair</w>      occurs 2,791 times, after  138 different tokens

The token tvs occurs hundreds of times, after only one other token: in the store's names it's always part of the same phrase, in products made for TVs. A model that backed off to raw frequency would think tvs a reasonable guess anywhere, and it almost never is. wide follows only a handful of tokens, mostly numbers' inch marks, and the inch mark itself, '', follows mostly digits. Meanwhile table and chair each follow well over a hundred different tokens: coffee table, side table, outdoor table. Their continuation counts say what their frequency doesn't: they turn up after well over a hundred different tokens.

Continuation counts are cheap to compute: for each token, keep the set of tokens seen before it, and count the set. They need no extra text and no tuning, only a different question asked of the same counts, and the next section builds on them.

This is the insight behind Kneser-Ney smoothing (Kneser and Ney, 1995): when the model falls back to a shorter context, it uses continuation counts instead of raw counts, so tokens that appear in many contexts get the probability, and tokens that appear in one fixed phrase don't.

Working rule: Weight tokens by how many different contexts they follow, not by how often they occur, whenever you back off from a long context to a short one. Use raw counts only at the highest order, where the context has actually been seen.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Smooth with Kneser-Ney

마지막으로 변경된 판: 2026.11.16판

Kneser-Ney smoothing combines interpolation of orders with continuation counts. At each order, Kneser-Ney subtracts a fixed discount d from every count it has seen, and gives the probability it took to the next order down, so a context seen often keeps nearly all its own prediction, and a context seen rarely leans on the shorter context, with the discount it frees going to tokens that turn up in many contexts.

The same formula applies at every order. Below the highest, the counts are continuation counts, except for n-grams that start a text, which keep their raw counts, as the KenLM toolkit does. For a context h and a token w, with c(h w) the count of the pair, c(h) the count of the context, and N(h) the number of different tokens seen after h:

P(w | h) = max(c(h w) − d, 0) / c(h) + d · N(h) / c(h) · P(w | shorter h)

The book's shared module lib.ngrams implements interpolated Kneser-Ney as Chen and Goodman describe it (Chen and Goodman, 1999), with a discount of 0.75, a common default. Its lowest order mixes in a uniform share over the vocabulary, so even a token never seen in training keeps a small probability.

Before trusting the module, work out one probability by hand: stool after a name that starts patio bar, order by order. Set it beside the module's answer, and beside the module's probabilities after this context, summed over every token:

kneser-ney-by-hand.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""Works out the probability of stool after a name that starts patio bar, order by order, with the chapter's formula.It checks the result against lib.ngrams and against the module's probabilities summed over the vocabulary: theby hand line should match the lib.ngrams line, and the summed line should be a probability distribution's total."""from lib import bpefrom lib.names import name_tokensfrom lib.ngrams import START, KneserNeymodel = KneserNey(name_tokens("train"), 4)     # a 4-gram model; below the top order its counts are continuation counts, except for n-grams that start a namecontext = tuple([START] + bpe.store_tokenizer().encode("patio bar"))     # a name that starts "patio bar"token, d = "stool</w>", model.d                # </w> ends a word; d is the discount, 0.75 by defaultdef by_hand(context, k):                         # the chapter's formula, one order at a time    if k == 0:        return 1 / (len(model.vocabulary) + 1)   # order 0 is uniform over the vocabulary, with one more slot for an unseen token    context = context[len(context) - (k - 1):] if k > 1 else ()     # the last k-1 tokens: the order-k model's own context    following, total = model.counts[k][context], model.totals[k].get(context, 0)    if total == 0:        return by_hand(context, k - 1)           # a context never seen gives all its weight to the order below    kept = max(following[token] - d, 0) / total                  # the discounted count    passed_down = d * len(following) / total                     # what the discount frees for the order below    print(f"order {k}: kept {kept:.4f}, passed down {passed_down:.4f}")    return kept + passed_down * by_hand(context, k - 1)print("by hand:", f"{by_hand(context, 4):.4f}")print("lib.ngrams:", f"{model.probability(context, token):.4f}")total = sum(model.probability(context, t) for t in model.vocabulary)print("probabilities after the context, summed:", f"{total:.4f}")
출력python kneser-ney-by-hand.py
order 4: kept 0.2500, passed down 0.7500
order 3: kept 0.8056, passed down 0.1667
order 2: kept 0.0630, passed down 0.1275
order 1: kept 0.0007, passed down 0.0318
by hand: 0.8621
lib.ngrams: 0.8621
probabilities after the context, summed: 1.0000

Worked by hand, the formula gives 0.8621 for stool, the same as the module's 0.8621. Each order keeps a share and passes the rest down; the shares add up to that answer. Over every token in the vocabulary, the module's probabilities after this context sum to 1.0000, as a probability distribution must.

Kneser-Ney models of every order from 1 to 5, scored on the validation names, show whether longer contexts now pay off:

kneser-ney-by-order.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""Builds Kneser-Ney models of orders 1 to 5 on the training names, scores each on the validation names, and estimates adiscount from the 4-gram counts. The lines to read are the validation perplexities, the 3-gram's set against the interpolated trigram's, and the discount line."""from lib.names import name_tokensfrom lib.ngrams import KneserNey, perplexitytrain, validation = name_tokens("train"), name_tokens("validation")for n in [1, 2, 3, 4, 5]:    model = KneserNey(train, n)    print(f"{n}-gram validation perplexity: {perplexity(model, validation):.1f}")model = KneserNey(train, 4)# the highest order keeps raw counts, not continuation counts, so these are how often each 4-gram occurs in trainingtop = [c for following in model.counts[4].values() for c in following.values()]once, twice = top.count(1), top.count(2)print("4-grams seen once:", f"{once:,}")print("4-grams seen twice:", f"{twice:,}")# Chen and Goodman's estimate of the discount: D = n1 / (n1 + 2 n2), with n1 and n2 the counts seen once and twiceprint("discount estimated from them:", f"{once / (once + 2 * twice):.2f}")
출력python kneser-ney-by-order.py
1-gram validation perplexity: 687.2
2-gram validation perplexity: 46.9
3-gram validation perplexity: 24.9
4-gram validation perplexity: 23.1
5-gram validation perplexity: 23.5
4-grams seen once: 164,374
4-grams seen twice: 18,359
discount estimated from them: 0.82

Kneser-Ney beats the hand-weighted interpolated trigram from the interpolation section at the same order, with no weights to choose. Its trigram scores 24.9, against 31.4 for the hand-weighted one.

Longer contexts now help on new text, instead of only on the text the model counted, up to a point: validation perplexity falls from 46.9 for the bigram to 23.1 for the 4-gram, then rises to 23.5 for the 5-gram. The step from trigram to 4-gram is small, and the 5-gram is worse, because by then most contexts are unseen and the extra order adds noise. The discount, too, can be estimated from the counts: Chen and Goodman give D = n₁ / (n₁ + 2 n₂), where n₁ and n₂ are the numbers of n-grams seen once and twice, and on the store's 4-grams it's 0.82, close to the default.

Working rule: Smooth n-gram models with interpolated Kneser-Ney and a discount of 0.75 unless held-out text says otherwise. Raise the order only while validation perplexity keeps falling noticeably, and stop when a step barely helps or hurts, as the step from 4-gram to 5-gram does. A variant, modified Kneser-Ney, uses three discounts instead of one and is the one toolkits such as KenLM build; it would likely lower the baseline a little, and this book keeps the single discount because it's easier to follow by hand.

세 토큰 문맥을 사용한 자동 완성

마지막으로 변경된 판: 2026.11.11판

더 긴 문맥은 1장에서 본 검색창의 최악의 제안을 바로잡을 것이다. 그 제안에서는 patio bar에 대한 제안이 bar에 대한 제안과 같았다. 4-gram Kneser-Ney 모델은 쇼핑객이 입력한 마지막 세 토큰을 보므로 patio가 문맥에서 사라지지 않는다.

bar에서 patio bar, outdoor bar으로 넘어가면 제안이 달라지는가? 각 접두어 뒤에 올 가능성이 가장 높은 다음 토큰 세 개를 4-gram 모델에서 뽑아 보자. 각 접두어는 쇼핑객의 입력이 그렇듯 이름의 시작으로 취급하며, 각 제안은 단어 끝 표시 없이 토큰만 보여 준다.

ngram-autocomplete.py Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1 통과
"""Asks the 4-gram model for its three likeliest next tokens after bar, patio bar and outdoor bar, each typed as the startof a name, then counts the training names that contain outdoor bar. The three suggestion lines carry the point, and the last linecounts the training names with outdoor bar in them."""from lib import bpefrom lib.names import name_tokensfrom lib.ngrams import START, KneserNeymodel = KneserNey(name_tokens("train"), 4)tokenizer = bpe.store_tokenizer()candidates = sorted(model.vocabulary)       # sorted first, so tokens with equal probability keep alphabetical orderfor typed in ["bar", "patio bar", "outdoor bar"]:    # padded with start markers, then the last three tokens: a short prefix still gives a full context    context = tuple([START] * 3 + tokenizer.encode(typed))[-3:]    ranked = sorted(candidates, key=lambda t: -model.probability(context, t))[:3]     # negated, so the likeliest sort first    print(f"{typed:<12} -> " + ", ".join(f"{t.replace('</w>', '')} {model.probability(context, t):.2f}" for t in ranked))# each name as one string of tokens, so the check below matches whole tokens rather than parts of themnames = [" ".join(t) for t in name_tokens("train")]print("training names containing 'outdoor bar':", sum("outdoor</w> bar</w>" in n for n in names))
출력python ngram-autocomplete.py
bar          -> & 0.23, unit 0.13, with 0.11
patio bar    -> stool 0.86, & 0.03, with 0.02
outdoor bar  -> & 0.23, with 0.19, stool 0.06
training names containing 'outdoor bar': 0

bar만 입력했을 때 모델은 &와 with 같은 연속을 제안한다. 이는 bar & counter stools처럼 시작하는 이름에 속하는 것이다. 단어의 위치를 무시한 1장의 바이그램은 stool을 가장 먼저 제안했다. patio bar 뒤에서는 stool을 높은 확률로 제안한다. 문맥이 예측을 바꾸었으며, 바이그램 모델은 이렇게 할 수 없다. 그러나 outdoor bar 뒤에서는 제안이 다시 &와 with 쪽으로 되돌아간다. 예제의 마지막 줄이 보여 주듯 학습 데이터의 어떤 이름도 outdoor bar을 포함하지 않아, 모델이 bar 뒤에 오는 것으로 후퇴하기 때문이다.

&나 with 같은 제안은 매장 이름에서 실제로 이어지는 연속이지만, 쇼핑객이 이를 제안받기를 원하는 경우는 드물다. 검색창은 1장에서 구두점에 대해 제안했듯이 이를 걸러 내거나, 단일 토큰 대신 전체 구로 완성할 수 있다. 이는 표시 방식의 선택이며, 모델의 확률은 어느 쪽이든 같다.

outdoor bar 줄은 개수 세기의 한계를 작게 보여 준다. 모델은 patio bar stool, outdoor patio, bar stool을 여러 번 보았으므로 야외 바에는 대개 스툴이 딸려 온다고 추측할 재료를 갖고 있다. 그러나 outdoor bar stool의 개수는 표의 별도 항목이며, 이를 patio bar stool과 이어 주는 것은 아무것도 없다. 모델에는 outdoor와 patio가 비슷하다는 개념이 필요한데, 개수 세기에는 그것이 없다. 4장에서 매장의 토큰에 그 개념을 부여한다.

작업 규칙: 제안에는 모델이 지원할 수 있는 가장 긴 문맥을 사용하고, outdoor bar처럼 학습 데이터에서 드문 문맥에 대한 제안은 흔한 문맥과 따로 확인한다. 개수 기반 모델이 일반적인 추측으로 후퇴하는 곳이 바로 거기이기 때문이다.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

The baseline every later model is measured against

마지막으로 변경된 판: 2026.11.16판

Every chapter from here on builds a model that learns, not one that counts, and each claims to predict the store's text better. A claim like that needs a baseline, a fixed yardstick: one model, one text, one tokenizer and one measure that every later model is compared against. A baseline should be the strongest simple model available, so that beating it means something.

The baseline for this book is the 4-gram Kneser-Ney model of this chapter, scored on the store's validation names with the store's tokenizer. Its validation perplexity is 23.1 per token, from the Kneser-Ney example. The 5-gram model is worse, at 23.5, and holds more counts, so the 4-gram is the baseline: the best of this chapter's counted models on the store's names. A stronger counted baseline exists, modified Kneser-Ney, and the book doesn't use it, for the reason the Kneser-Ney section gives.

The baseline also sets expectations. A perplexity near twenty means the model is, on average, choosing among about twenty likely tokens at each step of a product name. Some steps are nearly certain, such as table after coffee, and some are wide open, such as the first token of a product line's name. A neural model that beats the baseline should do so by predicting the open steps better, because it can relate one context to another, not by memorizing more.

Two things about the baseline will stay fixed for the rest of the book. Every perplexity is per token of the store's tokenizer, on the validation names, so the numbers in every chapter can be set side by side. And the test names stay untouched until the book's final comparison, so nothing in the book is tuned on them.

Working rule: Before building a learned model, build the strongest counted model you can and record its held-out perplexity as the baseline. Report every later model against it, on the same text and with the same tokenizer, and treat a learned model that doesn't beat it as a result to explain, never as a success. Later chapters find exactly that, and earn the learned models their place another way.

운영 환경에서 문제가 되는 점

N-gram 모델은 퍼플렉서티를 마지막 한 점까지 낮추는 것보다 속도와 투명성이 더 중요한 곳, 예를 들어 빠른 자동 완성에서 여전히 유용하다. 이 모델이 가져오는 문제는 다음과 같다.

  • 표가 커진다. 차수가 하나 늘 때마다 빈도 표가 하나 더 생기며, 서로 다른 n-gram의 수는 텍스트의 크기와 거의 같은 속도로 늘어난다. 한 번만 나타난 n-gram은 가지치기하고, 일부 드문 n-gram은 실제 정보를 담고 있으므로 가지치기 후의 검증 퍼플렉서티를 측정한다.
  • 드문 문맥에는 일반적인 답이 나온다. outdoor bar에서 보았듯이, 모델이 거의 본 적 없는 문맥은 더 짧은 문맥이 시사하는 쪽으로 되돌아간다. 흔한 접두어만이 아니라 드문 접두어에 대한 추천도 모니터링한다.
  • 할인값은 선택의 문제이다. 0.75라는 할인값은 타당한 기본값일 뿐 법칙이 아니다. 텍스트가 매장의 텍스트와 크게 다르다면 따로 떼어 둔 텍스트로 값을 조정한다.
  • 새 텍스트는 빈도를 바꾼다. 제품군은 생기고 사라지며, 모든 추천의 바탕이 되는 빈도는 마지막으로 다시 구축한 시점까지만 최신이다. 일정에 따라 다시 구축하고, 최근 텍스트에서의 검증 퍼플렉서티를 기준값과 비교하여 모델이 낡은 시점을 파악한다.
  • 확률의 합은 1이어야 한다. 직접 작성한 백오프 방식은 확률을 잃거나 이중으로 계산하는 경우가 많으며, 그러면 퍼플렉서티가 실제보다 좋거나 나빠 보인다. 몇 개의 문맥에서 각 문맥의 어휘 전체에 대한 확률의 합이 1인지 확인한다.

이 장에서 결정한 것

  • 이 책의 모든 모델은 저장소의 검증용 이름에 대해, 저장소의 토크나이저를 거쳐 토큰당 퍼플렉서티로 평가한다. 텍스트, 토크나이저, 척도를 하나로 통일했으므로 모든 장의 수치를 이름 기준으로 비교할 수 있다. 테스트용 이름은 11장에서 이루어지는 이 책의 최종 비교를 위해 남겨 둔다.
  • n-그램 모델은 할인값 0.75의 보간 크네저-네이 방식으로 평활화한다. 같은 차수에서 손으로 가중치를 정한 보간보다 성능이 좋았고, 조정할 가중치도 없다. 각 문맥을 그 횟수에 비례하여 신뢰하고, 많은 문맥 뒤에 나오는 토큰으로 백오프하기 때문이다.
  • 기준선은 4-그램 크네저-네이 모델이다. 검증용 이름에서 토큰당 23.1이다. 5-그램 모델은 점수가 더 나쁘고 횟수는 더 많이 저장하므로, 4-그램 모델을 기준 척도로 삼는다.
  • 자동 완성은 토큰 세 개 길이의 문맥을 쓰는 4-그램 모델의 제안을 사용한다. 이 방식은 patio bar을(를) 바로잡지만, outdoor bar처럼 드문 문맥에서는 여전히 일반적인 제안으로 되돌아간다. 학습된 모델이 더 잘해야 하는 부분이 바로 여기다.
  • 문맥이 길어져도 더 이상 도움이 되지 않는 것은 문맥이 중요하지 않아서가 아니라 본 적이 없어서다. 검증용 이름의 5-그램 중 절반이 넘는 수가 학습 데이터에 한 번도 나타나지 않으며, 모델은 본 적 없는 문맥을 비슷한 기존 문맥과 연결하지 못한다. 4장은 어떤 토큰끼리 비슷한지를 나타내는 벡터를 토큰에 부여하는 방식으로 이 문제를 다루기 시작한다.

이 판의 이 장

2026.11.16판예제 통과 2026년 11월 19이 장에서 7개 예제개 중 7개가 통과한다

영어판 2026.11.16판을 번역한 것이다.

이 절 인용

형식

각 형식은 이 번역이 따르는 영문 원서를 출처로 표기한다.

모든 형식은 이 판을 명시한다. 링크는 현재 판을 열며, 여기서 절의 제목은 본문이 마지막으로 바뀐 시점을 알려 준다. 접근 날짜와 검색 날짜는 이 자료를 여는 날짜다.

이 페이지의 내용

11장 중 3장: Counting further back

  1. Longer contexts fit the training text better
  2. 더 긴 문맥은 근거가 부족해진다
  3. Mix short and long contexts
  4. Count the contexts a token follows
  5. Smooth with Kneser-Ney
  6. 세 토큰 문맥을 사용한 자동 완성
  7. The baseline every later model is measured against
  8. 운영 환경에서 문제가 되는 점
  9. 이 장에서 결정한 것

맨 위로 돌아가기전체 장