Words as vectors
읽는 데 약 40분분 걸린다, 10절에 8개 예제이(가) 있음.
AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식
영어판 2026.11.16판을 번역한 것이다. 원문 읽기
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
By the end of this chapter you can turn every token of the store's text into a short vector of numbers, so that tokens used in similar ways get similar vectors, by two methods: compressing counts of which tokens appear together, and training a small model to predict each token's neighbours. You'll measure what those vectors know, where they fail, and how they let a model relate text it has never seen to text it has.
Chapter 3 ended on a limit that counting can't get past. Its 4-gram model had seen patio bar stool many times, and suggested stool after patio bar. After outdoor bar, which no training name contains, it fell back to generic suggestions, because a count of one context tells it nothing about another. A shopper, or any reader, knows that outdoor and patio mean nearly the same thing to the store, and a model needs some way to know it too. This chapter gives every token a vector, a list of numbers, and arranges the vectors so that tokens used alike sit close together.
Vectors need text that shows how words get used, and the store, an online seller of furniture and home goods, has plenty of it in its catalog's names and descriptions. Its training split, the 80% of products that models learn from, holds 2,744,320 tokens of that text, and every vector in this chapter comes from it alone, through the store's tokenizer from chapter 2.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
One-hot vectors know nothing
Every model so far has treated tokens as names: patio is one entry in a table and outdoor another, with nothing linking them. Written as vectors, that's a one-hot encoding, in which each token gets a vector as long as the vocabulary, with a 1 in its own position and 0 everywhere else. To measure how alike two vectors are, this book uses cosine similarity: the cosine of the angle between them, 1 when they point the same way, 0 when they're at right angles. It ignores how long the vectors are and compares only their directions.
Give the 2,000 most frequent tokens of the training text one-hot vectors, and compare patio with outdoor, with sofa and with itself. Those 2,000 are about half of the store tokenizer's vocabulary, which keeps every matrix in this chapter small enough to compute in seconds:
1"""One-hot vectors for the 2,000 most frequent tokens, and the cosine similarity of patio with outdoor, with sofa and with2itself. The prose's point rests on the patio–outdoor and patio–sofa lines: patio is as unrelated to outdoor as to sofa."""3import numpy as np45from lib.vectors import BREAK, training_ids # the book's shared module; BREAK marks the gap between two texts67ids, vocabulary = training_ids() # each training token as a number; RARE for tokens outside the vocabulary8index = {token: i for i, token in enumerate(vocabulary)}91011def one_hot(token):12 vector = np.zeros(len(vocabulary))13 vector[index[token]] = 1.0 # one column per token: 1 for this token, 0 everywhere else14 return vector151617def cosine(a, b):18 return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))192021print("numbers in each vector:", len(vocabulary))22print("cosine(patio, outdoor):", cosine(one_hot("patio</w>"), one_hot("outdoor</w>"))) # </w> marks a word's last token: patio</w> is the whole word23print("cosine(patio, sofa):", cosine(one_hot("patio</w>"), one_hot("sofa</w>")))24print("cosine(patio, patio):", cosine(one_hot("patio</w>"), one_hot("patio</w>")))25print("share of training tokens with a vector:", f"{(ids >= 0).sum() / (ids != BREAK).sum():.0%}") # ids >= 0 are the kept tokens; BREAK gaps aren't tokens
python one-hot-vectors.pynumbers in each vector: 2000
cosine(patio, outdoor): 0.0
cosine(patio, sofa): 0.0
cosine(patio, patio): 1.0
share of training tokens with a vector: 91%
A one-hot encoding says the two outdoor words are exactly as unrelated as a patio and a sofa. The cosine similarity of patio and outdoor is 0.0, and so is that of patio and sofa. Only a token compared with itself scores 1.0. Each vector has 2000 numbers. The 2,000 tokens that get vectors make up 91% of the training text; the rest are rarer, and this chapter leaves them out. Every pair of different tokens sits at right angles to every other, so nothing learned about one can carry over to another.
That's the same flaw as chapter 3's counts, in another form. A model that reads one-hot vectors has to learn about every token separately, from the contexts that token itself appears in. Rare tokens get little, and unseen combinations get nothing.
Working rule: Don't feed one-hot vectors to a model that needs to relate tokens to each other. Use them only as an index, as the next sections do, to look up a learned vector for each token.
토큰을 주변 토큰으로 기술하기
이 문제에서 벗어나는 방법은 단어를 그 단어가 어울리는 이웃으로 기술하는 것이다. patio와 outdoor처럼 같은 이웃 사이에 나타나는 두 단어는, 예를 들어 둘 다 furniture, chair, umbrella 근처에 나타난다면, 서로 함께 나타나는 일이 드물더라도 비슷한 의미를 가질 가능성이 크다. 그러므로 각 토큰을 그 이웃으로 기술한다. 모든 토큰 쌍에 대해 한 토큰이 다른 토큰의 작은 윈도(window) 안에, 여기서는 양쪽으로 두 토큰 이내에 얼마나 자주 나타나는지 세며, 두 상품의 텍스트 사이의 경계를 넘어서는 세지 않는다. 그러면 각 토큰의 횟수 행은 그 토큰이 어떤 토큰 근처에 나타나는지를 나타내는 벡터가 된다.
학습 분할의 이름과 설명에서 동시 출현 횟수를 세고, patio 근처에 가장 자주 놓이는 토큰을 나열해 보자.
1"""Counts how often each of the 2,000 kept tokens appears within two tokens of another, and lists the six tokens seen most2often near patio. The prose's point rests on that list: punctuation and common words come first."""3import numpy as np45from lib.vectors import BREAK, cooccurrence, training_ids # the book's shared module; BREAK marks the gap between two texts67ids, vocabulary = training_ids() # each training token as a number; RARE for tokens outside the vocabulary8counts = cooccurrence(ids, len(vocabulary)) # how often each token appears within 2 tokens of another9patio = vocabulary.index("patio</w>") # </w> marks a word's last token: patio</w> is the whole word, not a piece of a longer one1011print("training tokens:", f"{int((ids != BREAK).sum()):,}") # RARE tokens count here, since they are tokens; only the BREAK gaps don't12print("vocabulary kept:", f"{len(vocabulary):,}")13print("tokens near patio, counting repeats:", f"{int(counts[patio].sum()):,}") # the row's total, a token counted again at each repeat14for i in np.argsort(-counts[patio], kind="stable")[:6]: # the minus sorts most frequent first; ties keep vocabulary order15 print(f" near patio: {vocabulary[i]:<12} {int(counts[patio, i]):>6,}")
python cooccurrence-counts.pytraining tokens: 2,744,320
vocabulary kept: 2,000
tokens near patio, counting repeats: 8,790
near patio: ,</w> 646
near patio: or</w> 448
near patio: with</w> 433
near patio: your</w> 381
near patio: .</w> 340
near patio: furniture</w> 249
patio에서 두 칸 이내에 있는 8,790개의 토큰 가운데, 같은 토큰이 나타날 때마다 다시 세었을 때 가장 빈도가 높은 것들은 실망스럽다. 쉼표, or, with, your, 마침표가 차지하고, 파티오와 관련이 있는 첫 단어인 furniture는 그 뒤에 나온다. 이런 토큰은 어디에나 있으므로 거의 모든 것의 근처에 나타난다. 원시 횟수로 이루어진 행은 대부분 각 이웃이 얼마나 흔한지를 말해 줄 뿐, 그 행의 주인인 토큰에 어떤 특별한 점이 있는지는 말해 주지 않는다.
따라서 원시 횟수는 좋은 벡터가 되지 못한다. 자주 나타나는 두 토큰은 그 의미와 상관없이 쉼표나 with 같은 흔한 이웃을 공유하므로 서로 비슷해 보인다. 중요한 것은 어떤 이웃이 그 이웃의 흔한 정도로 예상되는 것보다 더 자주 그 토큰 곁에 나타나는가이다.
작업 규칙: 모델이 학습할 텍스트에서 동시 출현을 두 토큰에서 다섯 토큰 사이의 작은 윈도 안에서 세고, 문서 경계를 넘어서는 세지 않는다. 원시 횟수를 그대로 벡터로 쓰지 않는다. 다음 절에서 하듯이 가중치를 다시 조정하여 빈번한 이웃이 결과를 지배하지 못하게 한다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Weigh co-occurrence by surprise
The fix compares each count with what chance would predict. If patio and furniture appeared independently of each other, they'd co-occur about as often as their separate frequencies imply. Pointwise mutual information, or PMI, measures how far the real count departs from chance: the log of the ratio between how often two tokens co-occur and how often they would by chance (Church and Hanks, 1990). In symbols, for tokens a and b:
PMI(a, b) = log [ P(a, b) / (P(a) · P(b)) ]
where P(a, b) is the share of all co-occurrence counts that are of a and b together, and P(a) and P(b) are each token's share. PMI is positive when two tokens appear together more than chance predicts, and negative when less. Negative values are unreliable, since "less than chance" usually means "rarely seen at all", so positive PMI, or PPMI, replaces every negative value with zero. The book's shared module lib.vectors computes it from the co-occurrence counts of the previous section, in three lines: the count chance would predict for each pair, the log of the real count over it, and zero wherever that's negative or the pair never occurs:
expected = counts.sum(1, keepdims=True) * counts.sum(0, keepdims=True) / total
pmi = np.log(counts / expected)
return np.where(counts > 0, np.maximum(pmi, 0), 0.0)
Reweighting is meant to stop frequent neighbours dominating, so swap the raw counts for PPMI and list patio's strongest neighbours again:
1"""Reweights the co-occurrence counts with positive PMI, and lists the six tokens with the strongest weight beside patio.2The prose's point rests on that list: the commas and common words of the raw counts are gone."""3import numpy as np45from lib.vectors import cooccurrence, ppmi, training_ids67ids, vocabulary = training_ids() # each training token as a number; RARE for tokens outside the vocabulary8weights = ppmi(cooccurrence(ids, len(vocabulary))) # PPMI of the co-occurrence counts, within the same window of 2 tokens9patio = vocabulary.index("patio</w>") # </w> marks a word's last token: patio</w> is the whole word1011print("share of PPMI entries that are zero:", f"{(weights == 0).mean():.0%}") # zero for pairs that never co-occur and for pairs seen no more often than chance12for i in np.argsort(-weights[patio], kind="stable")[:6]: # the largest weights first; ties keep vocabulary order13 print(f" PPMI with patio: {vocabulary[i]:<12} {weights[patio, i]:.2f}")
python ppmi-neighbours.pyshare of PPMI entries that are zero: 86%
PPMI with patio: porch</w> 4.51
PPMI with patio: deck</w> 4.19
PPMI with patio: backyard</w> 3.97
PPMI with patio: sectional</w> 3.75
PPMI with patio: furniture</w> 3.59
PPMI with patio: garden</w> 3.53
The commas are gone. The strongest neighbours of patio by PPMI are porch, deck, backyard, sectional, furniture and garden: words a person would associate with a patio, ranked by how much more often than chance they appear near it. Most pairs of tokens appear together no more often than chance, which is why 86% of the PPMI matrix is zero.
PPMI vectors are already useful: two tokens with similar PPMI rows have similar distinctive neighbours. But each vector is still 2,000 numbers long, mostly zeros, and two tokens with related but not identical neighbours, say deck and porch, can share few nonzero entries.
Working rule: Reweight co-occurrence counts with positive PMI before using them as vectors. Expect PMI to overrate pairs of rare tokens, whose few co-occurrences look far above chance; that's one reason to keep only frequent tokens, as this chapter does, or to require a minimum count.
카운트를 밀집 벡터로 압축하기
대부분이 0인 2,000개의 숫자로 이루어진 PPMI 벡터는 공간은 넓지만 정보는 적다. 이를 짧은 밀집 벡터로 압축하면 공기 출현의 가장 강한 패턴은 남기고 잡음은 버리며, 정확히 같은 이웃을 거의 공유하지 않더라도 관련된 이웃을 가진 토큰들이 서로 가까이 놓인다. 이때 쓰는 도구가 특이값 분해(singular value decomposition, SVD)이다. SVD는 PPMI 행을 가장 잘 복원하는 64개의 방향을 찾고, 각 방향이 행을 얼마나 설명하는지에 따라 가중치를 준다. 토큰의 새 벡터는 이 방향들을 따라 가중된 64개의 좌표이며, 이것이 임베딩(embedding), 즉 토큰을 나타내는 밀집 벡터이다. lib.vectors.svd_vectors 메서드가 이를 계산하고 각 벡터를 단위 길이로 조정하며, lib.vectors.nearest 메서드는 임의의 벡터와 코사인 유사도가 가장 높은 토큰을 찾는다.
여섯 토큰의 가장 가까운 이웃 네 개를 보면 64개의 숫자가 무엇을 보존하는지 알 수 있다.
1"""Compresses the PPMI matrix into 64 numbers per token, and lists the four nearest neighbours of six tokens. The prose's2point rests on the first two lists and the last line: patio and outdoor are each other's nearest neighbour, at the cosine printed last."""3from lib.vectors import DIMENSIONS, nearest, svd_vectors # svd_vectors does the PPMI and SVD steps; nearest ranks tokens by cosine similarity45vectors, vocabulary = svd_vectors() # one 64-number row per token, each scaled to unit length6index = {token: i for i, token in enumerate(vocabulary)}78print("dimensions per token:", DIMENSIONS)9for token in ["patio</w>", "outdoor</w>", "sofa</w>", "oak</w>", "blue</w>", "48</w>"]: # </w> marks a word's last token10 neighbours = nearest(vectors, vocabulary, vectors[index[token]], k=4, skip=(token,)) # skip leaves out the token itself, which would otherwise be its own nearest at 1.0011 print(f"{token:<12} -> " + ", ".join(f"{t} {s:.2f}" for t, s in neighbours))12print("cosine(patio, outdoor):", f"{float(vectors[index['patio</w>']] @ vectors[index['outdoor</w>']]):.2f}") # unit-length vectors, so the dot product is the cosine
python svd-neighbours.pydimensions per token: 64
patio</w> -> outdoor</w> 0.84, bistro</w> 0.79, porch</w> 0.79, lounge</w> 0.77
outdoor</w> -> patio</w> 0.84, porch</w> 0.82, indoor</w> 0.80, furniture</w> 0.74
sofa</w> -> sectional</w> 0.93, chaise</w> 0.90, loveseat</w> 0.89, lounge</w> 0.87
oak</w> -> walnut</w> 0.96, veneers</w> 0.90, espresso</w> 0.89, weathered</w> 0.88
blue</w> -> green</w> 0.86, beige</w> 0.86, pur 0.80, abstract</w> 0.79
48</w> -> 36</w> 0.94, 32</w> 0.93, 24</w> 0.92, 30</w> 0.89
cosine(patio, outdoor): 0.84
이제 모든 토큰은 64개의 숫자를 가지며, 그 이웃은 가구점을 위해 만든 유의어 사전처럼 읽힌다. patio와 outdoor는 서로에게 가장 가까운 이웃이고 코사인 유사도는 0.84이다. 원-핫 벡터에서는 이 값이 0이었다. sofa는 sectional, chaise, loveseat 옆에 놓이고, oak는 목재 마감인 walnut, espresso 옆에 놓인다. blue는 다른 색상 옆에, 48는 인치 단위의 다른 크기 옆에 놓인다.
이 중 어느 것도 가르친 적이 없다. 이 방법은 sofa의 정의도, 색상 목록도, 48가 숫자라는 힌트도 본 적이 없다. 다만 이 토큰들이 상점 텍스트에서 같은 종류의 이웃 사이에 나타난다는 사실을 찾아냈고, SVD는 그 때문에 이들을 가까이 놓았다. 이것이 3장의 카운트에는 없던 성질이다. 모델이 patio bar stool에 대해 학습한 내용은 이제 임베딩을 통해 outdoor bar stool에도 적용될 수 있다.
작업 규칙: PPMI를 SVD로 50차원에서 300차원 사이로 압축하고, 이 절에서 한 방식대로 결과를 점검한다. 토큰 몇 개의 가장 가까운 이웃을 나열해 읽어 본다. 이웃이 대부분 빈도 높은 기능어라면 재가중이 잘못된 것이고, 서로 무관하다면 윈도나 어휘가 너무 작은 것이다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Learn vectors by predicting neighbours
The other way to get embeddings is to learn them by prediction, which is how every neural language model in this book gets its own. Skip-gram with negative sampling, the method behind word2vec, gives each token two vectors, one for when it's the centre token and one for when it's a neighbour, and trains them on one task: given a token, tell its real neighbours apart from random tokens (Mikolov et al., 2013a; Mikolov et al., 2013b). For a token and one of its real neighbours, the model scores the pair by the dot product of their vectors, turns the score into a probability with the logistic function, and nudges the vectors so the probability rises. For a few random noise tokens drawn from the vocabulary, the model nudges the probability down. Repeated over millions of pairs, tokens with the same neighbours end up with similar vectors. That's no coincidence: Levy and Goldberg showed that skip-gram with negative sampling, trained this way, in effect factorizes a PMI matrix, shifted down by the log of the number of noise tokens, much as the SVD factorizes PPMI (Levy and Goldberg, 2014).
Write v for the centre token's vector, u for the real neighbour's, and uₙ for each noise token's, and σ for the logistic function, σ(s) = 1 / (1 + e⁻ˢ), which turns a score into a probability. The skip-gram loss for one pair is the surprise at the real neighbour plus the surprise at each noise token's not being one:
L = −log σ(v·u) − Σₙ log σ(−v·uₙ)
and lib.skipgram.step lowers it with three lines of gradient descent, where real is σ(v·u), fake holds σ(v·uₙ) for each noise token, and rate is the step size:
np.add.at(inputs, centre, -rate * grad_v) # move v: towards u, away from each uₙ
np.add.at(outputs, context, -rate * (real - 1)[:, None] * v) # move u towards v
np.add.at(outputs, noise, -rate * fake[:, :, None] * v[:, None, :]) # move each uₙ away from v
Run three updates on one real pair, patio and umbrella, with sofa and 48 as noise tokens. The centre vectors start small and random and the neighbour vectors at zero, as lib.skipgram.train starts them:
1"""Three updates of skip-gram with negative sampling on one real pair, patio and umbrella, with sofa and 48 as noise tokens,2printing each pair's probability before and after. The prose's point rests on the four probability lines: umbrella's rises,3and the noise tokens' fall."""4import numpy as np56from lib.skipgram import sigmoid, step # one gradient step, and the logistic function7from lib.vectors import training_ids89_, vocabulary = training_ids()10rng = np.random.default_rng(2026) # a fixed seed, so the starting vectors are the same on every run11inputs = (rng.random((len(vocabulary), 8)) - 0.5) / 8 # each token's vector as a centre, small and random12outputs = np.zeros((len(vocabulary), 8)) # and as a neighbour: zero, as training starts13centre, context = [vocabulary.index("patio</w>")], [vocabulary.index("umbrella</w>")] # the real pair: patio as the centre, umbrella its real neighbour14noise = np.array([[vocabulary.index("sofa</w>"), vocabulary.index("48</w>")]]) # noise tokens, picked by hand here; training draws them at random151617def chances():18 real = sigmoid(inputs[centre[0]] @ outputs[context[0]])19 fake = sigmoid(outputs[noise[0]] @ inputs[centre[0]])20 return real, fake212223before = chances() # zero neighbour vectors give every score zero, so every probability is one half24for _ in range(3): # three updates on the same pair25 step(np.array(centre), np.array(context), noise, inputs, outputs, rate=1.0) # rate 1.0, far above the 0.025 that training starts from, so three steps show26after = chances()27print("P(umbrella is a real neighbour of patio), before:", f"{before[0]:.3f}")28print("P(umbrella is a real neighbour of patio), after 3 steps:", f"{after[0]:.3f}")29print("P(sofa, 48 are real neighbours), before:", ", ".join(f"{p:.3f}" for p in before[1]))30print("P(sofa, 48 are real neighbours), after 3 steps:", ", ".join(f"{p:.3f}" for p in after[1]))
python skipgram-step.pyP(umbrella is a real neighbour of patio), before: 0.500
P(umbrella is a real neighbour of patio), after 3 steps: 0.525
P(sofa, 48 are real neighbours), before: 0.500, 0.500
P(sofa, 48 are real neighbours), after 3 steps: 0.475, 0.475
Training pulls real neighbours together and pushes random tokens apart, and three updates show it in miniature. Before any update, the neighbour vectors are zero, so every score is zero, and σ(0) is one half: the model knows nothing. After three, umbrella scores 0.525 as a neighbour of patio, up from 0.500, and sofa and 48 have dropped to 0.475, 0.475. That's all training does, millions of times over.
Working rule: When you train embeddings, start the second set of vectors at zero and the first at small random values, as lib.skipgram.train does, and draw noise tokens in proportion to their frequency raised to the power 0.75, which Mikolov et al. found works better than plain frequency. Check one update by hand before running millions: a probability moving the wrong way means the gradient's sign is wrong.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Compare learned vectors with counted ones
Training skip-gram on all the store's training text runs millions of updates, and its exact result depends on the machine's floating-point arithmetic, which varies slightly between processors. So it runs once, in recordings/skipgram.py, with the book's own code and a fixed seed, and the examples read its saved vectors. Its manifest records the settings: a window of two tokens, 64 dimensions, five noise tokens per pair and two passes over the text. One setting is fixed in lib.skipgram rather than recorded: pairs centred on very frequent tokens, such as commas, are mostly skipped, with a threshold of 10⁻⁴, which keeps them from dominating training (Mikolov et al., 2013b, who suggest around 10⁻⁵ for larger text).
Ask the recorded skip-gram vectors for the same six tokens' neighbours, and compare them with the SVD's:
1"""Reads the skip-gram vectors and training losses that recordings/skipgram.py saved, and lists the same six tokens' four2nearest neighbours as the SVD example. The prose's point rests on the two loss lines and on outdoor's nearest neighbour."""3import json45import numpy as np67from lib.vectors import nearest89recorded = "data/recorded/skipgram/" # saved by recordings/skipgram.py, which trains once: its values depend on the machine10vectors = np.load(recorded + "vectors.npy")11vocabulary = json.load(open(recorded + "vocabulary.json", encoding="utf-8"))12losses = np.load(recorded + "losses.npy") # one loss per batch, in training order13index = {token: i for i, token in enumerate(vocabulary)}1415print("training loss, first batch:", f"{losses[0]:.2f}") # index 0: the first batch, when nothing has been learned yet16print("training loss, last 100 batches:", f"{losses[-100:].mean():.2f}")17for token in ["patio</w>", "outdoor</w>", "sofa</w>", "oak</w>", "blue</w>", "48</w>"]:18 neighbours = nearest(vectors, vocabulary, vectors[index[token]], k=4, skip=(token,)) # skip leaves out the token itself19 print(f"{token:<12} -> " + ", ".join(f"{t} {s:.2f}" for t, s in neighbours))
python skipgram-neighbours.pytraining loss, first batch: 4.16
training loss, last 100 batches: 2.05
patio</w> -> porch</w> 0.77, bistro</w> 0.69, garden</w> 0.69, couch</w> 0.67
outdoor</w> -> indoor</w> 0.75, patio</w> 0.66, garden</w> 0.65, living</w> 0.61
sofa</w> -> couch</w> 0.85, sleeper</w> 0.84, chaise</w> 0.80, sectional</w> 0.78
oak</w> -> walnut</w> 0.91, espresso</w> 0.84, weathered</w> 0.81, glossy</w> 0.80
blue</w> -> beige</w> 0.88, gray</w> 0.87, green</w> 0.85, brown</w> 0.80
48</w> -> 36</w> 0.94, 32</w> 0.92, 1.5</w> 0.89, 24</w> 0.89
The model learned to tell real neighbours from noise: its training loss fell from 4.16 on the first batch to 2.05 on the last hundred. Its neighbours agree with the SVD's on most tokens, sofa with sectional and chaise, oak with walnut and espresso, colours with colours, sizes with sizes, and differ in emphasis on others: skip-gram puts outdoor closest to indoor, because the two words appear in the same positions in descriptions such as indoor or outdoor use.
That difference is worth noticing. Vectors built from neighbours put together words used in the same places, and opposites often are: indoor and outdoor here, and, in the next section, small, whose nearest neighbours include larger as well as smaller. Similar vectors mean similar use, not similar meaning.
Working rule: Treat counted embeddings (PPMI and SVD) and learned ones (skip-gram) as close alternatives: on a few million tokens they give similar neighbours, and counted ones are deterministic and fast. Prefer learned embeddings when they're part of a larger model you're training anyway, as in every later chapter of this book. Never read a high similarity as synonymy without checking for opposites.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Analogies, and where they fail
Embeddings famously support analogies by arithmetic: if the difference between chair and chairs is a direction that means "plural", then adding it to table should land near tables. In vectors, the answer to "a is to b as c is to what?" is the token nearest to b − a + c, leaving out the three tokens in the question (Mikolov et al., 2013a). It's a neat test of whether the vectors have learned consistent relationships, and a good place to see their limits.
Three analogies on the skip-gram vectors show where the arithmetic holds and where it breaks, and the nearest neighbours of three tokens they raise show why:
1"""Three analogies on the skip-gram vectors, each answered by the tokens nearest b minus a plus c; whether lamps and light2have vectors at all; and the nearest tokens to a few single words. The prose rests on the three analogy lines, the lamps line and the neighbour lists."""3import json45import numpy as np67from lib.vectors import nearest89vectors = np.load("data/recorded/skipgram/vectors.npy") # saved by recordings/skipgram.py, which trains once10vocabulary = json.load(open("data/recorded/skipgram/vocabulary.json", encoding="utf-8"))11index = {token: i for i, token in enumerate(vocabulary)}1213for a, b, c in [("chair", "chairs", "table"), ("chair", "chairs", "lamp"), ("black", "white", "dark")]: # each reads: a is to b as c is to what14 a, b, c = (word + "</w>" for word in (a, b, c)) # the tokenizer marks each word's last token with </w>, so words are looked up in that form15 query = vectors[index[b]] - vectors[index[a]] + vectors[index[c]] # b - a + c: "c, changed the way a became b"16 answers = nearest(vectors, vocabulary, query, k=3, skip=(a, b, c)) # the words in the question can't be their own answer17 print(f"{a} : {b} :: {c} : ? -> " + ", ".join(t for t, _ in answers))18print("lamps in the vocabulary:", "lamps</w>" in index)19print("light in the vocabulary:", "light</w>" in index)20for token in ["light</w>", "dark</w>", "small</w>"]:21 print(f"{token:<12} -> " + ", ".join(t for t, _ in nearest(vectors, vocabulary, vectors[index[token]], k=4, skip=(token,))))
python analogies.pychair</w> : chairs</w> :: table</w> : ? -> tables</w>, cup</w>, stools</w>
chair</w> : chairs</w> :: lamp</w> : ? -> tops</w>, coffee</w>, tray</w>
black</w> : white</w> :: dark</w> : ? -> brown</w>, blue</w>, glossy</w>
lamps in the vocabulary: False
light in the vocabulary: True
light</w> -> lights</w>, bulb</w>, dimmable</w>, candle</w>
dark</w> -> brown</w>, walnut</w>, gray</w>, espresso</w>
small</w> -> smaller</w>, larger</w>, displaying</w>, storing</w>
Only the first analogy gets the answer you'd expect: chair is to chairs as table is to tables, the top answer. The second fails because lamps isn't among the 2,000 tokens that have vectors, so the right answer can't be found at all, and the arithmetic returns whatever is nearest. The third fails differently. light is in the vocabulary, but black is to white as dark is to brown, by the vectors' reckoning. The neighbour lists show why: light sits with lights, bulb, dimmable and candle, because in the store's text it's mostly a lamp, as in ceiling light, not a shade, while dark sits with brown, walnut and espresso, the finishes it describes. And small sits with larger as well as smaller: size words appear in the same places, whichever size they name.
Analogies are a demonstration, not a measure. They work for relationships the text expresses consistently and often, such as plurals of common words, and fail for rare words, for words with several senses, and for relationships the text rarely spells out. A model doesn't use analogies; it uses the vectors' positions, and those carry the same strengths and blind spots.
Working rule: Don't judge embeddings by analogies; judge them by how much they help the task they're for, as the next section and every later chapter do. When an analogy fails, check whether the answer is in the vocabulary and whether the words mean what you assume in your text.
벡터로 비슷한 이름 찾기
토큰 임베딩에 한 단계를 더하면 텍스트 전체의 임베딩을 얻을 수 있다. 텍스트를 이루는 토큰의 벡터를 평균하고 그 결과를 단위 길이로 조정하면, 비슷한 텍스트는 비슷한 벡터를 갖게 된다. 이것은 검색 엔진에서 의미 기반 검색(semantic retrieval)이라고 부르는 방식, 곧 단어가 일치하는지가 아니라 의미로 텍스트를 찾는 방식의 거친 형태이다. 이 책의 짝이 되는 책 A Primer on Search Relevance는 학습된 모델로 이를 제대로 구축한다(semantic retrieval). 여기서는 이 장이 해결하려고 했던 한 가지를 시험하는 데 쓴다. 3장의 카운트 기반 모델은 outdoor bar와 patio bar를 연관 짓지 못한다.
모든 학습 이름의 skip-gram 벡터를 평균하면 각 이름이 하나의 벡터가 된다. 1장에서 설명했듯이 드문 단어, 대개 제품 라인의 이름은 [name]로 출력된다. 두 가지 질의로 그 효과를 확인한다. 하나는 3장의 4-gram 모델이 풀지 못했던 접두어 outdoor bar이고, 다른 하나는 patio umbrella이다.
1"""Averages each training name's skip-gram token vectors into one unit-length vector, and prints the four training names2nearest to each of two queries. The prose's point rests on the first query: two patio bar stool names come up for outdoor bar."""3import json45import numpy as np67from lib import bpe8from lib.store import load_products, masked910vectors = np.load("data/recorded/skipgram/vectors.npy")11vocabulary = json.load(open("data/recorded/skipgram/vocabulary.json", encoding="utf-8"))12index, tokenizer = {t: i for i, t in enumerate(vocabulary)}, bpe.store_tokenizer()131415def name_vector(text): # the average of the text's token vectors, at unit length16 found = [index[t] for t in tokenizer.encode(text) if t in index] # tokens outside the 2,000 kept have no vector, so they're left out17 average = vectors[found].mean(0) if found else np.zeros(vectors.shape[1]) # a text with no kept tokens gets the zero vector18 return average / (np.linalg.norm(average) or 1.0) # `or 1.0`: a zero average stays zero, rather than dividing by zero192021names = load_products("train")["name"].tolist() # the training split's product names, the texts to compare22matrix = np.array([name_vector(n) for n in names])23for query in ["outdoor bar", "patio umbrella"]:24 best = np.argsort(-(matrix @ name_vector(query)), kind="stable")[:4] # unit vectors, so the dot product is the cosine; the four largest, most similar first25 print(f"{query}: " + " | ".join(masked(names[i]) for i in best)) # [name]: a rare word
python name-vectors.pyoutdoor bar: outdoor modern industrial 5 piece bar height dining set | [name] bar & bar stool | 29.5 '' patio bar stool | 28.5 '' patio bar stool
patio umbrella: premium tight weave outdoor patio umbrella cover | patio lantern | garden patio 88 [name] rattan deck box | outdoor patio chair
평균한 벡터는 patio bar stool 이름 두 개를 outdoor bar와 가장 가까운 이름에 올려놓으며, 둘 다 outdoor라는 단어를 포함하지 않는다. 앞의 모델은 이 연결을 만들 수 없었다. patio bar stool에 대한 카운트는 있었지만 outdoor bar에 대한 카운트는 없었기 때문이다. 이 연결은 벡터가 서로 가까운 outdoor와 patio를 거쳐 이어진다. patio umbrella의 경우 가장 가까운 이름은 파티오 및 야외 제품이고 최상위는 파라솔 덮개이지만, 파라솔 자체인 것은 하나도 없다. 평균을 내면 patio가 umbrella보다 더 크게 작용하기 때문이다.
평균은 투박하다. 단어 순서를 잊기 때문에 bar stool과 stool bar이 같은 벡터를 갖고, 자주 나오는 토큰이 중요한 토큰을 묻어 버리기도 한다. 그래도 이 책의 이후 모든 모델이 기반으로 삼는 성질은 보여 준다. 원-핫 토큰 대신 임베딩을 읽는 모델은 한 구절에 대해 배운 것을 다른 구절로 옮길 수 있다.
작업 규칙: 짧은 텍스트를 순서와 무관하게 빠르게 표현해야 할 때는 토큰 임베딩을 평균하고, 관련 항목은 찾아내되 순위는 느슨하게 매겨진다고 예상한다. 본격적인 의미 기반 검색에는 Primer처럼 그 목적으로 학습된 모델을 사용한다. 언어 모델링에서는 임베딩 자체를 모델에 주고 결합 방법을 모델이 학습하게 하며, 5장에서 이를 시작한다.
운영 환경에서 문제가 생기는 지점
임베딩은 학습에 쓴 텍스트의 습성을 그것을 사용하는 모든 곳으로 가져간다.
- 비슷하게 쓰인다고 의미가 비슷한 것은 아니다.
indoor와outdoor는 서로 가까이에 놓인다. 가까운 벡터를 서로 바꿔 써도 되는 것으로 취급하는 시스템은 반대말을 아무렇지 않게 맞바꾼다. 제품이 의존하는 단어들의 최근접 이웃을 확인해야 한다. - 드물거나 새로운 토큰은 벡터가 부실하거나 아예 없다. 이 장에서는 토큰 2,000개를 남겼다. 이 목록 밖의 토큰에는 벡터가 없고, 몇 번밖에 나타나지 않은 토큰의 벡터에는 잡음이 많다. 저장소의 토크나이저에는 그 두 배가 넘는 토큰이 있으며, 토크나이저가 조각으로 나누는 드문 단어는 그 조각 가운데 2,000개에 속하는 것에서만 벡터를 얻는다. 이후 모든 장의 모델처럼 자체 임베딩을 학습하는 모델은 어휘 전체를 포괄한다.
- 벡터는 그 출처가 된 텍스트를 반영한다. 저장소의 텍스트는
light를 램프로 설명하므로, 그 벡터는 조명 관련 단어들과 함께 놓인다. 한 도메인에서 학습한 임베딩은 그 도메인의 의미와 편향을 다른 도메인으로 가져간다. - 레시피가 아니라 벡터를 배포한다. 경사 하강법으로 임베딩을 학습하면 프로세서마다 벡터가 조금씩 달라진다. 이 책이 벡터를 기록해 두는 이유가 여기에 있다. 배포하는 벡터를 그것을 만든 코드와 설정과 함께 저장해야 한다.
- 임베딩은 시간이 지나면 낡는다. 카탈로그에는 새로운 제품 라인과 단어가 나타난다. 어휘가 바뀌면 임베딩을 다시 학습하거나 확장하고, 다시 학습할 때마다 중요한 토큰의 이웃 목록을 확인해야 한다.
이 장에서 결정한 것
- 토큰은 벡터의 코사인 유사도로 비교한다. 코사인 유사도는 길이를 무시하고 방향을 비교하며,
lib.vectors.nearest가 반환하는 값이 바로 이것이다. 모든 벡터를 먼저 단위 길이로 맞추므로 코사인은 단순한 내적이 되고, 원래 벡터가 더 긴 빈출 토큰이 크기 때문에 유리해지는 일도 없다. - 카운트 기반 임베딩은 양의 PMI를 SVD로 압축하여 만든다.
lib.vectors에서 토큰 두 개 크기의 윈도와 가장 빈번한 2,000개 토큰을 대상으로 한다. 이 임베딩은 상점의 텍스트에서patio를outdoor곁에 놓았는데, 원-핫 벡터로는 할 수 없는 일이었다. - 학습 기반 임베딩은 네거티브 샘플링을 쓰는 스킵그램으로 만든다. 정확한 값이 기계에 따라 달라지므로, 이 책의 코드와 고정된 시드로
recordings/skipgram.py에서 한 번만 학습한다. 이 임베딩은 대부분의 토큰에서 카운트 기반 임베딩과 일치한다. - 비슷한 벡터는 비슷한 쓰임을 뜻한다.
indoor와outdoor처럼 서로 반대되는 말도 가까운 벡터를 갖는다. 유추는 텍스트에 자주 나타나는 관계에서만 성립하므로, 이 책은 유추가 아니라 임베딩이 작업에 얼마나 도움이 되는지로 임베딩을 평가한다. - 이후의 모든 모델은 자신의 임베딩을 직접 학습한다. 다음 토큰을 예측하는 과정의 일부로 학습하며, 5장의 신경 언어 모델이 그 시작이다. 이 장의 벡터는 임베딩이 지식을 한 맥락에서 다른 맥락으로 옮겨 준다는 점을 보여 주며, 이 책의 나머지 모델은 이를 활용한다.
이 판의 이 장
2026.11.16판예제 통과 2026년 11월 19이 장에서 8개 예제개 중 8개가 통과한다
영어판 2026.11.16판을 번역한 것이다.