Words as vectors
閱讀約需 44 分鐘 分鐘,10 節 中共有 8 個範例。
此部分尚未翻譯,因此以英文呈現。
By the end of this chapter you can turn every token of the store's text into a short vector of numbers, so that tokens used in similar ways get similar vectors, by two methods: compressing counts of which tokens appear together, and training a small model to predict each token's neighbours. You'll measure what those vectors know, where they fail, and how they let a model relate text it has never seen to text it has.
Chapter 3 ended on a limit that counting can't get past. Its 4-gram model had seen patio bar stool many times, and suggested stool after patio bar. After outdoor bar, which no training name contains, it fell back to generic suggestions, because a count of one context tells it nothing about another. A shopper, or any reader, knows that outdoor and patio mean nearly the same thing to the store, and a model needs some way to know it too. This chapter gives every token a vector, a list of numbers, and arranges the vectors so that tokens used alike sit close together.
Vectors need text that shows how words get used, and the store, an online seller of furniture and home goods, has plenty of it in its catalog's names and descriptions. Its training split, the 80% of products that models learn from, holds 2,744,320 tokens of that text, and every vector in this chapter comes from it alone, through the store's tokenizer from chapter 2.
One-hot 向量什麼都不知道
到目前為止,每個模型都把詞元當成名稱來處理:patio 是表中的一個項目,outdoor 是另一個項目,兩者之間沒有任何連結。若寫成向量,這就是 one-hot 編碼:每個詞元得到一個與詞彙表等長的向量,在它自己的位置上為 1,其餘位置全為 0。要衡量兩個向量有多相似,本書使用餘弦相似度(cosine similarity):即兩向量夾角的餘弦值,方向相同時為 1,互成直角時為 0。它不考慮向量的長度,只比較方向。
為訓練文本中最常出現的 2,000 個詞元建立 one-hot 向量,然後將 patio 與 outdoor、與 sofa,以及與它自己相比較。這 2,000 個詞元約為商店文本分詞器的詞彙表的一半,使本章的每個矩陣都小到幾秒內即可算完:
1"""One-hot vectors for the 2,000 most frequent tokens, and the cosine similarity of patio with outdoor, with sofa and with2itself. The prose's point rests on the patio–outdoor and patio–sofa lines: patio is as unrelated to outdoor as to sofa."""3import numpy as np45from lib.vectors import BREAK, training_ids # the book's shared module; BREAK marks the gap between two texts67ids, vocabulary = training_ids() # each training token as a number; RARE for tokens outside the vocabulary8index = {token: i for i, token in enumerate(vocabulary)}91011def one_hot(token):12 vector = np.zeros(len(vocabulary))13 vector[index[token]] = 1.0 # one column per token: 1 for this token, 0 everywhere else14 return vector151617def cosine(a, b):18 return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))192021print("numbers in each vector:", len(vocabulary))22print("cosine(patio, outdoor):", cosine(one_hot("patio</w>"), one_hot("outdoor</w>"))) # </w> marks a word's last token: patio</w> is the whole word23print("cosine(patio, sofa):", cosine(one_hot("patio</w>"), one_hot("sofa</w>")))24print("cosine(patio, patio):", cosine(one_hot("patio</w>"), one_hot("patio</w>")))25print("share of training tokens with a vector:", f"{(ids >= 0).sum() / (ids != BREAK).sum():.0%}") # ids >= 0 are the kept tokens; BREAK gaps aren't tokens
python one-hot-vectors.pynumbers in each vector: 2000
cosine(patio, outdoor): 0.0
cosine(patio, sofa): 0.0
cosine(patio, patio): 1.0
share of training tokens with a vector: 91%
one-hot 編碼的結果是:這兩個戶外用詞之間毫無關聯的程度,與露台和沙發之間完全相同。patio 與 outdoor 的餘弦相似度是 0.0,patio 與 sofa 的餘弦相似度也是如此。只有詞元與自己比較時,分數才是 1.0。每個向量有 2000 個數字。獲得向量的這 2,000 個詞元占訓練文本的 91%;其餘的詞元較為罕見,本章不予處理。任兩個不同詞元的向量都互成直角,因此學到的關於某一個詞元的任何知識,都無法延伸到另一個詞元。
這與第 3 章的計數方法是同一個缺陷,只是換了一種形式。讀取 one-hot 向量的模型必須逐一學習每個詞元,而且只能依據該詞元自身出現過的上下文。罕見的詞元得到的資訊很少,未曾見過的組合則什麼也得不到。
**實用原則:**不要把 one-hot 向量餵給需要讓詞元彼此建立關聯的模型。只把它們當作索引使用,如後續各節所做的,用來查找每個詞元的學得向量。
以周圍的詞元描述一個詞元
解決之道,是從一個詞所交往的「夥伴」來描述它。兩個詞若出現在相同的鄰近詞之間,例如patio與outdoor都會出現在furniture、chair和umbrella附近,那麼即使它們很少一起出現,意思也可能相近。因此,我們用鄰近詞元來描述每個詞元:對每一對詞元,計算其中一個出現在另一個的小視窗(window)內的次數,這裡取前後各兩個詞元,而且不跨越兩件商品文字之間的間隔。每個詞元的計數列,就成為一個表示它出現在哪些詞元附近的向量。
在訓練切分的名稱與描述中計算共現次數,並列出最常出現在patio附近的詞元:
1"""Counts how often each of the 2,000 kept tokens appears within two tokens of another, and lists the six tokens seen most2often near patio. The prose's point rests on that list: punctuation and common words come first."""3import numpy as np45from lib.vectors import BREAK, cooccurrence, training_ids # the book's shared module; BREAK marks the gap between two texts67ids, vocabulary = training_ids() # each training token as a number; RARE for tokens outside the vocabulary8counts = cooccurrence(ids, len(vocabulary)) # how often each token appears within 2 tokens of another9patio = vocabulary.index("patio</w>") # </w> marks a word's last token: patio</w> is the whole word, not a piece of a longer one1011print("training tokens:", f"{int((ids != BREAK).sum()):,}") # RARE tokens count here, since they are tokens; only the BREAK gaps don't12print("vocabulary kept:", f"{len(vocabulary):,}")13print("tokens near patio, counting repeats:", f"{int(counts[patio].sum()):,}") # the row's total, a token counted again at each repeat14for i in np.argsort(-counts[patio], kind="stable")[:6]: # the minus sorts most frequent first; ties keep vocabulary order15 print(f" near patio: {vocabulary[i]:<12} {int(counts[patio, i]):>6,}")
python cooccurrence-counts.pytraining tokens: 2,744,320
vocabulary kept: 2,000
tokens near patio, counting repeats: 8,790
near patio: ,</w> 646
near patio: or</w> 448
near patio: with</w> 433
near patio: your</w> 381
near patio: .</w> 340
near patio: furniture</w> 249
在patio前後兩個位置之內共有8,790個詞元,同一詞元每次出現都重複計算。其中最常見的結果令人失望:逗號、or、with、your和句號,接著才出現第一個與露臺有關的詞furniture。這些詞元幾乎出現在所有詞的附近,因為它們無所不在。原始計數的一列,多半只反映各鄰近詞本身有多常見,而不是這一列所屬詞元有何特別之處。
因此,原始計數不適合作為向量。兩個經常出現的詞元,會因為共有相同的高頻鄰近詞(逗號和with)而看起來相似,無論它們實際的意思為何。重要的是:相較於各鄰近詞本身的常見程度,哪些鄰近詞與某個詞元同時出現的頻率高於預期。
實用原則: 在模型要學習的文字中,於小視窗(兩到五個詞元)內計算共現次數,且不跨越文件邊界。不要直接把原始計數當作向量:要重新加權,如下一節所做的,讓高頻鄰近詞不再佔據主導。
以意外程度加權共現
修正的方法,是把每個計數與機率預期的數值相比。若 patio 與 furniture 彼此獨立出現,兩者共現的次數應該大致等於各自頻率所隱含的次數。逐點互資訊(pointwise mutual information,PMI)衡量實際計數偏離機率預期的程度:取兩個詞元實際共現的頻率,除以它們在機率下預期共現的頻率,再取對數(Church and Hanks, 1990)。以符號表示,對詞元 a 與 b:
PMI(a, b) = log [ P(a, b) / (P(a) · P(b)) ]
其中 P(a, b) 是所有共現計數中 a 與 b 一起出現的比例,P(a) 與 P(b) 則是各詞元自身的比例。兩個詞元一起出現的次數多於機率預期時,PMI 為正;少於預期時,PMI 為負。負值並不可靠,因為「少於機率預期」通常只表示「幾乎沒看過」,所以正 PMI(positive PMI,PPMI)把每個負值都換成零。本書的共用模組 lib.vectors 以前一節的共現計數來計算它,只需三行:先算出每一對詞元在機率下預期的計數,再取實際計數除以預期計數的對數,凡是結果為負或該詞對從未出現之處一律設為零:
expected = counts.sum(1, keepdims=True) * counts.sum(0, keepdims=True) / total
pmi = np.log(counts / expected)
return np.where(counts > 0, np.maximum(pmi, 0), 0.0)
重新加權的目的,是避免高頻鄰居佔據主導地位,因此把原始計數換成 PPMI,再列出 patio 最強的鄰居:
1"""Reweights the co-occurrence counts with positive PMI, and lists the six tokens with the strongest weight beside patio.2The prose's point rests on that list: the commas and common words of the raw counts are gone."""3import numpy as np45from lib.vectors import cooccurrence, ppmi, training_ids67ids, vocabulary = training_ids() # each training token as a number; RARE for tokens outside the vocabulary8weights = ppmi(cooccurrence(ids, len(vocabulary))) # PPMI of the co-occurrence counts, within the same window of 2 tokens9patio = vocabulary.index("patio</w>") # </w> marks a word's last token: patio</w> is the whole word1011print("share of PPMI entries that are zero:", f"{(weights == 0).mean():.0%}") # zero for pairs that never co-occur and for pairs seen no more often than chance12for i in np.argsort(-weights[patio], kind="stable")[:6]: # the largest weights first; ties keep vocabulary order13 print(f" PPMI with patio: {vocabulary[i]:<12} {weights[patio, i]:.2f}")
python ppmi-neighbours.pyshare of PPMI entries that are zero: 86%
PPMI with patio: porch</w> 4.51
PPMI with patio: deck</w> 4.19
PPMI with patio: backyard</w> 3.97
PPMI with patio: sectional</w> 3.75
PPMI with patio: furniture</w> 3.59
PPMI with patio: garden</w> 3.53
逗號不見了。依 PPMI 排序,patio 最強的鄰居是 porch、deck、backyard、sectional、furniture 與 garden:這些都是人會與露台聯想在一起的詞,排序依據是它們在露台附近出現的次數比機率預期多出多少。多數詞元對一起出現的次數不超過機率預期,因此 PPMI 矩陣有 86% 為零。
PPMI 向量已經相當有用:兩個詞元的 PPMI 列若相似,它們具有辨識力的鄰居也相似。但每個向量仍有 2,000 個數字長,其中大半是零,而兩個鄰居相關但不完全相同的詞元,例如 deck 與 porch,可能只共有很少的非零項。
實用原則: 在把共現計數當作向量使用之前,先以正 PMI 重新加權。預期 PMI 會高估由罕見詞元組成的詞對,因為它們僅有的幾次共現看起來遠高於機率預期;這也是為什麼要像本章這樣只保留高頻詞元,或要求最低計數。
將計數壓縮成稠密向量
一個含有 2,000 個數值、且大多為零的 PPMI 向量,空間很大,資訊卻很少。把它壓縮成短而稠密的向量,可以保留共現關係中最強的模式、濾除雜訊,並讓鄰居相關的詞元彼此靠近,即使它們共有的完全相同鄰居很少。所用的工具是奇異值分解(singular value decomposition,SVD)。它會找出 64 個方向,這些方向合起來能最貼近地重建 PPMI 的各列,並依每個方向能解釋各列的程度加權。一個詞元沿這些方向的 64 個加權座標,就是它的新向量:嵌入(embedding),也就是代表某個詞元的稠密向量。方法 lib.vectors.svd_vectors 負責計算這些向量,並將每個向量縮放為單位長度;lib.vectors.nearest 則可找出與任一向量的餘弦相似度最高的詞元。
以下列出六個詞元各自最近的四個鄰近詞,可看出這 64 個數值保留了什麼:
1"""Compresses the PPMI matrix into 64 numbers per token, and lists the four nearest neighbours of six tokens. The prose's2point rests on the first two lists and the last line: patio and outdoor are each other's nearest neighbour, at the cosine printed last."""3from lib.vectors import DIMENSIONS, nearest, svd_vectors # svd_vectors does the PPMI and SVD steps; nearest ranks tokens by cosine similarity45vectors, vocabulary = svd_vectors() # one 64-number row per token, each scaled to unit length6index = {token: i for i, token in enumerate(vocabulary)}78print("dimensions per token:", DIMENSIONS)9for token in ["patio</w>", "outdoor</w>", "sofa</w>", "oak</w>", "blue</w>", "48</w>"]: # </w> marks a word's last token10 neighbours = nearest(vectors, vocabulary, vectors[index[token]], k=4, skip=(token,)) # skip leaves out the token itself, which would otherwise be its own nearest at 1.0011 print(f"{token:<12} -> " + ", ".join(f"{t} {s:.2f}" for t, s in neighbours))12print("cosine(patio, outdoor):", f"{float(vectors[index['patio</w>']] @ vectors[index['outdoor</w>']]):.2f}") # unit-length vectors, so the dot product is the cosine
python svd-neighbours.pydimensions per token: 64
patio</w> -> outdoor</w> 0.84, bistro</w> 0.79, porch</w> 0.79, lounge</w> 0.77
outdoor</w> -> patio</w> 0.84, porch</w> 0.82, indoor</w> 0.80, furniture</w> 0.74
sofa</w> -> sectional</w> 0.93, chaise</w> 0.90, loveseat</w> 0.89, lounge</w> 0.87
oak</w> -> walnut</w> 0.96, veneers</w> 0.90, espresso</w> 0.89, weathered</w> 0.88
blue</w> -> green</w> 0.86, beige</w> 0.86, pur 0.80, abstract</w> 0.79
48</w> -> 36</w> 0.94, 32</w> 0.93, 24</w> 0.92, 30</w> 0.89
cosine(patio, outdoor): 0.84
現在每個詞元都有 64 個數值,而它的鄰近詞讀起來就像專為家具店建立的同義詞典。patio 與 outdoor 互為最近鄰,餘弦相似度為 0.84,而它們的 one-hot 向量相似度為零。sofa 旁邊是 sectional、chaise 與 loveseat;oak 旁邊是 walnut 與 espresso,都是木材的表面處理;blue 旁邊是其他顏色;48 旁邊則是其他以英寸為單位的尺寸。
這些都不是人教給它的。這個方法從未見過 sofa 的定義、顏色清單,或是 48 是數字的任何提示。它發現這些詞元在商店的文字中出現在相同類型的鄰近詞之間,SVD 因此把它們放得很近。這正是第 3 章的計數所缺少的性質:模型對 patio bar stool 所學到的任何內容,現在都可以透過嵌入,延伸到 outdoor bar stool。
**實用原則:**以 SVD 將 PPMI 壓縮到 50 至 300 維,並比照本節的做法檢查結果:列出少數幾個詞元的最近鄰並逐一閱讀。如果鄰近詞大多是高頻的功能詞,表示重新加權出了問題;如果它們互不相關,則是視窗或詞彙表太小。
透過預測鄰近詞元來學習向量
取得嵌入的另一種方式,是透過預測來學習,本書中每一個神經語言模型的嵌入都是這樣得來的。帶負採樣的 Skip-gram(skip-gram with negative sampling)是 word2vec 背後的方法,它為每個詞元配置兩個向量,一個在該詞元作為中心詞元時使用,另一個在它作為鄰近詞元時使用,並以單一任務加以訓練:給定一個詞元,分辨哪些是它真正的鄰近詞元,哪些是隨機抽出的詞元(Mikolov et al., 2013a;Mikolov et al., 2013b)。對於一個詞元及其某個真正的鄰近詞元,模型以兩者向量的內積為這組配對評分,再用邏輯函數(logistic function)把分數轉成機率,並微調向量使該機率上升。對於從詞彙表中抽出的少數隨機雜訊(noise)詞元,模型則微調向量使機率下降。在數百萬組配對上重複這個過程後,擁有相同鄰近詞元的詞元,最終會得到相似的向量。這並非巧合:Levy 與 Goldberg 證明,以這種方式訓練的帶負採樣 skip-gram,實際上是在分解一個 PMI 矩陣,該矩陣整體下移了雜訊詞元數量的對數,這與 SVD 分解 PPMI 十分相似(Levy and Goldberg, 2014)。
以 v 表示中心詞元的向量,u 表示真正鄰近詞元的向量,uₙ 表示各雜訊詞元的向量,σ 表示邏輯函數 σ(s) = 1 / (1 + e⁻ˢ),它把分數轉成機率。一組配對的 skip-gram 損失,等於對真正鄰近詞元的意外程度,加上對每個雜訊詞元並非鄰近詞元的意外程度:
L = −log σ(v·u) − Σₙ log σ(−v·uₙ)
lib.skipgram.step 以三行梯度下降來降低這個損失,其中 real 為 σ(v·u),fake 存放每個雜訊詞元的 σ(v·uₙ),rate 為步長:
np.add.at(inputs, centre, -rate * grad_v) # move v: towards u, away from each uₙ
np.add.at(outputs, context, -rate * (real - 1)[:, None] * v) # move u towards v
np.add.at(outputs, noise, -rate * fake[:, :, None] * v[:, None, :]) # move each uₙ away from v
對一組真正的配對 patio 與 umbrella 執行三次更新,並以 sofa 與 48 作為雜訊詞元。中心向量從小的隨機值開始,鄰近向量從零開始,與 lib.skipgram.train 的起始方式相同:
1"""Three updates of skip-gram with negative sampling on one real pair, patio and umbrella, with sofa and 48 as noise tokens,2printing each pair's probability before and after. The prose's point rests on the four probability lines: umbrella's rises,3and the noise tokens' fall."""4import numpy as np56from lib.skipgram import sigmoid, step # one gradient step, and the logistic function7from lib.vectors import training_ids89_, vocabulary = training_ids()10rng = np.random.default_rng(2026) # a fixed seed, so the starting vectors are the same on every run11inputs = (rng.random((len(vocabulary), 8)) - 0.5) / 8 # each token's vector as a centre, small and random12outputs = np.zeros((len(vocabulary), 8)) # and as a neighbour: zero, as training starts13centre, context = [vocabulary.index("patio</w>")], [vocabulary.index("umbrella</w>")] # the real pair: patio as the centre, umbrella its real neighbour14noise = np.array([[vocabulary.index("sofa</w>"), vocabulary.index("48</w>")]]) # noise tokens, picked by hand here; training draws them at random151617def chances():18 real = sigmoid(inputs[centre[0]] @ outputs[context[0]])19 fake = sigmoid(outputs[noise[0]] @ inputs[centre[0]])20 return real, fake212223before = chances() # zero neighbour vectors give every score zero, so every probability is one half24for _ in range(3): # three updates on the same pair25 step(np.array(centre), np.array(context), noise, inputs, outputs, rate=1.0) # rate 1.0, far above the 0.025 that training starts from, so three steps show26after = chances()27print("P(umbrella is a real neighbour of patio), before:", f"{before[0]:.3f}")28print("P(umbrella is a real neighbour of patio), after 3 steps:", f"{after[0]:.3f}")29print("P(sofa, 48 are real neighbours), before:", ", ".join(f"{p:.3f}" for p in before[1]))30print("P(sofa, 48 are real neighbours), after 3 steps:", ", ".join(f"{p:.3f}" for p in after[1]))
python skipgram-step.pyP(umbrella is a real neighbour of patio), before: 0.500
P(umbrella is a real neighbour of patio), after 3 steps: 0.525
P(sofa, 48 are real neighbours), before: 0.500, 0.500
P(sofa, 48 are real neighbours), after 3 steps: 0.475, 0.475
訓練會把真正的鄰近詞元拉近,把隨機詞元推遠,三次更新就足以微縮地呈現這一點。在任何更新之前,鄰近向量為零,因此每個分數都是零,而 σ(0) 等於二分之一:模型一無所知。三次更新之後,umbrella 被評為 patio 的鄰近詞元的機率是 0.525,原本為 0.500,而 sofa 與 48 已降至 0.475, 0.475。訓練所做的就是這些,重複數百萬次。
實用原則: 訓練嵌入時,如 lib.skipgram.train 所做,第二組向量從零開始,第一組從小的隨機值開始,並依詞元頻率的 0.75 次方成比例地抽取雜訊詞元,Mikolov 等人發現這比單純依頻率抽取的效果更好。在執行數百萬次之前,先手動檢查一次更新:機率朝錯誤的方向變動,表示梯度的正負號有誤。
比較學習得來的向量與計數得來的向量
以商店的全部訓練文字訓練 skip-gram,需要執行數百萬次更新,而其確切結果取決於機器的浮點運算,不同處理器之間會有些微差異。因此訓練只執行一次,在 recordings/skipgram.py 中,使用本書自己的程式碼與固定的隨機種子,後續範例則讀取其儲存的向量。它的 資訊清單 記錄了各項設定:視窗為兩個 token、64 維、每組配對五個雜訊 token,以及對文字做兩輪訓練。有一項設定是固定在 lib.skipgram 中,而非記錄下來:以極高頻 token(例如逗號)為中心的配對大多會被略過,門檻為 10⁻⁴,這可避免它們主導訓練(Mikolov et al., 2013b,他們建議在較大的文字上採用約 10⁻⁵)。
向已記錄的 skip-gram 向量查詢同樣六個 token 的鄰近詞,並與 SVD 的結果比較:
1"""Reads the skip-gram vectors and training losses that recordings/skipgram.py saved, and lists the same six tokens' four2nearest neighbours as the SVD example. The prose's point rests on the two loss lines and on outdoor's nearest neighbour."""3import json45import numpy as np67from lib.vectors import nearest89recorded = "data/recorded/skipgram/" # saved by recordings/skipgram.py, which trains once: its values depend on the machine10vectors = np.load(recorded + "vectors.npy")11vocabulary = json.load(open(recorded + "vocabulary.json", encoding="utf-8"))12losses = np.load(recorded + "losses.npy") # one loss per batch, in training order13index = {token: i for i, token in enumerate(vocabulary)}1415print("training loss, first batch:", f"{losses[0]:.2f}") # index 0: the first batch, when nothing has been learned yet16print("training loss, last 100 batches:", f"{losses[-100:].mean():.2f}")17for token in ["patio</w>", "outdoor</w>", "sofa</w>", "oak</w>", "blue</w>", "48</w>"]:18 neighbours = nearest(vectors, vocabulary, vectors[index[token]], k=4, skip=(token,)) # skip leaves out the token itself19 print(f"{token:<12} -> " + ", ".join(f"{t} {s:.2f}" for t, s in neighbours))
python skipgram-neighbours.pytraining loss, first batch: 4.16
training loss, last 100 batches: 2.05
patio</w> -> porch</w> 0.77, bistro</w> 0.69, garden</w> 0.69, couch</w> 0.67
outdoor</w> -> indoor</w> 0.75, patio</w> 0.66, garden</w> 0.65, living</w> 0.61
sofa</w> -> couch</w> 0.85, sleeper</w> 0.84, chaise</w> 0.80, sectional</w> 0.78
oak</w> -> walnut</w> 0.91, espresso</w> 0.84, weathered</w> 0.81, glossy</w> 0.80
blue</w> -> beige</w> 0.88, gray</w> 0.87, green</w> 0.85, brown</w> 0.80
48</w> -> 36</w> 0.94, 32</w> 0.92, 1.5</w> 0.89, 24</w> 0.89
模型學會了分辨真正的鄰近詞與雜訊:其訓練損失從第一個批次的 4.16 降到最後一百個批次的 2.05。它的鄰近詞在多數 token 上與 SVD 一致,sofa 對應 sectional 與 chaise,oak 對應 walnut 與 espresso,顏色對應顏色,尺寸對應尺寸;在其他 token 上則重點有所不同:skip-gram 將 outdoor 放在最接近 indoor 的位置,因為這兩個詞在諸如 indoor or outdoor use 的描述中出現在相同的位置。
這項差異值得留意。由鄰近詞建立的向量,會把用在相同位置的詞放在一起,而反義詞往往正是如此:這裡的 indoor 與 outdoor,以及在下一節中的 small,其最近鄰包含 larger,也包含 smaller。向量相近代表用法相近,而非意義相近。
實用原則: 把計數得來的嵌入(PPMI 與 SVD)和學習得來的嵌入(skip-gram)視為相近的替代方案:在幾百萬個 token 的規模上,兩者給出相似的鄰近詞,而計數得來的嵌入具確定性且速度快。當嵌入本來就是你要訓練的較大模型的一部分時,則偏好學習得來的嵌入,本書後續每一章都是如此。切勿在未檢查反義詞的情況下,把高相似度讀成同義。
此部分尚未翻譯,因此以英文呈現。
Analogies, and where they fail
Embeddings famously support analogies by arithmetic: if the difference between chair and chairs is a direction that means "plural", then adding it to table should land near tables. In vectors, the answer to "a is to b as c is to what?" is the token nearest to b − a + c, leaving out the three tokens in the question (Mikolov et al., 2013a). It's a neat test of whether the vectors have learned consistent relationships, and a good place to see their limits.
Three analogies on the skip-gram vectors show where the arithmetic holds and where it breaks, and the nearest neighbours of three tokens they raise show why:
1"""Three analogies on the skip-gram vectors, each answered by the tokens nearest b minus a plus c; whether lamps and light2have vectors at all; and the nearest tokens to a few single words. The prose rests on the three analogy lines, the lamps line and the neighbour lists."""3import json45import numpy as np67from lib.vectors import nearest89vectors = np.load("data/recorded/skipgram/vectors.npy") # saved by recordings/skipgram.py, which trains once10vocabulary = json.load(open("data/recorded/skipgram/vocabulary.json", encoding="utf-8"))11index = {token: i for i, token in enumerate(vocabulary)}1213for a, b, c in [("chair", "chairs", "table"), ("chair", "chairs", "lamp"), ("black", "white", "dark")]: # each reads: a is to b as c is to what14 a, b, c = (word + "</w>" for word in (a, b, c)) # the tokenizer marks each word's last token with </w>, so words are looked up in that form15 query = vectors[index[b]] - vectors[index[a]] + vectors[index[c]] # b - a + c: "c, changed the way a became b"16 answers = nearest(vectors, vocabulary, query, k=3, skip=(a, b, c)) # the words in the question can't be their own answer17 print(f"{a} : {b} :: {c} : ? -> " + ", ".join(t for t, _ in answers))18print("lamps in the vocabulary:", "lamps</w>" in index)19print("light in the vocabulary:", "light</w>" in index)20for token in ["light</w>", "dark</w>", "small</w>"]:21 print(f"{token:<12} -> " + ", ".join(t for t, _ in nearest(vectors, vocabulary, vectors[index[token]], k=4, skip=(token,))))
python analogies.pychair</w> : chairs</w> :: table</w> : ? -> tables</w>, cup</w>, stools</w>
chair</w> : chairs</w> :: lamp</w> : ? -> tops</w>, coffee</w>, tray</w>
black</w> : white</w> :: dark</w> : ? -> brown</w>, blue</w>, glossy</w>
lamps in the vocabulary: False
light in the vocabulary: True
light</w> -> lights</w>, bulb</w>, dimmable</w>, candle</w>
dark</w> -> brown</w>, walnut</w>, gray</w>, espresso</w>
small</w> -> smaller</w>, larger</w>, displaying</w>, storing</w>
Only the first analogy gets the answer you'd expect: chair is to chairs as table is to tables, the top answer. The second fails because lamps isn't among the 2,000 tokens that have vectors, so the right answer can't be found at all, and the arithmetic returns whatever is nearest. The third fails differently. light is in the vocabulary, but black is to white as dark is to brown, by the vectors' reckoning. The neighbour lists show why: light sits with lights, bulb, dimmable and candle, because in the store's text it's mostly a lamp, as in ceiling light, not a shade, while dark sits with brown, walnut and espresso, the finishes it describes. And small sits with larger as well as smaller: size words appear in the same places, whichever size they name.
Analogies are a demonstration, not a measure. They work for relationships the text expresses consistently and often, such as plurals of common words, and fail for rare words, for words with several senses, and for relationships the text rarely spells out. A model doesn't use analogies; it uses the vectors' positions, and those carry the same strengths and blind spots.
Working rule: Don't judge embeddings by analogies; judge them by how much they help the task they're for, as the next section and every later chapter do. When an analogy fails, check whether the answer is in the vocabulary and whether the words mean what you assume in your text.
以向量尋找相似的名稱
詞元的嵌入向量只需再多一個步驟,就能得到整段文字的嵌入向量:將文字中各詞元的向量取平均,再縮放成單位長度,相似的文字就會得到相似的向量。這是搜尋引擎所謂語意檢索(semantic retrieval)的粗略版本,也就是依據意義而非比對字詞來尋找文字;本書的姊妹作《A Primer on Search Relevance》則以訓練過的模型完整建構這項技術(語意檢索)。在這裡,它用來檢驗本章打算解決的一件事:第 3 章的計數模型無法將 outdoor bar 與 patio bar 關聯起來。
將每個訓練名稱的 skip-gram 向量取平均,就能把每個名稱轉成一個向量。罕見的詞,最常見的是產品系列的名稱,會顯示為 [name],如第 1 章所說明。以下兩個查詢可以看出這樣做的效果:一個是 outdoor bar,也就是難倒第 3 章的 4-gram 模型的那個前綴,另一個是 patio umbrella:
1"""Averages each training name's skip-gram token vectors into one unit-length vector, and prints the four training names2nearest to each of two queries. The prose's point rests on the first query: two patio bar stool names come up for outdoor bar."""3import json45import numpy as np67from lib import bpe8from lib.store import load_products, masked910vectors = np.load("data/recorded/skipgram/vectors.npy")11vocabulary = json.load(open("data/recorded/skipgram/vocabulary.json", encoding="utf-8"))12index, tokenizer = {t: i for i, t in enumerate(vocabulary)}, bpe.store_tokenizer()131415def name_vector(text): # the average of the text's token vectors, at unit length16 found = [index[t] for t in tokenizer.encode(text) if t in index] # tokens outside the 2,000 kept have no vector, so they're left out17 average = vectors[found].mean(0) if found else np.zeros(vectors.shape[1]) # a text with no kept tokens gets the zero vector18 return average / (np.linalg.norm(average) or 1.0) # `or 1.0`: a zero average stays zero, rather than dividing by zero192021names = load_products("train")["name"].tolist() # the training split's product names, the texts to compare22matrix = np.array([name_vector(n) for n in names])23for query in ["outdoor bar", "patio umbrella"]:24 best = np.argsort(-(matrix @ name_vector(query)), kind="stable")[:4] # unit vectors, so the dot product is the cosine; the four largest, most similar first25 print(f"{query}: " + " | ".join(masked(names[i]) for i in best)) # [name]: a rare word
python name-vectors.pyoutdoor bar: outdoor modern industrial 5 piece bar height dining set | [name] bar & bar stool | 29.5 '' patio bar stool | 28.5 '' patio bar stool
patio umbrella: premium tight weave outdoor patio umbrella cover | patio lantern | garden patio 88 [name] rattan deck box | outdoor patio chair
平均後的向量讓兩個 patio bar stool 名稱排在最接近 outdoor bar 的名稱之列,而且兩者都不含 outdoor 這個詞。先前那個模型做不到這項關聯,因為它有 patio bar stool 的計數,卻沒有 outdoor bar 的計數。這項關聯是經由 outdoor 與 patio 建立的,這兩個詞的向量很接近。至於 patio umbrella,最接近的名稱是庭院與戶外產品,排名第一的是傘套,但沒有任何一個本身是傘:平均的做法使 patio 的影響超過 umbrella。
平均法相當粗糙:它會忘記詞序,因此 bar stool 與 stool bar 得到相同的向量,而且頻繁出現的詞元會淹沒真正重要的那一個。不過,它展現了本書後續每個模型所依據的特性:讀取嵌入向量而非 one-hot 詞元的模型,能將對某個詞組學到的內容延伸應用到另一個詞組。
**實用原則:**當你需要為短文字取得快速、與詞序無關的表示時,可以將詞元嵌入向量取平均,並預期它能找出相關的項目,但排序會較為鬆散。若要做真正的語意檢索,請使用專為此目的訓練的模型,如 Primer 所做的那樣。至於語言建模,則應把嵌入向量本身交給模型,讓它自行學習如何組合,第 5 章即開始這麼做。
正式上線後會出什麼問題
嵌入向量(embedding)會把訓練文本的習慣,帶進所有使用它們的地方:
- 用法相似不代表語意相似。
indoor與outdoor在向量空間中彼此相近。若系統把相近的向量視為可互換,就會輕易把意義相反的詞互相替換;請檢查產品所依賴的詞彙,看看它們的最近鄰是哪些詞。 - 罕見與新出現的詞元,向量品質不佳,或根本沒有向量。 本章保留了 2,000 個詞元。不在這份清單內的詞元沒有向量,只出現寥寥幾次的詞元,其向量則含有雜訊。該資料庫的斷詞器擁有超過此數量兩倍的詞元;罕見詞若被斷詞器切成數個片段,只有落在這 2,000 個詞元之中的片段才會取得向量。自行學習嵌入向量的模型(後續各章的模型皆是如此)則能涵蓋整個詞彙表。
- 向量反映它們的來源文本。 該資料庫的文本把
light描述為一盞燈,因此它的向量落在照明相關詞彙的附近。在某一領域訓練出的嵌入向量,會把該領域的詞義與偏見帶到另一個領域。 - 要交付向量,而不只是製作方法。 以梯度下降訓練嵌入向量,在不同處理器上得到的向量會略有差異,這也是本書要把向量記錄下來的原因。請保存你所交付的向量,以及產生這些向量的程式碼與設定。
- 嵌入向量會過時。 目錄中會出現新的產品線與新的詞彙;當詞彙發生漂移時,請重新訓練或擴充嵌入向量,並在每次重新訓練後,檢查重要詞元的鄰近詞清單。
本章的決定
- 詞元之間以其向量的餘弦相似度來比較。 餘弦相似度忽略長度,只比較方向,
lib.vectors.nearest回傳的就是這個值。每個向量會先縮放為單位長度,因此餘弦值就是單純的內積,而原始向量較長的高頻詞元,也不會因為長度而占便宜。 - 計數式嵌入來自以 SVD 壓縮的正 PMI(positive PMI), 視窗為兩個詞元,詞彙取最常見的 2,000 個詞元,實作於
lib.vectors。在商店的文字上,這種嵌入讓patio與outdoor靠得很近,這是獨熱向量(one-hot vector)做不到的。 - 學得的嵌入來自採用負採樣的 skip-gram, 以本書自己的程式碼和固定的隨機種子,在
recordings/skipgram.py中訓練一次,因為其確切數值會隨機器而異。對大多數詞元而言,它們與計數式嵌入的結果一致。 - 相似的向量代表相似的用法。 像
indoor與outdoor這類反義詞會得到相近的向量,而類比只對文字中經常表達的關係有效,所以本書以嵌入對任務的幫助程度來評判嵌入,絕不以類比來評判。 - 之後的每個模型都會學習自己的嵌入, 作為預測下一個詞元的一部分,從第 5 章的神經語言模型開始。本章的向量顯示嵌入能把知識從一個情境帶到另一個情境,本書其餘的模型則將這一點付諸應用。
本章,於本版次中
版次 2026.11.16範例於 2026年11 月 19 通過測試本章 8 個範例 個範例中有 8 個通過
譯自英文書,版次 2026.11.16。