從零開始打造語言模型
語言模型如何運作,逐一建構每個部分:下一個詞的預測、詞元、嵌入、神經網路模型、注意力機制與 transformer、在 CPU 上訓練小型模型、選擇詞彙、開放模型內部包含什麼,以及指令微調如何改變模型。
適合對象 本書寫給熟悉 Python 和 NumPy、但剛接觸語言模型運作方式的讀者:可能是呼叫 LLM API、想了解內部運作原理的開發者,也可能是轉往語言處理領域的資料科學家。
在 Kindle 或任何電子閱讀器上閱讀
此 EPUB 亦可於 Apple Books、Kobo 及多數電子閱讀器上閱讀。
傳送至 Kindle
將本版次的 EPUB 寄送至 繁體中文.
-
核准我們的地址
Amazon 只會從您已核准的地址將內容寄送到 Kindle,且每個寄送服務都須個別核准。請將我們的地址加入您的「已核准的個人文件電子郵件清單」。
books@rollingedition.com開啟 Amazon 的個人文件設定. 在 Amazon 上,路徑為「管理您的內容和裝置」、「偏好設定」、「個人文件設定」。
已傳送。幾分鐘內將會出現在你的 Kindle 書庫中。
如果未能送達,幾乎都是第 1 步的核准設定出了問題:請確認 books@rollingedition.com 已列入你的「已核准的個人文件電子郵件清單」。
您已儲存之書籍的每個新版次,也會送達您的 Kindle。
這不是 Kindle 地址。地址應以 @kindle.com 結尾,可在 Amazon 的個人文件設定中找到。
你今天已寄送 {count} 本書,這是單一帳戶一天的寄送上限。請明天再試。
今日的書籍寄送已達上限。請明日再試。
本書未能寄出。請於數分鐘後再試。
本版次檔案過大,無法以電子郵件寄送。
傳送至 Kindle 功能隨 Labs 提供。
這不是 Kindle 地址。地址應以 @kindle.com 結尾,可在 Amazon 的個人文件設定中找到。
本書未能寄出。請於數分鐘後再試。
或下載 EPUB 檔,並將其拖曳至 Amazon 的 Send to Kindle 頁面。
Kindle is a trademark of Amazon.com, Inc.
目錄
11 章,閱讀時間約 8 小時 47 分鐘,附有86 個範例。
-
1預測下一個詞
讀完本章後,你能夠只靠計數,就用商店的商品名稱建立一個完整的語言模型,用它在購物者輸入時建議下一個詞、生成文字,並用一個數字衡量它對從未見過的文字預測得有多好。
-
2Token
讀完本章後,你將能從零開始建構一個位元組對編碼(byte-pair encoding)的 tokenizer,透過衡量各種詞彙表大小所付出的代價來選定詞彙表大小,並且能讀懂一個開放語言模型的 tokenizer,讀到足以用手動方式套用它的合併規則。
-
3Counting further back
By the end of this chapter you can build an n-gram language model that looks several tokens back, smooth it with Kneser-Ney so it handles text it hasn't seen (smoothing gives some probability to what the counts never saw, as chapter 1 showed), measure the baseline that every neural model in this book is measured against, and see why counting, however clever, runs out of road.
-
4Words as vectors
By the end of this chapter you can turn every token of the store's text into a short vector of numbers, so that tokens used in similar ways get similar vectors, by two methods: compressing counts of which tokens appear together, and training a small model to predict each token's neighbours.
-
5神經語言模型
讀完本章後,你可以建立一個神經語言模型,根據前面一個視窗內的詞元來預測下一個詞元,手動計算它的梯度並加以驗證,用 PyTorch 訓練它,並與計數式基準模型做公平的比較。
-
6注意力
讀完本章後,你可以用手算出注意力:為每個 token 的查詢(query)與每個較早 token 的鍵(key)之間的匹配程度打分,把分數轉成權重,再對這些權重所選出的值(value)取平均。
-
7The transformer block
By the end of this chapter you can assemble a transformer block from its parts, joined by residual connections: attention, normalization, a gated feed-forward layer and position information.
-
8訓練小型模型
讀完本章後,你能從頭到尾訓練一個小型 transformer,並評斷其結果。
-
9Choosing the next token
By the end of this chapter you can turn a language model's probabilities into text, and choose how: always the likeliest token, the likeliest whole text by beam search, or a sample shaped by a temperature, a top-k or top-p cut, a repetition penalty and a minimum length.
-
10Inside an open model
By the end of this chapter you can read an open language model with what this book has built: count where its parameters go, work out what it costs to run, score text with it fairly against a model with a different tokenizer, find the attention patterns of chapter 6 in its heads, test what it copies, and learn what its size buys on the store's text, and what it doesn't.
-
11From predicting to following instructions
By the end of this chapter you can explain how a model that predicts the next token becomes one that follows requests, build the two training losses that do it, and measure what that change does and doesn't fix.