Cover of Language Models from the Ground Up, edition 2026.11.16, code FY8Q-PVVV Language Models from the Ground Up FY8Q-PVVV 24 editions since Oct 2026 Rolling Edition 2026.11.16

從零開始打造語言模型

版次 2026.11.16 範例於 2026年11 月 19 通過測試 86 個範例中有 86 個通過 變更內容

由 AI 翻譯。經人工以英文驗證。 我們如何製作書籍

譯自英文書,版次 2026.11.16。 閱讀原文

語言模型如何運作,逐一建構每個部分:下一個詞的預測、詞元、嵌入、神經網路模型、注意力機制與 transformer、在 CPU 上訓練小型模型、選擇詞彙、開放模型內部包含什麼,以及指令微調如何改變模型。

適合對象 本書寫給熟悉 Python 和 NumPy、但剛接觸語言模型運作方式的讀者:可能是呼叫 LLM API、想了解內部運作原理的開發者,也可能是轉往語言處理領域的資料科學家。

開始閱讀
版次歷史自 2026年十月 以來的 24 個版次每個版次及其變動內容
2026.10.12026.11.16
2026.11.16 · 2026年11 月 21散文修訂、新內容44 節 已變更

目錄

11 章,閱讀時間約 8 小時 47 分鐘,附有86 個範例。

  1. 1預測下一個詞

    讀完本章後,你能夠只靠計數,就用商店的商品名稱建立一個完整的語言模型,用它在購物者輸入時建議下一個詞、生成文字,並用一個數字衡量它對從未見過的文字預測得有多好。

  2. 2Token

    讀完本章後,你將能從零開始建構一個位元組對編碼(byte-pair encoding)的 tokenizer,透過衡量各種詞彙表大小所付出的代價來選定詞彙表大小,並且能讀懂一個開放語言模型的 tokenizer,讀到足以用手動方式套用它的合併規則。

  3. 3Counting further back

    By the end of this chapter you can build an n-gram language model that looks several tokens back, smooth it with Kneser-Ney so it handles text it hasn't seen (smoothing gives some probability to what the counts never saw, as chapter 1 showed), measure the baseline that every neural model in this book is measured against, and see why counting, however clever, runs out of road.

  4. 4Words as vectors

    By the end of this chapter you can turn every token of the store's text into a short vector of numbers, so that tokens used in similar ways get similar vectors, by two methods: compressing counts of which tokens appear together, and training a small model to predict each token's neighbours.

  5. 5神經語言模型

    讀完本章後,你可以建立一個神經語言模型,根據前面一個視窗內的詞元來預測下一個詞元,手動計算它的梯度並加以驗證,用 PyTorch 訓練它,並與計數式基準模型做公平的比較。

  6. 6注意力

    讀完本章後,你可以用手算出注意力:為每個 token 的查詢(query)與每個較早 token 的鍵(key)之間的匹配程度打分,把分數轉成權重,再對這些權重所選出的值(value)取平均。

  7. 7The transformer block

    By the end of this chapter you can assemble a transformer block from its parts, joined by residual connections: attention, normalization, a gated feed-forward layer and position information.

  8. 8訓練小型模型

    讀完本章後,你能從頭到尾訓練一個小型 transformer,並評斷其結果。

  9. 9Choosing the next token

    By the end of this chapter you can turn a language model's probabilities into text, and choose how: always the likeliest token, the likeliest whole text by beam search, or a sample shaped by a temperature, a top-k or top-p cut, a repetition penalty and a minimum length.

  10. 10Inside an open model

    By the end of this chapter you can read an open language model with what this book has built: count where its parameters go, work out what it costs to run, score text with it fairly against a model with a different tokenizer, find the attention patterns of chapter 6 in its heads, test what it copies, and learn what its size buys on the store's text, and what it doesn't.

  11. 11From predicting to following instructions

    By the end of this chapter you can explain how a model that predicts the next token becomes one that follows requests, build the two training losses that do it, and measure what that change does and doesn't fix.

引用本節

格式

各種格式皆引用英文原書,本譯本即依循該原書。

每種格式都會標示本版。其連結會開啟目前版次,節標題會說明該節內文最後一次變動的時間。存取日期與擷取日期即為你開啟此頁的當日。