量化路由器並測量有何變化

閱讀約需 39 分鐘 分鐘,5 節 中共有 2 個範例。

版次 2026.11.9 範例於 2026年11 月 20 通過測試 36 個範例中有 36 個通過 變更內容

由 AI 翻譯。經人工以英文驗證。 我們如何製作書籍

譯自英文書,版次 2026.11.9。 閱讀原文

讀完本章後,你可以用一次呼叫,將微調後路由器的骨幹權重儲存為 8 位元整數,計算固定樣本上預測改變的數量,並說明這個數量能顯示什麼、不能顯示什麼。你也會看到速度是如何測量的,以及一台機器上的錄製結果有何發現。

路由器的量化副本會佔用較少空間,問題在於它是否仍然給出相同的回答。第 5 章讓路由器停在一個轉接器 banking77-lora-v1,其權重為 float32。它的移交門檻為 0.63,是依照第 5 章的規則在開發集切分上選定的。量化是以較少的位元儲存權重。量化後的模型,其權重佔用的空間較少,但可能回傳不同的答案。本章在第 5 章評分過的 200 則測試訊息固定樣本 test_200 上測量後者,並不主張量化沒有代價。

這個副本可以取代 float32 轉接器,用於處理銀行客服訊息的收件匣,而在此之前有三個問題。副本的權重佔用多少位元組,與轉接器相比如何?副本對相同的訊息是否給出相同的回答?在負責收件匣的機器上,它是否比較快?前兩個問題可以在這裡測量,依據是模型的形狀與一個固定樣本。第三個問題來自在一台指定機器上的錄製結果,關於速度的一節會說明該錄製結果有何發現,以及它沒有涵蓋什麼。每個答案都是附有分母的計數,或是列明輸入的比值。

此部分尚未翻譯,因此以英文呈現。

One call to quantize the linear layers

最近一次變動於第 2026.11.9 版

A linear layer is a weight matrix that maps one vector to another. The router's backbone is made of them. Quantizing a layer stores each of its weights as an 8-bit integer, with a scale that maps the integers back to approximate float values. Fewer bits per weight means fewer bytes for the layer, and the answers can move, because the values the layer computes are no longer the exact ones.

The scale is what makes the scheme work for a whole matrix. A single scale for an entire matrix has to cover its largest weight, so the rows whose weights are small get a coarse grid. Jacob et al. (2018) report that a shared resolution gives channels with narrow ranges much larger relative error (arXiv:1712.05877). That is a CNN result, so here it is a reason to test per-row scales, not a measurement. A per-row scale lets each output row use the full range of the 8-bit integers for its own weights. That is why the byte count adds one float32 scale for each output row. The scales are few, but they are not free, so the byte count below includes them.

The helper in this book, int8_classifier, names its scheme in its docstring: per-row int8 weights, with float32 activations. The scheme is torchao's Int8WeightOnlyConfig, which the torchao source at v0.18.0 describes as "int8 weight-only symmetric per-channel quantization to linear layers". The helper applies it in three steps. It merges the adapter's weights into the base model with PEFT's merge_and_unload. It calls quantize_ from torchao.quantization with Int8WeightOnlyConfig, and passes a filter that selects every nn.Linear layer except the one named score. Then it returns the model in evaluation mode. The int8 copy, named banking77-lora-v1-int8 in the recordings, is the model the helper returns: a derived model built from the adapter, not a second adapter. The output layer, score, is the head that chapter 5 trained, and the section on it explains why the helper leaves it out.

The merge comes first for a reason. Chapter 5 described the adapter as a small set of matrices beside the base model's layers. Merging folds the adapter's update into the base weights, so each layer is one weight matrix, quantized once, and the layer count describes the router. Without the merge, the same call would also quantize the adapter's own matrices as separate int8 layers.

Older PyTorch code uses torch.ao.quantization.quantize_dynamic, which also stores Linear weights as int8 and quantizes activations as it runs, a different scheme. PyTorch 2.15 deprecates it, and its warning points to torchao's quantize_ (PyTorch v2.15.0, torch/ao/quantization/utils.py), which the torchao README at v0.18.0 shows as the migration. The examples in this book call no deprecated API themselves.

int8-layers-and-bytes.py Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0 通過
"""Counts the backbone Linear layers the int8 copy quantizes, and the weight bytes they store, from their shapes.The prose's saving rests on the last printed line, the int8 share of float32 weight bytes; the head assert checks that the score head is not quantized."""import osos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import torch  # noqa: E402torch.set_num_threads(2)  # the counts come from shapes, so the thread count changes no printed lineimport router_lib  # noqa: E402router_lib.setup_runtime()  # fixes the seeds, requires deterministic algorithms and sets float32 as the default dtypeRECORDED = router_lib.HERE / "data" / "recorded" / "lora-full"_, model = router_lib.int8_classifier(RECORDED / "adapter")  # merges the adapter, then quantizes every backbone Linear, not scorelayers = router_lib.backbone_linears(model)  # (name, layer) pairs, all except scorequantized = [layer for _, layer in layers if router_lib.is_int8(layer)]  # a torchao int8 weight, not a float oneweights = sum(layer.weight.numel() for _, layer in layers)rows = sum(layer.weight.shape[0] for _, layer in layers)  # weight shape is (output rows, input size)print(f"backbone Linear layers: {len(layers)}")print(f"backbone Linear layers quantized to int8: {len(quantized)}")head_quantized = router_lib.is_int8(model.score)assert not head_quantized, "the score head must stay float32"print(f"score head quantized: {'yes' if head_quantized else 'no'}")print(f"score head weight dtype: {model.score.weight.dtype}")print(f"weights in backbone Linear layers: {weights}")print(f"output rows in backbone Linear layers: {rows}")float_bytes = 4 * weightsint8_bytes = 1 * weights + 4 * rows  # one byte a weight, plus one float32 scale (4 bytes) for each output rowprint(f"weight bytes, int8 as a share of float32: {int8_bytes / float_bytes:.1%}")
輸出python int8-layers-and-bytes.py
backbone Linear layers: 210
backbone Linear layers quantized to int8: 210
score head quantized: no
score head weight dtype: torch.float32
weights in backbone Linear layers: 106168320
output rows in backbone Linear layers: 155520
weight bytes, int8 as a share of float32: 25.1%

Building the int8 copy and counting its layers and their weights shows what the helper quantized. The backbone has 210 linear layers, and the int8 copy quantizes 210 of them. Those layers hold 106168320 weights, in 155520 output rows.

The last line is arithmetic on shapes, and it is not a measurement of a file or a process. A float32 weight takes 4 bytes. An int8 weight takes 1 byte, and the calculation adds one float32 scale, 4 bytes, for each output row, because that is the convention the helper's scheme names. Over the weights in those layers, the int8 copy takes 25.1% of the float32 bytes. A quarter of that is the weights themselves, and the rest is the scales. The figure covers only the weights of these layers. It leaves out every other tensor the model holds, so it says nothing about the size of a file on disk, and nothing about the memory a running router uses.

The bytes matter on the serving machine only if its memory has to hold the weights. Whether it does depends on the machine and the rest of the process, which this chapter does not measure. The ratio tells a reader how much of the weight budget the copy frees.

Importing torchao prints warnings to standard error, and an example's first run may print a load report from transformers. They come from the libraries' own import code, not from a call the examples make, and nothing in the examples suppresses or acts on them. Some lines say torchao could not load its GPU-only kernel libraries, which the CPU path does not use. A SyntaxWarning from torchao's own source shows only on the first import, before Python caches the bytecode. Notices that register_constant() is deprecated come from torch and are triggered by torchao's import. The load report lists score.weight as MISSING: the base checkpoint has no classification head, so transformers starts a new one, and the adapter's saved head replaces it. None of these changes an output line. The book pins torchao at 0.18.0 in its requirements file, and every example uses that pin.

Working rule: Quantize only the layers you can name, leave the output layer in float32, and count the bytes from the shapes before you claim any saving. Report the saving as a ratio of weight bytes, with the count of layers beside it, and never as a file size or as a speed.

計算預測有變動的數量

最近一次變動於第 2026.11.9 版

量化會改變每一層所算出的數值,因此答案也可能隨之改變。檢驗量化副本的標準,是它對同樣的訊息是否給出同樣的答案。

以第 5 章清單中的門檻,用兩個副本分別對 test_200 評分,即可看出兩者的答案是否一致。對每個副本,這次執行會計算答對的數量、轉交人工的數量,以及未轉交訊息上的準確率。接著計算兩個副本所判定的團隊不同的訊息數,以及轉交決定不同的訊息數。同一次執行 int8-changed-predictions 還會對第三個副本評分,該副本連分數頭(score head)也一併量化,詳見分數頭一節。

兩個副本的處理方式刻意不同。float32 這一側是第 5 章載入時的轉接器(adapter),未經合併;int8 這一側則先合併再量化,因為輔助函式是先合併、後量化。所以下面的計數同時包含合併與量化兩者的影響。這次執行從未單獨對合併後的 float32 副本評分,本章也不主張單是合併就不會造成任何改變。

int8-changed-predictions.py Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0 通過
"""Scores test_200 with the float32 adapter, its int8 copy and the copy with the head in int8, at the threshold in chapter 5's manifest.The prose's point rests on the line that counts predictions changed by int8, read with the hand-off line under it."""import osos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import json  # noqa: E402import torch  # noqa: E402torch.set_num_threads(2)  # two threads for these forward passes; the speed section's recording uses fourimport router_lib  # noqa: E402router_lib.setup_runtime()  # fixes the seeds, requires deterministic algorithms and sets float32 as the default dtypeRECORDED = router_lib.HERE / "data" / "recorded" / "lora-full"threshold = json.loads((RECORDED / "manifest.json").read_text())["handoff_threshold"]  # chosen on dev in chapter 5, stored in the manifesttest = router_lib.read_rows("test")  # (text, label index) pairssample = router_lib.read_sample("test_200")  # fixed row indices, so every copy scores the same messagesgolds = [test[i][1] for i in sample]  # index 1 of each row is its gold teamtexts = [test[i][0] for i in sample]tokenizer, float_model = router_lib.load_classifier(RECORDED / "adapter")  # the adapter as chapter 5 loaded it, unmergedfloat_preds, float_confs = router_lib.classify_by_length(float_model, tokenizer, texts)  # results in the order of texts_, int8_model = router_lib.int8_classifier(RECORDED / "adapter")  # merged first, then backbone quantized, head float32int8_preds, int8_confs = router_lib.classify_by_length(int8_model, tokenizer, texts)_, head_model = router_lib.int8_classifier_all_linears(RECORDED / "adapter")  # as above, with the score head quantized toohead_preds, head_confs = router_lib.classify_by_length(head_model, tokenizer, texts)print(f"handoff threshold from manifest: {threshold:.2f}")print(f"messages on test_200: {len(texts)}")router_lib.print_copy_report("float32", float_preds, float_confs, golds, threshold)  # a handed-off message is wrong if its team is wrong; the last measure divides by kept messages onlyrouter_lib.print_copy_report("int8", int8_preds, int8_confs, golds, threshold)changed, handoffs = router_lib.count_changes(float_preds, float_confs, int8_preds, int8_confs, threshold)  # a changed prediction is a different team, handed off or not; a changed hand-off is a confidence on the other side of the thresholdprint(f"predictions changed by int8: {changed}/{len(texts)} ({changed / len(texts):.1%})")print(f"hand-off decisions changed by int8: {handoffs}/{len(texts)}")changed_head, _ = router_lib.count_changes(float_preds, float_confs, head_preds, head_confs, threshold)print(f"predictions changed with the head in int8 too: {changed_head}/{len(texts)}")
輸出python int8-changed-predictions.py
handoff threshold from manifest: 0.63
messages on test_200: 200
float32 accuracy on test_200: 172/200 (86.0%)
float32 handed off: 33/200 (16.5%)
float32 accuracy on messages not handed off: 156/167 (93.4%)
int8 accuracy on test_200: 172/200 (86.0%)
int8 handed off: 33/200 (16.5%)
int8 accuracy on messages not handed off: 156/167 (93.4%)
predictions changed by int8: 0/200 (0.0%)
hand-off decisions changed by int8: 0/200
predictions changed with the head in int8 too: 1/200

在這份樣本上,int8 副本改變了 0/200 (0.0%) 筆預測與 0/200 筆轉交決定。連分數頭也量化的副本,則在分數頭一節中計算。本段旁的輸出列出各副本的準確率、轉交數,以及未轉交訊息上的準確率,且每項都附有分母。

這三項指標即第 2 章所定義者,各有其分母。準確率是答對的訊息數除以全部 200 筆,因此被轉交的訊息若團隊判錯,仍算作錯誤。轉交率是低於門檻的訊息數除以全部 200 筆。未轉交訊息上的準確率,是保留下來的訊息中答對者所占的比例;若沒有任何訊息被保留,則此值無定義。預測變動數使用同樣的 200 筆訊息,所問的是這三項指標不回答的問題:兩個副本對每一筆訊息是否一致?一致不等於正確,這正是輸出中同時印出準確率的原因。

預測有變動並不必然是損失。它可能把錯的團隊改成對的,也可能把對的團隊改成錯的,光看數量無法分辨。要判讀方向,就要在兩個副本意見不一的每一筆訊息上,將各副本所判定的團隊與標準答案標籤比對。準確率可能掩蓋這一點:兩個副本可以有相同的準確率,卻是不同的訊息答對,所以相同的總數無法說明哪些訊息發生了變動。當計數為零時,不論哪個方向都沒有變動可供判讀。拿到非零計數的讀者,應先檢視這些變動,再看總數。

一旦某個副本開始服務佇列,一次變動就有具體的代價。假設 float32 轉接器把一則訊息送往 card_arrival,而 int8 副本把同一則訊息送往 card_delivery_estimate。兩者都是團隊,但其中只有一個可能是該訊息的標準答案標籤,所以這個改變不是修正了一個答案,就是弄壞了一個答案,除非兩個團隊都判錯。無論如何,收件匣都只會把該則訊息送到一個團隊。這項改變會落在負責該團隊佇列的客服人員身上,而在總數裡沒有人看得出來。這個計數的作用,就是在副本處理任何訊息之前,讓這種改變先顯現出來。

請把預測變動數視為一個結果:來自一個轉接器、一個門檻,而不是一項保證。換一份樣本、轉接器或門檻,可能會顯示出變動;而 200 筆的樣本,也可能漏掉在完整訊息流中很罕見的變動。

這個計數只涉及預測,別無其他。如輔助函式的文件字串所述,該副本的啟動值(activations)仍以 float32 保存,因此這個計數無法告訴你,連啟動值也量化的副本會是什麼情形。本章並未測量那種副本。

實用原則: 對兩個副本,在相同的訊息、相同的門檻下計算預測變動數,並將每個計數連同其分母,與準確率及轉交數並列印出。當計數為零時,就如實回報該樣本上為零,並註明樣本與所取自的轉接器。

將分數頭維持在 float32

最近一次變動於第 2026.11.2 版

分數頭是具有 77 個輸出的那一層,每個團隊對應一個輸出。它的 softmax 給出信心值,第 5 章會拿這個信心值與門檻比較,而其中最大的機率所對應的團隊,就是路由器回傳的團隊。輔助函式在呼叫時略過分數頭,因此分數頭仍維持 float32。本節說明本書為何做這個選擇,以及計數結果對此提供了什麼依據。

本章所提供服務的模型副本,其分數頭維持 float32,這個選擇依據的是 int8-changed-predictions 的一項計數。若分數頭也改用 int8,在這份樣本上會有 1/200 筆預測改變。這項計數並未依敏感度為各層排序,本章也不主張改變究竟是否來自分數頭。

有風險的是信心值略高於門檻的訊息。分數頭的變動可能使其中一筆降到門檻之下,該訊息便會交給人員處理,而不是交給團隊。前一節中改變的轉交決策數量,可以顯示某個副本是否造成這類變動;在這次執行中,int8 副本改變了 0/200 筆轉交決策。

輔助函式如何處理分數頭,記錄在 int8-layers-and-bytes。如該範例的斷言所示,分數頭並未量化,其權重為 torch.float32。這些印出的內容與第一節的各層計數出自同一次執行,因此兩者可以直接比較。

預測改變的計數涵蓋整個 int8 副本,以及分數頭也一併量化的副本。它並未逐層拆分這兩者,所以無法說明改變來自哪一層。要找出來源,必須一次只量化一層,並在開發集切分上重新計數。

**實用原則:**除非在開發集切分上,以選定的門檻計數後顯示,量化後的分數頭不會改變任何你所依賴的團隊判定與轉交決策,否則請將分數頭維持在 float32。請說明你是以哪個門檻計數。

在錄製紀錄中測量速度

最近一次變動於第 2026.11.2 版

權重所需的位元組變少,本身並不會讓模型變快,本章也不主張如此。int8 複本在特定 CPU 上是否計算得更快,取決於該 CPU 上執行的運算常式,因此本書在一台指明的機器上進行測量。

沒有任何範例會印出計時結果。計時結果會隨機器及其負載而變,若範例印出計時,每次執行的輸出都會不同,而本書的範例每次都必須得到相同的輸出。速度比值應該放在錄製紀錄中。

僅權重的 int8 節省了權重的位元組,而這個複本在 CPU 上是否也計算得更快,則是另一個問題。throughput 錄製紀錄就是用來檢驗這一點。它在其資訊清單所指明的機器上,對本章的 int8 複本 banking77-lora-v1-int8(由範例所用的同一個輔助函式建立)計時,並與 float32 轉接器及替身模型比較。在該機器上,使用四個執行緒、批次大小為 16 時,int8 複本每秒處理的訊息數是 float32 分類器的 0.9 倍,因此這份錄製紀錄並未提供採用 int8 提供服務的速度理由。第 10 章會連同測量所用的機器一併報告這些速率。這份錄製紀錄沒有對 360M 模型計時,本章也不對該機器以外的任何機器宣稱速度。

這類錄製紀錄會固定其模型與輸入,在指明的機器上執行一次,並將輸出連同指出腳本的資訊清單一起儲存。使用錄製紀錄的範例會讀取其已儲存的輸出,自己並不執行模型。就速度比較而言,錄製紀錄會讓兩個複本處理相同的訊息,且兩者使用相同的執行緒數,並把每個複本的計時各自儲存為獨立的輸出。比值是由這些已儲存的輸出算出,讀者可在旁邊看到機器與執行緒數。若讀者想得知自己機器上的比值,應在該機器上執行同一份錄製紀錄,並在報告結果時一併說明機器。

實用原則: 只從指明其執行機器的錄製紀錄中讀取速度比值,絕不從權重位元組推論。上述比值只描述它自己的那台機器,因此在把它當作你的速度之前,請先在你打算用來提供服務的機器上執行該錄製紀錄。

以交接決策為準來決定

最近一次變動於第 2026.11.2 版

要決定的是路由器提供哪一份副本。第 8 章預設提供 float32 轉接器,本章沒有更動這項預設。int8 副本的預測改變數為 0/200 (0.0%),交接決策改變數為 0/200。計數為零,只代表這份副本仍可列為候選。它所提供的節省,是某些層中權重位元組的比例。錄製結果在其機器上沒有發現 int8 副本有速度提升,本書也沒有任何實測的記憶體需求支持它,因此一份在此計數中沒有答案改變、也沒有實測效益的副本,目前沒有值得推薦之處。

保留 float32 是有代價的選擇。轉接器佔用的位元組比 int8 副本多,在小型機器上這可能有影響。本書所展示的範圍較窄:在這份樣本上,預測改變數為 0/200 (0.0%),而且只有計數為零,才能排除「預測改變」這項拒絕副本的理由。其餘理由是本章尚未量測的,因此由它們決定預設值。如果在提供服務的機器上錄製的結果顯示有重要的速度提升,並且實測到記憶體需求,採用 int8 就成為一個風險已經計數的工程考量。在那之前,float32 轉接器是未解問題最少的選擇。

交接是需要留意的部分。第 5 章選擇閾值,目的是讓未交接訊息的準確率在開發集上達到目標。信心值即使只有些微變動的副本,也可能交接出不同的一組訊息,這時閾值就不再發揮當初選定它的作用。這就是為什麼預測改變數要在所選閾值下計算,也是交接計數與它並列的原因。同樣的量測適用於任何經過量化的轉接器,例如重新訓練的 v2。

量化副本在正式環境中出錯,通常源於沒有人計數的變動。函式庫升級可能改變運算方式,訊息組成的變化也可能使部分訊息越過閾值。每當函式庫或訊息改變時,以及量化副本處理第一則訊息之前,都應在開發集上重新計數。

同樣的步驟也適用於讀者自己路由器的量化副本,而且除了樣本與閾值之外,不需要本書的其他內容。從轉接器的清單檔取得閾值,並確認清單檔載明它是在開發集上選定的。如輔助工具所做的,將主幹量化,並讓分類頭維持 float32。以兩份副本對固定樣本評分,計算準確率、交接數、未交接訊息的準確率,以及預測改變數,每項都附上分母。先分別檢視各方向的翻轉,再看總數。接著在所選閾值下於開發集上重新計數,因為閾值正是在那裡選定的。通過這些步驟的副本,在取代 float32 轉接器之前,仍須在錄製中量測其速度與記憶體。

實用原則: 預設提供 float32 轉接器。只有在提供服務的機器上的錄製結果指出速度提升,且有實測的記憶體需求支持時,才考慮 int8 副本,並且在它處理任何訊息之前,先在開發集上以所選閾值計算其預測改變數。

本章,於本版次中

版次 2026.11.9範例於 2026年11 月 20 通過測試本章 2 個範例 個範例中有 2 個通過

譯自英文書,版次 2026.11.9。

引用本節

格式

各種格式皆引用英文原書,本譯本即依循該原書。

每種格式都會標示本版。其連結會開啟目前版次,節標題會說明該節內文最後一次變動的時間。存取日期與擷取日期即為你開啟此頁的當日。

本頁內容

第 7 章,共 10 章: 量化路由器並測量有何變化

  1. One call to quantize the linear layers
  2. 計算預測有變動的數量
  3. 將分數頭維持在 float32
  4. 在錄製紀錄中測量速度
  5. 以交接決策為準來決定

返回頂端所有章節