量化路由器并测量发生了什么变化
阅读约需 39分钟 分钟,5节中含2个示例。
读完本章,你将能够一次调用就把微调后路由器的主干权重存为 8 位整数,统计固定样本上预测发生变化的数量,并说明这个数量能说明什么、不能说明什么。你还将了解速度是如何测量的,以及一台机器上的一次记录得出了什么结果。
量化后的路由器副本会占用更少的空间;问题在于它给出的答案是否仍然相同。第 5 章将路由器留作一个适配器 banking77-lora-v1,其权重为 float32。它的移交阈值是 0.63,由第 5 章中的规则在开发集上选定。量化是用更少的位数存储权重。量化后的模型,其权重占用的空间更小,但返回的答案可能不同。本章在第 5 章已评分的 200 条测试消息的固定样本 test_200 上测量后一点。本章并不声称量化没有代价。
这个副本可以取代 float32 适配器,用于为银行的支持消息做路由的收件箱,而在此之前有三个问题。副本的权重占用多少字节,与适配器相比如何?副本对相同的消息是否给出相同的答案?在为收件箱提供服务的机器上,它是否更快?前两个问题可以在这里根据模型的形状和一个固定样本来测量。第三个问题来自在一台有明确名称的机器上所做的记录,关于速度的一节会说明该记录得出了什么,以及它没有涵盖什么。每个答案都是带分母的计数,或是写明输入的比值。
这部分尚未翻译,因此以英文显示。
One call to quantize the linear layers
A linear layer is a weight matrix that maps one vector to another. The router's backbone is made of them. Quantizing a layer stores each of its weights as an 8-bit integer, with a scale that maps the integers back to approximate float values. Fewer bits per weight means fewer bytes for the layer, and the answers can move, because the values the layer computes are no longer the exact ones.
The scale is what makes the scheme work for a whole matrix. A single scale for an entire matrix has to cover its largest weight, so the rows whose weights are small get a coarse grid. Jacob et al. (2018) report that a shared resolution gives channels with narrow ranges much larger relative error (arXiv:1712.05877). That is a CNN result, so here it is a reason to test per-row scales, not a measurement. A per-row scale lets each output row use the full range of the 8-bit integers for its own weights. That is why the byte count adds one float32 scale for each output row. The scales are few, but they are not free, so the byte count below includes them.
The helper in this book, int8_classifier, names its scheme in its docstring: per-row int8 weights, with float32 activations. The scheme is torchao's Int8WeightOnlyConfig, which the torchao source at v0.18.0 describes as "int8 weight-only symmetric per-channel quantization to linear layers". The helper applies it in three steps. It merges the adapter's weights into the base model with PEFT's merge_and_unload. It calls quantize_ from torchao.quantization with Int8WeightOnlyConfig, and passes a filter that selects every nn.Linear layer except the one named score. Then it returns the model in evaluation mode. The int8 copy, named banking77-lora-v1-int8 in the recordings, is the model the helper returns: a derived model built from the adapter, not a second adapter. The output layer, score, is the head that chapter 5 trained, and the section on it explains why the helper leaves it out.
The merge comes first for a reason. Chapter 5 described the adapter as a small set of matrices beside the base model's layers. Merging folds the adapter's update into the base weights, so each layer is one weight matrix, quantized once, and the layer count describes the router. Without the merge, the same call would also quantize the adapter's own matrices as separate int8 layers.
Older PyTorch code uses torch.ao.quantization.quantize_dynamic, which also stores Linear weights as int8 and quantizes activations as it runs, a different scheme. PyTorch 2.15 deprecates it, and its warning points to torchao's quantize_ (PyTorch v2.15.0, torch/ao/quantization/utils.py), which the torchao README at v0.18.0 shows as the migration. The examples in this book call no deprecated API themselves.
1"""Counts the backbone Linear layers the int8 copy quantizes, and the weight bytes they store, from their shapes.2The prose's saving rests on the last printed line, the int8 share of float32 weight bytes; the head assert checks that the score head is not quantized."""34import os56os.environ["HF_HUB_OFFLINE"] = "1"7os.environ["TRANSFORMERS_OFFLINE"] = "1"89import torch # noqa: E4021011torch.set_num_threads(2) # the counts come from shapes, so the thread count changes no printed line1213import router_lib # noqa: E4021415router_lib.setup_runtime() # fixes the seeds, requires deterministic algorithms and sets float32 as the default dtype16RECORDED = router_lib.HERE / "data" / "recorded" / "lora-full"1718_, model = router_lib.int8_classifier(RECORDED / "adapter") # merges the adapter, then quantizes every backbone Linear, not score19layers = router_lib.backbone_linears(model) # (name, layer) pairs, all except score20quantized = [layer for _, layer in layers if router_lib.is_int8(layer)] # a torchao int8 weight, not a float one21weights = sum(layer.weight.numel() for _, layer in layers)22rows = sum(layer.weight.shape[0] for _, layer in layers) # weight shape is (output rows, input size)2324print(f"backbone Linear layers: {len(layers)}")25print(f"backbone Linear layers quantized to int8: {len(quantized)}")26head_quantized = router_lib.is_int8(model.score)27assert not head_quantized, "the score head must stay float32"28print(f"score head quantized: {'yes' if head_quantized else 'no'}")29print(f"score head weight dtype: {model.score.weight.dtype}")30print(f"weights in backbone Linear layers: {weights}")31print(f"output rows in backbone Linear layers: {rows}")32float_bytes = 4 * weights33int8_bytes = 1 * weights + 4 * rows # one byte a weight, plus one float32 scale (4 bytes) for each output row34print(f"weight bytes, int8 as a share of float32: {int8_bytes / float_bytes:.1%}")
python int8-layers-and-bytes.pybackbone Linear layers: 210
backbone Linear layers quantized to int8: 210
score head quantized: no
score head weight dtype: torch.float32
weights in backbone Linear layers: 106168320
output rows in backbone Linear layers: 155520
weight bytes, int8 as a share of float32: 25.1%
Building the int8 copy and counting its layers and their weights shows what the helper quantized. The backbone has 210 linear layers, and the int8 copy quantizes 210 of them. Those layers hold 106168320 weights, in 155520 output rows.
The last line is arithmetic on shapes, and it is not a measurement of a file or a process. A float32 weight takes 4 bytes. An int8 weight takes 1 byte, and the calculation adds one float32 scale, 4 bytes, for each output row, because that is the convention the helper's scheme names. Over the weights in those layers, the int8 copy takes 25.1% of the float32 bytes. A quarter of that is the weights themselves, and the rest is the scales. The figure covers only the weights of these layers. It leaves out every other tensor the model holds, so it says nothing about the size of a file on disk, and nothing about the memory a running router uses.
The bytes matter on the serving machine only if its memory has to hold the weights. Whether it does depends on the machine and the rest of the process, which this chapter does not measure. The ratio tells a reader how much of the weight budget the copy frees.
Importing torchao prints warnings to standard error, and an example's first run may print a load report from transformers. They come from the libraries' own import code, not from a call the examples make, and nothing in the examples suppresses or acts on them. Some lines say torchao could not load its GPU-only kernel libraries, which the CPU path does not use. A SyntaxWarning from torchao's own source shows only on the first import, before Python caches the bytecode. Notices that register_constant() is deprecated come from torch and are triggered by torchao's import. The load report lists score.weight as MISSING: the base checkpoint has no classification head, so transformers starts a new one, and the adapter's saved head replaces it. None of these changes an output line. The book pins torchao at 0.18.0 in its requirements file, and every example uses that pin.
Working rule: Quantize only the layers you can name, leave the output layer in float32, and count the bytes from the shapes before you claim any saving. Report the saving as a ratio of weight bytes, with the count of layers beside it, and never as a file size or as a speed.
统计预测发生变化的数量
量化会改变各层计算出的数值,因此模型给出的答案也可能随之改变。检验量化副本的标准是:它对同样的消息是否给出同样的回答。
用第 5 章清单中的阈值,让两个副本分别为 test_200 打分,即可看出二者的回答是否一致。对每个副本,这次运行统计正确回答数、转交数,以及未被转交的消息上的准确率。随后统计两个副本给出的团队不同的消息数,以及转交决定不同的消息数。同一次运行,即 int8-changed-predictions,还会为第三个副本打分,该副本的打分头也经过量化,相关内容见关于打分头的一节。
两个副本有意不按同样方式处理。float32 一侧是第 5 章加载的适配器,未经合并;int8 一侧则先合并,因为辅助函数是先合并再量化的。所以下面的统计结果同时包含合并与量化两方面的影响。这次运行从不单独为合并后的 float32 副本打分,本章也不声称仅合并本身不会带来任何变化。
1"""Scores test_200 with the float32 adapter, its int8 copy and the copy with the head in int8, at the threshold in chapter 5's manifest.2The prose's point rests on the line that counts predictions changed by int8, read with the hand-off line under it."""34import os56os.environ["HF_HUB_OFFLINE"] = "1"7os.environ["TRANSFORMERS_OFFLINE"] = "1"89import json # noqa: E4021011import torch # noqa: E4021213torch.set_num_threads(2) # two threads for these forward passes; the speed section's recording uses four1415import router_lib # noqa: E4021617router_lib.setup_runtime() # fixes the seeds, requires deterministic algorithms and sets float32 as the default dtype18RECORDED = router_lib.HERE / "data" / "recorded" / "lora-full"19threshold = json.loads((RECORDED / "manifest.json").read_text())["handoff_threshold"] # chosen on dev in chapter 5, stored in the manifest20test = router_lib.read_rows("test") # (text, label index) pairs21sample = router_lib.read_sample("test_200") # fixed row indices, so every copy scores the same messages22golds = [test[i][1] for i in sample] # index 1 of each row is its gold team23texts = [test[i][0] for i in sample]2425tokenizer, float_model = router_lib.load_classifier(RECORDED / "adapter") # the adapter as chapter 5 loaded it, unmerged26float_preds, float_confs = router_lib.classify_by_length(float_model, tokenizer, texts) # results in the order of texts27_, int8_model = router_lib.int8_classifier(RECORDED / "adapter") # merged first, then backbone quantized, head float3228int8_preds, int8_confs = router_lib.classify_by_length(int8_model, tokenizer, texts)29_, head_model = router_lib.int8_classifier_all_linears(RECORDED / "adapter") # as above, with the score head quantized too30head_preds, head_confs = router_lib.classify_by_length(head_model, tokenizer, texts)3132print(f"handoff threshold from manifest: {threshold:.2f}")33print(f"messages on test_200: {len(texts)}")34router_lib.print_copy_report("float32", float_preds, float_confs, golds, threshold) # a handed-off message is wrong if its team is wrong; the last measure divides by kept messages only35router_lib.print_copy_report("int8", int8_preds, int8_confs, golds, threshold)36changed, handoffs = router_lib.count_changes(float_preds, float_confs, int8_preds, int8_confs, threshold) # a changed prediction is a different team, handed off or not; a changed hand-off is a confidence on the other side of the threshold37print(f"predictions changed by int8: {changed}/{len(texts)} ({changed / len(texts):.1%})")38print(f"hand-off decisions changed by int8: {handoffs}/{len(texts)}")39changed_head, _ = router_lib.count_changes(float_preds, float_confs, head_preds, head_confs, threshold)40print(f"predictions changed with the head in int8 too: {changed_head}/{len(texts)}")
python int8-changed-predictions.pyhandoff threshold from manifest: 0.63
messages on test_200: 200
float32 accuracy on test_200: 172/200 (86.0%)
float32 handed off: 33/200 (16.5%)
float32 accuracy on messages not handed off: 156/167 (93.4%)
int8 accuracy on test_200: 172/200 (86.0%)
int8 handed off: 33/200 (16.5%)
int8 accuracy on messages not handed off: 156/167 (93.4%)
predictions changed by int8: 0/200 (0.0%)
hand-off decisions changed by int8: 0/200
predictions changed with the head in int8 too: 1/200
在这份样本上,int8 副本改变了 0/200 (0.0%) 个预测和 0/200 个转交决定。同时量化了打分头的副本,其统计见关于打分头的一节。本段旁边的输出给出了每个副本的准确率、转交数,以及未被转交的消息上的准确率,并各附分母。
这三个度量就是第 2 章所定义的那三个,各有其分母。准确率是正确的消息数除以全部 200 条消息,因此被转交的消息若团队判断错误,也按错误计。转交率是低于阈值的消息数除以全部 200 条。未被转交的消息上的准确率,是保留下来的消息中正确的消息数除以保留的消息数;若一条消息也没有保留,该值无定义。预测变化数使用同样的 200 条消息,所问的是这三个度量不能回答的问题:两个副本对每一条消息的判断是否一致?一致并不等于正确,所以输出中同时打印了准确率。
预测发生变化并不必然意味着损失。变化可能把错误的团队改成正确的,也可能把正确的改成错误的,仅凭这个数量无法分辨是哪一种。要看清方向,需要在两个副本意见不一的每条消息上,把各副本给出的团队与金标准标签比对。准确率可能掩盖这一点:两个副本的准确率可以相同,而各自答对的却是不同的消息,所以相同的总数并不说明哪些消息发生了变动。当数量为零时,两个方向上都没有可读的翻转。得到非零数量的读者,应先查看这些翻转,再看总数。
一旦某个副本开始为队列服务,翻转就会带来具体的代价。假设 float32 适配器把一条消息送往 card_arrival,而 int8 副本把同一条消息送往 card_delivery_estimate。两者都是团队,但其中只有一个可能是这条消息的金标准标签,所以这一变化要么修正了一个答案,要么破坏了一个答案,除非两个团队都是错的。无论哪种情形,收件箱都会把这条消息送往某一个团队。这一变化会影响到负责该团队队列的支持人员,而这一变化在总数里是看不出来的。这个统计数量使得这种变化在副本处理任何消息之前就能被看到。
应把预测变化数当作一个结果来读:它来自一个适配器、一个阈值,并不是保证。换一份样本、换一个适配器或换一个阈值,都可能显示出变化,而且 200 条的样本可能漏掉在完整消息流中很少出现的变化。
这个统计只涉及预测,别无其他。如辅助函数的文档字符串所说,该副本的激活值仍以 float32 保存,所以这个数量对于同时量化激活值的副本什么也说明不了。本章不测量那样的副本。
工作准则: 在相同的消息上、以相同的阈值,为两个副本统计预测变化数,并把每个数量连同其分母,与准确率和转交数一起打印出来。当数量为零时,应报告为该样本上的零,并指明样本以及所用的适配器。
让评分头保持 float32
评分头是有 77 个输出的那一层,每个输出对应一个团队。它的 softmax 给出第 5 章拿来与阈值比较的置信度,其中最大的概率所对应的团队就是路由器返回的团队。辅助函数在调用时没有处理这一层,所以评分头保持 float32。本节说明本书为什么这样选择,以及计数结果对这一选择提供了什么依据。
本章所服务的模型副本把评分头保持为 float32,这一选择依据的是 int8-changed-predictions 给出的一项计数。评分头也采用 int8 时,在这份样本上有 1/200 条预测发生变化。这项计数并没有按敏感度对各层排序,本章也不断言变化是否来自评分头。
处于风险中的是置信度略高于阈值的那些消息。评分头的变化可能把其中一部分拉到阈值以下,这样消息就会交给人工处理,而不是交给某个团队。上一节中变化的转人工决策数,可以说明某个副本是否改变了其中任何一条;在这次运行中,int8 副本改变了 0/200 条转人工决策。
辅助函数对评分头做了什么,见 int8-layers-and-bytes。正如该示例中的断言所示,评分头没有被量化,它的权重是 torch.float32。这些打印出的行与第一节中各层的计数来自同一次运行,因此两者可以直接对照。
预测变化数针对的是整个 int8 副本,以及评分头也被量化的副本。它没有对其中任何一个逐层拆分,所以无法说明变化来自哪一层。要找出来源,需要一次只量化一层,并在开发集上重新计数。
工作准则: 除非在开发集上按所选阈值做的计数表明,量化后的评分头没有改变你所依赖的任何团队和任何转人工决策,否则让评分头保持 float32。并说明你是按哪个阈值计数的。
在记录中测量速度
权重占用的字节变少,并不会让模型自动变快,本章也不作这样的断言。int8 副本在某一款 CPU 上是否计算得更快,取决于该 CPU 上运行的算术例程,所以本书在一台指明型号的机器上对此进行测量。
没有任何示例会打印耗时。耗时会随机器以及机器上的负载而变化,打印耗时的示例每次运行都会得到不同的输出,而本书的示例每次都必须得到相同的输出。速度比应当放在记录中。
仅权重的 int8 节省了权重字节,而该副本在 CPU 上是否也计算得更快,则是另一个问题。throughput 记录检验的就是这个问题。它在其清单所指明的机器上,对本章的 int8 副本 banking77-lora-v1-int8(由示例所用的同一个辅助函数构建)分别与 float32 适配器和替身模型进行计时比较。在那台机器上,使用四个线程、批大小为 16 时,int8 副本每秒处理的消息数是 float32 分类器的 0.9 倍,因此这份记录没有给出任何采用 int8 来提供服务的速度理由。第 10 章给出了这些速率,并注明了测量它们所用的机器。该记录没有对 360M 模型计时,本章也不对那台机器以外的任何机器声称有何速度。
这类记录会固定其模型和输入,在指明的机器上运行一次,并将输出与一份指明脚本的清单一并保存。使用记录的示例读取其保存的输出,自己并不运行模型。对于速度比较,记录让两个副本处理同样的消息,且两者使用相同数量的线程,并把每个副本的耗时分别保存为各自的输出。速度比由这些保存的输出算出,读者可以在旁边看到机器和线程数。若读者想得到自己机器上的速度比,应在该机器上运行同一份记录,并在报告结果时注明机器。
工作准则: 只从指明了运行机器的记录中读取速度比,绝不要由权重字节推断速度比。上面的速度比只描述它自己的那台机器,所以在把它当作你的速度之前,应先在你打算用来提供服务的机器上运行该记录。
结合移交来做决定
要决定的是路由器提供哪一份副本。第 8 章默认提供 float32 适配器,本章没有改变这一默认做法。int8 副本的预测变化数为 0/200 (0.0%),移交决策变化数为 0/200。计数为零,只能让这份副本继续作为候选。它能带来的节省,是某些层中权重字节数的比例。录制结果显示,int8 副本在其所在机器上没有速度提升,本书也没有实测的内存需求支持它。因此,一份在此计数中答案没有变化、又没有实测收益的副本,目前还没有任何值得推荐的理由。
保留 float32 是有代价的选择。适配器占用的字节数比 int8 副本更多,在小型机器上这可能有影响。本书目前展示的结论范围更窄:在这份样本上,预测变化数为 0/200 (0.0%),而且只有计数为零,才能排除“预测发生变化”这一拒绝该副本的理由。其余理由是本章尚未测量的,因此由它们决定默认选择。如果在服务机器上的录制显示出有意义的速度提升,并且测得了内存需求,那么选用 int8 就成为一个带有已计数风险的工程判断。在此之前,float32 适配器是悬而未决的问题最少的那一个。
移交是需要重点关注的部分。第 5 章选定阈值,是为了让未被移交的消息上的准确率在开发集上达到目标。置信度哪怕只有轻微变动的副本,也可能移交另一批消息,这时阈值就不再起到当初选定它时的作用。这就是预测变化数要在选定的阈值下统计,并且要同时给出移交计数的原因。同样的度量适用于任何经过量化的适配器,例如重新训练的 v2。
量化副本在生产中出问题,往往源于一个无人计数的变化。库升级可能改变运算方式,消息构成的变化也可能使其中一些消息越过阈值。每当库或消息发生变化时,都要在开发集上重新计数;量化副本提供第一条消息之前,也要重新计数。
同样的步骤也适用于读者自己的路由器的量化副本,而且除了样本和阈值之外,不需要本书的任何其他内容。阈值取自适配器的清单,并确认清单写明它是在开发集上选定的。量化主干网络,让头部保持 float32,与辅助函数的做法一致。用两份副本对固定样本评分,统计准确率、移交数、未移交消息上的准确率以及预测变化数,每项都要给出分母。先分别查看两个方向上的翻转,再看总数。然后在选定的阈值下于开发集上重新计数,因为阈值正是在那里选定的。通过这些步骤的副本,在取代 float32 适配器之前,仍要通过录制来测量它的速度和内存。
工作准则: 默认提供 float32 适配器。只有当服务机器上的录制显示出速度提升,并且有实测的内存需求支持时,才考虑 int8 副本;此外,在它提供任何消息之前,还要在开发集上按选定阈值统计其预测变化数。
本章,在本版次中
版次 2026.11.9示例于 2026年11月20日 通过本章 2个示例 个示例中有 2 个通过
译自英文版图书,版次 2026.11.9。