小模型并不总是正确的选择
阅读约需 44分钟 分钟,6节中含5个示例。
读完本章,你应当能够说明小型路由器在哪些情况下仍然不敌替身模型,在哪些情况下托管模型预计会表现更好;能够依据你给出并注明日期的输入,计算各个方案的月成本;能够找出自备机器成为更便宜方案时的业务量;还能够指出在任何成本之前就决定这一问题的各项要求。
小型路由器可以在一组消息中的大多数上胜过更大的模型,但对于某些消息、某些团队或某些业务量,它仍可能是错误的选择。银行的支持收件箱(第 1 章)有 77 个团队。在九章的篇幅中,本书为它构建了一个分类器头部,用一条会把不确定的消息转交出去的路由来提供这个头部的服务,并配上为路由器做版本管理、把转交的消息送给人工处理的各项检查,还尝试了蒸馏得到的学生模型和 int8 副本,对两者都做了测量,但都没有上线。测量路由器的各章,都是拿它与替身模型对比评分(第 1 章)。
这部分尚未翻译,因此以英文显示。
Where the small model still loses
Chapter 2 set the stand-in's score as the baseline. This section puts the router beside it on the same messages: all 1001 messages of test_1001, the fixed sample that every chapter's samples come from (chapter 2). The router is the classifier banking77-lora-v1, run live.
1"""The router and the stand-in on all 1,001 messages of test_1001, the committed test order. The prose rests on the2lines for the router's wrong answers sent as answers and for the messages the stand-in gets right and the router wrong.3The stand-in's answers are recorded; the router runs live on 2 threads, and its first 200 messages are the test_200 sample."""45import os67os.environ["HF_HUB_OFFLINE"] = "1"8os.environ["TRANSFORMERS_OFFLINE"] = "1"910import torch # noqa: E4021112torch.set_num_threads(2)1314import router_lib # noqa: E4021516router_lib.setup_runtime()17# Runs the router live on the sample and reads the stand-in's answers from its recording, then scores both.18c = router_lib.compare_router_and_standin("test_1001")19n = c["messages"]20print(f"messages on test_1001: {n}")21print(f"router banking77-lora-v1 correct: {c['router_correct']}/{n} ({c['router_correct'] / n:.1%})")22print(f"stand-in correct: {c['standin_correct']}/{n} ({c['standin_correct'] / n:.1%})")23print(f"router handed off: {c['handed_off']}/{n} ({c['handed_off'] / n:.1%})")24print(f"right for the stand-in, wrong for the router: {c['standin_right_router_wrong']}/{n}")25print(f"of those, handed off by the router: {c['of_those_handed_off']}/{c['standin_right_router_wrong']}")26print(f"wrong for the router and sent as an answer: {c['router_wrong_sent']}/{n} ({c['router_wrong_sent'] / n:.1%})")27print(f"right for the router, wrong for the stand-in: {c['router_right_standin_wrong']}/{n}")
python router-on-test-1001.pymessages on test_1001: 1001
router banking77-lora-v1 correct: 847/1001 (84.6%)
stand-in correct: 560/1001 (55.9%)
router handed off: 146/1001 (14.6%)
right for the stand-in, wrong for the router: 48/1001
of those, handed off by the router: 22/48
wrong for the router and sent as an answer: 76/1001 (7.6%)
right for the router, wrong for the stand-in: 335/1001
On the whole set the router is ahead, right on 847/1001 (84.6%) against the stand-in's 560/1001 (55.9%), which is the baseline from chapter 2. Yet the stand-in is right and the router wrong on 48/1001 messages. The router hands off some of those, so a person sees them, and the rest reach the caller as wrong answers. The reverse disagreement, where the router is right and the stand-in wrong, has its own count in the output.
The count that matters most for the inbox is the wrong answers the router sends to a caller: 76/1001 (7.6%). Those are messages it kept, because their confidence cleared the threshold. Chapter 5 chose that threshold on the dev split, and chapter 9 says where what the router hands off goes, which is a person.
1"""Where the router still loses: test_200 scored for the router and the stand-in. The prose rests on the count of2messages the stand-in gets right and the router wrong, and on the wrong answers the router sends as answers.3The stand-in's answers are recorded; the router runs live on 2 threads."""45import os67os.environ["HF_HUB_OFFLINE"] = "1"8os.environ["TRANSFORMERS_OFFLINE"] = "1"910import json # noqa: E4021112import torch # noqa: E40213torch.set_num_threads(2)1415import router_lib # noqa: E4021617router_lib.setup_runtime()18ADAPTER = router_lib.HERE / "data" / "recorded" / "lora-full"19# The threshold chapter 5 chose on dev, as the adapter's manifest records it.20threshold = json.loads((ADAPTER / "manifest.json").read_text())["handoff_threshold"]21# Keyed by test row index, not by position in the sample, so each lookup uses the sample's row numbers.22answers = router_lib.standin_rows()23test = router_lib.read_rows("test") # (text, label index) per row24sample = router_lib.read_sample("test_200")25golds = [test[i][1] for i in sample] # index 1 is the gold team, as an index26standin_preds = [answers[i]["label"] for i in sample] # the label is the team's index, the same form as the gold label27tokenizer, model = router_lib.load_classifier(ADAPTER / "adapter")28preds, confs = router_lib.classify(model, tokenizer, [test[i][0] for i in sample])29n = len(sample)30router_right = [p == g for p, g in zip(preds, golds)]31standin_right = [p == g for p, g in zip(standin_preds, golds)]32handed_off = [router_lib.handoff(c, threshold) for c in confs] # true below the threshold; a NaN confidence is handed off too33lost = [i for i in range(n) if standin_right[i] and not router_right[i]]34sent_wrong = sum(not r and not h for r, h in zip(router_right, handed_off)) # wrong and not handed off: what a caller receives35print(f"messages on test_200: {n}")36print(f"router banking77-lora-v1 correct: {sum(router_right)}/{n} ({sum(router_right) / n:.1%})")37print(f"stand-in correct: {sum(standin_right)}/{n} ({sum(standin_right) / n:.1%})")38print(f"router threshold from manifest: {threshold:.2f}")39print(f"router handed off: {sum(handed_off)}/{n} ({sum(handed_off) / n:.1%})")40print(f"right for the stand-in, wrong for the router: {len(lost)}/{n} ({len(lost) / n:.1%})")41print(f"of those, handed off by the router: {sum(handed_off[i] for i in lost)}/{len(lost)}")42print(f"wrong for the router and sent as an answer: {sent_wrong}/{n} ({sent_wrong / n:.1%})")43print(f"right for the router, wrong for the stand-in: {sum(r and not s for r, s in zip(router_right, standin_right))}/{n}")
python where-the-router-still-loses.pymessages on test_200: 200
router banking77-lora-v1 correct: 172/200 (86.0%)
stand-in correct: 109/200 (54.5%)
router threshold from manifest: 0.63
router handed off: 33/200 (16.5%)
right for the stand-in, wrong for the router: 6/200 (3.0%)
of those, handed off by the router: 3/6
wrong for the router and sent as an answer: 11/200 (5.5%)
right for the router, wrong for the stand-in: 69/200
test_200 is the first 200 messages of test_1001, a prefix the count script checks, so these counts are part of the full run and not a second sample. On it, the stand-in is right and the router wrong on 6/200 (3.0%) messages, and the router hands off 3/6 of them.
The hand-off holds back the wrong answers. The router sends 11/200 (5.5%) wrong answers on these 200 messages. On the messages it keeps, its accuracy is 156/167 (93.4%), and 33/200 (16.5%) of all 200 go to a person.
Forcing a small model onto the team list keeps every answer a team, and does not make the answer right. Chapter 4 asked the two smaller models for a team that way. The table gives each prompted variant's count on the 200 messages of test_200:
| Prompted variant | Right on test_200 |
|---|---|
| SmolLM2-135M-Instruct, raw prompt | 9/200 (4.5%) |
| SmolLM2-135M-Instruct, chat template | 11/200 (5.5%) |
| SmolLM2-360M-Instruct, raw prompt | 12/200 (6.0%) |
The raw 135M model gave one answer more often than any other (card_arrival, 165/200), and that team is the gold team for 0/200 of the messages. A model that leans on one answer is a prior, not a router. Chapter 4 shows why a prompt cannot make up for a model this size.
The stand-in fails in patterns too. Across all 1001 messages, its answers never name 4 of the 77 teams, so every message whose gold team is one of them goes to another team. Two confusions tie for its most common error. The clearest is the team Refund_not_showing_up, which is answered as request_refund on 13/13 of its messages. A pair like that is where a labelled set has to be read before any model is trusted on it, and chapter 2 lists the pairs for every candidate.
A student copies its teacher, including the teacher's mistakes. Chapter 6 distilled the stand-in's answers into a student. The teacher agreed with the gold label on 549/1000 (54.9%) of the training rows it labelled, so the student learned from labels that were wrong almost half the time. The student reached 69/200 (34.5%) on test_200, and the same recipe trained on the gold labels of the same rows reached 77/200 (38.5%).
The student also falls below its teacher, which reached 109/200 (54.5%) on test_200, and below the gold-label control, by a gap this sample cannot settle (chapter 6). The student is right where the control is wrong on 24/200 messages, and the control where the student is wrong on 32/200.
A new team is another place the small model loses. A classifier head has one output for each team it was trained on, so a team without an output cannot be routed by it. Adding one means labelled messages for that team, a training run with the recipe from chapter 5, and the checks from chapter 9 before the router sends a message to it. Whether a prompt change could add a new team is a question for the reader to test. This book did not measure a hosted model, and the table later in this chapter lists the question as open.
Put together, the small models here did three things: a prompted model leaned on one answer, a student copied its teacher's mistakes, and a team with no output could not be routed. None of these is a fault in the router's arithmetic. The stand-in did one thing more: it never named 4 teams and confused a pair. That is the stand-in's, not either option's. A larger threshold trades the router's 11/200 (5.5%) wrong answers on test_200 for hand-offs (chapter 5), but it gives no team an output and fixes neither the teacher's errors nor the stand-in's. Each changes what a caller sees: a wrong team, a handed-off message, or no team at all.
Working rule: Before you route with a small model, run it and the hosted model on the same fixed sample. Count the messages each gets right that the other gets wrong, and the wrong answers that would reach a caller. Count a prompted model's answers against the team list, and count how often its most common answer is the gold team. Do not route a team that has no labelled messages of its own, and do not train a student on a teacher's labels before you have counted how often the teacher agrees with the gold labels.
核算账单与机器
托管模型按消息付费。托管模型的月账单等于消息数乘以单条消息的成本,而单条消息的成本是其输入词元数乘以输入单价,加上输出词元数乘以输出单价。按每百万词元标价的价目表,除以一百万即得每个词元的价格。路由消息在输入一侧较长,因为提示词和消息本身都要计数;在输出一侧较短,因为答案只是一个团队名称。
机器无论忙闲都要付费。机器的月账单等于其运行小时数乘以每小时费率。一台机器一个月能路由的消息数,等于其吞吐量(单位为每秒消息数)乘以一个月的秒数,再乘以利用率,即该机器在一个月中处于忙碌状态的时间占比。因此,机器的单条消息成本取决于它路由了多少条消息,比较必须先有一个消息量,才能得出答案。
下面三个公式适用于任意一对方案。
hosted cost per month = messages × (input tokens × input price + output tokens × output price)
machine cost per month = hours × machine cost per hour
machine capacity per month = messages per second × seconds per month × utilisation
每个输入都需要一个取值,每个价格都需要标明日期和来源。本书不引用任何托管价格,因为没有一个价格同时具备读者可以核查的来源和日期。请在您进行比较的当天查阅服务商的价目页面,并把价格、日期和网址写在数字旁边。
输入词元还需要一个分词器。托管服务商对同一段文本的计数,可能与开源模型的计数不同,本书没有测量服务商的计数。路由提示词就是托管模型对每条消息会收到的内容:77 个团队名称的列表加上消息本身,由 router_prompt 构造。月度示例用 SmolLM2-135M-Instruct 的分词器对它计数,每条消息为 587 个词元,示例把这一计数标注为服务商计数的替代值。
吞吐量是必须由录制结果提供的输入,因为它取决于机器。录制结果会写明所用机器,并保存其测得的速率。在 a cloud machine with 4 vCPUs and 15 GiB of memory 上,使用 4 个线程,替代模型在每次调用处理一条消息时,每秒路由 0.32 条消息。路由器的分类器 banking77-lora-v1 以 float32 运行,在同样的调用规模下每秒路由 19.0 条消息。输出中也给出了它在 16 条一批时的速率。
只有每次调用一条的这一对是同等条件下的比较。分类器在 16 条一批时,一次前向传播处理 16 条消息;而替代模型每次调用处理一条消息,并带着完整的路由提示词。因此批处理的比值 104.0 混合了模型大小与批处理两个因素,本章不把它用于比较。比较采用每次调用一条的比值 60.8。每个副本在预热一条消息之后,对 test_200 的 200 条消息各进行一次计时遍历,分词包含在计时范围之内。比值由未经四舍五入的速率算出,因此并不恰好等于所印速率之商。
1"""The throughput rates and ratios as recorded, and the machine they were measured on. The prose rests on the2one-per-call ratio line, the like-for-like comparison; the batched lines differ in call size. Reads the recording only; no model runs."""34import json56import router_lib78folder = router_lib.HERE / "data" / "recorded" / "throughput"9ratios = json.loads((folder / "ratios.json").read_text())10rates = router_lib.recorded_rates()11manifest = json.loads((folder / "manifest.json").read_text())1213print(f"machine: {manifest['machine'].split(', ')[0]}") # the machine's description is cut at its first comma, which keeps the name and drops the rest14print(f"threads: {manifest['settings']['n_threads']}")15print(f"messages on test_200: {ratios['messages']}")16print(f"stand-in rate, one per call (messages a second): {rates['standin_one_message_per_call']}")17print(f"classifier fp32 rate, one per call (messages a second): {rates['classifier_fp32_one_message_per_call']}")18print(f"classifier fp32 rate, batches of 16 (messages a second): {rates['classifier_fp32_batches_of_16']}")19print(f"classifier int8 rate, batches of 16 (messages a second): {rates['classifier_int8_batches_of_16']}")20# Each ratio is computed from unrounded rates, so it is not the quotient of the printed rates.21print(f"fp32 over stand-in, one per call, throughput ratio: {ratios['classifier_fp32_one_at_a_time_over_standin']}")22print(f"fp32 over stand-in, batches of 16, throughput ratio: {ratios['classifier_fp32_over_standin']}")23print(f"int8 over stand-in, batches of 16, throughput ratio: {ratios['classifier_int8_over_standin']}")24print(f"int8 over fp32, batches of 16, throughput ratio: {ratios['classifier_int8_over_fp32']}")25print(f"predictions agreeing, fp32 and int8: {ratios['agreement_fp32_int8']}/{ratios['messages']}")
python cost-ratio-from-recording.pymachine: a cloud machine with 4 vCPUs and 15 GiB of memory
threads: 4
messages on test_200: 200
stand-in rate, one per call (messages a second): 0.32
classifier fp32 rate, one per call (messages a second): 19.0
classifier fp32 rate, batches of 16 (messages a second): 33.0
classifier int8 rate, batches of 16 (messages a second): 29.0
fp32 over stand-in, one per call, throughput ratio: 60.8
fp32 over stand-in, batches of 16, throughput ratio: 104.0
int8 over stand-in, batches of 16, throughput ratio: 91.8
int8 over fp32, batches of 16, throughput ratio: 0.9
predictions agreeing, fp32 and int8: 200/200
在这台机器上、这一线程数下,int8 副本并不比 float32 更快。它在 16 条一批时相对 float32 的比值为 0.9。第 7 章指出,字节数减少并不决定速度,而这次录制测量的正是速度。两个副本有 200/200 的预测一致。录制是在该线程数下进行的,而本书的服务预算是两个 vCPU,因此读者所用的速率应来自在自己机器上、以自己的线程数进行的录制。
账单需要路由器的速率,录制结果给出了一个:分类器在每次调用一条消息时的速率,计时方式与替代模型相同。路由接口每个请求处理一条消息,所以账单采用这一速率。读者应将它替换为在自己机器上测得的速率。
1"""The monthly bill for a hosted model and for the router on one machine, from inputs the reader sets. The hosted bill,2the machine's bill and the cheaper-route line are the point the prose rests on. The prices, volume, machine cost3and utilisation are example values, not published prices; the token counts are live and the router's rate is recorded."""45import os67os.environ["HF_HUB_OFFLINE"] = "1"8os.environ["TRANSFORMERS_OFFLINE"] = "1"910import torch # noqa: E4021112torch.set_num_threads(2)1314import router_lib # noqa: E4021516router_lib.setup_runtime()17inputs = router_lib.bill_inputs_recorded()18test = router_lib.read_rows("test")19messages = [test[i][0] for i in router_lib.read_sample("test_200")] # index 0 is the text; the label is index 12021# The hosted prompt, counted with the 135M model's tokenizer as a stand-in for the provider's.22input_tokens = router_lib.router_prompt_tokens(messages)23# Prices are per million tokens, hence the division at the end.24per_message = (input_tokens * inputs["price_per_million_input_tokens"]25 + inputs["output_tokens_per_message"] * inputs["price_per_million_output_tokens"]) / 1_000_00026hosted_month = inputs["messages_per_month"] * per_message2728machine_month = inputs["machine_cost_per_hour"] * inputs["hours_per_month"]29capacity = inputs["router_messages_per_second"] * 3600 * inputs["hours_per_month"] * inputs["utilisation"]3031print(f"example input, messages per month: {inputs['messages_per_month']}")32print(f"example input, price per million input tokens, USD: {inputs['price_per_million_input_tokens']}")33print(f"example input, price per million output tokens, USD: {inputs['price_per_million_output_tokens']}")34print(f"example input, output tokens per message: {inputs['output_tokens_per_message']}")35print(f"example input, machine cost per hour, USD: {inputs['machine_cost_per_hour']}")36print(f"example input, hours per month: {inputs['hours_per_month']}")37print(f"example input, utilisation: {inputs['utilisation']}")38print(f"recorded input, router messages per second, one per call: {inputs['router_messages_per_second']}")39print(f"input tokens per message, hosted prompt (counted with SmolLM2-135M-Instruct, a stand-in tokenizer): {input_tokens}")40print(f"example hosted price per message (example prices), USD: {per_message:.6f}")41print(f"example hosted bill per month (example prices), USD: {hosted_month:.2f}")42print(f"example machine bill per month (example cost per hour), USD: {machine_month:.2f}")43print(f"machine capacity per month (recorded rate, example utilisation): {capacity:.0f}")44print(f"example machine cost per message at capacity (example cost per hour), USD: {machine_month / capacity:.6f}")45print(f"example volume within machine capacity: {'yes' if inputs['messages_per_month'] <= capacity else 'no'}") # above capacity, one machine cannot carry the volume and this bill does not hold46print(f"example values, cheaper route at the input volume: {'hosted' if hosted_month < machine_month else 'machine'}")
python monthly-bill-from-labelled-inputs.pyexample input, messages per month: 100000
example input, price per million input tokens, USD: 1.0
example input, price per million output tokens, USD: 4.0
example input, output tokens per message: 6
example input, machine cost per hour, USD: 0.4
example input, hours per month: 730
example input, utilisation: 0.5
recorded input, router messages per second, one per call: 19.0
input tokens per message, hosted prompt (counted with SmolLM2-135M-Instruct, a stand-in tokenizer): 587
example hosted price per message (example prices), USD: 0.000611
example hosted bill per month (example prices), USD: 61.10
example machine bill per month (example cost per hour), USD: 292.00
machine capacity per month (recorded rate, example utilisation): 24966000
example machine cost per message at capacity (example cost per hour), USD: 0.000012
example volume within machine capacity: yes
example values, cheaper route at the input volume: hosted
在这些示例输入下,托管路线更便宜。在示例消息量下,托管账单为每月 61.10 美元,而自有机器无论消息量多少,每月都花费 292.00 美元。
按录制的速率和示例利用率,这台机器每月可以路由 24966000 条消息,在满负荷时其单条消息成本为 0.000012 美元。每月 100000 条的示例消息量远低于这一容量,两种账单在何处相交,将在盈亏平衡一节中求出。这些输入中的每小时成本适合录制所用的机器;两个 vCPU 的机器需要它自己录制的速率。
这份账单中的每一个输入都可能不再成立。价格会变,因此某个月读到的价格只是关于那个月的事实。这里的利用率是对尚未发生的流量所作的假设,而收件箱的消息量会随银行销售的产品和填满队列的事件而变化。某个季度里每月有一半时间闲置的机器,下个季度可能整月都在忙碌。因此每个月都要重新核算,使用实际消息量、标有日期的价格和机器实测的利用率。只做一次的比较,记录的只是做比较的那个月。
工作准则: 用您核查过的输入和已经定价的机器来比较两份账单。写明每个价格的来源及您读取它的日期。用您所比较的托管模型的分词器统计输入词元数,吞吐量取自在您打算购买的机器上进行的录制。凡是示例值,就在旁边注明,并在做决定之前把它替换掉。
让数据留在行内
本节不是成本比较,本书中的任何数字也不能决定这个问题。收件箱收到的每条消息都是客户的文字。托管模型回答这条消息时,文字会离开银行的机器,送到服务提供方的机器上。对每条消息,一次托管调用都会发送消息文本和 77 个团队名称的列表。银行可能出于自身的原因不愿这样做:客户的数据、合同、关于谁可以读取什么的规定。即使托管方案的账单更低,这些原因依然成立。
本书中的消息是 BANKING77 的公开消息,因此本书自己的运行结果并不能说明任何隐私问题。这一要求属于银行,只有银行才能说明它是否适用。
留在银行机器上的小模型,通过运行位置来满足这一要求。路由器、它的权重和它的日志都处在银行的控制之下,第 9 章中的版本管理和文件检查适用于该路由加载的每一个适配器。派生副本,例如 int8 副本,没有自己的清单和固定版本,因此在提供服务之前,两者都需要补上。留在行内本身也有代价,但这个代价不是账单。银行要运行这台机器,为它打补丁,并让路由的各项检查持续运行,其中包括无论其他部分如何变化都保留在路由中的转人工环节(第 8 章)。
账单是一个随业务量、价格和日期而变化的数字。要求则不会变化。所以应先确定要求,再只比较满足要求的方案。在银行机器上运行的小模型是这样的一个方案。在那里运行的更大的开源模型是另一个。替身模型也是其中之一,在这台机器上,它每次调用一条,每秒路由 0.32 条消息,而分类器为 19.0 条。这一差距是它在吞吐量上的实测代价,相差 60.8 倍,而不是金钱上的代价。如果没有任何方案能在收件箱预期的业务量下满足要求,成本比较就没有可决定的内容。满足要求的小模型仍然必须通过第一节中的准确率和转人工计数,满足要求并不能免除这些检查。
**工作准则:**先确定数据要求,再考虑成本。把它作为一项要求写下来,并写明负责它的人,不要把它算作节省或成本。当消息必须留在银行的机器上时,选择能在那里运行的模型,然后按本章第一节的方法统计它的准确率和转人工情况。
用表格来选择
上面的问题并不都与成本有关,而且大多数问题在任何人运行一次记录之前就能回答。下表把这些问题并列在一起。对每个问题,表中给出小型路由器的情形、托管模型的情形,以及应当在你的收件箱上统计什么。
| 问题 | 小型路由器 | 托管模型 | 要衡量什么 |
|---|---|---|---|
| 消息必须留在银行自己的机器上吗? | 满足:在银行的机器上运行 | 不满足:文本会发送给服务提供方 | 这是要求,不是计数:先做决定 |
| 每个答案都必须是一个团队吗? | 是,通过分类头或约束解码实现(第 4 章) | 只有在提示词和检查把答案限制在名单内时才是 | 在固定样本上,团队名单之外的答案数 |
| 团队名单多久变一次? | 新增团队需要带标注的消息和一次训练(第 9 章) | 修改提示词是否足够:需要测试的问题 | 新团队开始路由所需的带标注消息和步骤 |
| 每月有多少条消息? | 固定的月费,直到其容量上限 | 按每条消息计费,随消息量增长 | 上一节中的盈亏平衡数量 |
| 每个团队有多少条带标注的消息? | 足以用于训练,也足以测试每个团队(第 2 章) | 提示词只需很少,甚至不需要 | 每个团队的带标注消息数,在训练划分上统计 |
| 任务需要开放文本或广泛的知识吗? | 限于团队名单,并且它的开放文本答案在本书第一次试验中失败了 | 需要测试的问题 | 在同一固定样本上的准确率,并附上计数 |
这张表列出的是要统计的内容,而不是一份评分。请按顺序阅读各行:先是第一行的要求,然后是团队名单,再是消息量,因为其中每一项都可能在下一个问题提出之前就结束讨论。在最后一列中,以“这是要求”开头的单元格指明了先于计数的决定。其余每个单元格都指明一个在你的样本上得到的计数,并印出其分母。开放文本一行依据的是第 4 章中的第一次试验:135M 模型不受约束的答案在 0/20 条消息上是团队名称。
工作准则: 在做选择之前,用你的收件箱回答每一行。某一行要求计数时,在固定样本上统计,并印出其分母。某一行指明的是要求时,让要求先于计数起决定作用,并把它作为要求写进决定中。
了解盈亏平衡点
盈亏平衡点是指托管模型的月账单与机器的月账单相等时的业务量。低于该业务量时,在你设定的输入下,托管模型的成本更低。高于该业务量时,机器的成本更低,前提是该业务量处于机器的处理能力之内。无论机器忙碌还是空闲,其账单都要照付,因此盈亏平衡点等于机器账单除以托管模型每条消息的价格:
每月盈亏平衡消息数 = 机器月账单 ÷ 托管模型每条消息价格
该公式并不从托管价格中减去机器每条消息的成本。机器在满负荷时每条消息的成本,等于其固定账单除以其处理能力,而这笔账单已经在分子中计入一次。再减一次同样的成本,就等于把机器算了两遍。该公式也只在处理能力范围内成立。超出该范围,一台机器无法承载这一业务量,此时需要为第二台机器按其自身的账单和处理能力重新计算。
1"""The monthly volume at which one machine's bill equals the hosted bill for the router's prompts. The break-even line2is the point the prose rests on, and it holds only inside the capacity line, which the last line checks. The prices and3the machine's cost are example values, as in the bill; the router's rate is recorded on the machine."""45import math6import os78os.environ["HF_HUB_OFFLINE"] = "1"9os.environ["TRANSFORMERS_OFFLINE"] = "1"1011import torch # noqa: E4021213torch.set_num_threads(2)1415import router_lib # noqa: E4021617router_lib.setup_runtime()18inputs = router_lib.bill_inputs_recorded() # example prices and cost per hour, with the router's rate from the recording19test = router_lib.read_rows("test") # (text, label index) for every test row, in file order20messages = [test[i][0] for i in router_lib.read_sample("test_200")] # index 0 is the text; the label is index 12122# Mean count of the whole router prompt, team list included, with SmolLM2-135M's tokenizer.23input_tokens = router_lib.router_prompt_tokens(messages)24# Prices are per million tokens, hence the division at the end.25per_message = (input_tokens * inputs["price_per_million_input_tokens"]26 + inputs["output_tokens_per_message"] * inputs["price_per_million_output_tokens"]) / 1_000_00027machine_month = inputs["machine_cost_per_hour"] * inputs["hours_per_month"]28capacity = inputs["router_messages_per_second"] * 3600 * inputs["hours_per_month"] * inputs["utilisation"]2930# The machine's bill is the same each month up to its capacity, so the hosted bill31# equals it at messages = machine bill / hosted price per message.32# Rounded up: the first whole message count at which the machine is no dearer.33breakeven = math.ceil(machine_month / per_message)3435print(f"input tokens per message, hosted prompt (counted with SmolLM2-135M-Instruct, a stand-in tokenizer): {input_tokens}")36print(f"example hosted price per message (example prices), USD: {per_message:.6f}")37print(f"example machine bill per month (example cost per hour), USD: {machine_month:.2f}")38print(f"machine capacity per month (recorded rate, example utilisation): {capacity:.0f}")39print(f"example break-even messages per month (example prices and cost per hour): {breakeven}")40print(f"break-even within machine capacity: {'yes' if breakeven <= capacity else 'no'}")
python break-even-from-recorded-ratio.pyinput tokens per message, hosted prompt (counted with SmolLM2-135M-Instruct, a stand-in tokenizer): 587
example hosted price per message (example prices), USD: 0.000611
example machine bill per month (example cost per hour), USD: 292.00
machine capacity per month (recorded rate, example utilisation): 24966000
example break-even messages per month (example prices and cost per hour): 477906
break-even within machine capacity: yes
以示例数值计算,机器的账单为 292.00 美元,盈亏平衡点为每月 477906 条消息。该数量处于机器每月 24966000 条消息的处理能力之内(检查结果输出为 yes)。该处理能力采用记录下来的每次调用一条的速率,并取示例利用率。
盈亏平衡点与速率无关,只取决于账单和托管价格。每月 100000 条消息的业务量远低于盈亏平衡点,因此托管账单更低。
成本点并不等于决策,其余部分仍由收件箱的实际情况来决定。只有在 让数据留在楼内 一节中的要求已经确定之后,它才回答成本问题。如果消息必须留在内部,问题就变成哪个小模型能满足准确率和转人工的计数要求,而盈亏平衡点则说明在各种业务量下,这台机器与托管模型相比成本如何。如果要求允许使用托管模型,且业务量低于盈亏平衡点,那么在示例价格下,托管模型是成本更低的路线。高于盈亏平衡点时,机器是成本更低的路线,前提是业务量处于其处理能力之内,并且路由器的错误答案送达提问者的数量是客服人员可以接受的。无论账单如何,需要转人工的消息都要交给人工处理:在前 200 条消息中,路由器转交了其中的 33/200 (16.5%) 条。
**工作准则:**用你的机器账单和你查到的托管价格计算盈亏平衡点,并在旁边写明该价格的日期和来源。在把盈亏平衡点解读为节省之前,先核对你预期的业务量是否处于你在自己机器上测得的处理能力之内。超出该处理能力时,按第二台机器计算,再重新计算一次。把盈亏平衡点当作成本点来读,而不是当作决策。路由器能否上线,仍由前面各节中的要求和准确率计数来决定。
本书为收件箱做出的决定
在本书所指明的样本和机器上,下面这些数字与收件箱有关。每一项都注明了统计该数字的章。
- 准确率。 在
test_1001的全部样本上,路由器答对 847/1001 (84.6%) 条,替代模型答对 560/1001 (55.9%) 条(第 2 章)。其余的由路由器转交给人工处理,而仍有 76/1001 (7.6%) 条错误答案会送到来电者手中。在路由器上线之前,客服人员必须接受这个数量。 - 速度。 在所记录的机器上,每次调用处理一条消息时,分类器的速度为每秒 19.0 条消息,替代模型为每秒 0.32 条,比值为 60.8(速度如何测量,见第 7 章)。int8 副本没有提供采用它的速度理由:在 16 条一批的情况下,它相对于 float32 的比值为 0.9。因此,该路由继续使用 float32 适配器(第 8 章)。
- 成本。 按示例输入,每月 100000 条消息,在托管方案上花费 61.10 美元,在自有机器上花费 292.00 美元。盈亏平衡点为每月 477906 条消息,处于 24966000 的容量之内。这些是示例价格和示例的每小时成本,因此在把盈亏平衡点当作决定依据之前,请换成你自己的数值,并分别注明日期和来源。
- 数据要求。 如果消息必须留在银行的机器上,就不能使用托管模型,选择范围只限于能在本地运行的方案。应先确定这一点,正如关于把数据留在机构内部的那一节所说。
- 小模型不是首选的情形。 没有已标注消息的团队,在模型头部没有可输出的内容;新团队则需要第 9 章中的步骤。开放文本的回答在本书的第一次试验中失败了(第 4 章)。托管模型在开放文本和广泛知识方面预计表现更好,这是一个需要检验的问题。而在低于盈亏平衡点的业务量下,如果允许使用托管模型,按示例价格,托管模型是更便宜的方案。
- 所得认识。 数据要求和团队清单决定还剩下哪些方案。只有在此之后,才由账单在这些方案中做出选择。
收件箱的结论。 如果消息必须留在银行的机器上,而且每个团队都有已标注的消息,那么选择银行自有机器上的小型路由器,前提是业务量在银行自己的价格下高于盈亏平衡点,并且处于机器的容量之内。无论账单如何,转交给人工的环节都保留在该路由中。上线之前,请在银行自己的样本上统计送到来电者手中的错误答案的数量。
工作准则: 在做选择之前,请根据你自己的收件箱,用你自己的数字重新写一遍这份清单。如果结论不是来自你自己的统计数据、日期和要求,它就不是本书针对你的收件箱所下的结论。
本章,在本版次中
版次 2026.11.9示例于 2026年11月20日 通过本章 5个示例 个示例中有 5 个通过
译自英文版图书,版次 2026.11.9。