小型模型是錯誤選擇的時候

閱讀約需 44 分鐘 分鐘,6 節 中共有 5 個範例。

版次 2026.11.9 範例於 2026年11 月 20 通過測試 36 個範例中有 36 個通過 變更內容

由 AI 翻譯。經人工以英文驗證。 我們如何製作書籍

譯自英文書,版次 2026.11.9。 閱讀原文

讀完本章,你應當能說明小型路由器在哪些情況下仍輸給替身模型、哪些情況下預期託管模型會表現更好,能根據你列出並註明日期的輸入數據計算各個選項的每月成本,能找出自備機器成為較便宜途徑的使用量,也能在任何成本之前,指出決定這個問題的各項需求。

小型路由器可以在一組訊息的多數項目上勝過較大的模型,卻仍可能對某些訊息、某些團隊或某些使用量來說是錯誤的選擇。銀行的客服信箱(第 1 章)有 77 個團隊。在九章的篇幅中,本書為它建立了分類器頭,用一條會把不確定的訊息轉交出去的路由來提供該分類器頭的服務,並以相應的檢查機制為路由器標註版本,再把轉交的訊息送給真人處理;本書也試過蒸餾出來的學生模型與 int8 副本,兩者都做了量測,但都沒有上線。量測路由器的各章,都是以替身模型為基準來為路由器評分(第 1 章)。

此部分尚未翻譯,因此以英文呈現。

Where the small model still loses

最近一次變動於第 2026.11.2 版

Chapter 2 set the stand-in's score as the baseline. This section puts the router beside it on the same messages: all 1001 messages of test_1001, the fixed sample that every chapter's samples come from (chapter 2). The router is the classifier banking77-lora-v1, run live.

router-on-test-1001.py Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0 通過
"""The router and the stand-in on all 1,001 messages of test_1001, the committed test order. The prose rests on thelines for the router's wrong answers sent as answers and for the messages the stand-in gets right and the router wrong.The stand-in's answers are recorded; the router runs live on 2 threads, and its first 200 messages are the test_200 sample."""import osos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import torch  # noqa: E402torch.set_num_threads(2)import router_lib  # noqa: E402router_lib.setup_runtime()# Runs the router live on the sample and reads the stand-in's answers from its recording, then scores both.c = router_lib.compare_router_and_standin("test_1001")n = c["messages"]print(f"messages on test_1001: {n}")print(f"router banking77-lora-v1 correct: {c['router_correct']}/{n} ({c['router_correct'] / n:.1%})")print(f"stand-in correct: {c['standin_correct']}/{n} ({c['standin_correct'] / n:.1%})")print(f"router handed off: {c['handed_off']}/{n} ({c['handed_off'] / n:.1%})")print(f"right for the stand-in, wrong for the router: {c['standin_right_router_wrong']}/{n}")print(f"of those, handed off by the router: {c['of_those_handed_off']}/{c['standin_right_router_wrong']}")print(f"wrong for the router and sent as an answer: {c['router_wrong_sent']}/{n} ({c['router_wrong_sent'] / n:.1%})")print(f"right for the router, wrong for the stand-in: {c['router_right_standin_wrong']}/{n}")
輸出python router-on-test-1001.py
messages on test_1001: 1001
router banking77-lora-v1 correct: 847/1001 (84.6%)
stand-in correct: 560/1001 (55.9%)
router handed off: 146/1001 (14.6%)
right for the stand-in, wrong for the router: 48/1001
of those, handed off by the router: 22/48
wrong for the router and sent as an answer: 76/1001 (7.6%)
right for the router, wrong for the stand-in: 335/1001

On the whole set the router is ahead, right on 847/1001 (84.6%) against the stand-in's 560/1001 (55.9%), which is the baseline from chapter 2. Yet the stand-in is right and the router wrong on 48/1001 messages. The router hands off some of those, so a person sees them, and the rest reach the caller as wrong answers. The reverse disagreement, where the router is right and the stand-in wrong, has its own count in the output.

The count that matters most for the inbox is the wrong answers the router sends to a caller: 76/1001 (7.6%). Those are messages it kept, because their confidence cleared the threshold. Chapter 5 chose that threshold on the dev split, and chapter 9 says where what the router hands off goes, which is a person.

where-the-router-still-loses.py Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0 通過
"""Where the router still loses: test_200 scored for the router and the stand-in. The prose rests on the count ofmessages the stand-in gets right and the router wrong, and on the wrong answers the router sends as answers.The stand-in's answers are recorded; the router runs live on 2 threads."""import osos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import json  # noqa: E402import torch  # noqa: E402torch.set_num_threads(2)import router_lib  # noqa: E402router_lib.setup_runtime()ADAPTER = router_lib.HERE / "data" / "recorded" / "lora-full"# The threshold chapter 5 chose on dev, as the adapter's manifest records it.threshold = json.loads((ADAPTER / "manifest.json").read_text())["handoff_threshold"]# Keyed by test row index, not by position in the sample, so each lookup uses the sample's row numbers.answers = router_lib.standin_rows()test = router_lib.read_rows("test")  # (text, label index) per rowsample = router_lib.read_sample("test_200")golds = [test[i][1] for i in sample]  # index 1 is the gold team, as an indexstandin_preds = [answers[i]["label"] for i in sample]  # the label is the team's index, the same form as the gold labeltokenizer, model = router_lib.load_classifier(ADAPTER / "adapter")preds, confs = router_lib.classify(model, tokenizer, [test[i][0] for i in sample])n = len(sample)router_right = [p == g for p, g in zip(preds, golds)]standin_right = [p == g for p, g in zip(standin_preds, golds)]handed_off = [router_lib.handoff(c, threshold) for c in confs]  # true below the threshold; a NaN confidence is handed off toolost = [i for i in range(n) if standin_right[i] and not router_right[i]]sent_wrong = sum(not r and not h for r, h in zip(router_right, handed_off))  # wrong and not handed off: what a caller receivesprint(f"messages on test_200: {n}")print(f"router banking77-lora-v1 correct: {sum(router_right)}/{n} ({sum(router_right) / n:.1%})")print(f"stand-in correct: {sum(standin_right)}/{n} ({sum(standin_right) / n:.1%})")print(f"router threshold from manifest: {threshold:.2f}")print(f"router handed off: {sum(handed_off)}/{n} ({sum(handed_off) / n:.1%})")print(f"right for the stand-in, wrong for the router: {len(lost)}/{n} ({len(lost) / n:.1%})")print(f"of those, handed off by the router: {sum(handed_off[i] for i in lost)}/{len(lost)}")print(f"wrong for the router and sent as an answer: {sent_wrong}/{n} ({sent_wrong / n:.1%})")print(f"right for the router, wrong for the stand-in: {sum(r and not s for r, s in zip(router_right, standin_right))}/{n}")
輸出python where-the-router-still-loses.py
messages on test_200: 200
router banking77-lora-v1 correct: 172/200 (86.0%)
stand-in correct: 109/200 (54.5%)
router threshold from manifest: 0.63
router handed off: 33/200 (16.5%)
right for the stand-in, wrong for the router: 6/200 (3.0%)
of those, handed off by the router: 3/6
wrong for the router and sent as an answer: 11/200 (5.5%)
right for the router, wrong for the stand-in: 69/200

test_200 is the first 200 messages of test_1001, a prefix the count script checks, so these counts are part of the full run and not a second sample. On it, the stand-in is right and the router wrong on 6/200 (3.0%) messages, and the router hands off 3/6 of them.

The hand-off holds back the wrong answers. The router sends 11/200 (5.5%) wrong answers on these 200 messages. On the messages it keeps, its accuracy is 156/167 (93.4%), and 33/200 (16.5%) of all 200 go to a person.

Forcing a small model onto the team list keeps every answer a team, and does not make the answer right. Chapter 4 asked the two smaller models for a team that way. The table gives each prompted variant's count on the 200 messages of test_200:

Prompted variant Right on test_200
SmolLM2-135M-Instruct, raw prompt 9/200 (4.5%)
SmolLM2-135M-Instruct, chat template 11/200 (5.5%)
SmolLM2-360M-Instruct, raw prompt 12/200 (6.0%)

The raw 135M model gave one answer more often than any other (card_arrival, 165/200), and that team is the gold team for 0/200 of the messages. A model that leans on one answer is a prior, not a router. Chapter 4 shows why a prompt cannot make up for a model this size.

The stand-in fails in patterns too. Across all 1001 messages, its answers never name 4 of the 77 teams, so every message whose gold team is one of them goes to another team. Two confusions tie for its most common error. The clearest is the team Refund_not_showing_up, which is answered as request_refund on 13/13 of its messages. A pair like that is where a labelled set has to be read before any model is trusted on it, and chapter 2 lists the pairs for every candidate.

A student copies its teacher, including the teacher's mistakes. Chapter 6 distilled the stand-in's answers into a student. The teacher agreed with the gold label on 549/1000 (54.9%) of the training rows it labelled, so the student learned from labels that were wrong almost half the time. The student reached 69/200 (34.5%) on test_200, and the same recipe trained on the gold labels of the same rows reached 77/200 (38.5%).

The student also falls below its teacher, which reached 109/200 (54.5%) on test_200, and below the gold-label control, by a gap this sample cannot settle (chapter 6). The student is right where the control is wrong on 24/200 messages, and the control where the student is wrong on 32/200.

A new team is another place the small model loses. A classifier head has one output for each team it was trained on, so a team without an output cannot be routed by it. Adding one means labelled messages for that team, a training run with the recipe from chapter 5, and the checks from chapter 9 before the router sends a message to it. Whether a prompt change could add a new team is a question for the reader to test. This book did not measure a hosted model, and the table later in this chapter lists the question as open.

Put together, the small models here did three things: a prompted model leaned on one answer, a student copied its teacher's mistakes, and a team with no output could not be routed. None of these is a fault in the router's arithmetic. The stand-in did one thing more: it never named 4 teams and confused a pair. That is the stand-in's, not either option's. A larger threshold trades the router's 11/200 (5.5%) wrong answers on test_200 for hand-offs (chapter 5), but it gives no team an output and fixes neither the teacher's errors nor the stand-in's. Each changes what a caller sees: a wrong team, a handed-off message, or no team at all.

Working rule: Before you route with a small model, run it and the hosted model on the same fixed sample. Count the messages each gets right that the other gets wrong, and the wrong answers that would reach a caller. Count a prompted model's answers against the team list, and count how often its most common answer is the gold team. Do not route a team that has no labelled messages of its own, and do not train a student on a teacher's labels before you have counted how often the teacher agrees with the gold labels.

計算帳單與機器成本

最近一次變動於第 2026.11.9 版

託管模型依訊息數量計費。託管模型的每月帳單,等於訊息數量乘以單則訊息的成本;單則訊息的成本,是其輸入 token 數乘以輸入單價,加上輸出 token 數乘以輸出單價。以每百萬 token 標價的價目表,除以一百萬即得每個 token 的價格。路由訊息在輸入端很長,因為提示詞與訊息本身都要計入;在輸出端很短,因為答案只是一個團隊名稱。

機器不論忙或閒都要付費。機器的每月帳單,等於其使用時數乘以每小時費率。一台機器每月能路由的訊息數,等於其吞吐量(每秒訊息數)乘以一個月的秒數,再乘以使用率,也就是該月中處於忙碌狀態的比例。因此機器的單則訊息成本,取決於它路由了多少訊息;比較必須先有一個訊息量,才能得出答案。

下列三條公式適用於任何一組選項。

hosted cost per month = messages × (input tokens × input price + output tokens × output price)
machine cost per month = hours × machine cost per hour
machine capacity per month = messages per second × seconds per month × utilisation

每個輸入都需要一個數值,每個價格都需要註明日期與來源。本書不引用任何託管價格,因為手上沒有同時具備來源與日期、可供讀者查證的價格。請在你進行比較的當天,讀取供應商的價格頁面,並把價格、日期與網址寫在數字旁邊。

輸入 token 也需要一個分詞器。託管供應商對同一段文字的計數,可能與開源模型的計數不同,而本書並未測量過其中任何一家。路由提示詞,就是託管模型對每則訊息會收到的內容:77 個團隊名稱的清單加上該則訊息,由 router_prompt 建構而成。每月範例以 SmolLM2-135M-Instruct 的分詞器計數,得到每則訊息 587 個 token,範例並將此計數標示為替代供應商計數的代用值。

吞吐量是必須由錄製結果提供的輸入,因為它取決於機器。錄製結果會寫明其機器,並保存所測得的速率。在 a cloud machine with 4 vCPUs and 15 GiB of memory 上,以 4 個執行緒執行時,替代模型在每次呼叫處理一則訊息的情況下,每秒路由 0.32 則訊息。路由器的分類器 banking77-lora-v1 以 float32 執行,在相同的呼叫規模下,每秒路由 19.0 則訊息。它以 16 則為一批的速率,同樣列在輸出中。

只有每次呼叫一則的這一組是同條件比較。分類器以 16 則為一批時,一次前向傳遞處理 16 則訊息,而替代模型每次呼叫只處理一則訊息,且帶著完整的路由提示詞。因此批次比值 104.0 混雜了模型大小與批次處理兩項因素,本章不用它來做比較。比較採用的是每次呼叫一則的比值 60.8。每個副本在 test_200 的 200 則訊息上各執行一次計時,預先以一則訊息暖機,且分詞包含在計時範圍內。比值是由未經四捨五入的速率算出的,因此與所印出速率的商並不完全相等。

cost-ratio-from-recording.py Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0 通過
"""The throughput rates and ratios as recorded, and the machine they were measured on. The prose rests on theone-per-call ratio line, the like-for-like comparison; the batched lines differ in call size. Reads the recording only; no model runs."""import jsonimport router_libfolder = router_lib.HERE / "data" / "recorded" / "throughput"ratios = json.loads((folder / "ratios.json").read_text())rates = router_lib.recorded_rates()manifest = json.loads((folder / "manifest.json").read_text())print(f"machine: {manifest['machine'].split(', ')[0]}")  # the machine's description is cut at its first comma, which keeps the name and drops the restprint(f"threads: {manifest['settings']['n_threads']}")print(f"messages on test_200: {ratios['messages']}")print(f"stand-in rate, one per call (messages a second): {rates['standin_one_message_per_call']}")print(f"classifier fp32 rate, one per call (messages a second): {rates['classifier_fp32_one_message_per_call']}")print(f"classifier fp32 rate, batches of 16 (messages a second): {rates['classifier_fp32_batches_of_16']}")print(f"classifier int8 rate, batches of 16 (messages a second): {rates['classifier_int8_batches_of_16']}")# Each ratio is computed from unrounded rates, so it is not the quotient of the printed rates.print(f"fp32 over stand-in, one per call, throughput ratio: {ratios['classifier_fp32_one_at_a_time_over_standin']}")print(f"fp32 over stand-in, batches of 16, throughput ratio: {ratios['classifier_fp32_over_standin']}")print(f"int8 over stand-in, batches of 16, throughput ratio: {ratios['classifier_int8_over_standin']}")print(f"int8 over fp32, batches of 16, throughput ratio: {ratios['classifier_int8_over_fp32']}")print(f"predictions agreeing, fp32 and int8: {ratios['agreement_fp32_int8']}/{ratios['messages']}")
輸出python cost-ratio-from-recording.py
machine: a cloud machine with 4 vCPUs and 15 GiB of memory
threads: 4
messages on test_200: 200
stand-in rate, one per call (messages a second): 0.32
classifier fp32 rate, one per call (messages a second): 19.0
classifier fp32 rate, batches of 16 (messages a second): 33.0
classifier int8 rate, batches of 16 (messages a second): 29.0
fp32 over stand-in, one per call, throughput ratio: 60.8
fp32 over stand-in, batches of 16, throughput ratio: 104.0
int8 over stand-in, batches of 16, throughput ratio: 91.8
int8 over fp32, batches of 16, throughput ratio: 0.9
predictions agreeing, fp32 and int8: 200/200

在這台機器、這個執行緒數量下,int8 副本並未比 float32 更快。它以 16 則為一批時相對於 float32 的比值為 0.9。第 7 章曾說明位元組變少並不能決定速度,而這份錄製結果量測的正是速度。兩個副本有 200/200 的預測結果一致。這次錄製是在該執行緒數量下進行的,而本書的服務預算是兩個 vCPU,因此讀者的速率應取自其自己的機器、以其自己的執行緒數量所做的錄製。

帳單需要路由器的速率,錄製結果提供了一個:分類器在每次呼叫一則訊息下的速率,其計時方式與替代模型相同。該路由每次請求處理一則訊息,因此帳單採用這個速率。讀者應以自己機器上取得的速率取代它。

monthly-bill-from-labelled-inputs.py Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0 通過
"""The monthly bill for a hosted model and for the router on one machine, from inputs the reader sets. The hosted bill,the machine's bill and the cheaper-route line are the point the prose rests on. The prices, volume, machine costand utilisation are example values, not published prices; the token counts are live and the router's rate is recorded."""import osos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import torch  # noqa: E402torch.set_num_threads(2)import router_lib  # noqa: E402router_lib.setup_runtime()inputs = router_lib.bill_inputs_recorded()test = router_lib.read_rows("test")messages = [test[i][0] for i in router_lib.read_sample("test_200")]  # index 0 is the text; the label is index 1# The hosted prompt, counted with the 135M model's tokenizer as a stand-in for the provider's.input_tokens = router_lib.router_prompt_tokens(messages)# Prices are per million tokens, hence the division at the end.per_message = (input_tokens * inputs["price_per_million_input_tokens"]               + inputs["output_tokens_per_message"] * inputs["price_per_million_output_tokens"]) / 1_000_000hosted_month = inputs["messages_per_month"] * per_messagemachine_month = inputs["machine_cost_per_hour"] * inputs["hours_per_month"]capacity = inputs["router_messages_per_second"] * 3600 * inputs["hours_per_month"] * inputs["utilisation"]print(f"example input, messages per month: {inputs['messages_per_month']}")print(f"example input, price per million input tokens, USD: {inputs['price_per_million_input_tokens']}")print(f"example input, price per million output tokens, USD: {inputs['price_per_million_output_tokens']}")print(f"example input, output tokens per message: {inputs['output_tokens_per_message']}")print(f"example input, machine cost per hour, USD: {inputs['machine_cost_per_hour']}")print(f"example input, hours per month: {inputs['hours_per_month']}")print(f"example input, utilisation: {inputs['utilisation']}")print(f"recorded input, router messages per second, one per call: {inputs['router_messages_per_second']}")print(f"input tokens per message, hosted prompt (counted with SmolLM2-135M-Instruct, a stand-in tokenizer): {input_tokens}")print(f"example hosted price per message (example prices), USD: {per_message:.6f}")print(f"example hosted bill per month (example prices), USD: {hosted_month:.2f}")print(f"example machine bill per month (example cost per hour), USD: {machine_month:.2f}")print(f"machine capacity per month (recorded rate, example utilisation): {capacity:.0f}")print(f"example machine cost per message at capacity (example cost per hour), USD: {machine_month / capacity:.6f}")print(f"example volume within machine capacity: {'yes' if inputs['messages_per_month'] <= capacity else 'no'}")  # above capacity, one machine cannot carry the volume and this bill does not holdprint(f"example values, cheaper route at the input volume: {'hosted' if hosted_month < machine_month else 'machine'}")
輸出python monthly-bill-from-labelled-inputs.py
example input, messages per month: 100000
example input, price per million input tokens, USD: 1.0
example input, price per million output tokens, USD: 4.0
example input, output tokens per message: 6
example input, machine cost per hour, USD: 0.4
example input, hours per month: 730
example input, utilisation: 0.5
recorded input, router messages per second, one per call: 19.0
input tokens per message, hosted prompt (counted with SmolLM2-135M-Instruct, a stand-in tokenizer): 587
example hosted price per message (example prices), USD: 0.000611
example hosted bill per month (example prices), USD: 61.10
example machine bill per month (example cost per hour), USD: 292.00
machine capacity per month (recorded rate, example utilisation): 24966000
example machine cost per message at capacity (example cost per hour), USD: 0.000012
example volume within machine capacity: yes
example values, cheaper route at the input volume: hosted

在這些範例輸入下,託管路線較便宜。以範例訊息量計算,託管帳單為每月 61.10 美元,而自有機器不論訊息量多寡,每月成本都是 292.00 美元。

依錄製速率與範例使用率計算,這台機器每月能路由 24966000 則訊息,滿載時的單則訊息成本為 0.000012 美元。每月 100000 則的範例訊息量,遠低於這個容量;損益平衡一節會找出兩份帳單交會之處。這些輸入中的每小時成本適用於所錄製的機器;兩個 vCPU 的機器需要它自己的錄製速率。

這份帳單中的每個輸入都可能不再成立。價格會變動,因此某個月讀到的價格,只是關於那個月的事實。這裡的使用率,是對尚未發生的流量所做的假設,而收件匣的訊息量會隨銀行所販售的產品,以及填滿其佇列的事件而變動。一台在某一季有一半時間閒置的機器,下一季可能整月忙碌。所以每個月都要重新計算,採用實際的訊息量、註明日期的價格,以及機器實測的使用率。只做過一次的比較,只是它完成當月的紀錄。

實用原則: 以你查證過的輸入與你已估價的機器,來比較兩份帳單。每個價格都要寫明其來源與你讀取的日期。以你所比較的託管模型之分詞器計算輸入 token 數,並從你打算購買的那台機器上的錄製結果取得吞吐量。凡是範例值,就在旁邊註明,並在做決定之前替換掉。

讓資料留在機構內部

本節不是成本比較,本書中的任何數字都不能決定這件事。收件匣收到的每一則訊息,都是客戶的文字。託管模型回答訊息時,文字會離開銀行的機器,送到供應商的機器上。每一則訊息,託管呼叫都會送出訊息文字與 77 個團隊名稱的清單。銀行可能基於自身的理由而不願如此:客戶的資料、合約,以及誰可以讀取什麼的規定。即使託管方案的帳單較低,這些理由依然成立。

本書中的訊息是 BANKING77 的公開訊息,因此本書自己的執行結果完全看不出隱私方面的情形。這項要求屬於銀行,只有銀行能說明它是否適用。

留在銀行機器上的小型模型,藉由執行的位置來滿足這項要求。路由器、它的權重與它的記錄檔,都留在銀行的控管之下,而且第 9 章中的版本管理與檔案檢查,適用於該路由載入的每一個 adapter。衍生的副本,例如 int8 副本,本身沒有 manifest 或固定版本,所以在上線服務之前,兩者都需要具備。留在內部本身也有成本,但那不是帳單。銀行要自己維運機器、持續修補,並讓路由的各項檢查持續執行,包括無論其他部分如何改變都會保留在路由中的轉交機制(第 8 章)。

帳單是會隨著量、價格與日期變動的數字。要求則不會變動。所以先確定要求,再只比較符合要求的選項。在銀行機器上執行的小型模型是這樣的選項之一。在那裡執行的較大開放模型是另一個。stand-in 也是其中之一,在這台機器上,它每次呼叫處理一則訊息的路由速度是每秒 0.32 則訊息,分類器則是每秒 19.0 則。這個差距是它在吞吐量上實測的成本,相差 60.8 倍,而不是金錢上的成本。如果沒有任何選項能在收件匣預期的量下滿足要求,成本比較就沒有東西可以決定。符合要求的小型模型,仍然必須通過第一節中的準確率與轉交計數,滿足要求並不能讓它免除這些檢驗。

實用原則: 先確定資料要求,再考慮成本。把它當作要求寫下來,並寫明負責的人員,不要把它算成節省或成本。當訊息必須留在銀行的機器上時,選擇能在那裡執行的模型,然後像本章第一節那樣計算它的準確率與轉交次數。

用表格做選擇

最近一次變動於第 2026.11.2 版

上述問題並非全都與成本有關,而且多數問題在任何人執行錄製之前就能回答。下表將這些問題並列。針對每一個問題,表中列出小型路由器(small router)的情形、託管模型(hosted model)的情形,以及該在你的收件匣上計算什麼。

問題 小型路由器 託管模型 該測量什麼
訊息是否必須留在銀行的機器上? 符合:在銀行的機器上執行 不符合:文字會送往供應商 這是需求,不是計數:請先決定
每個答案是否都必須是一個團隊? 是,可透過分類頭或限制式解碼(constrained decoding)達成(第 4 章) 只有在提示與檢查機制將答案限制在名單內時才成立 在固定樣本上,落在團隊名單之外的答案數
團隊名單多久更動一次? 新增團隊需要已標註的訊息與一次訓練(第 9 章) 只改提示是否足夠:這是需要測試的問題 新團隊開始路由所需的已標註訊息與步驟
每月有多少訊息? 固定的月費,直到其容量上限 按每則訊息計費,隨量增加 最後一節的損益平衡數量
每個團隊有多少已標註的訊息? 足以訓練,也足以測試每個團隊(第 2 章) 只需少量或不需要,因為使用提示 每個團隊的已標註訊息數,以訓練分割(training split)計算
任務是否需要開放式文字或廣泛的知識? 僅限於團隊名單,且其開放式文字的答案在本書的第一次試驗中失敗 這是需要測試的問題 在同一個固定樣本上的準確率,並附上其計數

這張表列出的是該計算的項目,而不是評分表。請依序閱讀各列:先是第一列的需求,其次是團隊名單,再其次是訊息量,因為每一項都可能在下一個問題提出之前就讓判斷結束。在最後一欄中,以「這是需求」開頭的儲存格指出的是應先於計數做出的決定。其他每個儲存格指出的都是在你的樣本上取得的計數,並印出其分母。開放式文字那一列依據的是第 4 章的第一次試驗:135M 模型未受限制的答案,在 0/20 則訊息上是團隊名稱。

實用原則: 在做選擇之前,先根據你的收件匣回答每一列。凡是要求計數的列,就在固定樣本上計算並印出其分母。凡是指出需求的列,就讓需求先於計數做出決定,並將它作為需求寫入決策中。

掌握損益平衡點

最近一次變動於第 2026.11.9 版

損益平衡點是指託管模型的每月帳單與機器的每月帳單相等時的訊息量。低於這個量,在你設定的輸入值下,託管模型的成本較低。高於這個量,只要訊息量仍在機器的處理容量之內,機器的成本就較低。機器的帳單無論機器忙碌或閒置都要支付,因此損益平衡點是以這筆帳單除以託管模型每則訊息的價格:

每月損益平衡訊息數 = 機器每月帳單 ÷ 託管模型每則訊息價格

這個公式不會從託管價格中扣除機器每則訊息的成本。機器在滿載時每則訊息的成本,等於其固定帳單除以其處理容量,而這筆帳單已在分子中計入一次。若再扣除同樣的成本,就等於把機器算了兩次。這個公式也只在容量之內成立。超過容量後,單一機器無法承載該訊息量,就必須針對第二台機器重新計算,並採用它自己的帳單與容量。

break-even-from-recorded-ratio.py Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0 通過
"""The monthly volume at which one machine's bill equals the hosted bill for the router's prompts. The break-even lineis the point the prose rests on, and it holds only inside the capacity line, which the last line checks. The prices andthe machine's cost are example values, as in the bill; the router's rate is recorded on the machine."""import mathimport osos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import torch  # noqa: E402torch.set_num_threads(2)import router_lib  # noqa: E402router_lib.setup_runtime()inputs = router_lib.bill_inputs_recorded()  # example prices and cost per hour, with the router's rate from the recordingtest = router_lib.read_rows("test")  # (text, label index) for every test row, in file ordermessages = [test[i][0] for i in router_lib.read_sample("test_200")]  # index 0 is the text; the label is index 1# Mean count of the whole router prompt, team list included, with SmolLM2-135M's tokenizer.input_tokens = router_lib.router_prompt_tokens(messages)# Prices are per million tokens, hence the division at the end.per_message = (input_tokens * inputs["price_per_million_input_tokens"]               + inputs["output_tokens_per_message"] * inputs["price_per_million_output_tokens"]) / 1_000_000machine_month = inputs["machine_cost_per_hour"] * inputs["hours_per_month"]capacity = inputs["router_messages_per_second"] * 3600 * inputs["hours_per_month"] * inputs["utilisation"]# The machine's bill is the same each month up to its capacity, so the hosted bill# equals it at messages = machine bill / hosted price per message.# Rounded up: the first whole message count at which the machine is no dearer.breakeven = math.ceil(machine_month / per_message)print(f"input tokens per message, hosted prompt (counted with SmolLM2-135M-Instruct, a stand-in tokenizer): {input_tokens}")print(f"example hosted price per message (example prices), USD: {per_message:.6f}")print(f"example machine bill per month (example cost per hour), USD: {machine_month:.2f}")print(f"machine capacity per month (recorded rate, example utilisation): {capacity:.0f}")print(f"example break-even messages per month (example prices and cost per hour): {breakeven}")print(f"break-even within machine capacity: {'yes' if breakeven <= capacity else 'no'}")
輸出python break-even-from-recorded-ratio.py
input tokens per message, hosted prompt (counted with SmolLM2-135M-Instruct, a stand-in tokenizer): 587
example hosted price per message (example prices), USD: 0.000611
example machine bill per month (example cost per hour), USD: 292.00
machine capacity per month (recorded rate, example utilisation): 24966000
example break-even messages per month (example prices and cost per hour): 477906
break-even within machine capacity: yes

以範例值來看,機器的帳單為 292.00 美元,損益平衡點為每月 477906 則訊息。這個數量落在機器每月 24966000 則訊息的容量之內(檢查結果印出 yes)。該容量採用的是所記錄的每次呼叫一則的處理速率,並搭配範例的使用率。

損益平衡點與處理速率無關,只取決於帳單與託管價格。每月 100000 則訊息的訊息量遠低於損益平衡點,因此託管模型的帳單較低。

成本點不是決策,其餘的部分仍由收件匣決定。唯有在將資料留在內部一節的要求確定之後,才能回答成本問題。若訊息必須留在內部,問題就變成哪一個小型模型能達到準確率與轉交人工的數量要求,而損益平衡點則說明在各種訊息量下,該機器相對於託管模型的成本。若要求允許使用託管模型,且訊息量低於損益平衡點,則在範例價格下,託管模型是成本較低的做法。高於損益平衡點時,只要訊息量在機器容量之內,且路由器送達來電者手中的錯誤答案數量是客服人員可以接受的,機器就是成本較低的做法。無論帳單如何,需要轉交的訊息都交給人工處理:在前 200 則訊息中,路由器轉交了其中的 33/200 (16.5%) 則。

**實用原則:**以你的機器帳單與你查得的託管價格來計算損益平衡點,並在價格旁註明其日期與來源。在把損益平衡點解讀為節省的成本之前,先確認你預期的訊息量落在你在自己機器上實測的容量之內。超過該容量時,就把第二台機器算進去,再重新計算。請把損益平衡點視為成本上的臨界點,而非決策。路由器是否上線,仍由先前各節的要求與準確率數據決定。

本書為收件匣所做的決定

最近一次變動於第 2026.11.2 版

在本書所指明的樣本與機器上,以下是與收件匣有關的數字。每一項都附上計算該數字的章。

  • 準確度。 在整個 test_1001 上,路由器答對 847/1001 (84.6%),替代模型答對 560/1001 (55.9%)(第 2 章)。路由器會把其餘的交給真人處理,而仍有 76/1001 (7.6%) 個錯誤答案會送到來電者手上。在路由器上線之前,客服人員必須接受這個數目。
  • 速度。 在所記錄的機器上,每次呼叫處理一則訊息時,分類器的速度為每秒 19.0 則訊息,替代模型為 0.32,比值為 60.8(第 7 章說明速度如何衡量)。int8 複本沒有提供任何採用它來服務的速度理由:以 16 則為一批時,它相對於 float32 的比值為 0.9。因此,路由所服務的仍是 float32 轉接器(第 8 章)。
  • 成本。 在範例輸入下,每月 100000 則訊息,在託管端的費用為 61.10 美元,在自有機器上為 292.00 美元。損益平衡點為每月 477906 則訊息,落在 24966000 的容量之內。這些是範例價格與範例的每小時成本,因此在把損益平衡點當作決定來解讀之前,請換成您自己的數字,並各自附上日期與來源。
  • 資料要求。 如果訊息必須留在銀行的機器上,託管模型就不能採用,選擇只能在可於該處執行的選項之間進行。請先確定這一點,正如關於把資料留在建築物內的那一節所說。
  • 小型模型不是首選的情況。 沒有已標註訊息的團隊,輸出端就沒有可用的頭部,而新團隊需要第 9 章中的步驟。開放文字的答案在本書的第一次試驗中失敗了(第 4 章)。託管模型在開放文字與廣泛知識上預期會表現得更好,這是有待測試的問題。而在容許使用託管模型、且量低於損益平衡點時,以範例價格來看,託管模型是較便宜的路線。
  • 學到的事。 資料要求與團隊清單決定哪些選項會留下來。只有在那之後,帳單才在其中做出選擇。

收件匣的結論。 如果訊息必須留在銀行的機器上,而且每個團隊都有已標註的訊息,那麼在銀行自己的機器上使用小型路由器就是首選,前提是以銀行自己的價格計算時,量高於損益平衡點,且在該機器的容量之內。無論帳單怎麼說,轉交給真人的步驟都留在路由之中。在上線之前,請在銀行自己的樣本上計算送到來電者手上的錯誤答案數。

實用原則: 在您做選擇之前,請從您的收件匣出發,用您的數字重新寫一次這份清單。不是出自您的計數、日期與要求的結論,就不是本書針對您的收件匣所下的結論。

本章,於本版次中

版次 2026.11.9範例於 2026年11 月 20 通過測試本章 5 個範例 個範例中有 5 個通過

譯自英文書,版次 2026.11.9。

引用本節

格式

各種格式皆引用英文原書,本譯本即依循該原書。

每種格式都會標示本版。其連結會開啟目前版次,節標題會說明該節內文最後一次變動的時間。存取日期與擷取日期即為你開啟此頁的當日。

本頁內容

第 10 章,共 10 章: 小型模型是錯誤選擇的時候

  1. Where the small model still loses
  2. 計算帳單與機器成本
  3. 讓資料留在機構內部
  4. 用表格做選擇
  5. 掌握損益平衡點
  6. 本書為收件匣所做的決定

返回頂端所有章節