對小型模型下提示,並將其答案限制在各團隊內

閱讀約需 41 分鐘 分鐘,7 節 中共有 5 個範例。

版次 2026.11.9 範例於 2026年11 月 20 通過測試 36 個範例中有 36 個通過 變更內容

由 AI 翻譯。經人工以英文驗證。 我們如何製作書籍

譯自英文書,版次 2026.11.9。 閱讀原文

讀完本章後,你可以為小型開放模型撰寫路由器提示,限制它的答案只能是團隊名稱,並衡量這些答案有多少是正確的。你也會看到為什麼有效的答案不等於正確的答案,這也是下一章要訓練模型,而不是對模型提出更好問題的原因。

路由器需要每個答案都是團隊名稱,而第 1 章讓小型模型以自由文字作答時,模型並沒有回傳團隊。本章先以建構方式讓答案必定是團隊,接著用一個提示和一項限制,執行第 3 章的兩個開放模型 SmolLM2-135M-Instruct 與 SmolLM2-360M-Instruct,並衡量它們的回傳結果。135M 模型正是第 5 章要訓練的模型,因此這裡的計數就是訓練必須超越的基準。

本章會出現兩種數字,並將兩者分開處理。即時範例使用 test_40 的前 6 則訊息執行,該集合是固定樣本的前 40 則訊息,且兩個模型使用同樣的 6 則。這些範例用來展示機制。6 則訊息無法區分兩個模型,因此比較改用一份記錄:每個模型與提示對 test_200 的全部 200 則訊息各回答一次,這 200 則是該樣本的前 200 則,包含上述 6 則,答案並已儲存。讀取記錄時不會載入任何模型,其計數只來自已儲存的答案,不來自其他來源。

在任何錄製之前先定案的提示

最近一次變動於第 2026.11.9 版

本章的每個範例都使用同一份路由提示(router prompt)。它告訴模型這則訊息來自一家網路銀行,要求模型只回答一個客服團隊、而且只寫團隊名稱,列出 77 個團隊,並以 Team: 結尾,讓模型接下來寫出的字就是答案。未經此任務訓練的原始模型(raw model)只會拿到這些內容,因此措辭很重要。本章不調整這份提示。提示在任何錄製之前就已固定,錄製的清單檔(manifest)中記有提示的雜湊值。提示一旦更動,就是另一次錄製,其計數不能與此處的計數並列比較。

提示中的團隊清單,與限制條件所用的清單相同。下一節的前綴樹(trie)以相同的標籤名稱、相同的順序建立,因此提示與限制條件不會對「有哪些團隊」產生分歧。這項相依關係也反向成立。讀者若新增或更改團隊名稱,提示就隨之改變,錄製的雜湊值也跟著改變。本章的計數便不再描述那份提示,讀者必須重新錄製後才能比較。

原始形式(raw form)會把路由提示的文字原樣交給模型。聊天形式(chat form)則把路由提示的文字包進模型的聊天範本(chat template)中,範本會加上預設的系統提示。想知道範本改變了什麼,可以對同樣的六則訊息執行聊天形式:

constrained-decode-135m-chat.py Python 3.12.15 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0 通過
"""Route the first 6 messages of test_40 with the 135M model through its chat template, decoding only team names.The template adds the model's default system prompt. The prose quotes this run's accuracy line next to the raw form's, and the messages are never printed."""import os# no network: the pinned model must already be in the local cacheos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import collections  # noqa: E402import torch  # noqa: E402# two threads, so the sums add in the same order on every runtorch.set_num_threads(2)import router_lib  # noqa: E402# seeds 0 for random and torch, deterministic algorithms and float32, so greedy decoding repeatsrouter_lib.setup_runtime()SMALL = "HuggingFaceTB/SmolLM2-135M-Instruct"tokenizer, model = router_lib.load_model(SMALL)trie = router_lib.build_trie(tokenizer)# (text, label index) for every test row, in file order; the index is the gold teamrows = router_lib.read_rows("test")# row numbers of the first six messages of the fixed 40-message samplesample = router_lib.read_sample("test_40", 6)teams = set(router_lib.LABELS)results = []for i in sample:    # the chat template wraps the message in the model's chat format, with its default system prompt    text = router_lib.chat_prompt(tokenizer, rows[i][0])    # the template has already written the special tokens, so the tokenizer adds none    results.append(router_lib.constrained_team_text(tokenizer, model, trie, text, add_special_tokens=False))valid = [text in teams for _, text in results]# the walk's index, never the decoded text, is what countsright = [index == rows[i][1] for (index, _), i in zip(results, sample)]given = collections.Counter(text for _, text in results)# most given team; a tie in count goes to the team listed firsttop = min(given, key=lambda team: (-given[team], router_lib.LABELS.index(team)))print(f"messages: {len(sample)}")print(f"answers that are a team: {sum(valid)}/{len(sample)}")print(f"accuracy: {sum(right)}/{len(sample)} ({100 * sum(right) / len(sample):.1f}%)")print(f"distinct teams given: {len(given)}")print(f"most given team: {top}, {given[top]}/{len(sample)}")
輸出python constrained-decode-135m-chat.py
messages: 6
answers that are a team: 6/6
accuracy: 0/6 (0.0%)
distinct teams given: 3
most given team: cancel_transfer, 3/6

本書以原始形式作為預設。這項選擇在讀取任何錄製結果之前就已做出:比較中的兩個模型都以它來量測,所以比較的是模型之間的差異,而不是提示之間的差異。聊天形式是之後才錄製的,僅作為觀察。在 200 則訊息上,聊天形式答對 11/200 (5.5%) 則,原始形式答對 9/200 (4.5%) 則。200 則中相差兩則,這份樣本無法在任一方向上顯示出這是真正的差異,因此不能說聊天形式有幫助,也不能說原始形式有害,預設值不因這個計數而更改。前綴樹的名稱以空格開頭,在原始形式中它接在 Team: 之後,但在聊天範本中,答案開頭是換行字元,而非空格。因此聊天形式的計數對應的是這棵前綴樹與範本的組合,而不只是範本本身。這個結果不能證明原始提示對小型模型普遍優於聊天範本,僅憑 200 則訊息的一份樣本也無法下此定論。

實用原則: 對所有要比較的模型使用同一份提示,並採用比較所需的形式。在採用聊天範本之前,先用與原始形式相同的樣本量測它,而差距很小時,就如實報告為小差距。

為什麼自由文字會失敗

第 1 章要求原始模型在沒有任何限制的情況下回答一個團隊,模型便自由解碼:與本章相同,採用貪婪解碼,每一步取分數最高的詞元,但可選的詞元上沒有任何遮罩。該章的範例統計了這些回答。其中有 0/20 是團隊名稱,因此路由器沒有可以往下送的結果。

模型本身的解碼過程,並不能防止回答變成一個句子、一個不在清單上的團隊,或是某個團隊名稱的近似拼寫。在每一步,解碼器都可以挑選詞彙表中的任何詞元,所以回答可能偏向任何地方。第 1 章的替身並未呈現這種失敗,因為它所記錄的回答來自一個只允許團隊名稱的文法。在路由器中執行的模型,需要由解碼器本身提供同樣的保證。

看似可行的補救方式,是取模型寫出的內容,再比對到最接近的團隊名稱。這種補救只是猜測。近似拼寫可能對應到正確的團隊,也可能對應到字母相近的錯誤團隊,路由器便會把訊息送往模型從未指名的團隊。第 1 章已說明應有的做法:不是團隊的回答視為錯誤,在樣本上計數,並將訊息交給人工處理。限制可以在回答寫出之前就排除其中大部分。計數仍然保留,這樣一來,失效的限制會以數字呈現,而不是悄悄改變路由結果。

實用原則: 將不是團隊的回答視為失敗的路由。在模型負責任何路由之前,先在固定樣本上統計這類回答,而且不要把近似拼寫比對到最接近的團隊。在解碼時限制回答,並保留計數,使失效的限制以數字呈現。

此部分尚未翻譯,因此以英文呈現。

A token trie allows only team names

最近一次變動於第 2026.11.9 版

A model writes tokens, not words, and a team name is a short run of them. build_trie in router_lib, the book's shared module, stores the 77 names as a trie: a tree whose paths spell out each team's tokens, with the team's index at the node where its path ends. The function asserts that no two teams end at the same node, so each path names one team.

The decoder walks this tree as it writes. At the start, the allowed tokens are the first token of each team name. After each token, the allowed set narrows to the children of the node the decoder has reached. Where a path has ended on a team, the end-of-answer token joins the allowed set, so the answer can stop there. No one of the 77 names is the start of another, so every path ends at a leaf. The code would still handle a name that was: at that node the decoder could stop, or continue along the longer name. A walk of this kind cannot leave the list, because every token it may choose leads toward a team.

Greedy decoding then takes the highest-scoring of the allowed tokens at each step. Transformers applies the rule through a callback, prefix_allowed_tokens_fn, which returns the allowed tokens for each position. The processor starts from a mask of minus infinity for every token, sets the allowed tokens' entries to zero, and adds the mask to the scores with scores_processed = scores + mask. Every token that is not allowed keeps minus infinity, so greedy decoding cannot pick it (transformers 5.23.0, src/transformers/generation/logits_process.py).

Chapter 1's stand-in got the same guarantee from its library. recordings/common.py builds a grammar, root ::= "card_arrival" | ..., that lists the 77 names as literal strings, and the recordings applied it through the chat template. llama-cpp-python adds a grammar to its sampler before the greedy step. In llama_cpp/llama.py the line sampler.add_grammar(self._model, grammar) runs under if grammar is not None:, and sampler.add_greedy() runs after it, under elif temp == 0.0: (llama-cpp-python v0.3.36, llama_cpp/llama.py). So the stand-in's answers were limited to the names by the library, where this chapter's callback does the same job in the decoder.

Two details of that code shape the callback. The processor calls it as self._prefix_allowed_tokens_fn(batch_id, sent), so the batch index comes first. And an empty list is an error: the processor raises ValueError when the callback returns no tokens. The book's callback therefore never returns an empty list. When no path can continue, it returns the end-of-answer token, and the answer stops there. The end-of-answer token is the model's <|im_end|>, which is also its pad token (id 2). The decoder stops after 32 new tokens (max_new_tokens=32). Every one of the 600 recorded answers ended on a team. A node with no children always holds a team, so the empty-list case is a guard and not a case the 77 names produce.

First, the 135M model with the trie, on those six messages.

constrained-decode-135m.py Python 3.12.15 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0 通過
"""Route the first 6 messages of test_40 with the 135M model, decoding only team names, and count how many right answers that gives.The point rests on the count of answers that are team names, which the trie guarantees, and on the accuracy line, which the measurement section reads. The messages are read from the test split and are never printed."""import os# no network: the pinned model must already be in the local cacheos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import collections  # noqa: E402import torch  # noqa: E402# two threads, so the sums add in the same order on every runtorch.set_num_threads(2)import router_lib  # noqa: E402# seeds 0 for random and torch, deterministic algorithms and float32, so greedy decoding repeatsrouter_lib.setup_runtime()SMALL = "HuggingFaceTB/SmolLM2-135M-Instruct"tokenizer, model = router_lib.load_model(SMALL)trie = router_lib.build_trie(tokenizer)# (text, label index) for every test row, in file order; the index is the gold teamrows = router_lib.read_rows("test")# row numbers of the first six messages of the fixed 40-message samplesample = router_lib.read_sample("test_40", 6)teams = set(router_lib.LABELS)# (label index, team text); the index comes from the trie walk, not from parsing the textresults = [router_lib.constrained_team(tokenizer, model, trie, rows[i][0]) for i in sample]valid = [text in teams for _, text in results]# the walk's index, never the decoded text, is what countsright = [index == rows[i][1] for (index, _), i in zip(results, sample)]given = collections.Counter(text for _, text in results)# most given team; a tie in count goes to the team listed firsttop = min(given, key=lambda team: (-given[team], router_lib.LABELS.index(team)))print(f"messages: {len(sample)}")print(f"answers that are a team: {sum(valid)}/{len(sample)}")print(f"accuracy: {sum(right)}/{len(sample)} ({100 * sum(right) / len(sample):.1f}%)")print(f"distinct teams given: {len(given)}")print(f"most given team: {top}, {given[top]}/{len(sample)}")
輸出python constrained-decode-135m.py
messages: 6
answers that are a team: 6/6
accuracy: 1/6 (16.7%)
distinct teams given: 3
most given team: card_arrival, 4/6

Run on 6 messages, the model's answers are team names in every case, 6/6, which is the guarantee the trie gives. The accuracy on the same messages is 1/6 (16.7%). The measurement section takes that second number apart.

The router does not read the team out of the text. constrained_team returns the index stored at the node where the walk ended, and the text is decoded only so that a reader can see it. The index is fixed by the trie and not by a parser that runs after the model. That is the reason the constraint is worth the code: a parser that matches text to teams can fail in ways the trie cannot. These examples return the team only so that it can be counted. Nothing in this chapter routes a message. A route returns a team only through handoff, which sends an uncertain answer to a person, and the examples here do not call it.

Working rule: Constrain the answer to the labels with a token trie built from the label names, and count the answers that are labels on a fixed sample. The constraint makes every answer a team. It does not make any answer right, so count the right answers as a separate measure.

以貪婪解碼讓每個計數都能重現

最近一次變動於第 2026.11.2 版

這裡的每個範例都使用貪婪解碼:分數最高者勝出,不做任何取樣。這對計數很重要。取樣得到的答案每次執行都可能不同,如此一來,任何印出的計數都無法與記錄比對。這些範例透過 router_lib.setup_runtime 固定隨機種子與 float32,並設定兩個執行緒,因此在同一部機器上,每次執行都會得到相同的答案。每個範例都會在開頭自行呼叫 torch.set_num_threads(2),使其不受執行器啟動方式的影響。每個範例印出的輸出,在每次執行時都會與其記錄的輸出比對,任何一行有變動,檢查就會失敗。

在同一部機器上可重複,並不等於在所有地方都可重複。PyTorch 2.15.0+cpu 關於隨機性的說明指出,不保證在不同發行版本、個別提交或不同平台之間得到完全可重現的結果。因此,本書只對 examples/requirements.txt 中固定的版本,以及本書執行時所用類型的機器宣稱可重複性。使用其他平台的讀者,應預期需要重新核對記錄的各行。如果某一行記錄在該處比對失敗,請先閱讀變動的那一行,再動手修改範例。正確答案的數量不同,可能來自不同的機器、不同的函式庫版本,或是更動過的提示,而每一種原因所需的修正方式都不一樣。

平手時同樣需要固定的規則。上一節中的標籤評分,在分數相同時以較小的團隊索引決定先後,方式是透過其排序鍵 (score, -index);而被給予次數最多的團隊計數,在次數相同時則以團隊名稱決定。受限解碼沒有團隊分數。它的貪婪步驟會在允許的詞元中取第一個最大值,因此平手時由較小的詞元 id 勝出。

實用原則: 使用貪婪解碼,固定隨機種子與執行緒數,並在每次執行時將每一行印出的輸出與記錄的副本比對。不要把結果描述成在其執行時所用的版本與機器類型之外仍可重現。

衡量受限解碼的答案

最近一次變動於第 2026.11.2 版

實際範例涵蓋六則訊息。使用原始提示的 135M 模型答對 1/6 (16.7%),360M 模型答對 0/6 (0.0%)。六則訊息無法為這些模型排名:這兩個數字之間只差一個正確答案。135M 模型最常給出的答案是 card_arrival, 4/6,而在同樣六則訊息上,聊天格式的結果是 0/6 (0.0%)。

360M 的執行使用同一支程式,只是換成較大的模型。

constrained-decode-360m.py Python 3.12.15 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0 通過
"""Decode team names for the first 6 messages of test_40 with the 360M model, and count how many of them are right.The prose sets its accuracy line against the 135M model's, on the same six messages. The messages are read from the test split and are never printed."""import os# no network: the pinned model must already be in the local cacheos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import collections  # noqa: E402import torch  # noqa: E402# two threads, so the sums add in the same order on every runtorch.set_num_threads(2)import router_lib  # noqa: E402# seeds 0 for random and torch, deterministic algorithms and float32, so greedy decoding repeatsrouter_lib.setup_runtime()MEDIUM = "HuggingFaceTB/SmolLM2-360M-Instruct"tokenizer, model = router_lib.load_model(MEDIUM)trie = router_lib.build_trie(tokenizer)# (text, label index) for every test row, in file order; the index is the gold teamrows = router_lib.read_rows("test")# row numbers of the first six messages of the fixed 40-message samplesample = router_lib.read_sample("test_40", 6)teams = set(router_lib.LABELS)# (label index, team text); the index comes from the trie walk, not from parsing the textresults = [router_lib.constrained_team(tokenizer, model, trie, rows[i][0]) for i in sample]valid = [text in teams for _, text in results]# the walk's index, never the decoded text, is what countsright = [index == rows[i][1] for (index, _), i in zip(results, sample)]given = collections.Counter(text for _, text in results)# most given team; a tie in count goes to the team listed firsttop = min(given, key=lambda team: (-given[team], router_lib.LABELS.index(team)))print(f"messages: {len(sample)}")print(f"answers that are a team: {sum(valid)}/{len(sample)}")print(f"accuracy: {sum(right)}/{len(sample)} ({100 * sum(right) / len(sample):.1f}%)")print(f"distinct teams given: {len(given)}")print(f"most given team: {top}, {given[top]}/{len(sample)}")
輸出python constrained-decode-360m.py
messages: 6
answers that are a team: 6/6
accuracy: 0/6 (0.0%)
distinct teams given: 4
most given team: card_payment_wrong_exchange_rate, 3/6

為了進行比較,每個變體各回答 200 則訊息。錄製結果的計數列在表中,每一列使用的訊息都相同:

test_200 上的變體 團隊清單以外的答案 答對 給出的相異團隊數
135M,原始提示 0/200 9/200 (4.5%) 18
135M,聊天範本 0/200 11/200 (5.5%) 17
360M,原始提示 0/200 12/200 (6.0%) 8

此範例從錄製結果中讀取表中的計數:

prompting-on-test-200.py Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0 通過
"""Compare the three prompting variants on test_200, from the recording only. No model is loaded.The chapter's table takes its three columns from the first three lines printed for each variant, and the point rests on the correct count.recordings/prompting_eval.py wrote one answer per variant and message to examples/data/recorded/prompting-eval/answers.jsonl."""import collectionsimport jsonfrom pathlib import Pathimport torch# two threads, as in the other examplestorch.set_num_threads(2)import router_lib  # noqa: E402# seeds 0 for random and torch, deterministic algorithms and float32, as in the other examplesrouter_lib.setup_runtime()ANSWERS = Path(__file__).resolve().parent / "data" / "recorded" / "prompting-eval" / "answers.jsonl"teams = set(router_lib.LABELS)# (text, label index) for every test row, in file order; the index is the gold teamrows = router_lib.read_rows("test")# the 200 messages of the fixed sample; every variant answers all of themtotal = len(router_lib.read_sample("test_200"))by_variant = collections.defaultdict(list)with open(ANSWERS) as f:    for line in f:        answer = json.loads(line)        by_variant[answer["variant"]].append(answer)for variant in sorted(by_variant):    answers = by_variant[variant]    outside = sum(answer["team"] not in teams for answer in answers)    # answer["label"] is the team index the variant chose; rows[...][1] is the message's gold index    right = sum(answer["label"] == rows[answer["index"]][1] for answer in answers)    given = collections.Counter(answer["team"] for answer in answers)    # most given team; a tie in count goes to the team name that sorts first, not to the index    top = min(given, key=lambda team: (-given[team], team))    # the gold index becomes a team name, to compare with the most given team    gold_top = sum(router_lib.LABELS[rows[answer["index"]][1]] == top for answer in answers)    print(f"{variant} answers outside the team list: {outside}/{total}")    print(f"{variant} correct: {right}/{total} ({100 * right / total:.1f}%)")    print(f"{variant} distinct teams given: {len(given)}")    print(f"{variant} most given team: {top}, {given[top]}/{total}")    print(f"{variant} gold for most given team: {gold_top}/{total}")
輸出python prompting-on-test-200.py
135m-chat answers outside the team list: 0/200
135m-chat correct: 11/200 (5.5%)
135m-chat distinct teams given: 17
135m-chat most given team: topping_up_by_card, 55/200
135m-chat gold for most given team: 1/200
135m-raw answers outside the team list: 0/200
135m-raw correct: 9/200 (4.5%)
135m-raw distinct teams given: 18
135m-raw most given team: card_arrival, 165/200
135m-raw gold for most given team: 0/200
360m-raw answers outside the team list: 0/200
360m-raw correct: 12/200 (6.0%)
360m-raw distinct teams given: 8
360m-raw most given team: card_payment_wrong_exchange_rate, 94/200
360m-raw gold for most given team: 4/200

沒有任何一列接近可用的路由器,不過限制在每一列都成立:任何一列都沒有團隊清單以外的答案。360M 原始提示這一列的正確答案最多,為 12/200 (6.0%),而且各列之間的差距都不超過幾則訊息。

單看相異團隊數會造成誤導。135M 原始提示這一列給出 18 個相異團隊,360M 原始提示這一列則為 8 個,但 135M 這一列的多數答案都落在同一個團隊 card_arrival, 165/200,而 360M 原始提示這一列中最常給出的團隊只得到 card_payment_wrong_exchange_rate, 94/200。

閱讀一列時,先看第一欄。如果它不是零,表示限制沒有成立,在修正之前,其餘兩欄都沒有意義。第二欄對路由而言最重要:它計算團隊與標準標籤相符的訊息數,分母是整個樣本,答錯的一律算錯。第三欄說明該變體實際用到多少個團隊。只給出少數團隊的變體,若不是在猜幾個熱門團隊,就是學到了某種狹窄的規律,而這個計數無法分辨是哪一種。數量很大也不能排除集中的情形,135M 原始提示這一列就是例子。讀者若在自己的訊息樣本上計算同樣三個數字,就能在自己的資料上得到比較結果,而且每一欄的意義相同。

360M 模型是兩者中較大的一個,第 3 章已經計算過它大了多少。這張表顯示在此提示與此限制之下,這筆成本換來了什麼:360M 答對 12/200 (6.0%),而 135M 模型在同樣的原始形式下答對 9/200 (4.5%)。這只是較大的模型多答對幾則訊息,本章不認為這是值得多付出額外規模的理由。

本書沒有為這些計數計算區間,200 則中相差幾則的差距可能來自樣本本身,因此這張表只支持一項陳述:三個變體很接近,而且沒有一個是路由器。讀者若想在兩種提示之間做選擇,應在自己的訊息樣本上執行兩者,並計算同樣三項數字。

trie 只檢查答案的形式,不檢查其他事項。它不會閱讀提示以外的訊息,也不知道哪個團隊才是正確的。在團隊名稱仍可能被寫出的每一步,清單上的任何團隊都是允許的,因此解碼器在其中的選擇是模型的選擇,遮罩只決定哪些團隊可以被寫出。

以下是一個有效但錯誤的答案,只以團隊描述,因為本書從不引用訊息。有一則標準團隊為 card_not_working 的測試訊息,被 135M 原始提示這一列判為 card_arrival,這是清單上的團隊,所以限制讓它通過,清單以外答案的計數仍維持為零。

這些輸出支持的結論是:受限解碼讓每個答案都是團隊,但有效的答案不等於正確的答案。限制只會移除不是團隊的答案,它不會選出更好的團隊,提示的更動也沒有做到。第 5 章是下一步:它在已標註的訊息上訓練一個分類頭,並在同一個樣本上衡量訓練後的模型是否路由得更好。

在正式環境中,這種失敗不會以錯誤的形式出現。不是團隊的答案計數維持為零,因此只監看該計數的檢查會一直顯示正常。只有在固定且已標註的樣本上計算正確答案數,才能看出問題。

代價落在收到訊息的人身上。有效但錯誤的團隊會把訊息送到錯誤的目的地,支援人員在那裡看到不屬於自己的請求,必須找出正確的團隊並轉交,這與轉手所產生的工作相同,而訊息上沒有任何跡象顯示它被送錯。無法表示自己不確定的路由器,沒有給這些人任何可據以行動的資訊。受限解碼也不會回傳其他 76 個團隊的分數,因此建立在其上的轉手規則沒有各團隊的分數可供比較。

實用原則: 對你比較的每個變體,在同一個固定樣本上,同時回報正確答案數與有效答案數。所有答案都有效的路由器,仍可能在多數訊息上出錯,只有第二個計數能顯示這一點。

為什麼先驗不等於路由器

最近一次變動於第 2026.11.9 版

約束式解碼只回傳一個答案,沒有分數。標籤評分則為每個團隊給出一個分數。對每個團隊,標籤評分會把該團隊名稱的各個 token 在提示之後的對數機率加總,其中對數機率是模型賦予某個 token 的機率取對數的結果。總和最高的團隊,就是模型對該則訊息的首選團隊。

以三則訊息執行,並讀出各則的首選團隊:

label-scoring-prior.py Python 3.12.15 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0 通過
"""Score every team for the first 3 messages of test_40 with the 135M model, by the summed log-probability of its tokens.This shows the model's prior, the team it favours before any fine-tuning. The point rests on the most frequent top team and how many messages it tops. The messages are never printed."""import os# no network: the pinned model must already be in the local cacheos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import collections  # noqa: E402import torch  # noqa: E402# two threads, so the sums add in the same order on every runtorch.set_num_threads(2)import router_lib  # noqa: E402# seeds 0 for random and torch, deterministic algorithms and float32, so greedy decoding repeatsrouter_lib.setup_runtime()SMALL = "HuggingFaceTB/SmolLM2-135M-Instruct"tokenizer, model = router_lib.load_model(SMALL)rows = router_lib.read_rows("test")# row numbers of the first three messages of the fixed 40-message samplesample = router_lib.read_sample("test_40", 3)# each team's token ids, padded to the longest name, built once for all messagestable = router_lib.label_token_table(tokenizer)tops = []for i in sample:    scores = router_lib.label_scores(tokenizer, model, table, rows[i][0])    # index of the highest-scoring team; a tie goes to the lower index    tops.append(max(range(len(scores)), key=lambda k: (scores[k], -k)))counts = collections.Counter(tops)# most frequent top team; a tie in count goes to the lower indexbest = min(counts, key=lambda k: (-counts[k], k))print(f"messages: {len(sample)}")print(f"distinct top teams: {len(counts)}")print(f"most frequent top team name: {router_lib.LABELS[best]}")print(f"most frequent top team: {counts[best]} of {len(sample)}")
輸出python label-scoring-prior.py
messages: 3
distinct top teams: 1
most frequent top team name: card_arrival
most frequent top team: 3 of 3

在尚未經過任何訓練之前,模型對每則訊息都把同一個團隊排在第一位:card_arrival 在 3 of 3 則訊息中位居首位。該團隊就是模型的先驗,而三則訊息太少,無法對此多作推論。

如果每則訊息的首選團隊都相同,這個分數反映的就是模型的先驗,對任何單一訊息幾乎沒有說明力。200 則訊息的紀錄在約束式解碼中呈現了相同的先驗:card_arrival, 165/200 是 135M raw 該列最常給出的團隊,在樣本中是 0/200 則訊息的標準答案團隊。

這個總和偏好較短的名稱。團隊名稱的每個 token 都會加上一個不大於零的對數機率,因此 token 較多的團隊有更多項把總分往下拉。此範例只是把各項相加,沒有除以長度,本章也不宣稱已對此作了校正。使用標籤評分的讀者,在信任首選團隊之前,應先拿一份帶有標準答案標籤的樣本來檢查。

標籤評分確實為每個團隊給出一個分數,而這正是轉交規則所需要的。每則訊息都會對全部 77 個團隊名稱評分,程式碼在提示執行一次之後,以單一批次完成這些評分。第 5 章則是透過一個以已標註訊息訓練出來的分類頭,只做一次前向傳遞就得到每個團隊的分數。標籤評分是從本章通往該章的橋梁:它顯示模型本來就有逐團隊的分數,而在訓練之前,這些分數對訊息本身幾乎沒有說明力。

實用原則: 用標籤評分來觀察模型在訓練前偏好哪個團隊,不要用它來做路由。在這項任務上,不要用這種規模的提示式模型來路由訊息;200 則訊息的紀錄就是證據。改為以已標註的訊息訓練一個分類頭,如第 5 章所做的那樣。

本章的決定

最近一次變動於第 2026.11.2 版

本章的規則僅適用於這兩個模型、這個提示詞與這項限制,並以錄製的 200 則訊息為範圍。更大的模型、不同的提示詞或更長的樣本,都可能改變計數結果,而本書並未在該樣本上測試過其中任何一項。本章確實能夠證明的範圍較窄:在使用原始提示詞與 trie 的情況下,這種規模的模型會回傳有效的團隊,但正確的很少,而正確答案的數量才是需要留意的數字。

每項決定都在各自的節中測試:提示詞在第一節,trie 在該節,貪婪解碼在該節,兩項計數則在測量一節。如第一節所述,讀者若更動其中任何一項,在比較之前必須重新錄製。限制能讓每個答案都有效,但只有訓練才能讓答案正確。

本章做出了以下決定:

  • 路由器提示詞採用原始形式,比較中的兩個模型都使用它。
  • 受限解碼使用涵蓋 77 個團隊名稱的 token trie,搭配貪婪解碼與固定的隨機種子。
  • 每個變體都在同一個固定樣本上,同時回報有效答案的數量與正確答案的數量。
  • 即時範例用來說明各個部分如何運作,只有 200 則訊息的錄製資料才用來比較模型。

本章,於本版次中

版次 2026.11.9範例於 2026年11 月 20 通過測試本章 5 個範例 個範例中有 5 個通過

譯自英文書,版次 2026.11.9。

引用本節

格式

各種格式皆引用英文原書,本譯本即依循該原書。

每種格式都會標示本版。其連結會開啟目前版次,節標題會說明該節內文最後一次變動的時間。存取日期與擷取日期即為你開啟此頁的當日。

本頁內容

第 4 章,共 10 章: 對小型模型下提示,並將其答案限制在各團隊內

  1. 在任何錄製之前先定案的提示
  2. 為什麼自由文字會失敗
  3. A token trie allows only team names
  4. 以貪婪解碼讓每個計數都能重現
  5. 衡量受限解碼的答案
  6. 為什麼先驗不等於路由器
  7. 本章的決定

返回頂端所有章節