Prompt a small model and constrain its answers to the teams

阅读约需 44分钟 分钟,7节中含5个示例。

版次 2026.11.9 示例于 2026年11月20日 通过 36个示例中有36个通过 变化内容

AI 翻译。由人工以英文核验。 我们如何制作图书

译自英文版图书,版次 2026.11.9。 阅读原文

这部分尚未翻译,因此以英文显示。

By the end of this chapter, you can write a router prompt for a small open model, constrain its answer so that it can only be a team name, and measure how many of those answers are right. You will also see why a valid answer is not a correct one, which is why the next chapter trains a model instead of asking it better questions.

A router needs every answer to be a team name, and when chapter 1 let a small model answer in free text, the model did not return teams. This chapter first makes the answer a team by construction. It then runs the two open models from chapter 3, SmolLM2-135M-Instruct and SmolLM2-360M-Instruct, with one prompt and one constraint, and measures what they return. The 135M model is the one chapter 5 trains, so the counts here are the ones that training has to beat.

Two kinds of number appear, and the chapter keeps them apart. The live examples run on the first six messages of test_40, the first 40 messages of the fixed sample, and the same six for both models. They show the mechanism. Six messages cannot separate two models, so the comparison uses a recording: each model and prompt answered all 200 messages of test_200, the first 200 of that sample, which includes those six, once, and the answers were saved. Reading the recording loads no model, and its counts come from the saved answers and nothing else.

在任何录制之前确定提示词

最近一次修改见于版次 2026.11.9

本章的每个示例都使用同一个路由提示词。它告诉模型这条消息来自一家网络银行,要求只给出一个支持团队,且只写团队名称,列出 77 个团队,并以 Team: 结尾,这样模型接下来写出的内容就是答案。一个没有针对该任务训练过的原始模型除此之外什么也得不到,因此措辞很重要。本章不对它做调优。提示词在任何录制之前就已固定,录制的清单中保存着提示词的哈希值。提示词一旦改变,就是另一次录制,其计数不能与这里的计数放在一起比较。

提示词中的团队列表与约束所用的列表是同一份。下一节中的字典树由相同的标签名称按相同的顺序构建,因此提示词和约束不会在哪些团队存在这一点上产生分歧。这种依赖关系反过来同样成立。读者若增加或重命名某个团队,就改变了提示词,也就改变了录制的哈希值。此时本章的计数不再描述该提示词,读者必须重新录制后再做比较。

原始形式把路由提示词的文本原样传给模型。对话形式则把路由提示词的文本包裹在模型的对话模板中,模板会加入一段默认的系统提示词。要看清模板带来了什么变化,可以在同样的六条消息上运行对话形式:

constrained-decode-135m-chat.py Python 3.12.15 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0 通过
"""Route the first 6 messages of test_40 with the 135M model through its chat template, decoding only team names.The template adds the model's default system prompt. The prose quotes this run's accuracy line next to the raw form's, and the messages are never printed."""import os# no network: the pinned model must already be in the local cacheos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import collections  # noqa: E402import torch  # noqa: E402# two threads, so the sums add in the same order on every runtorch.set_num_threads(2)import router_lib  # noqa: E402# seeds 0 for random and torch, deterministic algorithms and float32, so greedy decoding repeatsrouter_lib.setup_runtime()SMALL = "HuggingFaceTB/SmolLM2-135M-Instruct"tokenizer, model = router_lib.load_model(SMALL)trie = router_lib.build_trie(tokenizer)# (text, label index) for every test row, in file order; the index is the gold teamrows = router_lib.read_rows("test")# row numbers of the first six messages of the fixed 40-message samplesample = router_lib.read_sample("test_40", 6)teams = set(router_lib.LABELS)results = []for i in sample:    # the chat template wraps the message in the model's chat format, with its default system prompt    text = router_lib.chat_prompt(tokenizer, rows[i][0])    # the template has already written the special tokens, so the tokenizer adds none    results.append(router_lib.constrained_team_text(tokenizer, model, trie, text, add_special_tokens=False))valid = [text in teams for _, text in results]# the walk's index, never the decoded text, is what countsright = [index == rows[i][1] for (index, _), i in zip(results, sample)]given = collections.Counter(text for _, text in results)# most given team; a tie in count goes to the team listed firsttop = min(given, key=lambda team: (-given[team], router_lib.LABELS.index(team)))print(f"messages: {len(sample)}")print(f"answers that are a team: {sum(valid)}/{len(sample)}")print(f"accuracy: {sum(right)}/{len(sample)} ({100 * sum(right) / len(sample):.1f}%)")print(f"distinct teams given: {len(given)}")print(f"most given team: {top}, {given[top]}/{len(sample)}")
输出python constrained-decode-135m-chat.py
messages: 6
answers that are a team: 6/6
accuracy: 0/6 (0.0%)
distinct teams given: 3
most given team: cancel_transfer, 3/6

本书默认使用原始形式。这一选择是在读取任何录制结果之前做出的:参与比较的两个模型都按它来测量,因此比较的是模型之间的差异,而不是提示词之间的差异。对话形式是在之后录制的,作为一项观察。在这 200 条消息上,它答对了 11/200 (5.5%) 条,原始形式答对了 9/200 (4.5%) 条。200 条中相差两条,这个样本无法在任何一个方向上显示出差异,因此既不能说明对话形式有帮助,也不能说明原始形式有损害,默认设置也不因这一计数而改变。字典树中的名称以一个空格开头,在原始形式中它接在 Team: 之后,而在对话模板中,接在后面的并不是空格,而是开启答案的换行。因此对话形式的计数既取决于模板,也取决于这棵字典树。这个结果并不能证明对小模型而言原始提示词总是优于对话模板,仅凭 200 条消息的一个样本也无法就此下结论。

工作准则: 对所有参与比较的模型使用同一个提示词,并采用比较所需要的形式。在采用某个对话模板之前,先在与原始形式相同的样本上对它进行测量,并把很小的差距如实报告为很小的差距。

自由文本为何会失败

第 1 章曾让原始模型在没有任何约束的情况下给出一个团队,模型是自由解码的:与本章一样采用贪心方式,即每一步取得分最高的词元,但对可选的词元没有施加掩码。该章的示例统计了这些回答。其中 0/20 是团队名称,因此路由器没有可以继续转发的内容。

模型自身的解码过程并不能阻止回答变成一句话、一个不在列表中的团队,或某个团队名称的近似拼写。解码器在每一步都可以选取词表中的任意词元,所以回答可以出现任何内容。第 1 章中的替代实现并没有表现出这种失败,因为它记录的回答来自一个只允许团队名称的语法。在路由器中运行的模型,需要解码器本身提供同样的保证。

看上去可行的补救办法,是把模型写出的内容与最接近的团队名称相匹配。这种补救只是猜测。近似拼写可能匹配到正确的团队,也可能匹配到字母相近的错误团队,路由器随后就会把消息发给模型从未提到的团队。第 1 章已经说明了应当怎么做:不是团队的回答算作错误,在样本上计数,消息交给人工处理。约束会在回答写出之前排除其中的大部分。计数仍要保留,这样,一旦约束失效,就会表现为一个数字,而不是路由中悄然出现的变化。

工作准则: 把不是团队的回答视为路由失败。在让模型路由任何内容之前,先在固定样本上统计这类回答,并且不要把近似拼写匹配到最接近的团队。在解码时约束回答,并保留计数,以便约束失效时表现为一个数字。

这部分尚未翻译,因此以英文显示。

A token trie allows only team names

最近一次修改见于版次 2026.11.9

A model writes tokens, not words, and a team name is a short run of them. build_trie in router_lib, the book's shared module, stores the 77 names as a trie: a tree whose paths spell out each team's tokens, with the team's index at the node where its path ends. The function asserts that no two teams end at the same node, so each path names one team.

The decoder walks this tree as it writes. At the start, the allowed tokens are the first token of each team name. After each token, the allowed set narrows to the children of the node the decoder has reached. Where a path has ended on a team, the end-of-answer token joins the allowed set, so the answer can stop there. No one of the 77 names is the start of another, so every path ends at a leaf. The code would still handle a name that was: at that node the decoder could stop, or continue along the longer name. A walk of this kind cannot leave the list, because every token it may choose leads toward a team.

Greedy decoding then takes the highest-scoring of the allowed tokens at each step. Transformers applies the rule through a callback, prefix_allowed_tokens_fn, which returns the allowed tokens for each position. The processor starts from a mask of minus infinity for every token, sets the allowed tokens' entries to zero, and adds the mask to the scores with scores_processed = scores + mask. Every token that is not allowed keeps minus infinity, so greedy decoding cannot pick it (transformers 5.23.0, src/transformers/generation/logits_process.py).

Chapter 1's stand-in got the same guarantee from its library. recordings/common.py builds a grammar, root ::= "card_arrival" | ..., that lists the 77 names as literal strings, and the recordings applied it through the chat template. llama-cpp-python adds a grammar to its sampler before the greedy step. In llama_cpp/llama.py the line sampler.add_grammar(self._model, grammar) runs under if grammar is not None:, and sampler.add_greedy() runs after it, under elif temp == 0.0: (llama-cpp-python v0.3.36, llama_cpp/llama.py). So the stand-in's answers were limited to the names by the library, where this chapter's callback does the same job in the decoder.

Two details of that code shape the callback. The processor calls it as self._prefix_allowed_tokens_fn(batch_id, sent), so the batch index comes first. And an empty list is an error: the processor raises ValueError when the callback returns no tokens. The book's callback therefore never returns an empty list. When no path can continue, it returns the end-of-answer token, and the answer stops there. The end-of-answer token is the model's <|im_end|>, which is also its pad token (id 2). The decoder stops after 32 new tokens (max_new_tokens=32). Every one of the 600 recorded answers ended on a team. A node with no children always holds a team, so the empty-list case is a guard and not a case the 77 names produce.

First, the 135M model with the trie, on those six messages.

constrained-decode-135m.py Python 3.12.15 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0 通过
"""Route the first 6 messages of test_40 with the 135M model, decoding only team names, and count how many right answers that gives.The point rests on the count of answers that are team names, which the trie guarantees, and on the accuracy line, which the measurement section reads. The messages are read from the test split and are never printed."""import os# no network: the pinned model must already be in the local cacheos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import collections  # noqa: E402import torch  # noqa: E402# two threads, so the sums add in the same order on every runtorch.set_num_threads(2)import router_lib  # noqa: E402# seeds 0 for random and torch, deterministic algorithms and float32, so greedy decoding repeatsrouter_lib.setup_runtime()SMALL = "HuggingFaceTB/SmolLM2-135M-Instruct"tokenizer, model = router_lib.load_model(SMALL)trie = router_lib.build_trie(tokenizer)# (text, label index) for every test row, in file order; the index is the gold teamrows = router_lib.read_rows("test")# row numbers of the first six messages of the fixed 40-message samplesample = router_lib.read_sample("test_40", 6)teams = set(router_lib.LABELS)# (label index, team text); the index comes from the trie walk, not from parsing the textresults = [router_lib.constrained_team(tokenizer, model, trie, rows[i][0]) for i in sample]valid = [text in teams for _, text in results]# the walk's index, never the decoded text, is what countsright = [index == rows[i][1] for (index, _), i in zip(results, sample)]given = collections.Counter(text for _, text in results)# most given team; a tie in count goes to the team listed firsttop = min(given, key=lambda team: (-given[team], router_lib.LABELS.index(team)))print(f"messages: {len(sample)}")print(f"answers that are a team: {sum(valid)}/{len(sample)}")print(f"accuracy: {sum(right)}/{len(sample)} ({100 * sum(right) / len(sample):.1f}%)")print(f"distinct teams given: {len(given)}")print(f"most given team: {top}, {given[top]}/{len(sample)}")
输出python constrained-decode-135m.py
messages: 6
answers that are a team: 6/6
accuracy: 1/6 (16.7%)
distinct teams given: 3
most given team: card_arrival, 4/6

Run on 6 messages, the model's answers are team names in every case, 6/6, which is the guarantee the trie gives. The accuracy on the same messages is 1/6 (16.7%). The measurement section takes that second number apart.

The router does not read the team out of the text. constrained_team returns the index stored at the node where the walk ended, and the text is decoded only so that a reader can see it. The index is fixed by the trie and not by a parser that runs after the model. That is the reason the constraint is worth the code: a parser that matches text to teams can fail in ways the trie cannot. These examples return the team only so that it can be counted. Nothing in this chapter routes a message. A route returns a team only through handoff, which sends an uncertain answer to a person, and the examples here do not call it.

Working rule: Constrain the answer to the labels with a token trie built from the label names, and count the answers that are labels on a fixed sample. The constraint makes every answer a team. It does not make any answer right, so count the right answers as a separate measure.

贪心解码,使每个计数都可重复

最近一次修改见于版次 2026.11.2

这里的每个示例都使用贪心解码:得分最高者胜出,不做任何采样。这对计数很重要。采样得到的答案每次运行都可能不同,这样一来,打印出的计数就无法与记录进行核对。示例通过 router_lib.setup_runtime 固定随机种子和 float32,并设置两个线程,因此在同一台机器上,运行会重复得到相同的答案。每个示例都在开头自行调用 torch.set_num_threads(2),这样它就不依赖运行器是如何启动的。每次运行时,每个示例打印的输出都会与其记录的输出比较,任何一行有变化,检查即告失败。

在同一台机器上可重复,并不等于在所有环境中都可重复。PyTorch 2.15.0+cpu 关于随机性的说明指出,不保证在不同版本、不同提交或不同平台之间得到完全可复现的结果。因此,本书只对 examples/requirements.txt 中固定的版本,并且只在本书运行时所用的那类机器上,声明结果可重复。使用其他平台的读者应预期需要重新核对记录的各行。如果在那里某一行记录未能通过,请先读一读发生变化的那一行,再去改动示例。正确答案的数量不同,可能源于机器不同、库版本不同或提示词有改动,而每种情况需要不同的修复方式。

平局同样需要固定的规则。上一节中的标签评分在得分相同时取团队索引较小者,这通过其排序键 (score, -index) 实现;而给出次数最多的团队的计数,在次数相同时按团队名称决定。约束解码没有团队得分。它的贪心步骤在允许的词元中取第一个最大值,因此平局时取词元 id 较小者。

**工作准则:**使用贪心解码,固定随机种子和线程数,并在每次运行时把每一行打印输出与记录的副本比较。不要在所运行的版本和机器类型之外,将某个结果描述为可重复。

衡量受约束的答案

最近一次修改见于版次 2026.11.2

在线示例涵盖六条消息。使用原始提示词的 135M 模型答对了 1/6 (16.7%),360M 模型答对了 0/6 (0.0%)。六条消息无法为这些模型排出高下:这两个计数之间的全部差别只是一个正确答案。135M 模型最常给出的答案是 card_arrival, 4/6,而在同样六条消息上,聊天形式得到的结果是 0/6 (0.0%)。

360M 的运行使用同一个程序,只是换成了更大的模型。

constrained-decode-360m.py Python 3.12.15 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0 通过
"""Decode team names for the first 6 messages of test_40 with the 360M model, and count how many of them are right.The prose sets its accuracy line against the 135M model's, on the same six messages. The messages are read from the test split and are never printed."""import os# no network: the pinned model must already be in the local cacheos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import collections  # noqa: E402import torch  # noqa: E402# two threads, so the sums add in the same order on every runtorch.set_num_threads(2)import router_lib  # noqa: E402# seeds 0 for random and torch, deterministic algorithms and float32, so greedy decoding repeatsrouter_lib.setup_runtime()MEDIUM = "HuggingFaceTB/SmolLM2-360M-Instruct"tokenizer, model = router_lib.load_model(MEDIUM)trie = router_lib.build_trie(tokenizer)# (text, label index) for every test row, in file order; the index is the gold teamrows = router_lib.read_rows("test")# row numbers of the first six messages of the fixed 40-message samplesample = router_lib.read_sample("test_40", 6)teams = set(router_lib.LABELS)# (label index, team text); the index comes from the trie walk, not from parsing the textresults = [router_lib.constrained_team(tokenizer, model, trie, rows[i][0]) for i in sample]valid = [text in teams for _, text in results]# the walk's index, never the decoded text, is what countsright = [index == rows[i][1] for (index, _), i in zip(results, sample)]given = collections.Counter(text for _, text in results)# most given team; a tie in count goes to the team listed firsttop = min(given, key=lambda team: (-given[team], router_lib.LABELS.index(team)))print(f"messages: {len(sample)}")print(f"answers that are a team: {sum(valid)}/{len(sample)}")print(f"accuracy: {sum(right)}/{len(sample)} ({100 * sum(right) / len(sample):.1f}%)")print(f"distinct teams given: {len(given)}")print(f"most given team: {top}, {given[top]}/{len(sample)}")
输出python constrained-decode-360m.py
messages: 6
answers that are a team: 6/6
accuracy: 0/6 (0.0%)
distinct teams given: 4
most given team: card_payment_wrong_exchange_rate, 3/6

为了进行比较,每个变体都回答了 200 条消息。表中是录制结果中的计数,每一行使用的消息相同:

test_200 上的变体 团队列表之外的答案 正确 给出的不同团队数
135M,原始提示词 0/200 9/200 (4.5%) 18
135M,聊天模板 0/200 11/200 (5.5%) 17
360M,原始提示词 0/200 12/200 (6.0%) 8

本示例从录制结果中读取表中的计数:

prompting-on-test-200.py Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0 通过
"""Compare the three prompting variants on test_200, from the recording only. No model is loaded.The chapter's table takes its three columns from the first three lines printed for each variant, and the point rests on the correct count.recordings/prompting_eval.py wrote one answer per variant and message to examples/data/recorded/prompting-eval/answers.jsonl."""import collectionsimport jsonfrom pathlib import Pathimport torch# two threads, as in the other examplestorch.set_num_threads(2)import router_lib  # noqa: E402# seeds 0 for random and torch, deterministic algorithms and float32, as in the other examplesrouter_lib.setup_runtime()ANSWERS = Path(__file__).resolve().parent / "data" / "recorded" / "prompting-eval" / "answers.jsonl"teams = set(router_lib.LABELS)# (text, label index) for every test row, in file order; the index is the gold teamrows = router_lib.read_rows("test")# the 200 messages of the fixed sample; every variant answers all of themtotal = len(router_lib.read_sample("test_200"))by_variant = collections.defaultdict(list)with open(ANSWERS) as f:    for line in f:        answer = json.loads(line)        by_variant[answer["variant"]].append(answer)for variant in sorted(by_variant):    answers = by_variant[variant]    outside = sum(answer["team"] not in teams for answer in answers)    # answer["label"] is the team index the variant chose; rows[...][1] is the message's gold index    right = sum(answer["label"] == rows[answer["index"]][1] for answer in answers)    given = collections.Counter(answer["team"] for answer in answers)    # most given team; a tie in count goes to the team name that sorts first, not to the index    top = min(given, key=lambda team: (-given[team], team))    # the gold index becomes a team name, to compare with the most given team    gold_top = sum(router_lib.LABELS[rows[answer["index"]][1]] == top for answer in answers)    print(f"{variant} answers outside the team list: {outside}/{total}")    print(f"{variant} correct: {right}/{total} ({100 * right / total:.1f}%)")    print(f"{variant} distinct teams given: {len(given)}")    print(f"{variant} most given team: {top}, {given[top]}/{total}")    print(f"{variant} gold for most given team: {gold_top}/{total}")
输出python prompting-on-test-200.py
135m-chat answers outside the team list: 0/200
135m-chat correct: 11/200 (5.5%)
135m-chat distinct teams given: 17
135m-chat most given team: topping_up_by_card, 55/200
135m-chat gold for most given team: 1/200
135m-raw answers outside the team list: 0/200
135m-raw correct: 9/200 (4.5%)
135m-raw distinct teams given: 18
135m-raw most given team: card_arrival, 165/200
135m-raw gold for most given team: 0/200
360m-raw answers outside the team list: 0/200
360m-raw correct: 12/200 (6.0%)
360m-raw distinct teams given: 8
360m-raw most given team: card_payment_wrong_exchange_rate, 94/200
360m-raw gold for most given team: 4/200

没有哪一行接近一个合格的路由器,不过约束在每一行都成立:任何一行都没有列表之外的答案。360M 原始提示词一行的正确答案最多,为 12/200 (6.0%),而且没有哪一行领先另一行超过几条消息。

仅看不同团队数会产生误导。135M 原始提示词一行给出 18 个不同团队,360M 原始提示词一行则为 8 个,但 135M 原始提示词一行的大多数答案都给了同一个团队 card_arrival, 165/200,而 360M 原始提示词一行中被给出最多的团队只得到 card_payment_wrong_exchange_rate, 94/200。

阅读一行时,先看第一列。如果它不为零,说明约束没有成立,在解决这个问题之前,其余两列都没有意义。第二列对路由而言才是关键:它统计所给团队就是标准标签的消息数,分母是整个样本,错误答案按错误计。第三列说明该变体到底用到了多少个团队。只给出少数几个团队的变体,要么是在猜几个常见团队,要么是学到了某种狭窄的规律,而这个计数无法分辨是哪一种。数量很大也不能排除答案集中,135M 原始提示词一行就是如此。读者在自己的消息样本上运行同样的三个计数,就能在自己的数据上得到一个比较,而且每一列的含义保持不变。

360M 模型是两者中较大的一个,第 3 章已经统计过它大了多少。表中显示了在这个提示词和这个约束下,这笔成本换来了什么:答对 12/200 (6.0%),而 135M 模型在同样的原始形式下答对 9/200 (4.5%)。较大的模型多答对了几条消息,本章不认为这是为额外规模付费的理由。

本书没有为这些计数计算区间,而 200 条中相差几条消息的差距可能来自样本本身,所以这张表只支持一个结论:三个变体很接近,而且没有一个是合格的路由器。想在两个提示词之间做选择的读者,应当在自己的消息样本上运行两者,并统计同样的三项。

字典树只检查答案的形状,别的什么都不检查。它不会读取提示词之外的消息,也不知道哪个团队才是对的。在团队名称仍有可能被写出的每一步,列表中的任何团队都是允许的,因此解码器在其中的选择就是模型的选择,而掩码只决定哪些团队可以被写出。

下面是一个有效但错误的答案,只用团队来描述,因为本书从不引用消息原文。一条标准团队为 card_not_working 的测试消息,被 135M 原始提示词一行答成了 card_arrival,这是列表中的团队,所以约束放行了它,列表之外的答案数仍然为零。

这些输出支持的结论是:受约束解码使每个答案都是一个团队,但有效的答案不等于正确的答案。约束只会去掉不是团队的答案,它不会选出更好的团队,提示词的改动也没有做到这一点。第 5 章是下一步:它在带标签的消息上训练一个头部,并在同一样本上衡量训练后的模型是否路由得更好。

在生产环境中,这种失败不会表现为错误。不是团队的答案的计数保持为零,因此只监视该计数的检查会一直显示正常。只有在固定的带标签样本上统计正确答案的数量,才能发现问题。

代价落在接收消息的人身上。一个有效但错误的团队会把消息送到错误的去处,支持人员在那里看到一个不属于自己的请求,必须找出正确的团队并转交,这与一次移交所产生的工作相同,而且消息上没有任何迹象表明它被错误路由。一个无法说明自己何时不确定的路由器,不会给这些人任何可据以行动的依据。受约束解码也不会为其余 76 个团队返回分数,因此建立在它之上的移交规则将没有可供比较的各团队分数。

**工作准则:**对每个参与比较的变体,在同一个固定样本上,报告正确答案的数量,并与有效答案的数量并列。一个答案全部有效的路由器,仍可能在大多数消息上出错,只有第二个计数才能显示这一点。

为什么先验不是路由器

最近一次修改见于版次 2026.11.9

约束解码只返回一个答案,不给出分数。标签打分则为每个团队给出一个分数。对每个团队,标签打分把该团队名称的各个词元在提示之后的对数概率相加,其中对数概率是模型赋予某个词元的概率的对数。总和最高的团队,就是模型对这条消息给出的首选团队。

在三条消息上运行它,并读出每条消息的首选团队:

label-scoring-prior.py Python 3.12.15 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0 通过
"""Score every team for the first 3 messages of test_40 with the 135M model, by the summed log-probability of its tokens.This shows the model's prior, the team it favours before any fine-tuning. The point rests on the most frequent top team and how many messages it tops. The messages are never printed."""import os# no network: the pinned model must already be in the local cacheos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import collections  # noqa: E402import torch  # noqa: E402# two threads, so the sums add in the same order on every runtorch.set_num_threads(2)import router_lib  # noqa: E402# seeds 0 for random and torch, deterministic algorithms and float32, so greedy decoding repeatsrouter_lib.setup_runtime()SMALL = "HuggingFaceTB/SmolLM2-135M-Instruct"tokenizer, model = router_lib.load_model(SMALL)rows = router_lib.read_rows("test")# row numbers of the first three messages of the fixed 40-message samplesample = router_lib.read_sample("test_40", 3)# each team's token ids, padded to the longest name, built once for all messagestable = router_lib.label_token_table(tokenizer)tops = []for i in sample:    scores = router_lib.label_scores(tokenizer, model, table, rows[i][0])    # index of the highest-scoring team; a tie goes to the lower index    tops.append(max(range(len(scores)), key=lambda k: (scores[k], -k)))counts = collections.Counter(tops)# most frequent top team; a tie in count goes to the lower indexbest = min(counts, key=lambda k: (-counts[k], k))print(f"messages: {len(sample)}")print(f"distinct top teams: {len(counts)}")print(f"most frequent top team name: {router_lib.LABELS[best]}")print(f"most frequent top team: {counts[best]} of {len(sample)}")
输出python label-scoring-prior.py
messages: 3
distinct top teams: 1
most frequent top team name: card_arrival
most frequent top team: 3 of 3

在任何训练之前,模型对每条消息都把同一个团队排在第一位:card_arrival 在 3 of 3 条消息中居首。该团队就是模型的先验,而三条消息太少,无法就此多说什么。

如果每条消息的首选团队都相同,那么这个分数就是模型的先验,对任何一条具体消息说明的东西很少。200 条消息的记录在约束解码中显示了同样的先验:card_arrival, 165/200 是 135M 原始模型那一行最偏爱的团队,它是样本中 0/200 条消息的标注(gold)团队。

这种求和偏向短名称。团队名称的每个词元都加上一个对数概率,而对数概率从不大于零,所以词元更多的团队有更多的项把总和拉低。该示例直接把各项相加,没有除以长度,本章也不声称对此做了校正。使用标签打分的读者,应当先用带有标准标签的样本检查首选团队,再决定是否信任它们。

标签打分确实为每个团队给出了一个分数,而这正是交接规则所需要的。每条消息都要与全部 77 个团队名称打分,代码在提示运行一次之后,以一个批次完成打分。第 5 章通过一个在带标签消息上训练的头,经单次前向传播得到每个团队的分数。标签打分是从本章通向那一章的桥梁:它表明模型本来就有每个团队的分数,而在训练之前,这些分数对消息说明的东西很少。

工作准则: 用标签打分来观察模型在训练之前偏爱哪个团队,不要用它来路由。在这项任务上,不要用这种规模的提示式模型来路由消息;200 条消息的记录就是证据。应当改为在带标签的消息上训练一个头,正如第 5 章所做的那样。

本章的决定

最近一次修改见于版次 2026.11.2

本章的规则仅适用于这两个模型、这个提示词和这项约束,并且仅以录制的 200 条消息为样本。更大的模型、不同的提示词或更长的样本可能改变计数,而本书没有在该样本上测试过其中任何一项。本章确实能够确立的结论范围更窄:在使用原始提示词和字典树的情况下,这种规模的模型会返回有效的团队,但正确的很少,而正确答案的数量是需要关注的数字。

每项决定都在各自的节中检验:提示词在第一节,字典树在相应的一节,贪婪解码在相应的一节,两个计数则在测量一节。正如第一节所说,读者若改动其中任何一项,在比较之前都必须重新录制。约束能使每个答案都有效,而只有训练才能使答案正确。

本章做出了以下决定:

  • 路由器提示词采用原始形式,参与比较的两个模型都使用它。
  • 约束解码使用覆盖 77 个团队名称的词元字典树,采用贪婪解码并固定随机种子。
  • 每个变体都报告其有效答案的数量和正确答案的数量,并且使用同一个固定样本。
  • 实时示例展示各个部分如何工作,只有 200 条消息的录制样本用于比较模型。

本章,在本版次中

版次 2026.11.9示例于 2026年11月20日 通过本章 5个示例 个示例中有 5 个通过

译自英文版图书,版次 2026.11.9。

引用本节

格式

每种格式均引用英文原书,本译本以其为准。

每种格式都会注明该版次。其链接会打开当前版次,节标题处会说明该节文本最后一次修改的时间。访问日期和检索日期即为您打开此页的当天。

本页内容

第 4 章,共 10 章: Prompt a small model and constrain its answers to the teams

  1. 在任何录制之前确定提示词
  2. 自由文本为何会失败
  3. A token trie allows only team names
  4. 贪心解码,使每个计数都可重复
  5. 衡量受约束的答案
  6. 为什么先验不是路由器
  7. 本章的决定

返回顶部全部章节