Prompt a small model and constrain its answers to the teams
About 18 minutes to read, with 5 examples in 7 sections.
Written with AI. Verified by people. How we make books
This book is open to contributions, so anyone can check it or correct it. Its repository · Report an error · How to fix one
Labs
Learn this chapter by doing it
Run every example exactly as the book did, change the code to see what happens, and try your own ideas on the book's code and data. Nothing to install, and your work is saved. Opens in about 30 seconds.
Run every example exactly as the book did, change the code to see what happens, and try your own ideas on the book's code and data. Nothing to install, and your work is saved. Labs runs it for you: 40 hours a month.
Your browser can translate this page from its menu, and the code stays as the book has it.
Or read it with your AI assistant:
By the end of this chapter, you can write a router prompt for a small open model, constrain its answer so that it can only be a team name, and measure how many of those answers are right. You will also see why a valid answer is not a correct one, which is why the next chapter trains a model instead of asking it better questions.
A router needs every answer to be a team name, and when chapter 1 let a small model answer in free text, the model did not return teams. This chapter first makes the answer a team by construction. It then runs the two open models from chapter 3, SmolLM2-135M-Instruct and SmolLM2-360M-Instruct, with one prompt and one constraint, and measures what they return. The 135M model is the one chapter 5 trains, so the counts here are the ones that training has to beat.
Two kinds of number appear, and the chapter keeps them apart. The live examples run on the first six messages of test_40, the first 40 messages of the fixed sample, and the same six for both models. They show the mechanism. Six messages cannot separate two models, so the comparison uses a recording: each model and prompt answered all 200 messages of test_200, the first 200 of that sample, which includes those six, once, and the answers were saved. Reading the recording loads no model, and its counts come from the saved answers and nothing else.
The prompt, settled before any recording
Every example in this chapter uses one router prompt. It tells the model that the message comes from an online bank, asks for one support team and the team name only, lists the 77 teams, and ends with Team:, so the model's next words are the answer. A raw model that has not been trained for the task gets nothing else, so the wording matters. This chapter does not tune it. The prompt was fixed before any recording, and the recording's manifest holds the prompt's hash. A changed prompt is a different recording, and its counts cannot be set beside the ones here.
The list of teams in the prompt is the same list the constraint uses. The trie in the next section is built from the same label names in the same order, so the prompt and the constraint cannot disagree about which teams exist. The same dependency works the other way. A reader who adds or renames a team changes the prompt, and so changes the recording's hash. The counts in this chapter then no longer describe that prompt, and the reader must record again before comparing.
The raw form passes the router prompt's text to the model as it stands. The chat form wraps the router prompt's text in the model's chat template, which adds a default system prompt. To see what the template changes, run the chat form on the same six messages:
1"""Route the first 6 messages of test_40 with the 135M model through its chat template, decoding only team names.2The template adds the model's default system prompt. The prose quotes this run's accuracy line next to the raw form's, and the messages are never printed."""34import os56# no network: the pinned model must already be in the local cache7os.environ["HF_HUB_OFFLINE"] = "1"8os.environ["TRANSFORMERS_OFFLINE"] = "1"910import collections # noqa: E4021112import torch # noqa: E4021314# two threads, so the sums add in the same order on every run15torch.set_num_threads(2)1617import router_lib # noqa: E4021819# seeds 0 for random and torch, deterministic algorithms and float32, so greedy decoding repeats20router_lib.setup_runtime()2122SMALL = "HuggingFaceTB/SmolLM2-135M-Instruct"2324tokenizer, model = router_lib.load_model(SMALL)25trie = router_lib.build_trie(tokenizer)26# (text, label index) for every test row, in file order; the index is the gold team27rows = router_lib.read_rows("test")28# row numbers of the first six messages of the fixed 40-message sample29sample = router_lib.read_sample("test_40", 6)30teams = set(router_lib.LABELS)3132results = []33for i in sample:34 # the chat template wraps the message in the model's chat format, with its default system prompt35 text = router_lib.chat_prompt(tokenizer, rows[i][0])36 # the template has already written the special tokens, so the tokenizer adds none37 results.append(router_lib.constrained_team_text(tokenizer, model, trie, text, add_special_tokens=False))38valid = [text in teams for _, text in results]39# the walk's index, never the decoded text, is what counts40right = [index == rows[i][1] for (index, _), i in zip(results, sample)]41given = collections.Counter(text for _, text in results)42# most given team; a tie in count goes to the team listed first43top = min(given, key=lambda team: (-given[team], router_lib.LABELS.index(team)))4445print(f"messages: {len(sample)}")46print(f"answers that are a team: {sum(valid)}/{len(sample)}")47print(f"accuracy: {sum(right)}/{len(sample)} ({100 * sum(right) / len(sample):.1f}%)")48print(f"distinct teams given: {len(given)}")49print(f"most given team: {top}, {given[top]}/{len(sample)}")
python constrained-decode-135m-chat.pymessages: 6
answers that are a team: 6/6
accuracy: 0/6 (0.0%)
distinct teams given: 3
most given team: cancel_transfer, 3/6
The book uses the raw form as its default. The choice was made before any recording was read: both models in the comparison are measured on it, so the comparison is between models, not between prompts. The chat form was recorded afterwards, as an observation. On the 200 messages it got 11/200 (5.5%) right, and the raw form got 9/200 (4.5%). A gap of two messages in 200 is not a difference this sample can show in either direction, so the chat form is not shown to help, and the raw form is not shown to hurt, and the default is not changed because of this count. The trie's names begin with a space, which follows Team: in the raw form but not the new line that opens the answer in the chat template. The chat count is therefore for this trie as well as for the template. The result is not evidence that raw prompts beat chat templates for small models in general, and one sample of 200 messages does not settle that.
Working rule: Use one prompt for every model you compare, in the form the comparison needs. Before you adopt a chat template, measure it on the same sample as the raw form, and report a small gap as a small gap.
Why free text fails
Chapter 1 asked the raw model for one team with no constraint, and the model decoded freely: greedy, taking the highest-scoring token at each step, as in this chapter, but with no mask on the tokens it could pick. Its example counted the answers. 0/20 were team names, so the router had nothing to send on.
Nothing in the model's own decoding stops an answer from being a sentence, a team that is not on the list, or a near-miss spelling of one. At each step the decoder may pick any token in the vocabulary, so the answer can go anywhere. Chapter 1's stand-in did not show this failure, because its recorded answers came from a grammar that allowed only the team names. A model that runs in the router needs the same guarantee from the decoder itself.
The tempting repair is to take whatever the model wrote and match it to the nearest team name. That repair is a guess. A near-miss spelling may match the right team, or a wrong team that is close in letters, and the router would then send the message to a team the model never named. Chapter 1 said what to do instead: an answer that is not a team counts as an error, is counted on the sample, and the message goes to a person. A constraint removes most of those answers before they are written. The count is still kept, so that a constraint which fails to hold shows up as a number and not as a quiet change in routing.
Working rule: Treat an answer that is not a team as a failed route. Count such answers on a fixed sample before a model routes anything, and do not match a near-miss to the closest team. Constrain the answer at decode time, and keep the count, so that a broken constraint shows up as a number.
A token trie allows only team names
A model writes tokens, not words, and a team name is a short run of them. build_trie in router_lib, the book's shared module, stores the 77 names as a trie: a tree whose paths spell out each team's tokens, with the team's index at the node where its path ends. The function asserts that no two teams end at the same node, so each path names one team.
The decoder walks this tree as it writes. At the start, the allowed tokens are the first token of each team name. After each token, the allowed set narrows to the children of the node the decoder has reached. Where a path has ended on a team, the end-of-answer token joins the allowed set, so the answer can stop there. No one of the 77 names is the start of another, so every path ends at a leaf. The code would still handle a name that was: at that node the decoder could stop, or continue along the longer name. A walk of this kind cannot leave the list, because every token it may choose leads toward a team.
Greedy decoding then takes the highest-scoring of the allowed tokens at each step. Transformers applies the rule through a callback, prefix_allowed_tokens_fn, which returns the allowed tokens for each position. The processor starts from a mask of minus infinity for every token, sets the allowed tokens' entries to zero, and adds the mask to the scores with scores_processed = scores + mask. Every token that is not allowed keeps minus infinity, so greedy decoding cannot pick it (transformers 5.23.0, src/transformers/generation/logits_process.py).
Chapter 1's stand-in got the same guarantee from its library. recordings/common.py builds a grammar, root ::= "card_arrival" | ..., that lists the 77 names as literal strings, and the recordings applied it through the chat template. llama-cpp-python adds a grammar to its sampler before the greedy step. In llama_cpp/llama.py the line sampler.add_grammar(self._model, grammar) runs under if grammar is not None:, and sampler.add_greedy() runs after it, under elif temp == 0.0: (llama-cpp-python v0.3.36, llama_cpp/llama.py). So the stand-in's answers were limited to the names by the library, where this chapter's callback does the same job in the decoder.
Two details of that code shape the callback. The processor calls it as self._prefix_allowed_tokens_fn(batch_id, sent), so the batch index comes first. And an empty list is an error: the processor raises ValueError when the callback returns no tokens. The book's callback therefore never returns an empty list. When no path can continue, it returns the end-of-answer token, and the answer stops there. The end-of-answer token is the model's <|im_end|>, which is also its pad token (id 2). The decoder stops after 32 new tokens (max_new_tokens=32). Every one of the 600 recorded answers ended on a team. A node with no children always holds a team, so the empty-list case is a guard and not a case the 77 names produce.
First, the 135M model with the trie, on those six messages.
1"""Route the first 6 messages of test_40 with the 135M model, decoding only team names, and count how many right answers that gives.2The point rests on the count of answers that are team names, which the trie guarantees, and on the accuracy line, which the measurement section reads. The messages are read from the test split and are never printed."""34import os56# no network: the pinned model must already be in the local cache7os.environ["HF_HUB_OFFLINE"] = "1"8os.environ["TRANSFORMERS_OFFLINE"] = "1"910import collections # noqa: E4021112import torch # noqa: E4021314# two threads, so the sums add in the same order on every run15torch.set_num_threads(2)1617import router_lib # noqa: E4021819# seeds 0 for random and torch, deterministic algorithms and float32, so greedy decoding repeats20router_lib.setup_runtime()2122SMALL = "HuggingFaceTB/SmolLM2-135M-Instruct"2324tokenizer, model = router_lib.load_model(SMALL)25trie = router_lib.build_trie(tokenizer)26# (text, label index) for every test row, in file order; the index is the gold team27rows = router_lib.read_rows("test")28# row numbers of the first six messages of the fixed 40-message sample29sample = router_lib.read_sample("test_40", 6)30teams = set(router_lib.LABELS)3132# (label index, team text); the index comes from the trie walk, not from parsing the text33results = [router_lib.constrained_team(tokenizer, model, trie, rows[i][0]) for i in sample]34valid = [text in teams for _, text in results]35# the walk's index, never the decoded text, is what counts36right = [index == rows[i][1] for (index, _), i in zip(results, sample)]37given = collections.Counter(text for _, text in results)38# most given team; a tie in count goes to the team listed first39top = min(given, key=lambda team: (-given[team], router_lib.LABELS.index(team)))4041print(f"messages: {len(sample)}")42print(f"answers that are a team: {sum(valid)}/{len(sample)}")43print(f"accuracy: {sum(right)}/{len(sample)} ({100 * sum(right) / len(sample):.1f}%)")44print(f"distinct teams given: {len(given)}")45print(f"most given team: {top}, {given[top]}/{len(sample)}")
python constrained-decode-135m.pymessages: 6
answers that are a team: 6/6
accuracy: 1/6 (16.7%)
distinct teams given: 3
most given team: card_arrival, 4/6
Run on 6 messages, the model's answers are team names in every case, 6/6, which is the guarantee the trie gives. The accuracy on the same messages is 1/6 (16.7%). The measurement section takes that second number apart.
The router does not read the team out of the text. constrained_team returns the index stored at the node where the walk ended, and the text is decoded only so that a reader can see it. The index is fixed by the trie and not by a parser that runs after the model. That is the reason the constraint is worth the code: a parser that matches text to teams can fail in ways the trie cannot. These examples return the team only so that it can be counted. Nothing in this chapter routes a message. A route returns a team only through handoff, which sends an uncertain answer to a person, and the examples here do not call it.
Working rule: Constrain the answer to the labels with a token trie built from the label names, and count the answers that are labels on a fixed sample. The constraint makes every answer a team. It does not make any answer right, so count the right answers as a separate measure.
Decode greedily so every count repeats
Every example here uses greedy decoding: the highest score wins, and nothing is sampled. That matters for the counts. A sampled answer would differ from one run to the next, and then no printed count could be checked against a recording. The examples fix their seeds and float32 through router_lib.setup_runtime and set two threads, so on one machine a run repeats its answers. Each example calls torch.set_num_threads(2) itself, at the top, so that it does not depend on how its runner was started. Each example's printed output is compared with its recorded output on every run, and a changed line fails the check.
Repeatable on one machine is not the same as repeatable everywhere. The PyTorch 2.15.0+cpu notes on randomness say that completely reproducible results are not guaranteed across releases, individual commits or platforms. So the book claims repeatability for the versions pinned in examples/requirements.txt, on the kind of machine the book ran on. A reader on another platform should expect to check the recorded lines again. If a recorded line fails there, read the changed line before touching the example. A different count of right answers can come from a different machine, a different library version or a changed prompt, and each of those needs a different fix.
Ties need a fixed rule as well. Label scoring in the last section breaks a tie in score by the lower team index, through its sort key (score, -index), and the most-given-team count breaks a tie in count by team name. Constrained decoding has no team scores. Its greedy step takes the first maximum among the allowed tokens, so a tie there goes to the lower token id.
Working rule: Use greedy decoding, fix the seeds and the thread count, and compare each printed line with a recorded copy on every run. Do not describe a result as reproducible beyond the versions and the kind of machine it was run on.
Measure the constrained answers
The live examples cover six messages. The 135M model with the raw prompt got 1/6 (16.7%) right, and the 360M model got 0/6 (0.0%). Six messages cannot rank these models: one right answer is the whole difference between those two counts. The 135M model's most common answer was card_arrival, 4/6, and the chat form on the same six messages got 0/6 (0.0%).
The 360M run is the same program with the larger model.
1"""Decode team names for the first 6 messages of test_40 with the 360M model, and count how many of them are right.2The prose sets its accuracy line against the 135M model's, on the same six messages. The messages are read from the test split and are never printed."""34import os56# no network: the pinned model must already be in the local cache7os.environ["HF_HUB_OFFLINE"] = "1"8os.environ["TRANSFORMERS_OFFLINE"] = "1"910import collections # noqa: E4021112import torch # noqa: E4021314# two threads, so the sums add in the same order on every run15torch.set_num_threads(2)1617import router_lib # noqa: E4021819# seeds 0 for random and torch, deterministic algorithms and float32, so greedy decoding repeats20router_lib.setup_runtime()2122MEDIUM = "HuggingFaceTB/SmolLM2-360M-Instruct"2324tokenizer, model = router_lib.load_model(MEDIUM)25trie = router_lib.build_trie(tokenizer)26# (text, label index) for every test row, in file order; the index is the gold team27rows = router_lib.read_rows("test")28# row numbers of the first six messages of the fixed 40-message sample29sample = router_lib.read_sample("test_40", 6)30teams = set(router_lib.LABELS)3132# (label index, team text); the index comes from the trie walk, not from parsing the text33results = [router_lib.constrained_team(tokenizer, model, trie, rows[i][0]) for i in sample]34valid = [text in teams for _, text in results]35# the walk's index, never the decoded text, is what counts36right = [index == rows[i][1] for (index, _), i in zip(results, sample)]37given = collections.Counter(text for _, text in results)38# most given team; a tie in count goes to the team listed first39top = min(given, key=lambda team: (-given[team], router_lib.LABELS.index(team)))4041print(f"messages: {len(sample)}")42print(f"answers that are a team: {sum(valid)}/{len(sample)}")43print(f"accuracy: {sum(right)}/{len(sample)} ({100 * sum(right) / len(sample):.1f}%)")44print(f"distinct teams given: {len(given)}")45print(f"most given team: {top}, {given[top]}/{len(sample)}")
python constrained-decode-360m.pymessages: 6
answers that are a team: 6/6
accuracy: 0/6 (0.0%)
distinct teams given: 4
most given team: card_payment_wrong_exchange_rate, 3/6
For the comparison, each variant answered 200 messages. The recording's counts are in the table, with the same messages in every row:
| Variant on test_200 | Answers outside the team list | Correct | Distinct teams given |
|---|---|---|---|
| 135M, raw prompt | 0/200 | 9/200 (4.5%) | 18 |
| 135M, chat template | 0/200 | 11/200 (5.5%) | 17 |
| 360M, raw prompt | 0/200 | 12/200 (6.0%) | 8 |
This example reads the table's counts from the recording:
1"""Compare the three prompting variants on test_200, from the recording only. No model is loaded.2The chapter's table takes its three columns from the first three lines printed for each variant, and the point rests on the correct count.34recordings/prompting_eval.py wrote one answer per variant and message to examples/data/recorded/prompting-eval/answers.jsonl.5"""67import collections8import json9from pathlib import Path1011import torch1213# two threads, as in the other examples14torch.set_num_threads(2)1516import router_lib # noqa: E4021718# seeds 0 for random and torch, deterministic algorithms and float32, as in the other examples19router_lib.setup_runtime()2021ANSWERS = Path(__file__).resolve().parent / "data" / "recorded" / "prompting-eval" / "answers.jsonl"22teams = set(router_lib.LABELS)23# (text, label index) for every test row, in file order; the index is the gold team24rows = router_lib.read_rows("test")25# the 200 messages of the fixed sample; every variant answers all of them26total = len(router_lib.read_sample("test_200"))2728by_variant = collections.defaultdict(list)29with open(ANSWERS) as f:30 for line in f:31 answer = json.loads(line)32 by_variant[answer["variant"]].append(answer)3334for variant in sorted(by_variant):35 answers = by_variant[variant]36 outside = sum(answer["team"] not in teams for answer in answers)37 # answer["label"] is the team index the variant chose; rows[...][1] is the message's gold index38 right = sum(answer["label"] == rows[answer["index"]][1] for answer in answers)39 given = collections.Counter(answer["team"] for answer in answers)40 # most given team; a tie in count goes to the team name that sorts first, not to the index41 top = min(given, key=lambda team: (-given[team], team))42 # the gold index becomes a team name, to compare with the most given team43 gold_top = sum(router_lib.LABELS[rows[answer["index"]][1]] == top for answer in answers)44 print(f"{variant} answers outside the team list: {outside}/{total}")45 print(f"{variant} correct: {right}/{total} ({100 * right / total:.1f}%)")46 print(f"{variant} distinct teams given: {len(given)}")47 print(f"{variant} most given team: {top}, {given[top]}/{total}")48 print(f"{variant} gold for most given team: {gold_top}/{total}")
python prompting-on-test-200.py135m-chat answers outside the team list: 0/200
135m-chat correct: 11/200 (5.5%)
135m-chat distinct teams given: 17
135m-chat most given team: topping_up_by_card, 55/200
135m-chat gold for most given team: 1/200
135m-raw answers outside the team list: 0/200
135m-raw correct: 9/200 (4.5%)
135m-raw distinct teams given: 18
135m-raw most given team: card_arrival, 165/200
135m-raw gold for most given team: 0/200
360m-raw answers outside the team list: 0/200
360m-raw correct: 12/200 (6.0%)
360m-raw distinct teams given: 8
360m-raw most given team: card_payment_wrong_exchange_rate, 94/200
360m-raw gold for most given team: 4/200
No row is close to a router, though the constraint holds in every one: no answer in any row is outside the team list. The 360M raw row has the most right answers, at 12/200 (6.0%), and no row beats another by more than a few messages.
Distinct teams alone mislead. The 135M raw row gives 18 distinct teams, against 8 for the 360M raw row, but most of the 135M row's answers go to one team, card_arrival, 165/200, while the 360M raw row's most given team gets only card_payment_wrong_exchange_rate, 94/200.
To read a row, start with the first column. If it is not zero, the constraint has not held, and the other two columns mean nothing until that is fixed. The second column is the one that matters for routing: it counts the messages whose team is the gold label, out of the whole sample, with the wrong answers counted as wrong. The third column says how many teams the variant uses at all. A variant that gives only a few teams is either guessing a few popular teams or has learned something narrow, and the count does not say which. A large count does not rule out concentration either, as the 135M raw row shows. A reader who runs the same three counts on a sample of their own messages gets a comparison on their own data, with the same meaning in each column.
The 360M model is the larger of the two, and chapter 3 counted how much larger it is. The table shows what that cost buys on this prompt and this constraint: 12/200 (6.0%) right, against 9/200 (4.5%) for the 135M model on the same raw form. That is a few more messages for a larger model, and the chapter does not count it as a reason to pay for the extra size.
The book computes no interval for these counts, and a gap of a few messages out of 200 could come from the sample, so the table supports one statement: the three variants are close, and none of them is a router. A reader who wants to choose between two prompts should run both on a sample of their own messages and count the same three things.
The trie checks the shape of the answer and nothing else. It does not read the message beyond the prompt, and it does not know which team is right. Any team that is on the list is allowed at every step where its name could still be written, so the decoder's choice among them is the model's choice, and the mask only decides which teams can be written at all.
Here is one valid wrong answer, described by its teams only, since messages are never quoted in this book. One test message whose gold team is card_not_working got card_arrival from the 135M raw row, a team on the list, so the constraint let it through and the count of answers outside the list stays at zero.
The lesson the outputs support is that constrained decoding makes every answer a team, but a valid answer is not a right one. The constraint only removes answers that are not teams. It does not choose a better team, and the prompt change did not either. Chapter 5 is the next step: it trains a head on the labelled messages and measures whether the trained model routes better on the same sample.
In production the failure does not show up as an error. The count of answers that are not teams stays at zero, so a check that watches only that count stays green. Only a count of right answers on a fixed, labelled sample shows the problem.
The cost falls on the people who receive the message. A valid wrong team sends the message to the wrong destination, where support staff see a request that is not theirs, and they must find the right team and move it, which is the same work a hand-off creates, with no sign on the message that it was misrouted. A router that cannot say when it is unsure gives those people nothing to act on. Constrained decoding also returns no score for the other 76 teams, so a hand-off rule built on it would have no per-team scores to compare.
Working rule: Report the count of right answers beside the count of valid answers, on one fixed sample, for every variant you compare. A router whose answers are all valid can still be wrong on most messages, and only the second count shows it.
Why a prior is not a router
Constrained decoding returns one answer and no scores. Label scoring gives every team a score. For each team, label scoring sums the log-probability of the team's tokens after the prompt, where a log-probability is the logarithm of the probability the model gives a token. The team with the highest sum is the model's top team for the message.
Run it on three messages and read the top team of each:
1"""Score every team for the first 3 messages of test_40 with the 135M model, by the summed log-probability of its tokens.2This shows the model's prior, the team it favours before any fine-tuning. The point rests on the most frequent top team and how many messages it tops. The messages are never printed."""34import os56# no network: the pinned model must already be in the local cache7os.environ["HF_HUB_OFFLINE"] = "1"8os.environ["TRANSFORMERS_OFFLINE"] = "1"910import collections # noqa: E4021112import torch # noqa: E4021314# two threads, so the sums add in the same order on every run15torch.set_num_threads(2)1617import router_lib # noqa: E4021819# seeds 0 for random and torch, deterministic algorithms and float32, so greedy decoding repeats20router_lib.setup_runtime()2122SMALL = "HuggingFaceTB/SmolLM2-135M-Instruct"2324tokenizer, model = router_lib.load_model(SMALL)25rows = router_lib.read_rows("test")26# row numbers of the first three messages of the fixed 40-message sample27sample = router_lib.read_sample("test_40", 3)28# each team's token ids, padded to the longest name, built once for all messages29table = router_lib.label_token_table(tokenizer)3031tops = []32for i in sample:33 scores = router_lib.label_scores(tokenizer, model, table, rows[i][0])34 # index of the highest-scoring team; a tie goes to the lower index35 tops.append(max(range(len(scores)), key=lambda k: (scores[k], -k)))3637counts = collections.Counter(tops)38# most frequent top team; a tie in count goes to the lower index39best = min(counts, key=lambda k: (-counts[k], k))4041print(f"messages: {len(sample)}")42print(f"distinct top teams: {len(counts)}")43print(f"most frequent top team name: {router_lib.LABELS[best]}")44print(f"most frequent top team: {counts[best]} of {len(sample)}")
python label-scoring-prior.pymessages: 3
distinct top teams: 1
most frequent top team name: card_arrival
most frequent top team: 3 of 3
Before any training, the model puts one team first for every message: card_arrival tops 3 of 3 messages. That team is the model's prior, and three messages are too few to say more about it.
If the top team is the same for every message, the score is the model's prior and says little about any one message. The 200-message recording shows the same prior in constrained decoding: card_arrival, 165/200, the 135M raw row's favourite, is the gold team of 0/200 messages in the sample.
The sum favours short names. Each token of a team's name adds a log-probability, which is never above zero, so a team with more tokens has more terms to pull its total down. The example adds the terms without dividing by length, and the chapter does not claim to correct for it. A reader who uses label scoring should check the top teams against a sample with gold labels before trusting them.
Label scoring does give a score for each team, which is what a hand-off rule needs. Each message is scored against all 77 team names, which the code scores in one batch after the prompt runs once. Chapter 5 gets a score for each team from a single forward pass through a head that is trained on labelled messages. Label scoring is the bridge from this chapter to that one: it shows that the model already has per-team scores, and that before training they say little about the message.
Working rule: Use label scoring to see which team a model favours before training, and do not route with it. Do not route messages with a prompted model of this size on this task; the 200-message recording is the evidence. Train a head on the labelled messages instead, as chapter 5 does.
What this chapter decided
The rules in this chapter are scoped to these two models, this prompt and this constraint, on the 200 messages of the recording. A larger model, a different prompt or a longer sample could move the counts, and the book has not tested any of them on that sample. What the chapter does establish is narrower: with the raw prompt and the trie, a model of this size returns valid teams and few right ones, and the count of right ones is the number to watch.
Each decision is tested in its own section: the prompt in the first, the trie in its section and greedy decoding in its section, and the two counts in the measurement section. A reader who changes any of them must record again before comparing, as the first section says. A constraint makes every answer valid, and only training can make it right.
This chapter made these decisions:
- The router prompt is the raw form, and both models in the comparison use it.
- Constrained decoding uses a token trie over the 77 team names, with greedy decoding and fixed seeds.
- Every variant is reported with its count of valid answers and its count of right answers, on one fixed sample.
- The live examples show how each piece works, and only the 200-message recording compares models.
This chapter, in this edition
Edition 2026.11.9Examples passed Nov 20, 20265 of 5 examples in this chapter pass
This book is open to contributions, so anyone can check it or correct it. Found something wrong? Report it with the section's ID, the last part of the link beside each heading, or fix it yourself with a pull request.