작은 모델이 잘못된 선택일 때

읽는 데 약 40분분 걸린다, 6절에 5개 예제이(가) 있음.

2026.11.9판 예제 통과 2026년 11월 20 36개 예제 중 36개 통과 변경 사항

AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식

영어판 2026.11.9판을 번역한 것이다. 원문 읽기

이 장을 마치면 작은 라우터가 어디에서 여전히 대역 모델에 뒤지는지, 어디에서 호스팅 모델이 더 나은 결과를 낼 것으로 기대되는지 말할 수 있다. 날짜와 함께 밝힌 입력으로 각 선택지의 월 비용을 계산할 수 있다. 자체 머신이 더 저렴한 경로가 되는 사용량을 찾을 수 있다. 어떤 비용보다 먼저 이 문제를 결정하는 요구 사항을 짚어 낼 수 있다.

작은 라우터는 메시지 대부분에서 더 큰 모델을 앞서면서도, 일부 메시지, 일부 팀, 일부 사용량에서는 여전히 잘못된 선택일 수 있다. 은행의 지원 수신함(1장)에는 팀이 77개 있다. 이 책은 아홉 개 장에 걸쳐 이 수신함을 위한 분류기 헤드를 만들었다. 확신이 없는 메시지를 넘기는 경로와, 라우터의 버전을 관리하고 넘겨진 메시지를 사람에게 보내는 점검 절차를 갖추어 이 헤드를 서비스했다. 그리고 증류한 학생 모델과 int8 사본을 시험하여 둘 다 출시하지 않은 채 성능을 측정했다. 라우터를 측정하는 장들은 라우터를 대역 모델과 비교해 점수를 매긴다(1장).

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Where the small model still loses

마지막으로 변경된 판: 2026.11.2판

Chapter 2 set the stand-in's score as the baseline. This section puts the router beside it on the same messages: all 1001 messages of test_1001, the fixed sample that every chapter's samples come from (chapter 2). The router is the classifier banking77-lora-v1, run live.

router-on-test-1001.py Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0 통과
"""The router and the stand-in on all 1,001 messages of test_1001, the committed test order. The prose rests on thelines for the router's wrong answers sent as answers and for the messages the stand-in gets right and the router wrong.The stand-in's answers are recorded; the router runs live on 2 threads, and its first 200 messages are the test_200 sample."""import osos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import torch  # noqa: E402torch.set_num_threads(2)import router_lib  # noqa: E402router_lib.setup_runtime()# Runs the router live on the sample and reads the stand-in's answers from its recording, then scores both.c = router_lib.compare_router_and_standin("test_1001")n = c["messages"]print(f"messages on test_1001: {n}")print(f"router banking77-lora-v1 correct: {c['router_correct']}/{n} ({c['router_correct'] / n:.1%})")print(f"stand-in correct: {c['standin_correct']}/{n} ({c['standin_correct'] / n:.1%})")print(f"router handed off: {c['handed_off']}/{n} ({c['handed_off'] / n:.1%})")print(f"right for the stand-in, wrong for the router: {c['standin_right_router_wrong']}/{n}")print(f"of those, handed off by the router: {c['of_those_handed_off']}/{c['standin_right_router_wrong']}")print(f"wrong for the router and sent as an answer: {c['router_wrong_sent']}/{n} ({c['router_wrong_sent'] / n:.1%})")print(f"right for the router, wrong for the stand-in: {c['router_right_standin_wrong']}/{n}")
출력python router-on-test-1001.py
messages on test_1001: 1001
router banking77-lora-v1 correct: 847/1001 (84.6%)
stand-in correct: 560/1001 (55.9%)
router handed off: 146/1001 (14.6%)
right for the stand-in, wrong for the router: 48/1001
of those, handed off by the router: 22/48
wrong for the router and sent as an answer: 76/1001 (7.6%)
right for the router, wrong for the stand-in: 335/1001

On the whole set the router is ahead, right on 847/1001 (84.6%) against the stand-in's 560/1001 (55.9%), which is the baseline from chapter 2. Yet the stand-in is right and the router wrong on 48/1001 messages. The router hands off some of those, so a person sees them, and the rest reach the caller as wrong answers. The reverse disagreement, where the router is right and the stand-in wrong, has its own count in the output.

The count that matters most for the inbox is the wrong answers the router sends to a caller: 76/1001 (7.6%). Those are messages it kept, because their confidence cleared the threshold. Chapter 5 chose that threshold on the dev split, and chapter 9 says where what the router hands off goes, which is a person.

where-the-router-still-loses.py Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0 통과
"""Where the router still loses: test_200 scored for the router and the stand-in. The prose rests on the count ofmessages the stand-in gets right and the router wrong, and on the wrong answers the router sends as answers.The stand-in's answers are recorded; the router runs live on 2 threads."""import osos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import json  # noqa: E402import torch  # noqa: E402torch.set_num_threads(2)import router_lib  # noqa: E402router_lib.setup_runtime()ADAPTER = router_lib.HERE / "data" / "recorded" / "lora-full"# The threshold chapter 5 chose on dev, as the adapter's manifest records it.threshold = json.loads((ADAPTER / "manifest.json").read_text())["handoff_threshold"]# Keyed by test row index, not by position in the sample, so each lookup uses the sample's row numbers.answers = router_lib.standin_rows()test = router_lib.read_rows("test")  # (text, label index) per rowsample = router_lib.read_sample("test_200")golds = [test[i][1] for i in sample]  # index 1 is the gold team, as an indexstandin_preds = [answers[i]["label"] for i in sample]  # the label is the team's index, the same form as the gold labeltokenizer, model = router_lib.load_classifier(ADAPTER / "adapter")preds, confs = router_lib.classify(model, tokenizer, [test[i][0] for i in sample])n = len(sample)router_right = [p == g for p, g in zip(preds, golds)]standin_right = [p == g for p, g in zip(standin_preds, golds)]handed_off = [router_lib.handoff(c, threshold) for c in confs]  # true below the threshold; a NaN confidence is handed off toolost = [i for i in range(n) if standin_right[i] and not router_right[i]]sent_wrong = sum(not r and not h for r, h in zip(router_right, handed_off))  # wrong and not handed off: what a caller receivesprint(f"messages on test_200: {n}")print(f"router banking77-lora-v1 correct: {sum(router_right)}/{n} ({sum(router_right) / n:.1%})")print(f"stand-in correct: {sum(standin_right)}/{n} ({sum(standin_right) / n:.1%})")print(f"router threshold from manifest: {threshold:.2f}")print(f"router handed off: {sum(handed_off)}/{n} ({sum(handed_off) / n:.1%})")print(f"right for the stand-in, wrong for the router: {len(lost)}/{n} ({len(lost) / n:.1%})")print(f"of those, handed off by the router: {sum(handed_off[i] for i in lost)}/{len(lost)}")print(f"wrong for the router and sent as an answer: {sent_wrong}/{n} ({sent_wrong / n:.1%})")print(f"right for the router, wrong for the stand-in: {sum(r and not s for r, s in zip(router_right, standin_right))}/{n}")
출력python where-the-router-still-loses.py
messages on test_200: 200
router banking77-lora-v1 correct: 172/200 (86.0%)
stand-in correct: 109/200 (54.5%)
router threshold from manifest: 0.63
router handed off: 33/200 (16.5%)
right for the stand-in, wrong for the router: 6/200 (3.0%)
of those, handed off by the router: 3/6
wrong for the router and sent as an answer: 11/200 (5.5%)
right for the router, wrong for the stand-in: 69/200

test_200 is the first 200 messages of test_1001, a prefix the count script checks, so these counts are part of the full run and not a second sample. On it, the stand-in is right and the router wrong on 6/200 (3.0%) messages, and the router hands off 3/6 of them.

The hand-off holds back the wrong answers. The router sends 11/200 (5.5%) wrong answers on these 200 messages. On the messages it keeps, its accuracy is 156/167 (93.4%), and 33/200 (16.5%) of all 200 go to a person.

Forcing a small model onto the team list keeps every answer a team, and does not make the answer right. Chapter 4 asked the two smaller models for a team that way. The table gives each prompted variant's count on the 200 messages of test_200:

Prompted variant Right on test_200
SmolLM2-135M-Instruct, raw prompt 9/200 (4.5%)
SmolLM2-135M-Instruct, chat template 11/200 (5.5%)
SmolLM2-360M-Instruct, raw prompt 12/200 (6.0%)

The raw 135M model gave one answer more often than any other (card_arrival, 165/200), and that team is the gold team for 0/200 of the messages. A model that leans on one answer is a prior, not a router. Chapter 4 shows why a prompt cannot make up for a model this size.

The stand-in fails in patterns too. Across all 1001 messages, its answers never name 4 of the 77 teams, so every message whose gold team is one of them goes to another team. Two confusions tie for its most common error. The clearest is the team Refund_not_showing_up, which is answered as request_refund on 13/13 of its messages. A pair like that is where a labelled set has to be read before any model is trusted on it, and chapter 2 lists the pairs for every candidate.

A student copies its teacher, including the teacher's mistakes. Chapter 6 distilled the stand-in's answers into a student. The teacher agreed with the gold label on 549/1000 (54.9%) of the training rows it labelled, so the student learned from labels that were wrong almost half the time. The student reached 69/200 (34.5%) on test_200, and the same recipe trained on the gold labels of the same rows reached 77/200 (38.5%).

The student also falls below its teacher, which reached 109/200 (54.5%) on test_200, and below the gold-label control, by a gap this sample cannot settle (chapter 6). The student is right where the control is wrong on 24/200 messages, and the control where the student is wrong on 32/200.

A new team is another place the small model loses. A classifier head has one output for each team it was trained on, so a team without an output cannot be routed by it. Adding one means labelled messages for that team, a training run with the recipe from chapter 5, and the checks from chapter 9 before the router sends a message to it. Whether a prompt change could add a new team is a question for the reader to test. This book did not measure a hosted model, and the table later in this chapter lists the question as open.

Put together, the small models here did three things: a prompted model leaned on one answer, a student copied its teacher's mistakes, and a team with no output could not be routed. None of these is a fault in the router's arithmetic. The stand-in did one thing more: it never named 4 teams and confused a pair. That is the stand-in's, not either option's. A larger threshold trades the router's 11/200 (5.5%) wrong answers on test_200 for hand-offs (chapter 5), but it gives no team an output and fixes neither the teacher's errors nor the stand-in's. Each changes what a caller sees: a wrong team, a handed-off message, or no team at all.

Working rule: Before you route with a small model, run it and the hosted model on the same fixed sample. Count the messages each gets right that the other gets wrong, and the wrong answers that would reach a caller. Count a prompted model's answers against the team list, and count how often its most common answer is the gold team. Do not route a team that has no labelled messages of its own, and do not train a student on a teacher's labels before you have counted how often the teacher agrees with the gold labels.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Count the bill and the machine

마지막으로 변경된 판: 2026.11.9판

A hosted model is paid per message. A hosted model's monthly bill is the number of messages times what one message costs, and one message costs its input tokens at the input price plus its output tokens at the output price. A price list written per million tokens gives the price per token when divided by one million. A routing message is long on the input side, because the prompt and the message are both counted, and short on the output side, because the answer is one team name.

A machine is paid for whether it is busy or not. A machine's monthly bill is its hours times its hourly rate. The number of messages a machine can route in a month is its throughput, in messages per second, times the seconds in a month, times its utilisation, which is the share of the month it is busy. The machine's cost per message therefore depends on how many messages it routes, and the comparison needs a volume before it gives an answer.

The three formulas below hold for any pair of options.

hosted cost per month = messages × (input tokens × input price + output tokens × output price)
machine cost per month = hours × machine cost per hour
machine capacity per month = messages per second × seconds per month × utilisation

Each input needs a value, and each price needs a date and a source. The book quotes no hosted price, because it has none with both a source and a date that a reader could check. Read your provider's price page on the day you run the comparison, and write the price, its date and its address beside the number.

Input tokens also need a tokenizer. A hosted provider's count may differ from an open model's count of the same text, and this book did not measure one. The routing prompt is what a hosted model would receive for each message: the list of 77 team names and the message, as router_prompt builds it. The monthly example counts it with the tokenizer of SmolLM2-135M-Instruct, which gives 587 tokens per message, and the example labels that count as a stand-in for the provider's.

Throughput is the input that a recording has to supply, because it depends on the machine. The recording names its machine and saves the rates it measured. On a cloud machine with 4 vCPUs and 15 GiB of memory, at 4 threads, the stand-in routed 0.32 messages a second at one message per call. The router's classifier, banking77-lora-v1, in float32, routed 19.0 messages a second at the same call size. Its rate in batches of 16 is in the output as well.

Only the one-per-call pair is like for like. The classifier in batches of 16 takes 16 messages in one forward pass, and the stand-in takes one message per call with its whole router prompt. So the batched ratio of 104.0 mixes model size with batching, and this chapter does not use it for the comparison. The comparison uses the one-per-call ratio of 60.8. Each copy got one timed pass over the 200 messages of test_200, after a warm-up of one message, with tokenization inside the timed span. The ratios come from unrounded rates, so they are not exactly the quotient of the printed rates.

cost-ratio-from-recording.py Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0 통과
"""The throughput rates and ratios as recorded, and the machine they were measured on. The prose rests on theone-per-call ratio line, the like-for-like comparison; the batched lines differ in call size. Reads the recording only; no model runs."""import jsonimport router_libfolder = router_lib.HERE / "data" / "recorded" / "throughput"ratios = json.loads((folder / "ratios.json").read_text())rates = router_lib.recorded_rates()manifest = json.loads((folder / "manifest.json").read_text())print(f"machine: {manifest['machine'].split(', ')[0]}")  # the machine's description is cut at its first comma, which keeps the name and drops the restprint(f"threads: {manifest['settings']['n_threads']}")print(f"messages on test_200: {ratios['messages']}")print(f"stand-in rate, one per call (messages a second): {rates['standin_one_message_per_call']}")print(f"classifier fp32 rate, one per call (messages a second): {rates['classifier_fp32_one_message_per_call']}")print(f"classifier fp32 rate, batches of 16 (messages a second): {rates['classifier_fp32_batches_of_16']}")print(f"classifier int8 rate, batches of 16 (messages a second): {rates['classifier_int8_batches_of_16']}")# Each ratio is computed from unrounded rates, so it is not the quotient of the printed rates.print(f"fp32 over stand-in, one per call, throughput ratio: {ratios['classifier_fp32_one_at_a_time_over_standin']}")print(f"fp32 over stand-in, batches of 16, throughput ratio: {ratios['classifier_fp32_over_standin']}")print(f"int8 over stand-in, batches of 16, throughput ratio: {ratios['classifier_int8_over_standin']}")print(f"int8 over fp32, batches of 16, throughput ratio: {ratios['classifier_int8_over_fp32']}")print(f"predictions agreeing, fp32 and int8: {ratios['agreement_fp32_int8']}/{ratios['messages']}")
출력python cost-ratio-from-recording.py
machine: a cloud machine with 4 vCPUs and 15 GiB of memory
threads: 4
messages on test_200: 200
stand-in rate, one per call (messages a second): 0.32
classifier fp32 rate, one per call (messages a second): 19.0
classifier fp32 rate, batches of 16 (messages a second): 33.0
classifier int8 rate, batches of 16 (messages a second): 29.0
fp32 over stand-in, one per call, throughput ratio: 60.8
fp32 over stand-in, batches of 16, throughput ratio: 104.0
int8 over stand-in, batches of 16, throughput ratio: 91.8
int8 over fp32, batches of 16, throughput ratio: 0.9
predictions agreeing, fp32 and int8: 200/200

On this machine, at this thread count, the int8 copy was no faster than float32. Its ratio over float32 in batches of 16 is 0.9. Chapter 7 said that fewer bytes do not settle speed, and this recording measures the speed. The two copies agreed on 200/200 predictions. The recording ran at that thread count, and this book's serving budget is two vCPUs, so a reader's rates should come from a recording on their own machine at their own thread count.

The bill needs a rate for the router, and the recording gives one: the classifier at one message per call, timed as the stand-in was. The route handles one message per request, so the bill uses that rate. A reader replaces it with one taken on their own machine.

monthly-bill-from-labelled-inputs.py Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0 통과
"""The monthly bill for a hosted model and for the router on one machine, from inputs the reader sets. The hosted bill,the machine's bill and the cheaper-route line are the point the prose rests on. The prices, volume, machine costand utilisation are example values, not published prices; the token counts are live and the router's rate is recorded."""import osos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import torch  # noqa: E402torch.set_num_threads(2)import router_lib  # noqa: E402router_lib.setup_runtime()inputs = router_lib.bill_inputs_recorded()test = router_lib.read_rows("test")messages = [test[i][0] for i in router_lib.read_sample("test_200")]  # index 0 is the text; the label is index 1# The hosted prompt, counted with the 135M model's tokenizer as a stand-in for the provider's.input_tokens = router_lib.router_prompt_tokens(messages)# Prices are per million tokens, hence the division at the end.per_message = (input_tokens * inputs["price_per_million_input_tokens"]               + inputs["output_tokens_per_message"] * inputs["price_per_million_output_tokens"]) / 1_000_000hosted_month = inputs["messages_per_month"] * per_messagemachine_month = inputs["machine_cost_per_hour"] * inputs["hours_per_month"]capacity = inputs["router_messages_per_second"] * 3600 * inputs["hours_per_month"] * inputs["utilisation"]print(f"example input, messages per month: {inputs['messages_per_month']}")print(f"example input, price per million input tokens, USD: {inputs['price_per_million_input_tokens']}")print(f"example input, price per million output tokens, USD: {inputs['price_per_million_output_tokens']}")print(f"example input, output tokens per message: {inputs['output_tokens_per_message']}")print(f"example input, machine cost per hour, USD: {inputs['machine_cost_per_hour']}")print(f"example input, hours per month: {inputs['hours_per_month']}")print(f"example input, utilisation: {inputs['utilisation']}")print(f"recorded input, router messages per second, one per call: {inputs['router_messages_per_second']}")print(f"input tokens per message, hosted prompt (counted with SmolLM2-135M-Instruct, a stand-in tokenizer): {input_tokens}")print(f"example hosted price per message (example prices), USD: {per_message:.6f}")print(f"example hosted bill per month (example prices), USD: {hosted_month:.2f}")print(f"example machine bill per month (example cost per hour), USD: {machine_month:.2f}")print(f"machine capacity per month (recorded rate, example utilisation): {capacity:.0f}")print(f"example machine cost per message at capacity (example cost per hour), USD: {machine_month / capacity:.6f}")print(f"example volume within machine capacity: {'yes' if inputs['messages_per_month'] <= capacity else 'no'}")  # above capacity, one machine cannot carry the volume and this bill does not holdprint(f"example values, cheaper route at the input volume: {'hosted' if hosted_month < machine_month else 'machine'}")
출력python monthly-bill-from-labelled-inputs.py
example input, messages per month: 100000
example input, price per million input tokens, USD: 1.0
example input, price per million output tokens, USD: 4.0
example input, output tokens per message: 6
example input, machine cost per hour, USD: 0.4
example input, hours per month: 730
example input, utilisation: 0.5
recorded input, router messages per second, one per call: 19.0
input tokens per message, hosted prompt (counted with SmolLM2-135M-Instruct, a stand-in tokenizer): 587
example hosted price per message (example prices), USD: 0.000611
example hosted bill per month (example prices), USD: 61.10
example machine bill per month (example cost per hour), USD: 292.00
machine capacity per month (recorded rate, example utilisation): 24966000
example machine cost per message at capacity (example cost per hour), USD: 0.000012
example volume within machine capacity: yes
example values, cheaper route at the input volume: hosted

At these example inputs, the hosted route is the cheaper one. The hosted bill is 61.10 USD a month for the example volume, and a machine of your own costs 292.00 USD a month whatever the volume.

At the recorded rate and the example utilisation, the machine can route 24966000 messages a month, and at full capacity its cost per message is 0.000012 USD. The example volume of 100000 messages a month sits far below that capacity, and the break-even section finds where the two bills cross. The cost per hour in these inputs suits the recorded machine; a two-vCPU machine needs its own recorded rate.

Every input in this bill can stop being true. Prices change, so a price read in one month is a fact about that month. The utilisation here is an assumption about traffic that has not happened yet, and an inbox's volume moves with the products the bank sells and with the events that fill its queue. A machine that was idle half the month in one quarter can be busy all of the next. So count again each month, with the actual volume, the price on its date and the utilisation the machine measured. A comparison made once is a record of the month it was made in.

Working rule: Compare the two bills with inputs you have checked and a machine you have priced. Write each price with its source and the date you read it. Count the input tokens with the tokenizer of the hosted model you compare with, and take the throughput from a recording on the machine you would buy. Where a value is an example, say so beside it, and replace it before you decide.

데이터를 건물 안에 두기

이 절은 비용 비교가 아니며, 이 책의 어떤 수치도 이 문제를 결정하지 않는다. 받은편지함에 들어오는 각 메시지는 고객이 쓴 글이다. 호스팅 모델이 이 글에 답하면, 글은 은행의 기계를 떠나 제공자의 기계로 간다. 호출 한 번마다 호스팅 호출은 메시지 본문과 77개 팀 이름의 목록을 보낸다. 은행은 고객 데이터, 계약, 누가 무엇을 읽을 수 있는가에 관한 규정 등 자체적인 이유로 이를 원하지 않을 수 있다. 이러한 이유는 호스팅 쪽 요금이 더 낮더라도 그대로 유효하다.

이 책의 메시지는 BANKING77의 공개 메시지이므로, 이 책의 실행 결과는 프라이버시에 관해 아무것도 보여 주지 않는다. 이 요건은 은행에 속하며, 그 요건이 적용되는지는 은행만 말할 수 있다.

은행의 기계 안에 머무는 작은 모델은 실행되는 위치로 이 요건을 충족한다. 라우터와 그 가중치, 로그는 은행의 통제 아래에 있으며, 9장의 버전 관리와 파일 검사는 경로가 불러오는 모든 어댑터에 적용된다. int8 사본처럼 파생된 사본에는 자체 매니페스트도 고정(pin)도 없으므로, 서비스에 쓰기 전에 둘 다 마련해야 한다. 안에 머무는 데에는 청구서가 아닌 고유한 비용이 따른다. 은행은 기계를 운영하고, 패치를 유지하며, 다른 것이 바뀌어도 경로에 남는 인계를 포함해 경로의 검사가 계속 돌아가게 해야 한다(8장).

청구서는 물량, 가격, 날짜에 따라 움직이는 숫자이다. 요건은 움직이지 않는다. 그러므로 요건을 먼저 정하고, 그 요건을 충족하는 선택지만 비교한다. 은행의 기계에서 실행되는 작은 모델이 그런 선택지의 하나이다. 그곳에서 실행되는 더 큰 오픈 모델이 또 하나이다. 대역(stand-in)도 하나이며, 이 기계에서 호출당 하나씩 처리했을 때 초당 0.32개의 메시지를 라우팅했고, 분류기는 19.0개였다. 이 차이는 처리량으로 측정한 비용이며 60.8배에 달하고, 금전적 비용은 아니다. 받은편지함이 예상하는 물량에서 요건을 충족하는 선택지가 없다면, 비용 비교는 결정할 것이 없다. 요건을 충족하는 작은 모델도 첫 번째 절의 정확도와 인계 집계를 통과해야 하며, 요건을 충족했다고 해서 그 집계가 면제되지는 않는다.

작업 규칙: 비용보다 데이터 요건을 먼저 정한다. 요건을 소유한 사람들과 함께 요건으로 적어 두고, 절감이나 비용으로 세지 않는다. 메시지가 은행의 기계 안에 머물러야 한다면 그곳에서 실행되는 모델을 고르고, 그 뒤에 이 장의 첫 번째 절에서처럼 정확도와 인계를 집계한다.

표로 선택하기

마지막으로 변경된 판: 2026.11.2판

위의 질문들이 모두 비용에 관한 것은 아니며, 대부분은 누구도 녹화본을 실행하기 전에 답할 수 있다. 아래 표는 그 질문들을 나란히 놓은 것이다. 각 질문에 대해 소형 라우터는 어떤 모습인지, 호스팅 모델은 어떤 모습인지, 받은편지함에서 무엇을 세어야 하는지를 적었다.

질문 소형 라우터 호스팅 모델 측정할 것
메시지가 은행의 기계 안에 머물러야 하는가? 충족한다: 은행의 기계에서 실행된다 충족하지 못한다: 텍스트가 제공자에게 전달된다 개수가 아닌 요구 사항: 가장 먼저 결정한다
모든 답이 팀이어야 하는가? 그렇다. 헤드 또는 제약 디코딩으로 보장한다(4장) 프롬프트와 검사로 답을 목록 안에 묶어 둘 때만 가능하다 고정 표본에서 팀 목록 밖에 있는 답의 수
팀 목록은 얼마나 자주 바뀌는가? 새 팀에는 레이블이 붙은 메시지와 학습 실행이 필요하다(9장) 프롬프트 변경만으로 충분한지는 시험해 볼 문제이다 새 팀을 라우팅하는 데 필요한 레이블된 메시지와 단계
한 달에 메시지가 몇 건인가? 처리 용량까지는 월 고정 요금이다 메시지당 요금이며 물량에 따라 늘어난다 마지막 절의 손익분기 건수
팀마다 레이블된 메시지가 몇 건인가? 학습하고 팀별로 시험하기에 충분한 양(2장) 프롬프트라면 적거나 없어도 된다 학습 분할에서 센 팀별 레이블된 메시지 수
과업에 자유 텍스트나 폭넓은 지식이 필요한가? 팀 목록으로 한정되며, 자유 텍스트 답은 이 책의 첫 번째 시험에서 실패했다 시험해 볼 문제이다 같은 고정 표본에서의 정확도와 그 건수

이 표는 점수표가 아니라 세어야 할 것의 목록이다. 행을 순서대로 읽는다. 첫 행의 요구 사항, 그다음 팀 목록, 그다음 물량 순이다. 각 행이 다음 질문을 던지기 전에 문제를 끝낼 수 있기 때문이다. 마지막 열에서 "요구 사항"으로 시작하는 칸은 개수보다 먼저 내려야 하는 결정을 가리킨다. 나머지 칸은 모두 여러분의 표본에서 센 개수를 가리키며, 분모를 함께 적어야 한다. 자유 텍스트 행은 4장의 첫 번째 시험에 근거한다. 135M 모델의 제약 없는 답은 0/20건의 메시지에서 팀 이름이었다.

작업 규칙: 선택하기 전에 모든 행에 여러분의 받은편지함 기준으로 답한다. 행이 개수를 요구하면 고정 표본에서 세고 분모를 함께 적는다. 행이 요구 사항을 가리키면 개수보다 요구 사항이 먼저 결정하게 하고, 그것을 요구 사항으로 결정문에 적는다.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Know the break-even

마지막으로 변경된 판: 2026.11.9판

The break-even is the volume at which the hosted model's monthly bill and the machine's are equal. Below it, the hosted model costs less at the inputs you set. Above it, the machine costs less, provided the volume is inside the machine's capacity. The machine's bill is paid whether it is busy or idle, so the break-even divides that bill by the hosted price per message:

break-even messages per month = machine bill per month ÷ hosted price per message

The formula does not subtract the machine's cost per message from the hosted price. The machine's cost per message at capacity is its fixed bill divided by its capacity, and that bill is already counted once in the numerator. Subtracting the same cost again would count the machine twice. The formula also holds only within capacity. Above it, one machine cannot carry the volume, and the count is made again for a second machine, with its own bill and its own capacity.

break-even-from-recorded-ratio.py Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0 통과
"""The monthly volume at which one machine's bill equals the hosted bill for the router's prompts. The break-even lineis the point the prose rests on, and it holds only inside the capacity line, which the last line checks. The prices andthe machine's cost are example values, as in the bill; the router's rate is recorded on the machine."""import mathimport osos.environ["HF_HUB_OFFLINE"] = "1"os.environ["TRANSFORMERS_OFFLINE"] = "1"import torch  # noqa: E402torch.set_num_threads(2)import router_lib  # noqa: E402router_lib.setup_runtime()inputs = router_lib.bill_inputs_recorded()  # example prices and cost per hour, with the router's rate from the recordingtest = router_lib.read_rows("test")  # (text, label index) for every test row, in file ordermessages = [test[i][0] for i in router_lib.read_sample("test_200")]  # index 0 is the text; the label is index 1# Mean count of the whole router prompt, team list included, with SmolLM2-135M's tokenizer.input_tokens = router_lib.router_prompt_tokens(messages)# Prices are per million tokens, hence the division at the end.per_message = (input_tokens * inputs["price_per_million_input_tokens"]               + inputs["output_tokens_per_message"] * inputs["price_per_million_output_tokens"]) / 1_000_000machine_month = inputs["machine_cost_per_hour"] * inputs["hours_per_month"]capacity = inputs["router_messages_per_second"] * 3600 * inputs["hours_per_month"] * inputs["utilisation"]# The machine's bill is the same each month up to its capacity, so the hosted bill# equals it at messages = machine bill / hosted price per message.# Rounded up: the first whole message count at which the machine is no dearer.breakeven = math.ceil(machine_month / per_message)print(f"input tokens per message, hosted prompt (counted with SmolLM2-135M-Instruct, a stand-in tokenizer): {input_tokens}")print(f"example hosted price per message (example prices), USD: {per_message:.6f}")print(f"example machine bill per month (example cost per hour), USD: {machine_month:.2f}")print(f"machine capacity per month (recorded rate, example utilisation): {capacity:.0f}")print(f"example break-even messages per month (example prices and cost per hour): {breakeven}")print(f"break-even within machine capacity: {'yes' if breakeven <= capacity else 'no'}")
출력python break-even-from-recorded-ratio.py
input tokens per message, hosted prompt (counted with SmolLM2-135M-Instruct, a stand-in tokenizer): 587
example hosted price per message (example prices), USD: 0.000611
example machine bill per month (example cost per hour), USD: 292.00
machine capacity per month (recorded rate, example utilisation): 24966000
example break-even messages per month (example prices and cost per hour): 477906
break-even within machine capacity: yes

With the example values, the machine's bill of 292.00 USD breaks even at 477906 messages a month. That count sits inside the machine's capacity of 24966000 messages a month (the check prints yes). That capacity uses the recorded one-per-call rate with the example utilisation.

The break-even does not depend on the rate, only on the bill and the hosted price. The volume of 100000 messages a month sits well below it, so the hosted bill is the lower one.

A cost point is not a decision, and the inbox still decides the rest. It answers the cost question only after the requirement in keep the data inside the building is settled. If the messages must stay inside, the question is which small model meets the accuracy and hand-off counts, and the break-even says what that machine costs against a hosted model at each volume. If the requirement lets a hosted model in, and the volume is below the break-even, the hosted model is the lower-cost route at the example prices. Above the break-even, the machine is the lower-cost route, provided the volume is inside its capacity and the router's wrong answers reaching a caller are numbers the support staff can accept. Hand-offs go to a person whatever the bill says: on the first 200 messages, the router hands off 33/200 (16.5%) of them.

Working rule: Compute the break-even with your machine's bill and the hosted price you read, with that price's date and source written beside it. Check that the volume you expect sits inside the capacity you measured on your machine before you read the break-even as a saving. Above that capacity, count a second machine and compute again. Read the break-even as a cost point, not a decision. The requirements and the accuracy counts from the earlier sections still decide whether the router goes live.

이 책이 받은편지함을 위해 내린 결정

마지막으로 변경된 판: 2026.11.2판

책이 이름을 든 표본과 기계에서, 받은편지함에 관련된 수치는 다음과 같다. 각 수치에는 그것을 센 장을 함께 적었다.

  • 정확도. test_1001 전체에서 라우터는 847/1001 (84.6%)건을 맞혔고, 대용 모델은 560/1001 (55.9%)건을 맞혔다(2장). 라우터는 나머지를 사람에게 넘기며, 그래도 76/1001 (7.6%)건의 오답이 호출자에게 전달된다. 이것이 라우터를 가동하기 전에 지원 담당자들이 받아들여야 하는 건수이다.
  • 속도. 기록된 기계에서 호출당 메시지 한 건으로 실행했을 때, 분류기는 초당 19.0건을 처리했고 대용 모델은 초당 0.32건을 처리했다. 비율은 60.8이다(7장에서 속도를 어떻게 측정하는지 설명한다). int8 사본은 속도 면에서 서비스할 이유를 주지 못했다. 16건 배치에서 float32에 대한 비율은 0.9이었다. 따라서 경로가 서비스하는 것은 계속 float32 어댑터이다(8장).
  • 비용. 예제 입력에서 한 달 100000건의 메시지는 호스팅 쪽에서 61.10 USD, 자체 기계에서 292.00 USD가 든다. 손익분기점은 한 달 477906건이며, 처리 용량 24966000 안에 있다. 이것은 예제 가격과 예제 시간당 비용이므로, 손익분기점을 결정으로 읽기 전에 날짜와 출처를 붙인 자신의 값으로 바꾸어야 한다.
  • 데이터 요건. 메시지가 은행의 기계 안에 머물러야 한다면 호스팅 모델은 제외되며, 선택은 그곳에서 실행되는 선택지 사이에서 이루어진다. 건물 안에 데이터를 두는 문제를 다룬 절이 말하듯, 이것을 먼저 정해야 한다.
  • 작은 모델이 선택지가 아닌 경우. 레이블이 붙은 메시지가 없는 팀에는 헤드에 출력이 없으며, 새 팀에는 9장의 단계가 필요하다. 자유 서술형 답변은 이 책의 첫 시험에서 실패했다(4장). 호스팅 모델은 자유 서술과 폭넓은 지식에서 더 낫다고 예상되는데, 이는 시험해 볼 문제이다. 그리고 호스팅 모델이 허용되는 곳에서 물량이 손익분기점보다 적다면, 예제 가격에서는 호스팅 모델이 더 싼 경로이다.
  • 알게 된 것. 데이터 요건과 팀 목록이 어떤 선택지가 남는지를 정한다. 그다음에야 비용이 그 가운데에서 고른다.

받은편지함에 대한 판정. 메시지가 은행의 기계 안에 머물러야 하고 모든 팀에 레이블이 붙은 메시지가 있다면, 은행 자체 기계 위의 작은 라우터가 선택이다. 단, 양이 은행 자체 가격에서의 손익분기점을 넘고 기계의 처리 용량 안에 있어야 한다. 사람에게 넘기는 단계는 비용이 무엇을 말하든 경로에 남는다. 가동하기 전에 은행 자체 표본에서 호출자에게 전달되는 오답을 세어야 한다.

작업 규칙: 선택하기 전에 이 목록을 자신의 받은편지함과 자신의 수치로 다시 작성하라. 자신의 집계, 날짜, 요건에서 나오지 않은 판정은 자신의 받은편지함에 대한 이 책의 판정이 아니다.

이 판의 이 장

2026.11.9판예제 통과 2026년 11월 20이 장에서 5개 예제개 중 5개가 통과한다

영어판 2026.11.9판을 번역한 것이다.

이 절 인용

형식

각 형식은 이 번역이 따르는 영문 원서를 출처로 표기한다.

모든 형식은 이 판을 명시한다. 링크는 현재 판을 열며, 여기서 절의 제목은 본문이 마지막으로 바뀐 시점을 알려 준다. 접근 날짜와 검색 날짜는 이 자료를 여는 날짜다.

이 페이지의 내용

10장 중 10장: 작은 모델이 잘못된 선택일 때

  1. Where the small model still loses
  2. Count the bill and the machine
  3. 데이터를 건물 안에 두기
  4. 표로 선택하기
  5. Know the break-even
  6. 이 책이 받은편지함을 위해 내린 결정

맨 위로 돌아가기전체 장