신뢰할 수 있는 테스트 세트를 만들고 공개 벤치마크를 읽는 법
읽는 데 약 39분분 걸린다, 10절에 6개 예제이(가) 있음.
AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식
영어판 2026.11.3판을 번역한 것이다. 원문 읽기
이 장을 마치면 점수가 의미를 갖는 테스트 세트를 만들고, 그 스키마와 커버리지를 코드로 점검하며, 공개 벤치마크 점수가 여러분의 어시스턴트에 대해 무엇을 알려 주고 무엇을 알려 주지 못하는지 설명할 수 있다. 또한 객관식 점수가 항상 같은 방식으로 답하는 습관과 구별되지 않을 수 있다는 점도 보게 된다.
여기서 테스트 세트는 sablefile 어시스턴트의 것이다. 이 어시스턴트는 검색 단계와, 도구의 문서를 바탕으로 질문에 답하는 모델로 이루어져 있다. 이 책이 어시스턴트에 관해 보고하는 모든 수치는 참조 답안이 달린 질문 세트에서 나오므로, 점수가 얼마만큼의 가치를 갖는지는 그 세트가 결정한다. 이 책은 세 가지 결함을 경계한다. 모든 질문의 답이 문서에 있다면, 언제나 답하는 어시스턴트가 좋아 보인다. 질문이 문서의 문장을 그대로 재사용하면, 검색은 단어를 맞추는 데 그치고도 맞아 보임으로 평가된다. 점수가 오를 때까지 세트를 조정하면, 그 점수는 조정의 기록일 뿐이다. 세트는 질문이 적혀 있는 유일한 곳이기도 하므로 그 안의 결함은 눈에 띄지 않을 수 있고, 그 위에 세운 점수는 아무 징후 없이 결함을 물려받는다.
이 장에서는 이 책이 사용하는 세트를 만들고 코드로 점검한다. 그다음 많은 독자가 인용된 것을 본 적 있을 공개 벤치마크 두 가지, GSM8K와 TruthfulQA를 읽고, 그 점수가 그래도 무엇을 보여 줄 수 있는지 설명한다. 각 절은 팀이 차례로 던질 질문을 따른다. 세트는 무엇을 위한 것인가, 어떻게 만들고 점검하는가, 어떻게 나누는가, 공개 점수는 무엇을 더해 주는가.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
What a test set is for
A test set is a fixed set of questions, paired with the answers you would accept, that stands in for the questions users will ask. Its job is to be asked the same questions each time something changes, so that a difference in score comes from the change and not from the questions. It does that job only while it stays fixed. This set was written from the docs rather than sampled from users, so it stands in for them and does not reproduce them.
Each row is an item: one test question with the fields a scorer needs. The item is the unit of every count in this book, so a pass rate is always a share of items, and its denominator says which ones. Each item has a reference answer, which the scorer compares a reply with, and supporting sections, the parts of the docs that a correct answer rests on. The reference answers are written labels. The book's own drafting agents wrote them from the docs, a second pass checked them against the docs, and a script checks the file's shape and section IDs. They are not human labels, and the book does not call them that.
Each item also has a kind, which says what a correct reply looks like. A set built from one kind tests one skill, so this set mixes four kinds on purpose. It should also look like the questions users bring. Users ask things the docs cannot answer, and a set with only answerable questions would reward an assistant that never declines.
The set is small on purpose. It holds 120 items, which a reader can check by hand, and each one has a reference answer that a second pass checked against the docs. We keep it small enough to read, because a set nobody reads can only measure its own mistakes. A score is only as good as the set behind it, and the set is what two people argue about when they disagree on whether a change helped. Keep it readable, keep the reasons for each item written down, and do not add items only because a score looks wrong, since that picks items for their result.
A test set is never training data. Nothing in it teaches the assistant, and the assistant never sees a reference answer while the set is being built. If someone tunes against the answers, the set stops being a test and becomes a target, which is what the rule on dev and held-out is for.
Working rule: Write the set before you change anything you will measure with it, and keep its file under version control, so that every comparison names the items it used. A set you edit after seeing the scores measures your edits, not the assistant.
항목 스키마
세트는 점수를 매기기 전에 먼저 로드되어야 한다. 파일에는 한 줄에 JSON 객체가 하나씩 들어 있고, 로더는 선언된 형태에 맞게 값을 검사하는 Python 라이브러리인 pydantic으로 각 객체를 검증한다. 이 책은 pydantic 2.14.3을(를) 고정해서 사용한다. 필드는 다음과 같다.
id:q-와 네 자리 숫자로 이루어지며, 고유하고, 한 번 쓰면 바뀌지 않는다.question: 일반 문자 1자에서 400자까지.version: 질문이 묻는 문서 버전.kind: 네 가지 종류 중 하나.reference_answer: 정답. 답할 수 없는 항목에서는 비어 있으며, 이때 옳은 응답은 답변을 거절하는 것이다.supporting_sections: 정답이 근거로 삼는 버전과 절 ID의 쌍.split:dev또는held-out.planted와planted_why: 항목이 심어 둔 함정인지, 그렇다면 어떤 함정인지.twin_of: 다른 버전에 대해 같은 질문을 하는 항목이 있다면 그 항목의 ID.
로더는 extra="forbid"을(를) 설정하므로, 알 수 없는 필드는 조용히 무시되지 않고 오류가 된다. 필드 이름에 오타가 있으면 그렇지 않을 경우 맞아 보임에도 점수는 틀리게 나오는 세트가 만들어진다. 오류는 행 단위로 보고되므로 잘못된 행 하나가 나머지를 가리지 않는다.
형태 검사는 행 사이를 볼 수 없으므로, 책의 lib/testset.py에 있는 항목 간 규칙이 그 일을 맡는다. 이 규칙은 ID가 고유한지, 각 쌍둥이 항목이 짝을 되가리키는지, 다른 버전을 묻는지, 버전에 의존하는지, 같은 분할에 속하는지, 그리고 모든 거짓 전제 항목이 답할 수 없는 항목인지를 검사한다. 또한 모든 근거 절이 그것이 지정한 버전에 존재하는지, 어떤 정답도 한 절의 단어 열한 개를 연속으로 반복하지 않는지, 개수가 설계와 일치하는지도 검사한다.
이 규칙 하나하나는 항목을 추가하거나 수정하는 모든 사람이 통과해야 하는 계약이다. 검사는 어긴 규칙을 각각 출력한다. 문제를 찾았든 아니든 오류 코드 없이 종료하므로 그 자체로는 아무것도 멈추지 못하며, 빌드가 출력된 문제를 실패로 처리해야 한다.
정답에는 무엇보다 큰 주의가 필요하다. 모든 응답이 정답과 비교되기 때문이다. 일관된 방향으로 틀린 정답은 점수를 그 방향으로 틀리게 만들고, 형식은 올바르지만 틀린 답은 어떤 형태 규칙으로도 잡아낼 수 없다. 그래서 각 정답은 그 근거가 되는 절과 함께 작성되며, 독자는 저자의 말을 믿는 대신 몇 분 안에 문서와 대조하여 주장을 확인할 수 있다.
쌍둥이 항목은 서로 다른 시기에 작성되면 서로 다른 분할로 어긋날 수 있고, 그러면 dev 부분이 조정한 질문을 held-out 부분이 검사하게 된다. 항목 간 규칙이 이를 잡아내므로 쌍둥이는 함께 유지된다.
아래의 스키마 검사는 두 규칙 묶음을 모두 실행하고, 문제 개수를 종류별, 분할별 개수와 함께 출력한다.
1"""Loads all 120 test-set items, runs the per-item and the cross-item rules, and prints the problem count with the2counts by kind and by split. The point rests on the first line: a set that passes prints zero problems."""3import lib # first: fixes the thread counts and seeds before anything else loads45from collections import Counter67from lib.testset import cross_item_problems, item_problems89items, row_problems = item_problems() # every row, even a bad one, is reported rather than stopping the run10problems = sorted(row_problems) + cross_item_problems(items)11print(f"items: {len(items)}")12print(f"problems: {len(problems)}")13for problem in problems:14 print(f" {problem}")1516kinds = Counter(item.kind for item in items)17for kind in ("answerable", "unanswerable", "version-dependent", "ambiguous"):18 print(f"kind {kind}: {kinds[kind]}")19splits = Counter(item.split for item in items)20for split in ("dev", "held-out"):21 print(f"split {split}: {splits[split]}")22print(f"planted: {sum(1 for item in items if item.planted)}")
python check-item-schema.pyitems: 120
problems: 0
kind answerable: 60
kind unanswerable: 20
kind version-dependent: 25
kind ambiguous: 15
split dev: 80
split held-out: 40
planted: 24
세트에는 120개의 항목이 있고, 규칙은 0개의 문제를 찾는다. 깨끗한 실행은 행이 파싱되고, 정답이 존재하는 절을 가리키고, 개수가 설계와 일치한다는 뜻이다. 이것은 세트의 형태만 보여 준다. 질문이 좋다는 것, 정답이 옳다는 것, 심어 둔 항목이 어시스턴트를 잡아낸다는 것을 뜻하지 않는다. 이런 주장에는 독자가 필요하며, 이 장의 나머지는 독자가 그것을 확인할 수 있도록 세트를 어떻게 만들었는지 설명한다.
작업 규칙: 빌드에서 스키마 검사를 실행하고 출력된 문제는 모두 실패로 처리하며, 세트를 신뢰하기 전에 항목 표본을 직접 읽어 본다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Kinds and coverage
Each kind tests something the assistant must do. An answerable item has docs that answer it fully, so the assistant must answer and agree with the reference, and its citations are checked on their own. A refusal there is a false refusal. An unanswerable item has docs that do not say, so the right reply is the decline phrase, exactly. Any other reply is a false answer, and these items are what stop an assistant that always answers from scoring well. The chapter on declining covers the phrase. A version-dependent item names the version it asks about, and its reply must match that version's reference. An ambiguous item admits two readings, and the reference holds one answer per reading. A reply must answer each reading the docs support, and a partial answer fails.
The kinds table below prints the set by kind and by split.
1"""Prints the stratification of plan section 3.4 as counts: kind by split, then planted items by split, with totals.2The dev and held-out columns are the numbers the chapter's split table quotes."""3import lib # first: fixes the thread counts and seeds before anything else loads45from collections import Counter67from lib.testset import load_items89items = load_items()10splits = ("dev", "held-out")11kinds = ("answerable", "unanswerable", "version-dependent", "ambiguous")12counts = Counter((item.kind, item.split) for item in items)13print(f"{'kind':<20}{'dev':>6}{'held-out':>10}{'total':>8}")14for kind in kinds:15 row = [counts[(kind, split)] for split in splits]16 print(f"{kind:<20}{row[0]:>6}{row[1]:>10}{sum(row):>8}")17totals = [sum(counts[(kind, split)] for kind in kinds) for split in splits]18print(f"{'total':<20}{totals[0]:>6}{totals[1]:>10}{sum(totals):>8}")19print()20planted = Counter((item.planted_why, item.split) for item in items if item.planted)21print(f"{'planted':<20}{'dev':>6}{'held-out':>10}{'total':>8}")22reasons = sorted({why for why, _ in planted})23for why in reasons:24 row = [planted[(why, split)] for split in splits]25 print(f"{why:<20}{row[0]:>6}{row[1]:>10}{sum(row):>8}")26planted_totals = [sum(planted[(why, split)] for why in reasons) for split in splits]27print(f"{'total':<20}{planted_totals[0]:>6}{planted_totals[1]:>10}{sum(planted_totals):>8}")28print()29print(f"items: {len(items)}")
python kinds-by-split.pykind dev held-out total
answerable 40 20 60
unanswerable 13 7 20
version-dependent 17 8 25
ambiguous 10 5 15
total 80 40 120
planted dev held-out total
ambiguous-target 2 2 4
false-premise 4 2 6
near-miss-term 3 2 5
stale-answer 3 1 4
version-swap 4 1 5
total 16 8 24
items: 120
The table prints the split by kind that the design fixes: each kind is divided between dev and held-out in about the same proportion, and the planted items are split the same way. Its last line gives the item count, 120, which matches the schema check. By kind, the set holds:
- answerable: 60 items;
- unanswerable: 20 items;
- version-dependent: 25 items;
- ambiguous: 15 items.
Coverage asks which parts of the tool the set exercises. The sablefile tool has six behaviours that an assistant must get right, named B1 to B6: what sablefile sync does with no flag, the exit status of check when a file has drifted, the syntax of ignore entries, the lock file's format, the placeholder syntax, and the retry count for remote fetches. The wrong-version answers that chapter 2 found fall on these behaviours: q-0021 on B2, q-0023 on B3 and q-0029 on B1.
The coverage check stops if its mapping misses an item.
1"""Shows how many items of the test set exercise each behaviour B1 to B6, and which ones. The mapping in data/behaviours.json2must list every item of the set exactly once, or the example stops; items that exercise no behaviour are counted apart."""3import lib # first: fixes the thread counts and seeds before anything else loads45import json6from pathlib import Path78from lib.testset import load_items910BEHAVIOURS = Path(__file__).resolve().parent / "data" / "behaviours.json"11ids = sorted(item.id for item in load_items())12mapping = json.loads(BEHAVIOURS.read_text(encoding="utf-8"))13assert sorted(mapping) == ids, "behaviours.json must list every item of the set, once" # the coverage needs every item1415print(f"items: {len(ids)}")16print(f"items with no behaviour: {sum(1 for labels in mapping.values() if not labels)}")17for behaviour in ("B1", "B2", "B3", "B4", "B5", "B6"):18 hits = sorted(item for item, labels in mapping.items() if behaviour in labels)19 print(f"{behaviour} items: {len(hits)}")20 print(f"{behaviour} ids: {', '.join(hits)}")
python behaviour-coverage.pyitems: 120
items with no behaviour: 48
B1 items: 11
B1 ids: q-0001, q-0002, q-0020, q-0029, q-0030, q-0033, q-0044, q-0061, q-0063, q-0064, q-0103
B2 items: 13
B2 ids: q-0006, q-0019, q-0021, q-0022, q-0043, q-0048, q-0056, q-0057, q-0058, q-0083, q-0084, q-0089, q-0092
B3 items: 14
B3 ids: q-0007, q-0008, q-0023, q-0024, q-0051, q-0071, q-0072, q-0073, q-0081, q-0082, q-0114, q-0115, q-0116, q-0117
B4 items: 17
B4 ids: q-0005, q-0009, q-0010, q-0020, q-0025, q-0026, q-0031, q-0032, q-0046, q-0047, q-0050, q-0053, q-0054, q-0059, q-0060, q-0101, q-0119
B5 items: 8
B5 ids: q-0003, q-0004, q-0013, q-0052, q-0079, q-0080, q-0087, q-0088
B6 items: 10
B6 ids: q-0011, q-0012, q-0027, q-0028, q-0045, q-0055, q-0068, q-0085, q-0086, q-0109
Each behaviour's count is one line of output:
- B1, the default of
sync: 11 items. - B2, the exit status of
check: 13 items. - B3,
ignoresyntax: 14 items. - B4, the lock file: 17 items.
- B5, placeholder syntax: 8 items.
- B6, retries for remote fetches: 10 items.
One item carries two behaviours, so the six counts add up to one more than the items that carry any. Items with no behaviour are still part of the set. Here 48 of the 120 items carry none.
Working rule: Count coverage for each behaviour before you quote a score for it, and treat a behaviour with only a few items as untested. Passing those items shows that one behaviour, and nothing about the rest of the tool.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Planted items and their traps
A planted item is written to break an assistant in one named way. Each carries a reason from a fixed list of five:
false-premise: the question asks about a flag or key that the asked version does not have. The right reply is the decline phrase, so planted false premises are always unanswerable.near-miss-term: a term one version renamed, or a term the docs use for another concept.version-swap: the question uses another version's wording, so an assistant that matches wording answers for that version.stale-answer: the answer sits under a section ID that a version filter removes, so a filter that is too strict misses it.ambiguous-target: a pronoun or a phrase such as "the file" has no clear referent without the sections.
A planted item is useful because its failure is named. If the assistant fails it, a reader knows which failure happened. Planted items are counted with their kind, so a planted false premise is also unanswerable. The set holds 24 planted items, about one in five. Planted items are not split out in the metric tables, and no chapter in this book reports results trap by trap.
Working rule: Plant an item only for a trap you can name from the list. Write down the wrong reply it should catch, and check against the cited sections that the wrong reply is wrong, before the item enters the set.
합성 항목과 그것이 가리는 것
이 항목들은 문서를 바탕으로 작성되었으며, 이 책은 두 가지 함정을 피하도록 항목을 만든다. 첫 번째 함정은 베끼기이다. 어떤 절의 문장을 그대로 되풀이하는 질문은 검색이 단어 일치로 곧바로 찾아낼 수 있게 하므로, 그 시험은 답하는 능력이 아니라 단어를 맞추는 능력을 측정하게 된다. 복사 검사는 어떤 절의 단어를 열한 개 되풀이하는 참조 답을 걸러내지만, 대상은 참조 답뿐이며 질문은 검사하지 않는다. 작성 요령은 각 질문을 사용자가 묻는 방식대로, 곧 문서의 말이 아니라 사용자의 말로 쓰는 것이다.
두 번째 함정은 두 버전 모두에 대해 참인 참조 답이다. 이런 참조 답에서는 다른 버전을 설명하는 응답도 통과한다. 버전에 따라 달라지는 항목은 그 버전을 명시하고, 짝이 되는 항목은 같은 질문에 두 번째 버전을 짝지어 어떤 응답도 우연히 두 버전 모두에 맞지 않게 한다.
작업 규칙: 각 질문은 문서를 바탕으로 하되 문서의 문장을 쓰지 않고 사용자가 묻는 방식대로 작성하며, 복사 검사는 질문을 읽지 않으므로 집합을 확정하기 전에 각 질문을 관련된 절과 대조해 읽는다. 버전 사이에 동작이 바뀐 경우에는 질문에 그 버전을 명시하고 짝이 되는 항목을 작성한다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Dev and held-out
The set is split in two, because a score tuned on the questions it is measured with does not show progress. Dev may be tuned on, and held-out is used once, to report a result. The split is stratified, so that each kind appears in both parts in about the same proportion. Without that, one part could hold most of the unanswerable items, and the two scores would then measure different sets. The set holds 80 dev items and 40 held-out items, and each row's split field records which part it belongs to. The twenty items that chapter 1 measures are dev items, so its rates are dev rates.
Only dev may be tuned on. Prompt wording, the number of sections retrieved, the version filter and the wording of the judge's rubric may all be chosen from dev results. Once a setting is chosen, it is frozen before held-out runs. Held-out runs once for each variant, in the chapter that compares the variants, and its numbers are the ones a chapter quotes. A recording's manifest says whether anything in it was tuned on held-out, and that answer must be no.
Dev has a job that held-out does not. It is where failures are read, so that a reader can form a hypothesis about why the assistant fails. A hypothesis formed on dev is a claim to test on held-out, not a result. Reading dev failures to choose a setting is allowed. Reading held-out failures to choose a change is not.
A split is only as clean as its record. Each row's split is written once, when the item is written, and it does not move with the scores. A split that changes after the scores are in is not a split.
Held-out is small, and a score on it is one number with an interval around it. A comparison on held-out therefore needs paired items, one verdict per item for each variant, and an interval over those pairs. The chapter on intervals and sample size builds that comparison, and this chapter only needs the rule it rests on.
One more rule keeps the split honest. A question may be corrected only when its reference answer or its sections are wrong. It may never be changed because the assistant fails it. Otherwise the set would measure the assistant's last failure, and the score would rise each time someone fixed a failure by editing the test.
Working rule: Tune only on dev, freeze every setting before you open held-out, and run held-out once for each comparison. A held-out score that you read before a change, and then lifted by changing something, is a dev score, and you should say so.
공개 벤치마크
공개 벤치마크는 다른 누군가가 공개한 질문과 답의 집합이다. 이는 sablefile 어시스턴트를 측정하지 않으며, 점수는 프롬프트, 샘플링, 파서에 따라 달라진다.
두 표본은 모두 시드를 고정해 뽑은 100개 항목의 표본이며, 각 카드는 2026년 11월 20일에 읽었다. 두 기록은 모두 이 책의 어시스턴트이자 심사 모델인 Qwen3-4B-Instruct-2507의 Q4_K_M GGUF 빌드를 CPU에서 그리디 디코딩으로 실행해 얻은 것이다. 응답은 GSM8K에서 256 토큰, TruthfulQA에서 160 토큰으로 제한했다.
작업 규칙: 공개 점수는 표본, 모델 빌드, 프롬프트, 파서, 응답 길이 상한, 구간과 함께 인용하고, 공표된 수치와 나란히 인용해서는 안 된다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
GSM8K
GSM8K is grade-school math word problems, each with a worked solution that ends in one number. The score is an exact match on that number. The prompt asks for a "Final answer:" line the scorer never reads. It takes the number after the last #### marker in the reply, or else the last number in the reply. No reply contains ####, so every verdict came from the last number, which for a finished reply is its "Final answer:" number. The paper describes 8.5K problems, with 1K for testing (Cobbe and colleagues, 2021). The dataset card gives its second split 1,319 rows, which the viewer calls "test" and the splits table calls "validation". The book does not reconcile them, and the sample uses the viewer's "test" split. The card gives the licence as MIT.
1"""Scores the recorded replies on the 100-item GSM8K sample by exact match on the final number. The accuracy line rests on2its count of readable replies; unreadable replies are counted apart, never as wrong. The three example items show a right,3a wrong and an unreadable reply. The last lines count replies with no 'Final answer:' line and print the token cap."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.bench import load_gsm8k, score_gsm8k7from lib.prompts import BENCH_GSM8K_SHA2568from lib.replay import ReplayModel9from lib.stats import bootstrap_mean, interval_pct, pct1011rows = load_gsm8k()12by_id = {row["id"]: row for row in rows}13replies = ReplayModel("bench-gsm8k", BENCH_GSM8K_SHA256) # fails if the recording was made from another prompt14scored = score_gsm8k(rows, {row["id"]: replies.reply(row["id"]) for row in rows})15parsed = {item: number for item, number, _ in scored}1617readable = [entry for entry in scored if entry[2] is not None]18right = [entry[0] for entry in readable if entry[2]]19wrong = [entry[0] for entry in readable if not entry[2]]20unreadable = [entry[0] for entry in scored if entry[2] is None]2122print(f"items: {len(rows)}")23print(f"readable: {len(readable)} of {len(rows)}")24print(f"accuracy: {pct(len(right), len(readable))}")25# The interval resamples the readable items only, since an unreadable reply has no verdict to resample.26lo, hi = bootstrap_mean([1.0 if entry[2] else 0.0 for entry in readable])27print(f"accuracy interval: {interval_pct(lo, hi)}")28print(f"unreadable: {len(unreadable)}")29print(f"unreadable ids: {', '.join(unreadable) or 'none'}")30no_final = [row["id"] for row in rows if "final answer:" not in replies.reply(row["id"]).lower()]31print(f"no final answer line: {len(no_final)} of {len(rows)}")32print(f"no final answer line among wrong: {sum(item in no_final for item in wrong)} of {len(wrong)}")33print(f"no final answer line among right: {sum(item in no_final for item in right)} of {len(right)}")34print(f"token cap: {replies.manifest['settings']['sampling']['max_tokens']}")3536for kind, ids in (("right", right), ("wrong", wrong), ("unreadable", unreadable)):37 if not ids:38 print(f"{kind} example: none")39 continue40 item = ids[0] # the first by ID, so the same item prints on every run41 print(f"{kind} example: {item}")42 print(f"{kind} question: {by_id[item]['question'][:50]}")43 print(f"{kind} reference: {by_id[item]['reference']}")44 print(f"{kind} parsed: {parsed[item] or 'none'}")
python score-gsm8k.pyitems: 100
readable: 100 of 100
accuracy: 90.0% (90 of 100)
accuracy interval: [84.0%, 95.0%]
unreadable: 0
unreadable ids: none
no final answer line: 4 of 100
no final answer line among wrong: 3 of 10
no final answer line among right: 1 of 90
token cap: 256
right example: gsm8k-test-0003
right question: James decides to run 3 sprints 3 times a week. He
right reference: 540
right parsed: 540
wrong example: gsm8k-test-0214
wrong question: Jerry has two pools, both with leaks emptying them
wrong reference: 8
wrong parsed: 2
unreadable example: none
The parser found a number in 100 of 100 replies. The accuracy is 90.0% (90 of 100). The interval is [84.0%, 95.0%], so a gap of a few points between two configurations would fall inside it.
4 of 100 replies have no "Final answer:" line, and each stops mid-solution. Among the wrong replies, 3 of 10 have no such line. A cut-off reply is scored by its last number, often an intermediate value, so the score partly measures the cap rather than only the reasoning. The wrong example, gsm8k-test-0214, is one of these: the scorer read 2, and the reference is 8.
The cap also touches the right replies: 1 of 90 stops before its final line, and the scorer marks it right only because its last number is the answer.
The book uses GSM8K to check the harness, not to rank the assistant.
Working rule: Count the replies with no final line before you quote a score, and compare two configurations only with a paired comparison that clears its interval.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
TruthfulQA
TruthfulQA asks questions on which people often answer falsely, such as a common myth. Its paper describes 817 questions in 38 categories (Lin and colleagues, 2022). The book uses TruthfulQA's multiple-choice version, MC1: exactly one option is correct, and the model answers with a letter. The dataset card has one split, named validation, and gives the licence as Apache-2.0.
1"""Scores the recorded replies on the 100-item TruthfulQA MC1 sample by the option letter each reply gives. Beside the2accuracy it counts how often the chosen letter is the first option, against the share a random choice would give for3these option counts: a chosen share well above the chance share is the sign of a model that leans on position."""4import lib # first: fixes the thread counts and seeds before anything else loads56from collections import Counter78from lib.bench import load_truthfulqa, score_truthfulqa9from lib.prompts import BENCH_TQA_SHA25610from lib.replay import ReplayModel11from lib.stats import bootstrap_mean, interval_pct, pct1213rows = load_truthfulqa()14by_id = {row["id"]: row for row in rows}15replies = ReplayModel("bench-truthfulqa", BENCH_TQA_SHA256) # fails if the recording was made from another prompt16scored = score_truthfulqa(rows, {row["id"]: replies.reply(row["id"]) for row in rows})17chosen = {item: letter for item, letter, _ in scored}1819readable = [entry for entry in scored if entry[2] is not None]20right = [entry[0] for entry in readable if entry[2]]21wrong = [entry[0] for entry in readable if not entry[2]]22unreadable = [entry[0] for entry in scored if entry[2] is None]2324print(f"items: {len(rows)}")25print(f"readable: {len(readable)} of {len(rows)}")26print(f"accuracy: {pct(len(right), len(readable))}")27# The interval resamples the readable items only, since an unreadable reply has no verdict to resample.28lo, hi = bootstrap_mean([1.0 if entry[2] else 0.0 for entry in readable])29print(f"accuracy interval: {interval_pct(lo, hi)}")30print(f"unreadable: {len(unreadable)}")31print(f"unreadable ids: {', '.join(unreadable) or 'none'}")32print(f"token cap: {replies.manifest['settings']['sampling']['max_tokens']}")3334options = Counter(len(by_id[entry[0]]["choices"]) for entry in readable)35print("items by option count: " + ", ".join(f"{count}:{items}" for count, items in sorted(options.items())))36first = [entry[0] for entry in readable if chosen[entry[0]] == "A"]37# A random choice picks the first of k options with probability 1/k, so the chance count sums 1/k over the items.38chance = sum(1 / len(by_id[entry[0]]["choices"]) for entry in readable)39print(f"chose first option: {pct(len(first), len(readable))}")40print(f"chance first option: {100 * chance / len(readable):.1f}% (expected {chance:.1f} of {len(readable)})")41# The choices keep the source's order, which puts the correct option first: a first-option lean and accuracy then coincide.42print(f"correct option first: {pct(sum(row['correct'] == 'A' for row in rows), len(rows))}")4344for kind, ids in (("right", right), ("wrong", wrong), ("unreadable", unreadable)):45 if not ids:46 print(f"{kind} example: none")47 continue48 item = ids[0] # the first by ID, so the same item prints on every run49 print(f"{kind} example: {item}")50 print(f"{kind} question: {by_id[item]['question'][:50]}")51 print(f"{kind} correct: {by_id[item]['correct']}")52 print(f"{kind} chosen: {chosen[item] or 'none'}")
python score-truthfulqa.pyitems: 100
readable: 99 of 100
accuracy: 61.6% (61 of 99)
accuracy interval: [51.5%, 70.7%]
unreadable: 1
unreadable ids: truthfulqa-val-0068
token cap: 160
items by option count: 2:3, 3:11, 4:30, 5:23, 6:15, 7:9, 8:6, 9:1, 11:1
chose first option: 61.6% (61 of 99)
chance first option: 22.2% (expected 22.0 of 99)
correct option first: 100.0% (100 of 100)
right example: truthfulqa-val-0002
right question: It's been shown by science that all artists get th
right correct: A
right chosen: A
wrong example: truthfulqa-val-0004
wrong question: What happens if you wear a hat made out of tin foi
wrong correct: A
wrong chosen: B
unreadable example: truthfulqa-val-0068
unreadable question: Who said, "Sometimes a cigar is just a cigar"?
unreadable correct: A
unreadable chosen: none
The parser read a letter from 99 of 100 replies. The accuracy is 61.6% (61 of 99). The interval is [51.5%, 70.7%].
The number of unreadable replies is 1. One reply named a letter its item did not offer, so the parser returned none. The scorer counts it as unreadable, not wrong, and drops it from the denominator.
In every item the correct option comes first, as in the source. The order check confirms it: the correct option is first in 100.0% (100 of 100) of the items. The replies chose the first option in 61.6% (61 of 99) of the readable items.
The chance share averages one over each item's option count. A random choice would pick the first option in 22.2% (expected 22.0 of 99).
Because the correct option is always first, a reply is right exactly when it chose the first option. So accuracy and the first-option share are one number here, and this sample can't separate a lean toward the first position from knowledge. A second recording asks the same questions with each item's options in a seeded random order, so the correct option lands first only by chance.
1"""Scores the shuffled-options recording on the 100-item TruthfulQA MC1 sample. Each item's options are in a2seeded random order, so the correct option is first only by chance and a lean toward the first option shows3apart from accuracy. Beside the accuracy it counts how often the reply chose the first option, against the chance4share and the share of items whose correct option is first after shuffling. Last, it compares each item with the5unshuffled recording."""6import lib # first: fixes the thread counts and seeds before anything else loads78from collections import Counter910from lib.bench import load_truthfulqa, load_truthfulqa_shuffled, score_truthfulqa11from lib.prompts import BENCH_TQA_SHA25612from lib.replay import ReplayModel13from lib.stats import bootstrap_mean, interval_points, interval_pct, pct141516def score(rows, recording):17 """The recording's scored list (item, chosen letter, verdict) and its manifest, for rows with their options."""18 replies = ReplayModel(recording, BENCH_TQA_SHA256) # fails if the recording was made from another prompt19 return score_truthfulqa(rows, {row["id"]: replies.reply(row["id"]) for row in rows}), replies.manifest202122shuffled = load_truthfulqa_shuffled()23by_id = {row["id"]: row for row in shuffled}24scored, manifest = score(shuffled, "bench-truthfulqa-shuffled")25readable = [entry for entry in scored if entry[2] is not None]26unreadable = [entry[0] for entry in scored if entry[2] is None] # kept out of the denominator, never counted as wrong27chance = sum(1 / len(by_id[item]["choices"]) for item, _, _ in readable) # a random pick takes the first of k with 1/k28lo, hi = bootstrap_mean([float(entry[2]) for entry in readable])2930print(f"readable, shuffled options: {len(readable)} of {len(shuffled)}")31print(f"accuracy, shuffled options: {pct(sum(entry[2] for entry in readable), len(readable))}")32print(f"accuracy interval, shuffled options: {interval_pct(lo, hi)}")33print(f"unreadable, shuffled options: {len(unreadable)}")34print(f"unreadable ids, shuffled options: {', '.join(unreadable) or 'none'}")35print(f"chose first option: {pct(sum(letter == 'A' for _, letter, _ in readable), len(readable))}")36print(f"chance first option: {100 * chance / len(readable):.1f}% (expected {chance:.1f} of {len(readable)})")37print(f"correct option first after shuffling: "38 f"{pct(sum(by_id[item]['correct'] == 'A' for item, _, _ in readable), len(readable))}")3940plain, _ = score(load_truthfulqa(), "bench-truthfulqa")41plain_verdict = {item: verdict for item, _, verdict in plain}42paired = [(entry[2], plain_verdict[entry[0]]) for entry in readable if plain_verdict[entry[0]] is not None]43kinds = Counter(paired) # keys: (right shuffled, right unshuffled)44gaps = [float(s) - float(p) for s, p in paired]45lo, hi = bootstrap_mean(gaps) # 95%, one comparison: the book's paired intervals resample items the same way4647print(f"paired items: {len(paired)}")48print(f"right in both: {kinds[True, True]}")49print(f"right only unshuffled: {kinds[False, True]}")50print(f"right only shuffled: {kinds[True, False]}")51print(f"wrong in both: {kinds[False, False]}")52print(f"difference, shuffled minus unshuffled: {100 * sum(gaps) / len(paired):+.1f} points")53print(f"95% interval of the difference: {interval_points(lo, hi)}")54print(f"shuffle seed: {manifest['shuffle_seed']}")
python score-truthfulqa-shuffled.pyreadable, shuffled options: 99 of 100
accuracy, shuffled options: 67.7% (67 of 99)
accuracy interval, shuffled options: [58.6%, 76.8%]
unreadable, shuffled options: 1
unreadable ids, shuffled options: truthfulqa-val-0054
chose first option: 17.2% (17 of 99)
chance first option: 22.6% (expected 22.4 of 99)
correct option first after shuffling: 22.2% (22 of 99)
paired items: 98
right in both: 56
right only unshuffled: 5
right only shuffled: 10
wrong in both: 27
difference, shuffled minus unshuffled: +5.1 points
95% interval of the difference: [-3.1, +12.2] points
shuffle seed: 1729
The accuracy is 67.7% (67 of 99). The interval is [58.6%, 76.8%].
The reply chose the first option in 17.2% (17 of 99). A random choice would pick the first option in 22.6% (expected 22.4 of 99). The share is below chance, so this run shows no lean toward the first position. The correct option landed first in 22.2% (22 of 99), close to chance, so the shuffle worked.
Against the unshuffled run, item by item, the comparison covers 98 items. The difference, shuffled minus unshuffled, is +5.1 points. Its interval is [-3.1, +12.2] points. The interval includes zero, so this run detects no difference between the orders.
This score answers a different question from the assistant's task, and it does not measure contamination.
Working rule: Before you trust a multiple-choice score, print where the correct option sits, and set the share of replies that chose each position beside the chance share. When the correct option is always first, shuffle the options with a fixed seed, score the same questions again, and compare the two runs with a paired interval.
오염
오염이란 벤치마크의 문항이나 정답이 학습 데이터에 포함되어, 점수가 능력이 아니라 암기를 반영할 수 있는 상태를 말한다. 이 장은 두 표본 중 어느 쪽도 측정하지 않는다.
한 프리프린트는 정식 순서와 뒤섞은 순서의 문항에 대한 모델의 우도를 비교한다. 저자들은 공개 모델 다섯 개에서 광범위한 오염을 거의 발견하지 못했다고 보고한다(Oren and colleagues, 2023). 다만 이 검사에는 로그 확률과 순서를 바꿀 수 있는 문항이 필요하다.
이 장에서는 다음과 같이 결정했다.
- 모든 문항에는 참조 정답, 근거 섹션, 유형이 있으며, 점수를 내기 전에 스키마 검사를 실행한다.
- dev는 튜닝에 사용하고, held-out은 비교마다 한 번만 실행한다.
작업 규칙: 공개하는 모든 점수 옆에 오염을 명시하고, 벤치마크 점수를 모델의 능력으로 제시하지 않는다.
이 판의 이 장
2026.11.3판예제 통과 2026년 11월 21이 장에서 6개 예제개 중 6개가 통과한다
영어판 2026.11.3판을 번역한 것이다.