建立一組站得住腳的測試集,並解讀公開基準測試
閱讀約需 59 分鐘 分鐘,10 節 中共有 6 個範例。
讀完本章後,你能建立一組分數有意義的測試集,用程式碼檢查它的結構(schema)與涵蓋範圍,並說明公開基準測試的分數能告訴你、又不能告訴你自己的助理哪些事。你也會看到,選擇題的分數為何可能與「習慣用同一種方式作答」無從區分。
這裡的測試集屬於 sablefile 助理:它由一個檢索步驟與一個模型組成,根據該工具的文件回答問題。本書對這個助理報告的每一個數字,都來自一組附有參考答案的問題,因此這組問題決定了分數的價值。本書要防範三種缺陷。若每個問題在文件中都有答案,一個永遠作答的助理看起來就會表現良好。若問題沿用文件本身的句子,檢索只是比對字詞,也會看起來正確。若測試集被反覆調整到分數上升為止,那個分數記錄的就是調整的過程。此外,測試集是這些問題唯一被寫下來的地方,其中的缺陷可能不被察覺,建立在它之上的分數也會承襲這個缺陷,而且毫無徵兆。
本章建立本書使用的測試集,並用程式碼加以檢查。接著解讀兩個公開基準測試:GSM8K 與 TruthfulQA,許多讀者想必看過有人引用它們的分數;本章也說明這些分數仍能顯示什麼。各節依團隊會依序提出的問題安排:測試集的用途為何、如何建立與檢查、如何切分,以及公開分數能補充什麼。
此部分尚未翻譯,因此以英文呈現。
What a test set is for
A test set is a fixed set of questions, paired with the answers you would accept, that stands in for the questions users will ask. Its job is to be asked the same questions each time something changes, so that a difference in score comes from the change and not from the questions. It does that job only while it stays fixed. This set was written from the docs rather than sampled from users, so it stands in for them and does not reproduce them.
Each row is an item: one test question with the fields a scorer needs. The item is the unit of every count in this book, so a pass rate is always a share of items, and its denominator says which ones. Each item has a reference answer, which the scorer compares a reply with, and supporting sections, the parts of the docs that a correct answer rests on. The reference answers are written labels. The book's own drafting agents wrote them from the docs, a second pass checked them against the docs, and a script checks the file's shape and section IDs. They are not human labels, and the book does not call them that.
Each item also has a kind, which says what a correct reply looks like. A set built from one kind tests one skill, so this set mixes four kinds on purpose. It should also look like the questions users bring. Users ask things the docs cannot answer, and a set with only answerable questions would reward an assistant that never declines.
The set is small on purpose. It holds 120 items, which a reader can check by hand, and each one has a reference answer that a second pass checked against the docs. We keep it small enough to read, because a set nobody reads can only measure its own mistakes. A score is only as good as the set behind it, and the set is what two people argue about when they disagree on whether a change helped. Keep it readable, keep the reasons for each item written down, and do not add items only because a score looks wrong, since that picks items for their result.
A test set is never training data. Nothing in it teaches the assistant, and the assistant never sees a reference answer while the set is being built. If someone tunes against the answers, the set stops being a test and becomes a target, which is what the rule on dev and held-out is for.
Working rule: Write the set before you change anything you will measure with it, and keep its file under version control, so that every comparison names the items it used. A set you edit after seeing the scores measures your edits, not the assistant.
項目結構描述
在任何評分之前,測試集必須先能載入。檔案每行存放一個 JSON 物件,載入器以 pydantic 逐一驗證;pydantic 是一個 Python 函式庫,用來檢查數值是否符合宣告的結構。本書固定使用 pydantic 2.14.3。欄位如下:
id:q-加四位數字,不可重複,寫入後不得更動。question:1 到 400 個純文字字元。version:該問題所詢問的文件版本。kind:四種類型之一。reference_answer:標準答案;若為無法回答的項目則留空,此時正確的回應是拒答。supporting_sections:答案所依據的版本與章節 ID 配對。split:dev或held-out。planted與planted_why:該項目是否為預先設置的陷阱,以及是哪一種陷阱。twin_of:針對另一個版本提出相同問題的項目之 ID(若有的話)。
載入器設定了 extra="forbid",因此未知欄位會被視為錯誤,而不是被默默忽略。否則欄位名稱中的一個拼字錯誤,就會產生一份看起來正確、評分卻有誤的測試集。錯誤會逐列回報,所以一列有問題不會掩蓋其餘各列。
結構檢查無法跨列比對,因此本書 lib/testset.py 中的跨項目規則負責這部分工作。這些規則檢查 ID 是否唯一,每個孿生項目是否回頭指向它的對應項目、是否詢問另一個版本、是否與版本相關,且位於同一個分割中,以及每個錯誤前提項目是否皆為無法回答。它們也檢查每個依據的節在其指名的版本中是否存在,沒有任何標準答案連續重複某一節中的十一個單字,以及各項計數是否符合設計。
這些規則各是一份契約,任何新增或編輯項目的人都必須通過。檢查會印出每一條被違反的規則。無論是否發現問題,它結束時都以成功(零)狀態碼結束,因此它本身不會阻止任何事,建置流程必須把印出的問題視為失敗。
標準答案最需要謹慎,因為每個回應都要與它比對。一個朝固定方向出錯的標準答案,會使分數朝同一方向出錯,而任何結構規則都抓不到格式正確但內容錯誤的答案。這就是每份標準答案都連同其依據的節一起撰寫的原因,讀者可以在幾分鐘內對照文件查核該主張,而不必只憑作者的說法。
孿生項目若在不同時間撰寫,可能漂移到不同的分割,結果 dev 部分用來調校的問題,卻是 held-out 部分用來檢驗的問題。跨項目規則能抓到這種情況,因此孿生項目會維持在一起。
下方的結構檢查會執行兩組規則,並印出問題數量,以及依類型與依分割統計的數量。
1"""Loads all 120 test-set items, runs the per-item and the cross-item rules, and prints the problem count with the2counts by kind and by split. The point rests on the first line: a set that passes prints zero problems."""3import lib # first: fixes the thread counts and seeds before anything else loads45from collections import Counter67from lib.testset import cross_item_problems, item_problems89items, row_problems = item_problems() # every row, even a bad one, is reported rather than stopping the run10problems = sorted(row_problems) + cross_item_problems(items)11print(f"items: {len(items)}")12print(f"problems: {len(problems)}")13for problem in problems:14 print(f" {problem}")1516kinds = Counter(item.kind for item in items)17for kind in ("answerable", "unanswerable", "version-dependent", "ambiguous"):18 print(f"kind {kind}: {kinds[kind]}")19splits = Counter(item.split for item in items)20for split in ("dev", "held-out"):21 print(f"split {split}: {splits[split]}")22print(f"planted: {sum(1 for item in items if item.planted)}")
python check-item-schema.pyitems: 120
problems: 0
kind answerable: 60
kind unanswerable: 20
kind version-dependent: 25
kind ambiguous: 15
split dev: 80
split held-out: 40
planted: 24
測試集共有 120 個項目,規則找到 0 個問題。乾淨的執行結果表示各列都能解析、標準答案指向存在的節,且計數符合設計。這只說明測試集的結構。它並不表示問題出得好、標準答案正確,或預先設置的項目能抓出助理的錯誤。這些主張需要由人來讀,本章其餘部分說明測試集是如何建構的,好讓讀者能自行查核。
實用原則: 在建置流程中執行結構檢查,並將任何印出的問題視為失敗;在信任測試集之前,先以人工閱讀一部分項目的樣本。
此部分尚未翻譯,因此以英文呈現。
Kinds and coverage
Each kind tests something the assistant must do. An answerable item has docs that answer it fully, so the assistant must answer and agree with the reference, and its citations are checked on their own. A refusal there is a false refusal. An unanswerable item has docs that do not say, so the right reply is the decline phrase, exactly. Any other reply is a false answer, and these items are what stop an assistant that always answers from scoring well. The chapter on declining covers the phrase. A version-dependent item names the version it asks about, and its reply must match that version's reference. An ambiguous item admits two readings, and the reference holds one answer per reading. A reply must answer each reading the docs support, and a partial answer fails.
The kinds table below prints the set by kind and by split.
1"""Prints the stratification of plan section 3.4 as counts: kind by split, then planted items by split, with totals.2The dev and held-out columns are the numbers the chapter's split table quotes."""3import lib # first: fixes the thread counts and seeds before anything else loads45from collections import Counter67from lib.testset import load_items89items = load_items()10splits = ("dev", "held-out")11kinds = ("answerable", "unanswerable", "version-dependent", "ambiguous")12counts = Counter((item.kind, item.split) for item in items)13print(f"{'kind':<20}{'dev':>6}{'held-out':>10}{'total':>8}")14for kind in kinds:15 row = [counts[(kind, split)] for split in splits]16 print(f"{kind:<20}{row[0]:>6}{row[1]:>10}{sum(row):>8}")17totals = [sum(counts[(kind, split)] for kind in kinds) for split in splits]18print(f"{'total':<20}{totals[0]:>6}{totals[1]:>10}{sum(totals):>8}")19print()20planted = Counter((item.planted_why, item.split) for item in items if item.planted)21print(f"{'planted':<20}{'dev':>6}{'held-out':>10}{'total':>8}")22reasons = sorted({why for why, _ in planted})23for why in reasons:24 row = [planted[(why, split)] for split in splits]25 print(f"{why:<20}{row[0]:>6}{row[1]:>10}{sum(row):>8}")26planted_totals = [sum(planted[(why, split)] for why in reasons) for split in splits]27print(f"{'total':<20}{planted_totals[0]:>6}{planted_totals[1]:>10}{sum(planted_totals):>8}")28print()29print(f"items: {len(items)}")
python kinds-by-split.pykind dev held-out total
answerable 40 20 60
unanswerable 13 7 20
version-dependent 17 8 25
ambiguous 10 5 15
total 80 40 120
planted dev held-out total
ambiguous-target 2 2 4
false-premise 4 2 6
near-miss-term 3 2 5
stale-answer 3 1 4
version-swap 4 1 5
total 16 8 24
items: 120
The table prints the split by kind that the design fixes: each kind is divided between dev and held-out in about the same proportion, and the planted items are split the same way. Its last line gives the item count, 120, which matches the schema check. By kind, the set holds:
- answerable: 60 items;
- unanswerable: 20 items;
- version-dependent: 25 items;
- ambiguous: 15 items.
Coverage asks which parts of the tool the set exercises. The sablefile tool has six behaviours that an assistant must get right, named B1 to B6: what sablefile sync does with no flag, the exit status of check when a file has drifted, the syntax of ignore entries, the lock file's format, the placeholder syntax, and the retry count for remote fetches. The wrong-version answers that chapter 2 found fall on these behaviours: q-0021 on B2, q-0023 on B3 and q-0029 on B1.
The coverage check stops if its mapping misses an item.
1"""Shows how many items of the test set exercise each behaviour B1 to B6, and which ones. The mapping in data/behaviours.json2must list every item of the set exactly once, or the example stops; items that exercise no behaviour are counted apart."""3import lib # first: fixes the thread counts and seeds before anything else loads45import json6from pathlib import Path78from lib.testset import load_items910BEHAVIOURS = Path(__file__).resolve().parent / "data" / "behaviours.json"11ids = sorted(item.id for item in load_items())12mapping = json.loads(BEHAVIOURS.read_text(encoding="utf-8"))13assert sorted(mapping) == ids, "behaviours.json must list every item of the set, once" # the coverage needs every item1415print(f"items: {len(ids)}")16print(f"items with no behaviour: {sum(1 for labels in mapping.values() if not labels)}")17for behaviour in ("B1", "B2", "B3", "B4", "B5", "B6"):18 hits = sorted(item for item, labels in mapping.items() if behaviour in labels)19 print(f"{behaviour} items: {len(hits)}")20 print(f"{behaviour} ids: {', '.join(hits)}")
python behaviour-coverage.pyitems: 120
items with no behaviour: 48
B1 items: 11
B1 ids: q-0001, q-0002, q-0020, q-0029, q-0030, q-0033, q-0044, q-0061, q-0063, q-0064, q-0103
B2 items: 13
B2 ids: q-0006, q-0019, q-0021, q-0022, q-0043, q-0048, q-0056, q-0057, q-0058, q-0083, q-0084, q-0089, q-0092
B3 items: 14
B3 ids: q-0007, q-0008, q-0023, q-0024, q-0051, q-0071, q-0072, q-0073, q-0081, q-0082, q-0114, q-0115, q-0116, q-0117
B4 items: 17
B4 ids: q-0005, q-0009, q-0010, q-0020, q-0025, q-0026, q-0031, q-0032, q-0046, q-0047, q-0050, q-0053, q-0054, q-0059, q-0060, q-0101, q-0119
B5 items: 8
B5 ids: q-0003, q-0004, q-0013, q-0052, q-0079, q-0080, q-0087, q-0088
B6 items: 10
B6 ids: q-0011, q-0012, q-0027, q-0028, q-0045, q-0055, q-0068, q-0085, q-0086, q-0109
Each behaviour's count is one line of output:
- B1, the default of
sync: 11 items. - B2, the exit status of
check: 13 items. - B3,
ignoresyntax: 14 items. - B4, the lock file: 17 items.
- B5, placeholder syntax: 8 items.
- B6, retries for remote fetches: 10 items.
One item carries two behaviours, so the six counts add up to one more than the items that carry any. Items with no behaviour are still part of the set. Here 48 of the 120 items carry none.
Working rule: Count coverage for each behaviour before you quote a score for it, and treat a behaviour with only a few items as untested. Passing those items shows that one behaviour, and nothing about the rest of the tool.
此部分尚未翻譯,因此以英文呈現。
Planted items and their traps
A planted item is written to break an assistant in one named way. Each carries a reason from a fixed list of five:
false-premise: the question asks about a flag or key that the asked version does not have. The right reply is the decline phrase, so planted false premises are always unanswerable.near-miss-term: a term one version renamed, or a term the docs use for another concept.version-swap: the question uses another version's wording, so an assistant that matches wording answers for that version.stale-answer: the answer sits under a section ID that a version filter removes, so a filter that is too strict misses it.ambiguous-target: a pronoun or a phrase such as "the file" has no clear referent without the sections.
A planted item is useful because its failure is named. If the assistant fails it, a reader knows which failure happened. Planted items are counted with their kind, so a planted false premise is also unanswerable. The set holds 24 planted items, about one in five. Planted items are not split out in the metric tables, and no chapter in this book reports results trap by trap.
Working rule: Plant an item only for a trap you can name from the list. Write down the wrong reply it should catch, and check against the cited sections that the wrong reply is wrong, before the item enters the set.
合成題目及其隱藏的問題
這些題目是依據文件撰寫的,本書針對兩個陷阱來建構它們。第一個是照抄。若問題重複了某一節的句子,檢索便會立刻在字詞上找到相符之處,測試所衡量的就成了字詞比對,而不是回答能力。抄襲檢查會擋下與某一節重複十一個字詞的參考答案,但它只涵蓋參考答案,不涵蓋問題。撰寫時的實用原則是:以使用者的方式來提問,用使用者的字詞,而不是文件的字詞。
第二個陷阱是參考答案對兩個版本都成立,以致描述另一版本的回覆也能通過。與版本相關的題目要指明其版本,而它的孿生題則以同一個問題搭配第二個版本,如此一來,沒有任何回覆能碰巧同時適用於兩者。
實用原則: 以使用者的方式提問,內容出自文件,但不沿用其句子;在題組凍結之前,對照相關各節逐題檢視問題,因為抄襲檢查不會讀問題。對於在不同版本之間有所改變的行為,要在問題中指明版本,並寫出它的孿生題。
此部分尚未翻譯,因此以英文呈現。
Dev and held-out
The set is split in two, because a score tuned on the questions it is measured with does not show progress. Dev may be tuned on, and held-out is used once, to report a result. The split is stratified, so that each kind appears in both parts in about the same proportion. Without that, one part could hold most of the unanswerable items, and the two scores would then measure different sets. The set holds 80 dev items and 40 held-out items, and each row's split field records which part it belongs to. The twenty items that chapter 1 measures are dev items, so its rates are dev rates.
Only dev may be tuned on. Prompt wording, the number of sections retrieved, the version filter and the wording of the judge's rubric may all be chosen from dev results. Once a setting is chosen, it is frozen before held-out runs. Held-out runs once for each variant, in the chapter that compares the variants, and its numbers are the ones a chapter quotes. A recording's manifest says whether anything in it was tuned on held-out, and that answer must be no.
Dev has a job that held-out does not. It is where failures are read, so that a reader can form a hypothesis about why the assistant fails. A hypothesis formed on dev is a claim to test on held-out, not a result. Reading dev failures to choose a setting is allowed. Reading held-out failures to choose a change is not.
A split is only as clean as its record. Each row's split is written once, when the item is written, and it does not move with the scores. A split that changes after the scores are in is not a split.
Held-out is small, and a score on it is one number with an interval around it. A comparison on held-out therefore needs paired items, one verdict per item for each variant, and an interval over those pairs. The chapter on intervals and sample size builds that comparison, and this chapter only needs the rule it rests on.
One more rule keeps the split honest. A question may be corrected only when its reference answer or its sections are wrong. It may never be changed because the assistant fails it. Otherwise the set would measure the assistant's last failure, and the score would rise each time someone fixed a failure by editing the test.
Working rule: Tune only on dev, freeze every setting before you open held-out, and run held-out once for each comparison. A held-out score that you read before a change, and then lifted by changing something, is a dev score, and you should say so.
公開基準測試
公開基準測試是由他人發布的一組問題與答案。它並不衡量 sablefile 助理,其分數取決於提示、取樣方式與解析器。
兩份樣本皆為以固定種子抽取的一百個項目,其資料卡於 2026 年 11 月 20 日閱讀。兩份紀錄皆來自 Qwen3-4B-Instruct-2507,即本書的助理與評審,使用其 Q4_K_M GGUF 建置版本,在 CPU 上以貪婪解碼執行。GSM8K 的回覆上限為 256 個 token,TruthfulQA 則為 160 個。
實用原則: 引用公開分數時,必須同時附上其樣本、模型建置版本、提示、解析器、回覆上限與區間,且絕不與已發布的數字並列。
此部分尚未翻譯,因此以英文呈現。
GSM8K
GSM8K is grade-school math word problems, each with a worked solution that ends in one number. The score is an exact match on that number. The prompt asks for a "Final answer:" line the scorer never reads. It takes the number after the last #### marker in the reply, or else the last number in the reply. No reply contains ####, so every verdict came from the last number, which for a finished reply is its "Final answer:" number. The paper describes 8.5K problems, with 1K for testing (Cobbe and colleagues, 2021). The dataset card gives its second split 1,319 rows, which the viewer calls "test" and the splits table calls "validation". The book does not reconcile them, and the sample uses the viewer's "test" split. The card gives the licence as MIT.
1"""Scores the recorded replies on the 100-item GSM8K sample by exact match on the final number. The accuracy line rests on2its count of readable replies; unreadable replies are counted apart, never as wrong. The three example items show a right,3a wrong and an unreadable reply. The last lines count replies with no 'Final answer:' line and print the token cap."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.bench import load_gsm8k, score_gsm8k7from lib.prompts import BENCH_GSM8K_SHA2568from lib.replay import ReplayModel9from lib.stats import bootstrap_mean, interval_pct, pct1011rows = load_gsm8k()12by_id = {row["id"]: row for row in rows}13replies = ReplayModel("bench-gsm8k", BENCH_GSM8K_SHA256) # fails if the recording was made from another prompt14scored = score_gsm8k(rows, {row["id"]: replies.reply(row["id"]) for row in rows})15parsed = {item: number for item, number, _ in scored}1617readable = [entry for entry in scored if entry[2] is not None]18right = [entry[0] for entry in readable if entry[2]]19wrong = [entry[0] for entry in readable if not entry[2]]20unreadable = [entry[0] for entry in scored if entry[2] is None]2122print(f"items: {len(rows)}")23print(f"readable: {len(readable)} of {len(rows)}")24print(f"accuracy: {pct(len(right), len(readable))}")25# The interval resamples the readable items only, since an unreadable reply has no verdict to resample.26lo, hi = bootstrap_mean([1.0 if entry[2] else 0.0 for entry in readable])27print(f"accuracy interval: {interval_pct(lo, hi)}")28print(f"unreadable: {len(unreadable)}")29print(f"unreadable ids: {', '.join(unreadable) or 'none'}")30no_final = [row["id"] for row in rows if "final answer:" not in replies.reply(row["id"]).lower()]31print(f"no final answer line: {len(no_final)} of {len(rows)}")32print(f"no final answer line among wrong: {sum(item in no_final for item in wrong)} of {len(wrong)}")33print(f"no final answer line among right: {sum(item in no_final for item in right)} of {len(right)}")34print(f"token cap: {replies.manifest['settings']['sampling']['max_tokens']}")3536for kind, ids in (("right", right), ("wrong", wrong), ("unreadable", unreadable)):37 if not ids:38 print(f"{kind} example: none")39 continue40 item = ids[0] # the first by ID, so the same item prints on every run41 print(f"{kind} example: {item}")42 print(f"{kind} question: {by_id[item]['question'][:50]}")43 print(f"{kind} reference: {by_id[item]['reference']}")44 print(f"{kind} parsed: {parsed[item] or 'none'}")
python score-gsm8k.pyitems: 100
readable: 100 of 100
accuracy: 90.0% (90 of 100)
accuracy interval: [84.0%, 95.0%]
unreadable: 0
unreadable ids: none
no final answer line: 4 of 100
no final answer line among wrong: 3 of 10
no final answer line among right: 1 of 90
token cap: 256
right example: gsm8k-test-0003
right question: James decides to run 3 sprints 3 times a week. He
right reference: 540
right parsed: 540
wrong example: gsm8k-test-0214
wrong question: Jerry has two pools, both with leaks emptying them
wrong reference: 8
wrong parsed: 2
unreadable example: none
The parser found a number in 100 of 100 replies. The accuracy is 90.0% (90 of 100). The interval is [84.0%, 95.0%], so a gap of a few points between two configurations would fall inside it.
4 of 100 replies have no "Final answer:" line, and each stops mid-solution. Among the wrong replies, 3 of 10 have no such line. A cut-off reply is scored by its last number, often an intermediate value, so the score partly measures the cap rather than only the reasoning. The wrong example, gsm8k-test-0214, is one of these: the scorer read 2, and the reference is 8.
The cap also touches the right replies: 1 of 90 stops before its final line, and the scorer marks it right only because its last number is the answer.
The book uses GSM8K to check the harness, not to rank the assistant.
Working rule: Count the replies with no final line before you quote a score, and compare two configurations only with a paired comparison that clears its interval.
此部分尚未翻譯,因此以英文呈現。
TruthfulQA
TruthfulQA asks questions on which people often answer falsely, such as a common myth. Its paper describes 817 questions in 38 categories (Lin and colleagues, 2022). The book uses TruthfulQA's multiple-choice version, MC1: exactly one option is correct, and the model answers with a letter. The dataset card has one split, named validation, and gives the licence as Apache-2.0.
1"""Scores the recorded replies on the 100-item TruthfulQA MC1 sample by the option letter each reply gives. Beside the2accuracy it counts how often the chosen letter is the first option, against the share a random choice would give for3these option counts: a chosen share well above the chance share is the sign of a model that leans on position."""4import lib # first: fixes the thread counts and seeds before anything else loads56from collections import Counter78from lib.bench import load_truthfulqa, score_truthfulqa9from lib.prompts import BENCH_TQA_SHA25610from lib.replay import ReplayModel11from lib.stats import bootstrap_mean, interval_pct, pct1213rows = load_truthfulqa()14by_id = {row["id"]: row for row in rows}15replies = ReplayModel("bench-truthfulqa", BENCH_TQA_SHA256) # fails if the recording was made from another prompt16scored = score_truthfulqa(rows, {row["id"]: replies.reply(row["id"]) for row in rows})17chosen = {item: letter for item, letter, _ in scored}1819readable = [entry for entry in scored if entry[2] is not None]20right = [entry[0] for entry in readable if entry[2]]21wrong = [entry[0] for entry in readable if not entry[2]]22unreadable = [entry[0] for entry in scored if entry[2] is None]2324print(f"items: {len(rows)}")25print(f"readable: {len(readable)} of {len(rows)}")26print(f"accuracy: {pct(len(right), len(readable))}")27# The interval resamples the readable items only, since an unreadable reply has no verdict to resample.28lo, hi = bootstrap_mean([1.0 if entry[2] else 0.0 for entry in readable])29print(f"accuracy interval: {interval_pct(lo, hi)}")30print(f"unreadable: {len(unreadable)}")31print(f"unreadable ids: {', '.join(unreadable) or 'none'}")32print(f"token cap: {replies.manifest['settings']['sampling']['max_tokens']}")3334options = Counter(len(by_id[entry[0]]["choices"]) for entry in readable)35print("items by option count: " + ", ".join(f"{count}:{items}" for count, items in sorted(options.items())))36first = [entry[0] for entry in readable if chosen[entry[0]] == "A"]37# A random choice picks the first of k options with probability 1/k, so the chance count sums 1/k over the items.38chance = sum(1 / len(by_id[entry[0]]["choices"]) for entry in readable)39print(f"chose first option: {pct(len(first), len(readable))}")40print(f"chance first option: {100 * chance / len(readable):.1f}% (expected {chance:.1f} of {len(readable)})")41# The choices keep the source's order, which puts the correct option first: a first-option lean and accuracy then coincide.42print(f"correct option first: {pct(sum(row['correct'] == 'A' for row in rows), len(rows))}")4344for kind, ids in (("right", right), ("wrong", wrong), ("unreadable", unreadable)):45 if not ids:46 print(f"{kind} example: none")47 continue48 item = ids[0] # the first by ID, so the same item prints on every run49 print(f"{kind} example: {item}")50 print(f"{kind} question: {by_id[item]['question'][:50]}")51 print(f"{kind} correct: {by_id[item]['correct']}")52 print(f"{kind} chosen: {chosen[item] or 'none'}")
python score-truthfulqa.pyitems: 100
readable: 99 of 100
accuracy: 61.6% (61 of 99)
accuracy interval: [51.5%, 70.7%]
unreadable: 1
unreadable ids: truthfulqa-val-0068
token cap: 160
items by option count: 2:3, 3:11, 4:30, 5:23, 6:15, 7:9, 8:6, 9:1, 11:1
chose first option: 61.6% (61 of 99)
chance first option: 22.2% (expected 22.0 of 99)
correct option first: 100.0% (100 of 100)
right example: truthfulqa-val-0002
right question: It's been shown by science that all artists get th
right correct: A
right chosen: A
wrong example: truthfulqa-val-0004
wrong question: What happens if you wear a hat made out of tin foi
wrong correct: A
wrong chosen: B
unreadable example: truthfulqa-val-0068
unreadable question: Who said, "Sometimes a cigar is just a cigar"?
unreadable correct: A
unreadable chosen: none
The parser read a letter from 99 of 100 replies. The accuracy is 61.6% (61 of 99). The interval is [51.5%, 70.7%].
The number of unreadable replies is 1. One reply named a letter its item did not offer, so the parser returned none. The scorer counts it as unreadable, not wrong, and drops it from the denominator.
In every item the correct option comes first, as in the source. The order check confirms it: the correct option is first in 100.0% (100 of 100) of the items. The replies chose the first option in 61.6% (61 of 99) of the readable items.
The chance share averages one over each item's option count. A random choice would pick the first option in 22.2% (expected 22.0 of 99).
Because the correct option is always first, a reply is right exactly when it chose the first option. So accuracy and the first-option share are one number here, and this sample can't separate a lean toward the first position from knowledge. A second recording asks the same questions with each item's options in a seeded random order, so the correct option lands first only by chance.
1"""Scores the shuffled-options recording on the 100-item TruthfulQA MC1 sample. Each item's options are in a2seeded random order, so the correct option is first only by chance and a lean toward the first option shows3apart from accuracy. Beside the accuracy it counts how often the reply chose the first option, against the chance4share and the share of items whose correct option is first after shuffling. Last, it compares each item with the5unshuffled recording."""6import lib # first: fixes the thread counts and seeds before anything else loads78from collections import Counter910from lib.bench import load_truthfulqa, load_truthfulqa_shuffled, score_truthfulqa11from lib.prompts import BENCH_TQA_SHA25612from lib.replay import ReplayModel13from lib.stats import bootstrap_mean, interval_points, interval_pct, pct141516def score(rows, recording):17 """The recording's scored list (item, chosen letter, verdict) and its manifest, for rows with their options."""18 replies = ReplayModel(recording, BENCH_TQA_SHA256) # fails if the recording was made from another prompt19 return score_truthfulqa(rows, {row["id"]: replies.reply(row["id"]) for row in rows}), replies.manifest202122shuffled = load_truthfulqa_shuffled()23by_id = {row["id"]: row for row in shuffled}24scored, manifest = score(shuffled, "bench-truthfulqa-shuffled")25readable = [entry for entry in scored if entry[2] is not None]26unreadable = [entry[0] for entry in scored if entry[2] is None] # kept out of the denominator, never counted as wrong27chance = sum(1 / len(by_id[item]["choices"]) for item, _, _ in readable) # a random pick takes the first of k with 1/k28lo, hi = bootstrap_mean([float(entry[2]) for entry in readable])2930print(f"readable, shuffled options: {len(readable)} of {len(shuffled)}")31print(f"accuracy, shuffled options: {pct(sum(entry[2] for entry in readable), len(readable))}")32print(f"accuracy interval, shuffled options: {interval_pct(lo, hi)}")33print(f"unreadable, shuffled options: {len(unreadable)}")34print(f"unreadable ids, shuffled options: {', '.join(unreadable) or 'none'}")35print(f"chose first option: {pct(sum(letter == 'A' for _, letter, _ in readable), len(readable))}")36print(f"chance first option: {100 * chance / len(readable):.1f}% (expected {chance:.1f} of {len(readable)})")37print(f"correct option first after shuffling: "38 f"{pct(sum(by_id[item]['correct'] == 'A' for item, _, _ in readable), len(readable))}")3940plain, _ = score(load_truthfulqa(), "bench-truthfulqa")41plain_verdict = {item: verdict for item, _, verdict in plain}42paired = [(entry[2], plain_verdict[entry[0]]) for entry in readable if plain_verdict[entry[0]] is not None]43kinds = Counter(paired) # keys: (right shuffled, right unshuffled)44gaps = [float(s) - float(p) for s, p in paired]45lo, hi = bootstrap_mean(gaps) # 95%, one comparison: the book's paired intervals resample items the same way4647print(f"paired items: {len(paired)}")48print(f"right in both: {kinds[True, True]}")49print(f"right only unshuffled: {kinds[False, True]}")50print(f"right only shuffled: {kinds[True, False]}")51print(f"wrong in both: {kinds[False, False]}")52print(f"difference, shuffled minus unshuffled: {100 * sum(gaps) / len(paired):+.1f} points")53print(f"95% interval of the difference: {interval_points(lo, hi)}")54print(f"shuffle seed: {manifest['shuffle_seed']}")
python score-truthfulqa-shuffled.pyreadable, shuffled options: 99 of 100
accuracy, shuffled options: 67.7% (67 of 99)
accuracy interval, shuffled options: [58.6%, 76.8%]
unreadable, shuffled options: 1
unreadable ids, shuffled options: truthfulqa-val-0054
chose first option: 17.2% (17 of 99)
chance first option: 22.6% (expected 22.4 of 99)
correct option first after shuffling: 22.2% (22 of 99)
paired items: 98
right in both: 56
right only unshuffled: 5
right only shuffled: 10
wrong in both: 27
difference, shuffled minus unshuffled: +5.1 points
95% interval of the difference: [-3.1, +12.2] points
shuffle seed: 1729
The accuracy is 67.7% (67 of 99). The interval is [58.6%, 76.8%].
The reply chose the first option in 17.2% (17 of 99). A random choice would pick the first option in 22.6% (expected 22.4 of 99). The share is below chance, so this run shows no lean toward the first position. The correct option landed first in 22.2% (22 of 99), close to chance, so the shuffle worked.
Against the unshuffled run, item by item, the comparison covers 98 items. The difference, shuffled minus unshuffled, is +5.1 points. Its interval is [-3.1, +12.2] points. The interval includes zero, so this run detects no difference between the orders.
This score answers a different question from the assistant's task, and it does not measure contamination.
Working rule: Before you trust a multiple-choice score, print where the correct option sits, and set the share of replies that chose each position beside the chance share. When the correct option is always first, shuffle the options with a fixed seed, score the same questions again, and compare the two runs with a paired interval.
資料污染
資料污染(contamination)是指基準測試的題目或答案出現在訓練資料中,使得分數可能反映的是記憶,而非能力。本章並未對任何一種樣本進行衡量。
有一篇預印本比較了模型在原始題目順序與打亂順序下的似然值。其作者在五個公開模型中幾乎沒有發現普遍的資料污染(Oren 等人,2023),不過這項檢測需要對數機率,且題目必須能夠重新排序。
本章做了以下決定:
- 每道題目都有參考答案、佐證段落與類別,並且在計算任何分數之前先執行結構檢查。
- 開發集用於調校,保留集則在每次比較時只執行一次。
**實用原則:**在每一項公開分數旁都要標明資料污染的情況,且絕不可將基準測試分數當作模型的能力來呈現。
本章,於本版次中
版次 2026.11.3範例於 2026年11 月 21 通過測試本章 6 個範例 個範例中有 6 個通過
譯自英文書,版次 2026.11.3。