修改助理之前,先量測它
閱讀約需 49 分鐘 分鐘,11 節 中共有 11 個範例。
讀完本章,你將能以一組固定的問題量測文件問答助理,連同信賴區間一併解讀結果,並判斷二十個問題在什麼情況下不足以區分兩份提示詞。
本書中的助理負責回答 sablefile 的相關問題。sablefile 是一個命令列工具,用來讓專案中產生的檔案與其來源範本保持同步。使用者會詢問如何執行某個指令,或某項設定有什麼作用,助理則依據該工具的文件作答,並為每項陳述標示所依據的節。假設團隊中有人想換一份提示詞。他們用三個問題試了一下,覺得答案變好了,就直接上線。三個問題無法顯示這項修改是否真的有幫助。而修改之前助理表現如何,也沒有人說得出來,因為當時沒有留下任何紀錄。
因此,第一件要建立的事是量測:一組問題,附上各自應得的答案;一種每次都以相同方式讓助理作答的執行方法;以及一個附帶其不確定範圍的數字。修改要等這些就緒之後再進行。本章以最簡單的形式建立這套量測,並在結尾說明這種形式在哪裡走到極限。
工具與其文件
評估的穩定程度,取決於它所衡量的對象。sablefile 的文件有 1.0、1.1 與 2.0 三個版本,彼此在某些地方並不一致。對某個版本正確的答案,對另一個版本可能是錯的,因此每個問題都必須先指明版本,才能談其他細節。
先從文件著手。載入三個版本,計算各版本的節數,並計算三個版本共有的節 ID 數量。
1"""Loads the three doc versions and prints each one's section count, and how many section IDs all three share.23The shared IDs are the sections whose concept keeps one ID across versions (plan section 2.2)."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.docs import load_sections, versions78sections = load_sections() # every version, sorted by version and section ID9all_versions = versions() # the version labels present, sorted10for version in all_versions:11 print(f"sections in version {version}: {sum(1 for s in sections if s.version == version)}")1213ids = {version: {s.section_id for s in sections if s.version == version} for version in all_versions}14shared = set.intersection(*ids.values()) # IDs present in every version15print(f"section IDs in all {len(all_versions)} versions: {len(shared)}")16print(f"section IDs in 2.0 but in neither 1.0 nor 1.1: {len(ids['2.0'] - ids['1.0'] - ids['1.1'])}")
python list-the-docs.pysections in version 1.0: 35
sections in version 1.1: 36
sections in version 2.0: 40
section IDs in all 3 versions: 35
section IDs in 2.0 but in neither 1.0 nor 1.1: 4
輸出以各版本的節數開頭:1.0 有 35 節,1.1 有 36 節,2.0 有 40 節。
多數節在各版本之間沿用相同的 ID,三個版本中都出現的 ID 共有 35 個。最後一行對後續內容最為重要:有 4 個 ID 只存在於 2.0,因此依據舊版頁面作答的助理,對這些 ID 沒有任何可引用的內容。
各版本在工具的行為上也有差異,而這裡要談的差異出在 sync,也就是寫出產生檔案的那個指令。1.1 的文件指出,不帶任何旗標執行 sync 會寫出檔案,並將每個檔案記錄在 sablefile.lock 中,也就是專案根目錄下的鎖定檔。2.0 的文件則指出,不帶旗標執行 sync 只會印出計畫而不寫入任何東西,必須加上 --apply 才會寫出檔案,而鎖定檔位於 .sablefile/lock.json。關於 sync 的問題,在 1.1 與 2.0 各有不同的正確答案。讀了錯誤版本頁面的助理,語氣篤定,答案卻是錯的。
本書衡量的是答案。檢索排序、追蹤、成本、延遲、攻擊,以及智慧代理的任務集,屬於其他書籍的範疇,會在相關之處指出。
實用原則: 在每個問題中寫明版本,並以該版本自己的文件作為其答案的唯一來源。當兩個版本用詞相同、答案卻不同時,將它們拆成兩個項目,各對應一個版本。切勿讓最新版本的頁面取代較舊版本的頁面。
二十個項目
測試集是一份問題清單,每個問題都附有應得的答案。二十個項目是刻意取的小規模。你可以逐題閱讀,對照文件檢查其預期答案,並說明該題為何存在。一份沒有人讀過的測試集,無法告訴你它衡量的是什麼。
這些項目分成四種類型。可回答的項目,其答案就在文件頁面上。無法回答的項目所問的是文件未涵蓋的內容,正確的回應是拒答。依版本而異的項目,其答案會隨版本不同而改變,其中有數題各有一個孿生題:同一個問題,但問的是另一個版本。語意模糊的項目可能有兩種解讀,其參考答案同時涵蓋兩種解讀。
每個項目依 ID 順序列出,並附上其版本、類型、孿生題,以及是否為刻意植入。
1"""Prints the twenty chapter items in id order: version, kind, planted flag and question, then the counts by kind.23The counts are the set's make-up that the chapter describes (plan section 3.4)."""4import lib # first: fixes the thread counts and seeds before anything else loads56from collections import Counter78from lib.answers import chapter_items910items = chapter_items() # sorted by id11for item in items:12 twin = item.twin_of if item.twin_of else "none"13 planted = f"planted ({item.planted_why})" if item.planted else "not planted"14 print(f"{item.id} | version {item.version} | {item.kind} | {planted} | twin {twin} | {item.question}")1516kinds = Counter(item.kind for item in items)17for kind in ("answerable", "unanswerable", "version-dependent", "ambiguous"):18 print(f"items of kind {kind}: {kinds[kind]}")19print(f"planted items: {sum(1 for item in items if item.planted)}")20print(f"items in the dev split: {sum(1 for item in items if item.split == 'dev')}")
python twenty-questions.pyq-0001 | version 1.1 | version-dependent | not planted | twin q-0002 | In sablefile 1.1, if I run sync with no flags, are the target files written straight away?
q-0002 | version 2.0 | version-dependent | not planted | twin q-0001 | In sablefile 2.0, if I run sync with no flags, are the target files written?
q-0003 | version 1.0 | version-dependent | planted (version-swap) | twin q-0004 | In sablefile 1.0, is ${project} the right way to insert the project name into a template?
q-0004 | version 2.0 | version-dependent | not planted | twin q-0003 | In sablefile 2.0, is ${project} the right way to insert the project name into a template?
q-0005 | version 2.0 | answerable | planted (near-miss-term) | twin none | Does sablefile 2.0 still read sablefile.lock in my project root?
q-0006 | version 2.0 | answerable | not planted | twin none | Is there a way to make check exit with status 0 even when a file has drifted in 2.0?
q-0007 | version 1.1 | answerable | not planted | twin none | In 1.1, will an ignore entry of docs/**/*.tmp catch a file two folders down, like docs/x/y/a.tmp?
q-0008 | version 2.0 | answerable | not planted | twin none | In 2.0, is there a way to re-include something that an earlier ignore line had excluded?
q-0009 | version 1.0 | answerable | not planted | twin none | What does one line of the 1.0 lock file hold?
q-0010 | version 2.0 | answerable | not planted | twin none | Will syncing the same content twice in 2.0 produce a different lock file each time?
q-0011 | version 1.1 | answerable | not planted | twin none | How many times does 1.1 retry a remote template fetch that keeps failing, and can a flag change that?
q-0012 | version 2.0 | answerable | not planted | twin none | I want sync to retry a slow remote template only twice in 2.0. What do I pass?
q-0013 | version 1.0 | answerable | not planted | twin none | In 1.0, which table in sablefile.toml holds the values a placeholder can name?
q-0014 | version 1.1 | answerable | not planted | twin none | What does sablefile init put into a new sablefile.toml in 1.1?
q-0015 | version 1.0 | unanswerable | planted (false-premise) | twin none | In sablefile 1.0, what does the --dry-run flag on sync print?
q-0016 | version 2.0 | unanswerable | not planted | twin none | Does sablefile 2.0 run on Windows?
q-0017 | version 2.0 | unanswerable | not planted | twin none | How fast is sablefile check on a project with several thousand generated files?
q-0018 | version 1.1 | unanswerable | not planted | twin none | Can sablefile 1.1 fetch a remote template from a private URL that needs a login token?
q-0019 | version 2.0 | ambiguous | planted (ambiguous-target) | twin none | If I edit the file and then run check, will it fail?
q-0020 | version 2.0 | ambiguous | not planted | twin none | Does sync change the lock file?
items of kind answerable: 10
items of kind unanswerable: 4
items of kind version-dependent: 4
items of kind ambiguous: 2
planted items: 4
items in the dev split: 20
最後幾行列出測試集依類型的組成,其中可回答的項目占最大宗。4 個項目是刻意植入的。刻意植入的項目是為了抓出某一種特定錯誤而撰寫的,測試集會為它加上標記,因此該項目的失敗可以對照其陷阱來解讀。
每個陷阱都很單純。q-0003 詢問在 1.0 中 ${project} 是否為正確的佔位符,而 1.0 的語法是 {{project}}。它的孿生題 q-0004 對 2.0 提出同樣的問題,而 2.0 中 ${project} 才是正確的。q-0005 詢問 2.0 是否仍在專案根目錄讀取 sablefile.lock,這裡用的是 1.x 的檔名,而這個檔名在該版本中只與遷移有關。q-0015 詢問 --dry-run 旗標在 1.0 中的作用,但 1.0 的文件並未提到這個旗標。q-0019 詢問檔案在編輯後是否未通過檢查,卻沒有指明是哪個檔案,因此有兩種解讀。
實用原則: 在執行助理之前,先為每個項目寫下其版本、類型與預期答案,並為任何為抓出特定錯誤而撰寫的項目加上標記。讓測試集小到可以一次讀完,並且絕不要為了配合助理的回答而更動預期答案。
此部分尚未翻譯,因此以英文呈現。
Retrieve, then answer
The assistant doesn't read all the docs for every question. For each question it takes the three sections most likely to hold the answer, from the index of the version the question names, and answers from those three alone. A wrong answer can therefore come from two places: the right section was never retrieved, or the model misread the section it was given. A measurement that can't tell these apart can't say what to fix.
The index is built with BM25, which scores each section by the words it shares with the question. The scoring runs through bm25s at 0.3.16. Take one question: whether an ignore entry such as docs/**/*.tmp catches a file two folders down in 1.1. Its output shows the words the index keeps from the question, and the top three sections, ordered by score and then by section ID.
1"""Shows BM25 retrieval for one question (q-0007, version 1.1): the words kept, then the top three sections.23The sections come from the version's own index, ranked by score and then by section ID, so a tie has one order.4The tokeniser drops one-letter words, so the `3` of `3.5mm` is lost and only `5mm` is matched."""5import lib # first: fixes the thread counts and seeds before anything else loads67import bm25s89from lib.docs import load_sections10from lib.retrieval import Retriever11from lib.testset import load_items1213item = next(item for item in load_items() if item.id == "q-0007")14print(f"question {item.id} (version {item.version}): {item.question}")1516# The words the index actually matches on: the same tokeniser, stopwords and token pattern as lib.retrieval.17kept = bm25s.tokenize([item.question], stopwords="en", return_ids=False, show_progress=False)[0]18print(f"words kept for matching: {' '.join(kept)}")19dropped = bm25s.tokenize(["a 3.5mm jack"], stopwords="en", return_ids=False, show_progress=False)[0]20print(f"words kept from the text 'a 3.5mm jack': {' '.join(dropped)}")2122hits = Retriever(load_sections(item.version)).retrieve(item.question, 3) # top 3 of the version's index23for rank, hit in enumerate(hits, start=1):24 print(f"rank {rank}: {hit.section_id} | score {hit.score:.3f}")
python retrieve-top-sections.pyquestion q-0007 (version 1.1): In 1.1, will an ignore entry of docs/**/*.tmp catch a file two folders down, like docs/x/y/a.tmp?
words kept for matching: ignore entry docs tmp catch file two folders down like docs tmp
words kept from the text 'a 3.5mm jack': 5mm jack
rank 1: ignore-recursive | score 9.156
rank 2: sync-create-dirs | score 3.386
rank 3: ignore-syntax | score 3.348
Read the two lines of kept words before the ranking. The tokeniser drops every token of one character. The a in the question goes, and so do both 1s of 1.1, so the version number never reaches the index. That does no harm here, because each question is searched only in the index of its own version. The same rule takes the 3 out of 3.5mm: what is left of a 3.5mm jack is 5mm jack, so a question about a 3.5 mm jack can match only on 5mm. Neither loss raises an error, and the second one changes what the index can find.
The ranking puts ignore-recursive first, the section on recursive ignore entries, and the next two are the section on creating folders and the one on ignore syntax. Ties are broken by section ID, so the same question gives the same three sections on every run.
Working rule: Check retrieval before you check the answer. For each item, record the sections the assistant was given next to its reply. A wrong reply that lacks the right section is a retrieval problem, and a wrong reply that had the right section is a problem with the prompt or the model. Keep the wording of questions plain, because a tokeniser that drops one-character tokens will lose a measurement or a version number.
此部分尚未翻譯,因此以英文呈現。
Replay the answers
A model's reply to the same question can change with its runtime, its settings and the hardware it runs on. If every measurement calls the model again, each one measures something slightly different, and a change in the number might be noise. So the book records each reply once, from a pinned open model, and replays the recording. A replay returns the same text every time.
The recording is a folder of replies, one per item. It was made with a 4-bit build of Qwen3-4B-Instruct-2507, an open model pinned to one file and one revision, run at temperature zero on a CPU. It holds a reply for every item in the full set, and this chapter reads twenty of them by ID. Replay one of them: the reply to q-0006, which asks whether check can exit with status 0 when a file has drifted in 2.0.
1"""Replays the recorded baseline reply to q-0006 and parses it: the sections it was given, the reply, its citations.23Nothing is generated here: the reply is the recorded text, read back as the same string."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.answers import Assistant, chapter_items7from lib.contract import format_problems8from lib.prompts import P1_SHA256, P1_SYSTEM910assistant = Assistant("assistant-baseline", P1_SYSTEM, P1_SHA256) # the prompt hash must match the manifest11item = next(item for item in chapter_items() if item.id == "q-0006")12print(f"question {item.id} (version {item.version}, {item.kind}): {item.question}")13for rank, section in enumerate(assistant.sections(item), start=1): # the top 3, in rank order14 print(f"section in context, rank {rank}: {section.section_id}")1516reply = assistant.reply(item)17print("recorded reply:")18print(reply.text)19print(f"citations, in order: {' '.join(reply.citations)}")20print(f"format problems: {', '.join(format_problems(reply, context_ids=assistant.context_ids(item), expect_decline=False)) or 'none'}")21# A reply to an answerable item must answer and cite; this is the one fact the prose relies on here.22assert reply.citations and not reply.declined, "q-0006 should be answered, with at least one citation"
python replay-one-answer.pyquestion q-0006 (version 2.0, answerable): Is there a way to make check exit with status 0 even when a file has drifted in 2.0?
section in context, rank 1: check-exit-status
section in context, rank 2: no-fail-flag
section in context, rank 3: quickstart-check
recorded reply:
Yes, there is a way to make `check` exit with status 0 even when a file has drifted in 2.0. Use the `--no-fail` flag with `sablefile check`. This flag ensures that the check exits with status 0 even when files have drifted, without changing which files are reported. [sec:no-fail-flag]
citations, in order: no-fail-flag
format problems: none
The output lists the three sections the assistant was given, in rank order, and then prints the reply exactly as it was recorded. The reply says yes, and it cites no-fail-flag, which was among the three. Check the citation against that list. The reply cites a section it was given, so it didn't reach outside its context. An assert also requires the reply to be an answer with at least one citation, since a decline on this answerable item would be an error.
To put a live model in place of the recording, pass live=True to Assistant. That loads the pinned Qwen3-0.6B on the CPU and answers each question as it's asked. It is a smaller model than the 4B build the recording was made with, so its live replies, and any rate built on them, will differ from the recorded ones. This book quotes no live replies. Its numbers come from the recording, so you can check each one against a file.
Working rule: Replay for every comparison. Record again only when the model, the prompt or the sections the assistant sees have changed, and note in the recording what changed. Each recording names its prompt by hash, and the example checks that hash against the recording's manifest, so one recording can't stand in for a different prompt.
兩項程式檢查
有些錯誤不需要評審來判斷。引用了提示中並未出現的某一節,或是一則拒絕回答的回覆卻又同時給出了答案,這類問題只要閱讀回覆的文字就能找出。以程式進行的檢查執行起來不花成本,而且每次得到的結果都相同,因此本書在請任何模型來評判回覆之前,會先執行這兩項檢查。
引用
助理必須以引用格式為每一項陳述標註出處,也就是以方括號括起來的節 ID,而且只能引用提供給它的 ID。引用可能以兩種方式出錯:ID 可能根本不存在,也可能存在於另一個版本中,而版本互換的題目正是為了誘發這種錯誤而設計的。凡是引用了提示中沒有的任何節,無論錯在哪一種情況,檢查都會判定為 unknown-section。檢查也會標記沒有任何引用的回答。
引用檢查與格式檢查會對二十則基準回覆執行,輸出會統計每一種失敗的次數。
1"""Runs the citation and format checks on the baseline replies to the twenty chapter items, and counts each problem.23A citation is a problem when its section was not in the item's prompt, which covers a section of another version."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.answers import Assistant, chapter_items7from lib.contract import PROBLEMS, format_problems, parse_reply, section_ids_by_version8from lib.prompts import P1_SHA256, P1_SYSTEM9from lib.stats import pct1011assistant = Assistant("assistant-baseline", P1_SYSTEM, P1_SHA256)12items = chapter_items()13counts = {name: 0 for name in PROBLEMS} # items with each problem, by its fixed name14items_with_problems = 015answered = 016answered_uncited = 017citations = 018citations_outside = 0 # cited IDs that were not in the item's prompt19for item in items:20 reply = assistant.reply(item)21 problems = format_problems(reply, context_ids=assistant.context_ids(item), expect_decline=item.kind == "unanswerable")22 items_with_problems += bool(problems)23 for name in problems:24 counts[name] += 125 if not reply.declined and reply.text:26 answered += 127 answered_uncited += not reply.citations28 citations += len(reply.citations)29 citations_outside += sum(1 for cid in reply.citations if cid not in assistant.context_ids(item))30assert citations_outside == 0, "every citation must point at a section that was in its item's prompt"31# A zero rate shows nothing unless the check can fire: a 1.1 section cited for a 2.0 item must be flagged as unknown.32planted_item = next(item for item in items if item.version == "2.0")33planted = sorted(section_ids_by_version()["1.1"] - assistant.context_ids(planted_item))[0]34planted_problems = format_problems(parse_reply(f"Yes. [sec:{planted}]"), context_ids=assistant.context_ids(planted_item), expect_decline=False)35assert "unknown-section" in planted_problems, "the check must fire on a planted bad citation"3637print(f"items: {len(items)}")38print(f"items with a format problem: {pct(items_with_problems, len(items))}")39for name in sorted(counts): # sorted by problem name40 print(f"items with problem {name}: {counts[name]}")41print(f"answered items: {answered}")42print(f"answered items with no citation: {pct(answered_uncited, answered)}")43print(f"citations: {citations}")44print(f"citations to a section not in the prompt: {pct(citations_outside, citations)}")
python check-citations.pyitems: 20
items with a format problem: 0.0% (0 of 20)
items with problem answered-declined-expected: 0
items with problem declined-with-text: 0
items with problem empty: 0
items with problem no-citation: 0
items with problem unknown-section: 0
answered items: 15
answered items with no citation: 0.0% (0 of 15)
citations: 16
citations to a section not in the prompt: 0.0% (0 of 16)
基準回覆總共給出 16 個引用,每一個都指向提供給它的節。輸出對此給出了比率:引用提示以外之節的比率為 0.0% (0 of 16),已作答但沒有引用的題目比率為 0.0% (0 of 15)。
實用原則: 對每一則回覆都執行引用檢查,並將其比率與作答率分開回報,因為兩者計算的是不同的錯誤。只有在檢查有能力發現問題的前提下,零比率才算是一項結果,所以請保留一則引用了提示以外內容的測試用回覆,並對它執行檢查,以證明檢查仍然會觸發。
此部分尚未翻譯,因此以英文呈現。
Declines
When the docs don't answer a question, the assistant must reply with the decline phrase, "The docs don't say.", and nothing else. When the docs do answer, it must not use that sentence. These are two different errors, so each gets its own rate. A false refusal is an answerable item declined, and a false answer is an unanswerable item answered. Version-dependent and ambiguous items are left out of both rates. A version-dependent item has a rate of its own, version-correctness, which chapter 9 builds. An ambiguous item is judged by the pass rate alone, since a reply's text alone can't show which version or which reading was meant.
Both decline errors are counted over the baseline's replies.
1"""Counts the two decline errors over the baseline replies: answerable items declined, and unanswerable items answered.23Version-dependent and ambiguous items are left out of both rates: a version-dependent item is scored by its own rate,4version-correctness, and an ambiguous item has no rate yet. The last lines name any excluded item the baseline declined."""5import lib # first: fixes the thread counts and seeds before anything else loads67from lib.answers import Assistant, chapter_items8from lib.prompts import P1_SHA256, P1_SYSTEM9from lib.stats import pct1011assistant = Assistant("assistant-baseline", P1_SYSTEM, P1_SHA256)12items = chapter_items()13answerable = [item for item in items if item.kind == "answerable"]14unanswerable = [item for item in items if item.kind == "unanswerable"]1516false_refusals = [item.id for item in answerable if assistant.reply(item).declined] # already in id order17false_answers = [item.id for item in unanswerable if not assistant.reply(item).declined]18print(f"answerable items: {len(answerable)}")19print(f"false refusal rate (answerable items declined): {pct(len(false_refusals), len(answerable))}")20for item_id in false_refusals:21 print(f"false refusal: {item_id}")22print(f"unanswerable items: {len(unanswerable)}")23print(f"false answer rate (unanswerable items answered): {pct(len(false_answers), len(unanswerable))}")24for item_id in false_answers:25 print(f"false answer: {item_id}")26print(f"items not counted here (version-dependent and ambiguous): {len(items) - len(answerable) - len(unanswerable)}")27excluded = [item for item in items if item.kind in ("version-dependent", "ambiguous")]28declined_excluded = [item.id for item in excluded if assistant.reply(item).declined] # the decline is not counted29print(f"declined, not counted here: {', '.join(declined_excluded) or 'none'}")
python check-declines.pyanswerable items: 10
false refusal rate (answerable items declined): 0.0% (0 of 10)
unanswerable items: 4
false answer rate (unanswerable items answered): 0.0% (0 of 4)
items not counted here (version-dependent and ambiguous): 6
declined, not counted here: q-0004
Among the answerable and unanswerable items, the baseline makes neither error. The false refusal rate is 0.0% (0 of 10) of the 10 answerable items. The false answer rate is 0.0% (0 of 4) of the 4 unanswerable ones, so every unanswerable item got the decline, q-0015 among them.
The rates leave out 6 items, and the last line names the one of them that the baseline declined: q-0004. Item q-0004 asks whether ${project} is the right placeholder in 2.0, where it is. The baseline replied with the decline phrase, but the 2.0 docs answer it, in the placeholders section, so the decline is a wrong reply. The rate above doesn't count it as a false refusal, because that rate covers answerable items only. Version-correctness does count it as a miss.
Working rule: Keep the two decline rates apart, and leave version-dependent and ambiguous items out of both. A reply counts as a correct decline only when it is the decline phrase and nothing else, so a reply that gives the sentence and then an answer counts as the error it is. Read each false refusal by hand before you change the prompt, since one of them may point at a real gap in the docs. Read each decline on a version-dependent item too, since the rate doesn't count those.
單一評審
引用與拒答檢查的是回覆的形式,卻無法檢查回覆是否正確,而這正是團隊最關心的問題。為此,本書使用評審:由一個模型讀取問題、一份書面的參考答案與助理的回覆,然後回傳 PASS 或 FAIL。這裡的評審是 Qwen3-4B-Instruct-2507,與撰寫回覆的是同一個模型。評審與助理若是同一個模型,可能共有盲點,評審也就會漏掉助理因相同原因而犯的錯誤。本書在這個設定下無法測試這一點。本書的做法是:讓同一個模型在校準用的評分標準下,與公開的人工評分相比對,並說明把那些比率套用到這個評審時隱含了什麼假設,內容見第 6 章。
評審的規則很短,值得完整讀過。回覆只有在說出參考答案所說的內容時才算正確。用字不同無妨,加入不與參考答案牴觸的額外細節也無妨。回覆若說了參考答案無法支持的內容,或與之牴觸,即為 FAIL。若參考答案有多種解讀,回覆必須回答每一種,只回答一部分即為 FAIL。若參考答案是「none」,表示文件並未回答該問題,唯一正確的回覆就是拒答。回覆若既給出答案又附上拒答用語,即為 FAIL。評審在第一行寫出判定,其後附上簡短的理由。提示要求只寫一句話,但評審並不總是在一句話後就停筆。
以下是評審對 q-0005(那個差一點就答對的題目)所下的判定。助理對它的回覆指出 2.0 不會讀取 sablefile.lock,並含糊地提到遷移。
1"""Reads the reference-guided judge's verdict on the baseline reply to q-0005, a planted near-miss item, from its recording.23The judge sees the question, the written reference and the reply; the first line of its reply is the verdict."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.answers import Assistant, chapter_items7from lib.contract import parse_verdict8from lib.prompts import P1_SHA256, P1_SYSTEM, REF_SHA2569from lib.replay import ReplayModel1011assistant = Assistant("assistant-baseline", P1_SYSTEM, P1_SHA256)12item = next(item for item in chapter_items() if item.id == "q-0005")13print(f"question {item.id} (version {item.version}, {item.kind}, planted: {item.planted_why}): {item.question}")14print(f"reference answer: {item.reference_answer}")15reply = assistant.reply(item)16print(f"assistant reply: {reply.text}")1718judge = ReplayModel("judge-answers-baseline", REF_SHA256) # the prompt hash must match the judge's manifest19raw = judge.reply(item.id, 0)20verdict = parse_verdict(raw)21assert verdict is not None, "the judge's first line must be a verdict"22print(f"judge verdict: {verdict}")23reason = [line.strip() for line in raw.splitlines() if line.strip()][1:] # the line after the verdict24print(f"judge reason: {reason[0] if reason else 'none'}")
python judge-one-answer.pyquestion q-0005 (version 2.0, answerable, planted: near-miss-term): Does sablefile 2.0 still read sablefile.lock in my project root?
reference answer: Only as a fallback. If the new file is missing and an old sablefile.lock is present, a 2.0 command reads the old file but warns you about it. Once the new file exists, 2.0 reads only that one, and sablefile never deletes the old file.
assistant reply: No, sablefile 2.0 does not read `sablefile.lock` in the project root. Version 2.0 moves the lock file to a new place. The first 2.0 command that reads an older lock file handles it as described in lock-migration. [sec:install-upgrade]
judge verdict: FAIL
judge reason: The response incorrectly states that sablefile 2.0 "does not read `sablefile.lock`" and that it "moves the lock file to a new place," which contradicts the reference answer that says sablefile 2.0 reads the old lock file as a fallback and only reads the new file once it exists. The reference explicitly states that it reads the old file as a fallback and warns users, not that it moves or stops reading it.
參考答案說的則不同:2.0 只在沒有新檔案時,才把舊檔案當作後備來讀取,而且讀取時會警告使用者。評審的判定是 FAIL,其理由指出回覆與參考答案牴觸。文件中 2.0 遷移一節所說的與參考答案一致,因此就這一題而言,評審是對的。不過,一次正確的判定並不能證明評審可靠。
這正是評審需要對照真人評分來檢查的原因。評審與你自己的標註一致,並不表示它已經過校準。Shankar 等人的《誰來驗證驗證者?》(Who Validates the Validators?)(2024)是一項針對九位參與者的研究,發現人們在評分輸出的同時也在修正自己的評分準則,因此所定下的準則會隨著閱讀更多輸出而改變。只寫一次的評分標準,可能與評分者後來真正的意思漸行漸遠。第 5 章說明如何把標準寫下來,這件事本身就值得做:書面的評分標準讓第二位讀者能據以核對判定,也讓日後的判定有可比較的依據。與人工評分的比對見第 6 章。
**實用原則:**為每一題提供書面的參考答案,並且在信任由這些判定算出的比率之前,像看 PASS 判定一樣仔細閱讀 FAIL 判定。不要把與自己標註一致視為校準。在評審尚未對照人工評分檢查之前,請把它的比率報告為評審的比率,而不是助理品質的衡量。
通過率與其區間
通過率是一個計數除以一個分母,而分母決定了這個比率的意義。同樣的比率若建立在大得多的資料集上,便是另一種主張;而二十個題目的資料集,只要有一個題目改變,比率就會大幅變動。此處的分母是資料集中的每一個題目,正確的拒答也算通過。
pass rate = passed items / items
區間回答的問題與通過率不同:如果資料集是以同樣方式抽出的另外二十個題目,比率會移動多少?所用的方法是百分位數自助法(percentile bootstrap)。自助法本身出自 Efron:他在 1979 年發表於 Annals of Statistics 的論文 Bootstrap Methods: Another Look at the Jackknife,將重抽樣提出為估計統計量變異程度的方法。百分位數自助法是建立在重抽樣之上最簡單的區間,即重抽樣所得比率的中間 95%。這個程序從資料集中以放回方式抽取題目,不考慮題目的類型,因此一次抽樣可以包含任意組合的類型。它計算每次抽樣的通過率,並將抽樣重複許多次。這個區間顯示二十個題目能說明多少。
先計算基準版本的通過率及其區間,再依類型拆解通過率。
1"""The baseline's pass rate over the twenty chapter items, judged by the reference-guided judge, with its 95% interval.23Every item is in the denominator, and a correct decline counts as a pass. The interval resamples items."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.answers import chapter_items7from lib.judging import passed, verdicts8from lib.stats import bootstrap_mean, interval_pct, pct910items = chapter_items()11found = verdicts("judge-answers-baseline", [item.id for item in items]) # keyed in sorted id order12scores = passed(found) # 1.0 for PASS, 0.0 for FAIL, in the same order13print(f"items: {len(items)}")14print(f"pass rate: {pct(int(scores.sum()), len(items))}")15lo, hi = bootstrap_mean(scores) # level 0.95 for one comparison; seed SEED, 2,000 resamples16print(f"95% interval of the pass rate: {interval_pct(lo, hi)}")1718kind_of = {item.id: item.kind for item in items}19for kind in ("answerable", "unanswerable", "version-dependent", "ambiguous"):20 ids = [item_id for item_id in found if kind_of[item_id] == kind]21 passes = sum(1 for item_id in ids if found[item_id] == "PASS")22 print(f"pass rate, {kind}: {pct(passes, len(ids))}")
python pass-rate-interval.pyitems: 20
pass rate: 85.0% (17 of 20)
95% interval of the pass rate: [70.0%, 100.0%]
pass rate, answerable: 90.0% (9 of 10)
pass rate, unanswerable: 100.0% (4 of 4)
pass rate, version-dependent: 75.0% (3 of 4)
pass rate, ambiguous: 50.0% (1 of 2)
基準版本的通過率為 85.0% (17 of 20),其 95% 區間為 [70.0%, 100.0%]。區間在比率兩側的距離相同,上端落在比率的最大值(上限)。它之所以到達上限,是因為超過 2.5% 的抽樣不含任何失敗的題目,所以中間 95% 的頂端就是該上限。這表示區間無法排除一個從不失敗的助理,而這正是單一數字所隱藏的。
可回答的題目通過率為 90.0% (9 of 10),不可回答的題目為 100.0% (4 of 4),依版本而異的題目為 75.0% (3 of 4),有歧義的題目為 50.0% (1 of 2)。三個失敗分屬三種不同的類型。可回答類型中的那一個失敗是 q-0005,已在前文解讀。依版本而異類型中的那一個失敗,是在 q-0004 上的拒答,已在前文解讀。有歧義類型中的那一個失敗是 q-0019,評審在基準版本的回覆下判定其未通過,下一節的範例會說明這一點。
只有可回答類型的題目數超過少數幾個:共有 10 個。其他每個類型至多只有 4 個題目,因此它們的數字只是關於少數幾個題目的備註,不是結果。
區間只涵蓋由抽取題目所產生的變異。它無法說明這二十個題目是否代表真實使用者提出的問題,任何區間都無法說明這一點。第 3 章討論資料集在分數有意義之前必須包含什麼,以及為什麼這一份是依據文件撰寫,而不是從使用者抽樣而來。
**實用原則:**報告每個通過率時,一併列出其分母、區間,以及各題目類型的通過率。只有少數題目的類型視為備註,不視為結果,並說明它依據多少個題目。
兩個提示詞
團隊提出的提示詞是要衡量的變更。P1 是基準提示詞。P2 沿用相同的基本規則,只改變回覆的形式:它要求模型先引述其答案所依據的那一句話,再給出答案,並在每一行附上出處。問題在於這段引述是否有幫助。兩個通過率若建立在不同的題目上,衡量到的多半是題目本身,因此比較時必須讓同樣的題目通過兩個提示詞。每個題目於是得到兩個判定,差異則逐題計算。這就是成對比較。
兩個提示詞在同樣的二十個題目上執行,輸出會報告差異及其區間。
1"""Compares the baseline prompt (P1) with prompt-b (P2) on the twenty chapter items: each pass rate, the paired difference, its interval and n.23The word 'better' is used only when the whole interval is above zero, and 'worse' only when it is below zero. A difference4whose interval includes zero is reported as no difference detected."""5import lib # first: fixes the thread counts and seeds before anything else loads67from lib.answers import chapter_items8from lib.judging import paired9from lib.stats import interval_points, pct1011result = paired("judge-answers-baseline", "judge-answers-prompt-b", [item.id for item in chapter_items()])12first, second = result.first, result.second # 1.0 or 0.0 per item, in id order13print(f"P1 (baseline) pass rate: {pct(int(first.sum()), result.n)}")14print(f"P2 (prompt-b) pass rate: {pct(int(second.sum()), result.n)}")15print(f"items that P2 passes and P1 fails: {int(((second == 1) & (first == 0)).sum())}")16print(f"items that P1 passes and P2 fails: {int(((second == 0) & (first == 1)).sum())}")17print(f"paired items (n): {result.n}")18print(f"difference, P2 minus P1: {100 * result.difference:+.1f} points")19print(f"95% interval of the difference: {interval_points(result.diff_lo, result.diff_hi)}")20if result.diff_lo > 0:21 reading = f"P2 is better at n = {result.n}"22elif result.diff_hi < 0:23 reading = f"P2 is worse at n = {result.n}"24else:25 reading = f"no difference detected at n = {result.n}"26print(f"reading: {reading}")27# The two prompts tie: the same number of items pass under each, so the pass rates and the reading say the same thing.28assert int(first.sum()) == int(second.sum()), "the two prompts must tie: the same pass count under each"
python compare-two-prompts.pyP1 (baseline) pass rate: 85.0% (17 of 20)
P2 (prompt-b) pass rate: 85.0% (17 of 20)
items that P2 passes and P1 fails: 1
items that P1 passes and P2 fails: 1
paired items (n): 20
difference, P2 minus P1: +0.0 points
95% interval of the difference: [-15.0, +15.0] points
reading: no difference detected at n = 20
在這組題目上,兩個提示詞打成平手。P1 的通過率為 85.0% (17 of 20),P2 的通過率為 85.0% (17 of 20)。
成對檢視顯示了平手背後的情形。P2 通過了 1 個 P1 未通過的題目,P1 則通過了 1 個 P2 未通過的題目。這些是評審的判定,並不是關於提示詞的事實。第一個是 q-0019,評審對它判定 PASS,違反了評審自己的規則 3。差異(P2 減 P1)為 +0.0 points,其區間為 [-15.0, +15.0] points。區間包含零,因此在這個規模下,範例回報為未偵測到差異。
以下列出評審在兩個提示詞下對 q-0019 的判定,並在旁邊印出該 PASS 所違反的規則。
1"""Shows one judge verdict that breaks the judge's own rule: on q-0019, the judge passes P2's reply, which answers only one of the2two readings in the reference. The rule it breaks is rule 3 of the judge's prompt, printed below from that prompt.34The hand verdict is a person's reading, written into this file. It is printed beside the judge's verdict and is the only5number here that is not a recorded one; the count of passes with it is taken from the judge's own verdicts."""6import lib # first: fixes the thread counts and seeds before anything else loads78from lib.answers import Assistant, chapter_items9from lib.judging import verdicts10from lib.prompts import P1_SHA256, P1_SYSTEM, P2_SHA256, P2_SYSTEM, REF_SYSTEM11from lib.replay import ReplayModel1213ITEM_ID = "q-0019"14HAND = {"q-0019": "FAIL"} # read by hand against rule 3: the reply answers reading 1 and not reading 21516item = next(item for item in chapter_items() if item.id == ITEM_ID)17print(f"question {item.id} (version {item.version}, {item.kind}, planted: {item.planted_why}): {item.question}")18print(f"reference answer: {item.reference_answer}")19print(f"P2 reply: {Assistant('assistant-prompt-b', P2_SYSTEM, P2_SHA256).reply(item).text}")2021raw = ReplayModel("judge-answers-prompt-b", None).reply(item.id, 0) # the judge's recorded reply to P2's answer22print(f"judge verdict on P2's reply: {verdicts('judge-answers-prompt-b', [item.id])[item.id]}")23reason = [line.strip() for line in raw.splitlines() if line.strip()][1:] # the line after the verdict24print(f"judge reason: {reason[0] if reason else 'none'}")25print(f"judge verdict on P1's reply to the same item: {verdicts('judge-answers-baseline', [item.id])[item.id]}")26print(f"hand verdict on P2's reply, under rule 3: {HAND[item.id]}")27rule_3 = next(line for line in REF_SYSTEM.splitlines() if line.startswith("3. "))28print(f"rule it breaks: {rule_3}")2930ids = [item.id for item in chapter_items()]31judged = verdicts("judge-answers-prompt-b", ids)32passes = sum(verdict == "PASS" for verdict in judged.values())33by_hand = passes - sum(1 for item_id in HAND if judged[item_id] == "PASS" and HAND[item_id] == "FAIL")34print(f"P2 passes, as judged: {passes} of {len(ids)}")35print(f"P2 passes, with {ITEM_ID} read by hand: {by_hand} of {len(ids)}")
python judge-gets-one-wrong.pyquestion q-0019 (version 2.0, ambiguous, planted: ambiguous-target): If I edit the file and then run check, will it fail?
reference answer: Reading 1: if you edited a generated file (a target), check prints DRIFT and the path for it and exits with status 1, because its hash no longer matches the lock.
Reading 2: if you edited a template, check prints nothing and exits with status 0, since a template change is not drift.
P2 reply: Quote: "Run `sablefile check` after a sync. If no output appears, no generated file has drifted since the last sync. To see what drift looks like, open a generated file, change one line, and run the check again. The check prints `DRIFT` and the path of the file." [sec:quickstart-check]
Answer: Yes, if you edit the file and then run check, it will print `DRIFT` and the path of the file, indicating a drift [sec:quickstart-check].
judge verdict on P2's reply: PASS
judge reason: The response correctly identifies that editing a generated file and running check will result in a DRIFT message and path output, aligning with Reading 1 of the reference. It does not contradict or omit any part of the reference.
judge verdict on P1's reply to the same item: FAIL
hand verdict on P2's reply, under rule 3: FAIL
rule it breaks: 3. If the reference has several readings, the response must answer every reading. A partial answer is incorrect.
P2 passes, as judged: 17 of 20
P2 passes, with q-0019 read by hand: 16 of 20
請把這份輸出當作評審判定的紀錄來讀。q-0019 的參考答案有兩種解讀,而 P2 的回覆只回答了第一種。評審提示詞的規則 3 規定部分回答視為未通過,評審也判 P1 對同一題的回覆未通過,那份回覆同樣只回答了第一種解讀。對 P2 的回覆,評審卻判定通過,理由是該回覆涵蓋了參考答案,沒有遺漏任何內容。以人工逐一檢讀,P2 通過的題目是 16 of 20 個,不是 17 of 20 個。這個平手只差一次評審誤判,就不再是平手。這正是此處的通過率只是評審所給的通過率的緣故,也是第 6 章要用公開的人工評分來檢驗評審的原因。
實用原則: 以相同的題目、成對地比較提示詞,並連同區間報告差異。只有整個區間都在零之上,才說某個提示詞較好;只有整個區間都在零之下,才說它較差。當區間包含零時,應說在這個規模下未偵測到差異,這與說兩個提示詞相等並不相同。
二十個問題無法顯示的事
這最後一個範例沿用同樣的兩個提示詞,並探問這二十個項目能對它們說明什麼。結果是,二十個項目對這兩個提示詞幾乎說明不了什麼。
1"""Shows that twenty items cannot tell the two prompts apart: the interval of the paired difference includes zero.23The assert is the chapter's claim, checked against the numbers; the example prints no verdict word."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.answers import chapter_items7from lib.judging import paired8from lib.stats import interval_points910result = paired("judge-answers-baseline", "judge-answers-prompt-b", [item.id for item in chapter_items()])11print(f"paired items (n): {result.n}")12differing = int((result.first != result.second).sum())13print(f"items where the two verdicts differ: {differing}")14print(f"difference, P2 minus P1: {100 * result.difference:+.1f} points")15print(f"95% interval of the difference: {interval_points(result.diff_lo, result.diff_hi)}")16print(f"interval width: {100 * (result.diff_hi - result.diff_lo):.1f} points")17# Zero inside the interval is the claim: no difference that twenty items can show.18assert result.diff_lo <= 0.0 <= result.diff_hi, "the interval of the difference must include zero"19# Two items carry the whole comparison: the pass rates differ only where these two verdicts differ.20assert differing == 2, "exactly two items must have differing verdicts"
python twenty-cannot-tell.pypaired items (n): 20
items where the two verdicts differ: 2
difference, P2 minus P1: +0.0 points
95% interval of the difference: [-15.0, +15.0] points
interval width: 30.0 points
兩個提示詞只在 2 個項目上得到不同的判定,因此差異只能以粗略的步幅變動,一次一個項目。差異的區間寬度為 30.0 points。程式碼中的一個 assert 要求區間必須包含零,所以一旦數字不再支持這項主張,它就會明確地失敗。這麼寬的區間只能偵測到很大的變化,而對使用者仍然重要的較小變化,則可能藏在區間之內。
在正式環境中,同一種錯誤會以三種形式出現:相信一個無法支撐其所附主張的數字。團隊在最先寫好的二十個項目上選了數字較高的提示詞,報告又省略區間,於是沒有人看得出這個數字的承載力有多低。評審因為判定看起來合理而被信任,卻沒有人對照參考答案去讀它的 FAIL 理由。一組已錄製的回覆,在助理所見的各節內容已經改變之後仍被保留,量測的是舊系統,數字卻依然好看。第一種情況,你會在報告於每個比率旁附上區間時察覺。第二種情況,你會在每次執行都有人對照參考答案閱讀評審的 FAIL 理由時察覺。第三種情況,只有在錄製時也對所給的文件做雜湊,才看得出來。本書的清單檔對提示詞與測試集做了雜湊,所以重播時會拒絕提示詞的變更,但各節內容的變更卻不會被發現。
後面有兩章接續下一步。第 3 章建立本章所欠缺的較大測試集,第 8 章則推算一項差異需要多少個項目才能顯現。
本章做了以下決定:
- 每個問題都註明其版本,答案只取自該版本的文件。
- 本章的測試集為二十個項目,逐一完整閱讀,並標明刻意植入的項目。
- 每筆錄製針對每個模型、提示詞與文件各做一次,並在每次比較時重播。
- 在評審閱讀任何回覆之前,先以程式碼檢查引用與拒答。
- 評審對照書面參考答案評分,在校準檢驗之前,它的比率只算作該評審的比率。
- 通過率須附上分母、區間,以及各類項目的比率。
- 提示詞在相同的項目上比較,只有整個區間完全落在零的一側,才能認定其中一個較好或較差。
實用原則: 在每項比較旁都要報告區間。二十個項目可以找出有問題的答案,卻無法為兩個提示詞排出高下。
本章,於本版次中
版次 2026.11.3範例於 2026年11 月 21 通過測試本章 11 個範例 個範例中有 11 個通過
譯自英文書,版次 2026.11.3。