修改助手之前先做测量
阅读约需 45分钟 分钟,11节中含11个示例。
读完本章,你将能够用一组固定的问题来测量文档助手,连同区间一起解读结果,并判断二十个问题何时不足以区分两个提示词。
本书中的助手回答有关 sablefile 的问题。sablefile 是一个命令行工具,用来让项目中生成的文件与其所依据的模板保持同步。用户会询问如何运行某个命令,或者某项设置有什么作用,助手则依据该工具的文档作答,并为每条陈述标出所依据的文档小节。假设团队里有人想换一个提示词。他们在三个问题上试了试,答案看起来更好,于是就上线了。三个问题无法说明这次改动是否有帮助。改动之前助手表现如何,同样没有人说得清,因为当时什么都没有记录下来。
因此,首先要建立的是一套测量:一组问题,附上它们应得的答案;一种每次都以同样方式在这些问题上运行助手的办法;以及一个附带其大小(样本量)的数字。修改放在这之后。本章以最简单的形式搭建这套测量,并在结尾指出这种形式的局限所在。
工具及其文档
评测的稳定程度取决于被测对象本身。sablefile 的文档有三个版本,即 1.0、1.1 和 2.0,它们在若干地方互相矛盾。对一个版本正确的回答,对另一个版本可能是错的,因此每个问题都必须先写明版本,然后才能确定其余内容。
先从文档入手。加载这三个版本,统计各版本的节数,并统计三个版本共有的节 ID 数。
1"""Loads the three doc versions and prints each one's section count, and how many section IDs all three share.23The shared IDs are the sections whose concept keeps one ID across versions (plan section 2.2)."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.docs import load_sections, versions78sections = load_sections() # every version, sorted by version and section ID9all_versions = versions() # the version labels present, sorted10for version in all_versions:11 print(f"sections in version {version}: {sum(1 for s in sections if s.version == version)}")1213ids = {version: {s.section_id for s in sections if s.version == version} for version in all_versions}14shared = set.intersection(*ids.values()) # IDs present in every version15print(f"section IDs in all {len(all_versions)} versions: {len(shared)}")16print(f"section IDs in 2.0 but in neither 1.0 nor 1.1: {len(ids['2.0'] - ids['1.0'] - ids['1.1'])}")
python list-the-docs.pysections in version 1.0: 35
sections in version 1.1: 36
sections in version 2.0: 40
section IDs in all 3 versions: 35
section IDs in 2.0 but in neither 1.0 nor 1.1: 4
输出首先给出各版本的节数:1.0 中为 35,1.1 中为 36,2.0 中为 40。
大多数节在各版本之间保持 ID 不变,因为有 35 个 ID 在三个版本中都出现。最后一行对后文最为重要:有 4 个 ID 只存在于 2.0 中,因此依据旧版页面作答的助手对这些 ID 无从引用。
各版本在工具行为上也有差异,这里要紧的差异在于 sync,即写出生成文件的命令。1.1 的文档说明,不带任何标志的 sync 会写出这些文件,并把每个文件记录在 sablefile.lock 中,这个锁文件位于项目根目录。2.0 的文档则说明,不带标志的 sync 只打印一份计划,不写入任何内容;--apply 才会写出文件;锁文件位于 .sablefile/lock.json。关于 sync 的问题,对 1.1 与对 2.0 各有不同的正确答案。读了错误版本页面的助手,语气笃定,答案却是错的。
本书衡量的是回答。检索排序、追踪、成本、延迟、攻击以及智能体的任务集属于其他书的范围,相关之处会指明出处。
工作准则: 把版本写进每一个问题,并且只以该版本自己的文档作为其答案的来源。当两个版本措辞相同而答案不同时,把它们拆成两个条目,每个版本各一个。绝不能让最新版本的页面替代旧版本的页面。
二十个条目
测试集是一份问题清单,每个问题都附有它应当得到的答案。二十个条目的规模是有意取小的:你可以逐条阅读,对照文档核对其预期答案,并说明它为什么在这里。一份无人阅读的测试集,无法告诉你它在度量什么。
这些条目分为四类。可回答条目的答案就在文档页面上。不可回答条目询问文档未涉及的内容,正确的回复是拒答。版本相关条目的答案随版本而变,其中若干条目有一个孪生条目:同一个问题,针对另一个版本提出。有歧义条目可以有两种理解,其参考答案同时涵盖两种读法。
每个条目按 ID 顺序打印,同时给出其版本、类别、孪生条目,以及它是否为植入条目。
1"""Prints the twenty chapter items in id order: version, kind, planted flag and question, then the counts by kind.23The counts are the set's make-up that the chapter describes (plan section 3.4)."""4import lib # first: fixes the thread counts and seeds before anything else loads56from collections import Counter78from lib.answers import chapter_items910items = chapter_items() # sorted by id11for item in items:12 twin = item.twin_of if item.twin_of else "none"13 planted = f"planted ({item.planted_why})" if item.planted else "not planted"14 print(f"{item.id} | version {item.version} | {item.kind} | {planted} | twin {twin} | {item.question}")1516kinds = Counter(item.kind for item in items)17for kind in ("answerable", "unanswerable", "version-dependent", "ambiguous"):18 print(f"items of kind {kind}: {kinds[kind]}")19print(f"planted items: {sum(1 for item in items if item.planted)}")20print(f"items in the dev split: {sum(1 for item in items if item.split == 'dev')}")
python twenty-questions.pyq-0001 | version 1.1 | version-dependent | not planted | twin q-0002 | In sablefile 1.1, if I run sync with no flags, are the target files written straight away?
q-0002 | version 2.0 | version-dependent | not planted | twin q-0001 | In sablefile 2.0, if I run sync with no flags, are the target files written?
q-0003 | version 1.0 | version-dependent | planted (version-swap) | twin q-0004 | In sablefile 1.0, is ${project} the right way to insert the project name into a template?
q-0004 | version 2.0 | version-dependent | not planted | twin q-0003 | In sablefile 2.0, is ${project} the right way to insert the project name into a template?
q-0005 | version 2.0 | answerable | planted (near-miss-term) | twin none | Does sablefile 2.0 still read sablefile.lock in my project root?
q-0006 | version 2.0 | answerable | not planted | twin none | Is there a way to make check exit with status 0 even when a file has drifted in 2.0?
q-0007 | version 1.1 | answerable | not planted | twin none | In 1.1, will an ignore entry of docs/**/*.tmp catch a file two folders down, like docs/x/y/a.tmp?
q-0008 | version 2.0 | answerable | not planted | twin none | In 2.0, is there a way to re-include something that an earlier ignore line had excluded?
q-0009 | version 1.0 | answerable | not planted | twin none | What does one line of the 1.0 lock file hold?
q-0010 | version 2.0 | answerable | not planted | twin none | Will syncing the same content twice in 2.0 produce a different lock file each time?
q-0011 | version 1.1 | answerable | not planted | twin none | How many times does 1.1 retry a remote template fetch that keeps failing, and can a flag change that?
q-0012 | version 2.0 | answerable | not planted | twin none | I want sync to retry a slow remote template only twice in 2.0. What do I pass?
q-0013 | version 1.0 | answerable | not planted | twin none | In 1.0, which table in sablefile.toml holds the values a placeholder can name?
q-0014 | version 1.1 | answerable | not planted | twin none | What does sablefile init put into a new sablefile.toml in 1.1?
q-0015 | version 1.0 | unanswerable | planted (false-premise) | twin none | In sablefile 1.0, what does the --dry-run flag on sync print?
q-0016 | version 2.0 | unanswerable | not planted | twin none | Does sablefile 2.0 run on Windows?
q-0017 | version 2.0 | unanswerable | not planted | twin none | How fast is sablefile check on a project with several thousand generated files?
q-0018 | version 1.1 | unanswerable | not planted | twin none | Can sablefile 1.1 fetch a remote template from a private URL that needs a login token?
q-0019 | version 2.0 | ambiguous | planted (ambiguous-target) | twin none | If I edit the file and then run check, will it fail?
q-0020 | version 2.0 | ambiguous | not planted | twin none | Does sync change the lock file?
items of kind answerable: 10
items of kind unanswerable: 4
items of kind version-dependent: 4
items of kind ambiguous: 2
planted items: 4
items in the dev split: 20
最后几行按类别给出测试集的构成,其中可回答条目占比最大。植入条目有 4 个。植入条目是为捕捉某一种特定错误而写的,测试集对其加了标注,因此在该条目上的失败可以对照其陷阱来解读。
每个陷阱都很简单。q-0003 询问 ${project} 在 1.0 中是否为正确的占位符,而 1.0 的语法是 {{project}}。它的孪生条目 q-0004 就 2.0 提出同样的问题,在 2.0 中 ${project} 才是正确的。q-0005 询问 2.0 是否仍然在项目根目录读取 sablefile.lock,这里用的是 1.x 的文件名,而在 2.0 中,该文件名只与迁移有关。q-0015 询问 --dry-run 标志在 1.0 中的作用,而 1.0 文档中并没有这样的标志。q-0019 询问某个文件在你编辑之后是否未通过检查,却没有说明是哪个文件,因此有两种读法。
工作准则: 在运行助手之前,为每个条目写好其版本、类别和预期答案,并对为捕捉特定错误而写的条目加以标注。让测试集保持在一次就能读完的规模,并且绝不要为了迎合助手的回答而修改预期答案。
这部分尚未翻译,因此以英文显示。
Retrieve, then answer
The assistant doesn't read all the docs for every question. For each question it takes the three sections most likely to hold the answer, from the index of the version the question names, and answers from those three alone. A wrong answer can therefore come from two places: the right section was never retrieved, or the model misread the section it was given. A measurement that can't tell these apart can't say what to fix.
The index is built with BM25, which scores each section by the words it shares with the question. The scoring runs through bm25s at 0.3.16. Take one question: whether an ignore entry such as docs/**/*.tmp catches a file two folders down in 1.1. Its output shows the words the index keeps from the question, and the top three sections, ordered by score and then by section ID.
1"""Shows BM25 retrieval for one question (q-0007, version 1.1): the words kept, then the top three sections.23The sections come from the version's own index, ranked by score and then by section ID, so a tie has one order.4The tokeniser drops one-letter words, so the `3` of `3.5mm` is lost and only `5mm` is matched."""5import lib # first: fixes the thread counts and seeds before anything else loads67import bm25s89from lib.docs import load_sections10from lib.retrieval import Retriever11from lib.testset import load_items1213item = next(item for item in load_items() if item.id == "q-0007")14print(f"question {item.id} (version {item.version}): {item.question}")1516# The words the index actually matches on: the same tokeniser, stopwords and token pattern as lib.retrieval.17kept = bm25s.tokenize([item.question], stopwords="en", return_ids=False, show_progress=False)[0]18print(f"words kept for matching: {' '.join(kept)}")19dropped = bm25s.tokenize(["a 3.5mm jack"], stopwords="en", return_ids=False, show_progress=False)[0]20print(f"words kept from the text 'a 3.5mm jack': {' '.join(dropped)}")2122hits = Retriever(load_sections(item.version)).retrieve(item.question, 3) # top 3 of the version's index23for rank, hit in enumerate(hits, start=1):24 print(f"rank {rank}: {hit.section_id} | score {hit.score:.3f}")
python retrieve-top-sections.pyquestion q-0007 (version 1.1): In 1.1, will an ignore entry of docs/**/*.tmp catch a file two folders down, like docs/x/y/a.tmp?
words kept for matching: ignore entry docs tmp catch file two folders down like docs tmp
words kept from the text 'a 3.5mm jack': 5mm jack
rank 1: ignore-recursive | score 9.156
rank 2: sync-create-dirs | score 3.386
rank 3: ignore-syntax | score 3.348
Read the two lines of kept words before the ranking. The tokeniser drops every token of one character. The a in the question goes, and so do both 1s of 1.1, so the version number never reaches the index. That does no harm here, because each question is searched only in the index of its own version. The same rule takes the 3 out of 3.5mm: what is left of a 3.5mm jack is 5mm jack, so a question about a 3.5 mm jack can match only on 5mm. Neither loss raises an error, and the second one changes what the index can find.
The ranking puts ignore-recursive first, the section on recursive ignore entries, and the next two are the section on creating folders and the one on ignore syntax. Ties are broken by section ID, so the same question gives the same three sections on every run.
Working rule: Check retrieval before you check the answer. For each item, record the sections the assistant was given next to its reply. A wrong reply that lacks the right section is a retrieval problem, and a wrong reply that had the right section is a problem with the prompt or the model. Keep the wording of questions plain, because a tokeniser that drops one-character tokens will lose a measurement or a version number.
这部分尚未翻译,因此以英文显示。
Replay the answers
A model's reply to the same question can change with its runtime, its settings and the hardware it runs on. If every measurement calls the model again, each one measures something slightly different, and a change in the number might be noise. So the book records each reply once, from a pinned open model, and replays the recording. A replay returns the same text every time.
The recording is a folder of replies, one per item. It was made with a 4-bit build of Qwen3-4B-Instruct-2507, an open model pinned to one file and one revision, run at temperature zero on a CPU. It holds a reply for every item in the full set, and this chapter reads twenty of them by ID. Replay one of them: the reply to q-0006, which asks whether check can exit with status 0 when a file has drifted in 2.0.
1"""Replays the recorded baseline reply to q-0006 and parses it: the sections it was given, the reply, its citations.23Nothing is generated here: the reply is the recorded text, read back as the same string."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.answers import Assistant, chapter_items7from lib.contract import format_problems8from lib.prompts import P1_SHA256, P1_SYSTEM910assistant = Assistant("assistant-baseline", P1_SYSTEM, P1_SHA256) # the prompt hash must match the manifest11item = next(item for item in chapter_items() if item.id == "q-0006")12print(f"question {item.id} (version {item.version}, {item.kind}): {item.question}")13for rank, section in enumerate(assistant.sections(item), start=1): # the top 3, in rank order14 print(f"section in context, rank {rank}: {section.section_id}")1516reply = assistant.reply(item)17print("recorded reply:")18print(reply.text)19print(f"citations, in order: {' '.join(reply.citations)}")20print(f"format problems: {', '.join(format_problems(reply, context_ids=assistant.context_ids(item), expect_decline=False)) or 'none'}")21# A reply to an answerable item must answer and cite; this is the one fact the prose relies on here.22assert reply.citations and not reply.declined, "q-0006 should be answered, with at least one citation"
python replay-one-answer.pyquestion q-0006 (version 2.0, answerable): Is there a way to make check exit with status 0 even when a file has drifted in 2.0?
section in context, rank 1: check-exit-status
section in context, rank 2: no-fail-flag
section in context, rank 3: quickstart-check
recorded reply:
Yes, there is a way to make `check` exit with status 0 even when a file has drifted in 2.0. Use the `--no-fail` flag with `sablefile check`. This flag ensures that the check exits with status 0 even when files have drifted, without changing which files are reported. [sec:no-fail-flag]
citations, in order: no-fail-flag
format problems: none
The output lists the three sections the assistant was given, in rank order, and then prints the reply exactly as it was recorded. The reply says yes, and it cites no-fail-flag, which was among the three. Check the citation against that list. The reply cites a section it was given, so it didn't reach outside its context. An assert also requires the reply to be an answer with at least one citation, since a decline on this answerable item would be an error.
To put a live model in place of the recording, pass live=True to Assistant. That loads the pinned Qwen3-0.6B on the CPU and answers each question as it's asked. It is a smaller model than the 4B build the recording was made with, so its live replies, and any rate built on them, will differ from the recorded ones. This book quotes no live replies. Its numbers come from the recording, so you can check each one against a file.
Working rule: Replay for every comparison. Record again only when the model, the prompt or the sections the assistant sees have changed, and note in the recording what changed. Each recording names its prompt by hash, and the example checks that hash against the recording's manifest, so one recording can't stand in for a different prompt.
两项代码检查
有些错误无需评判者来判断。引用了提示中并未出现的某一节,或者在拒绝的同时又给出了答案,这类问题通过阅读回复文本即可发现。代码检查运行时不产生任何成本,每次得到的结果也相同,因此本书在请任何模型评判回复之前,先运行这两项检查。
引用
助手必须以引用形式为每一条陈述标注出处,引用形式即方括号中的节编号,且只能引用已提供给它的编号。引用可能以两种方式出错:编号可能根本不存在,也可能存在于另一个版本之中,而版本替换类题目正是为诱发这后一种错误而设计的。无论出错的方式是哪一种,只要引用的节不在提示中,检查就将其判为unknown-section。检查还会标出没有任何引用的回答。
引用检查与格式检查针对二十条基线回复运行,输出会统计每一类失败的数量。
1"""Runs the citation and format checks on the baseline replies to the twenty chapter items, and counts each problem.23A citation is a problem when its section was not in the item's prompt, which covers a section of another version."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.answers import Assistant, chapter_items7from lib.contract import PROBLEMS, format_problems, parse_reply, section_ids_by_version8from lib.prompts import P1_SHA256, P1_SYSTEM9from lib.stats import pct1011assistant = Assistant("assistant-baseline", P1_SYSTEM, P1_SHA256)12items = chapter_items()13counts = {name: 0 for name in PROBLEMS} # items with each problem, by its fixed name14items_with_problems = 015answered = 016answered_uncited = 017citations = 018citations_outside = 0 # cited IDs that were not in the item's prompt19for item in items:20 reply = assistant.reply(item)21 problems = format_problems(reply, context_ids=assistant.context_ids(item), expect_decline=item.kind == "unanswerable")22 items_with_problems += bool(problems)23 for name in problems:24 counts[name] += 125 if not reply.declined and reply.text:26 answered += 127 answered_uncited += not reply.citations28 citations += len(reply.citations)29 citations_outside += sum(1 for cid in reply.citations if cid not in assistant.context_ids(item))30assert citations_outside == 0, "every citation must point at a section that was in its item's prompt"31# A zero rate shows nothing unless the check can fire: a 1.1 section cited for a 2.0 item must be flagged as unknown.32planted_item = next(item for item in items if item.version == "2.0")33planted = sorted(section_ids_by_version()["1.1"] - assistant.context_ids(planted_item))[0]34planted_problems = format_problems(parse_reply(f"Yes. [sec:{planted}]"), context_ids=assistant.context_ids(planted_item), expect_decline=False)35assert "unknown-section" in planted_problems, "the check must fire on a planted bad citation"3637print(f"items: {len(items)}")38print(f"items with a format problem: {pct(items_with_problems, len(items))}")39for name in sorted(counts): # sorted by problem name40 print(f"items with problem {name}: {counts[name]}")41print(f"answered items: {answered}")42print(f"answered items with no citation: {pct(answered_uncited, answered)}")43print(f"citations: {citations}")44print(f"citations to a section not in the prompt: {pct(citations_outside, citations)}")
python check-citations.pyitems: 20
items with a format problem: 0.0% (0 of 20)
items with problem answered-declined-expected: 0
items with problem declined-with-text: 0
items with problem empty: 0
items with problem no-citation: 0
items with problem unknown-section: 0
answered items: 15
answered items with no citation: 0.0% (0 of 15)
citations: 16
citations to a section not in the prompt: 0.0% (0 of 16)
基线共给出16处引用,每一处都指向它已获得的节。输出据此给出了比率:引用提示之外的节的情形为0.0% (0 of 16),已作答但没有引用的题目为0.0% (0 of 15)。
**工作准则:**对每条回复都运行引用检查,并将其比率与作答率分开报告,因为二者统计的是不同的错误。只有当检查本来能够发现问题时,零比率才算得上一个结果,因此应保留一条引用提示之外内容的固定回复,并对它运行检查,以证明检查仍然有效。
拒答
当文档无法回答某个问题时,助手必须只回复拒答用语“The docs don't say.”,不得附带其他内容。当文档能够回答时,助手不得使用这句话。这是两种不同的错误,因此各有各的比率。误拒是指可回答的条目被拒答,误答是指不可回答的条目被作答。依赖版本的条目和有歧义的条目不计入这两个比率。依赖版本的条目有自己的比率,即版本正确率,第 9 章会构建它。有歧义的条目只按通过率评判,因为仅凭回复的文本无法看出所指的是哪个版本或哪种理解。
这两种拒答错误都是根据基线的回复统计的。
1"""Counts the two decline errors over the baseline replies: answerable items declined, and unanswerable items answered.23Version-dependent and ambiguous items are left out of both rates: a version-dependent item is scored by its own rate,4version-correctness, and an ambiguous item has no rate yet. The last lines name any excluded item the baseline declined."""5import lib # first: fixes the thread counts and seeds before anything else loads67from lib.answers import Assistant, chapter_items8from lib.prompts import P1_SHA256, P1_SYSTEM9from lib.stats import pct1011assistant = Assistant("assistant-baseline", P1_SYSTEM, P1_SHA256)12items = chapter_items()13answerable = [item for item in items if item.kind == "answerable"]14unanswerable = [item for item in items if item.kind == "unanswerable"]1516false_refusals = [item.id for item in answerable if assistant.reply(item).declined] # already in id order17false_answers = [item.id for item in unanswerable if not assistant.reply(item).declined]18print(f"answerable items: {len(answerable)}")19print(f"false refusal rate (answerable items declined): {pct(len(false_refusals), len(answerable))}")20for item_id in false_refusals:21 print(f"false refusal: {item_id}")22print(f"unanswerable items: {len(unanswerable)}")23print(f"false answer rate (unanswerable items answered): {pct(len(false_answers), len(unanswerable))}")24for item_id in false_answers:25 print(f"false answer: {item_id}")26print(f"items not counted here (version-dependent and ambiguous): {len(items) - len(answerable) - len(unanswerable)}")27excluded = [item for item in items if item.kind in ("version-dependent", "ambiguous")]28declined_excluded = [item.id for item in excluded if assistant.reply(item).declined] # the decline is not counted29print(f"declined, not counted here: {', '.join(declined_excluded) or 'none'}")
python check-declines.pyanswerable items: 10
false refusal rate (answerable items declined): 0.0% (0 of 10)
unanswerable items: 4
false answer rate (unanswerable items answered): 0.0% (0 of 4)
items not counted here (version-dependent and ambiguous): 6
declined, not counted here: q-0004
在可回答和不可回答的条目中,基线两种错误都没有出现。误拒率是 0.0% (0 of 10),即 10 个可回答条目中的比例。误答率为 0.0% (0 of 4),分母是 4 个不可回答条目,也就是说每个不可回答的条目都得到了拒答,q-0015 也在其中。
这两个比率之外还有 6 个条目未计入,最后一行指出了其中被基线拒答的那一个:q-0004。条目 q-0004 询问 ${project} 在 2.0 中是否是正确的占位符,而它确实是。基线回复了拒答用语,但 2.0 的文档在 placeholders 一节中回答了这个问题,所以这次拒答是错误的回复。上面的比率不把它算作误拒,因为该比率只涵盖可回答的条目。版本正确率则把它算作一次失误。
工作准则: 把两种拒答率分开,并把依赖版本的条目和有歧义的条目排除在两者之外。只有回复恰好是拒答用语、不含其他内容时,才算正确的拒答;先给出这句话再给出答案的回复,应按其本来面目计为错误。在修改提示词之前,逐条人工阅读每一次误拒,因为其中某一次可能指向文档中真实存在的缺口。依赖版本的条目上的每一次拒答也要阅读,因为比率并不统计这些条目。
一个评判者
引用和拒答检查的是回复的形式,但无法检查回复是否正确,而这恰恰是团队所关心的问题。为此,本书使用评判者:这是一个模型,它读取问题、一份书面参考答案和助手的回复,然后返回 PASS 或 FAIL。这里的评判者是 Qwen3-4B-Instruct-2507,与撰写这些回复的是同一个模型。评判者和助手若是同一个模型,就可能共有盲点,评判者也会因为同样的原因漏掉助手所犯的错误。本书在这一设置下无法对此进行检验。它所做的是:让同一个模型在校准评分细则之下,对照公开的人工评分进行检查,并在第 6 章中说明,把那些比率用于这个评判者需要假定什么。
评判者的规则很短,值得完整阅读。只有当回复所说的与参考答案一致时,回复才算正确。措辞不同没有问题,补充与参考答案不矛盾的额外细节也没有问题。回复若说了参考答案不支持的内容,或与参考答案相矛盾,则判为不通过。若参考答案有不止一种解读,回复必须回答其中的每一种,只回答一部分则判为不通过。若参考答案为 "none",则文档没有给出答案,唯一正确的回复就是拒答。回复若既给出了答案,又带有拒答用语,则判为不通过。评判者在第一行写出裁决,随后写一句简短的理由。提示要求只写一句话,评判者并不总是在一句话处停笔。
下面是评判者对 q-0005 这一险些答对的情形所作的裁决。助手对它的回复称 2.0 不读取 sablefile.lock,并含糊地指向了迁移说明。
1"""Reads the reference-guided judge's verdict on the baseline reply to q-0005, a planted near-miss item, from its recording.23The judge sees the question, the written reference and the reply; the first line of its reply is the verdict."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.answers import Assistant, chapter_items7from lib.contract import parse_verdict8from lib.prompts import P1_SHA256, P1_SYSTEM, REF_SHA2569from lib.replay import ReplayModel1011assistant = Assistant("assistant-baseline", P1_SYSTEM, P1_SHA256)12item = next(item for item in chapter_items() if item.id == "q-0005")13print(f"question {item.id} (version {item.version}, {item.kind}, planted: {item.planted_why}): {item.question}")14print(f"reference answer: {item.reference_answer}")15reply = assistant.reply(item)16print(f"assistant reply: {reply.text}")1718judge = ReplayModel("judge-answers-baseline", REF_SHA256) # the prompt hash must match the judge's manifest19raw = judge.reply(item.id, 0)20verdict = parse_verdict(raw)21assert verdict is not None, "the judge's first line must be a verdict"22print(f"judge verdict: {verdict}")23reason = [line.strip() for line in raw.splitlines() if line.strip()][1:] # the line after the verdict24print(f"judge reason: {reason[0] if reason else 'none'}")
python judge-one-answer.pyquestion q-0005 (version 2.0, answerable, planted: near-miss-term): Does sablefile 2.0 still read sablefile.lock in my project root?
reference answer: Only as a fallback. If the new file is missing and an old sablefile.lock is present, a 2.0 command reads the old file but warns you about it. Once the new file exists, 2.0 reads only that one, and sablefile never deletes the old file.
assistant reply: No, sablefile 2.0 does not read `sablefile.lock` in the project root. Version 2.0 moves the lock file to a new place. The first 2.0 command that reads an older lock file handles it as described in lock-migration. [sec:install-upgrade]
judge verdict: FAIL
judge reason: The response incorrectly states that sablefile 2.0 "does not read `sablefile.lock`" and that it "moves the lock file to a new place," which contradicts the reference answer that says sablefile 2.0 reads the old lock file as a fallback and only reads the new file once it exists. The reference explicitly states that it reads the old file as a fallback and warns users, not that it moves or stops reading it.
参考答案所说的则不同:2.0 只有在不存在新文件时,才把旧文件作为后备来读取,并且在这样做时会向你发出警告。评判者的裁决是 FAIL,其理由指出该回复与参考答案相矛盾。文档中关于 2.0 迁移的一节所说的与参考答案一致,所以在这一条目上评判者是对的。不过,一次正确的裁决并不能说明评判者可靠。
这正是评判者需要对照人工评分来检查的原因。与你自己的标注相符的评判者,并不因此就算已经校准。Shankar 等人的 Who Validates the Validators?(2024)是一项有九名参与者的研究,它发现人们在给输出打分的过程中会不断完善自己的评分标准,因此随着读到的输出越来越多,他们最终确定的标准也会改变。一份一次写成的评分细则,可能会偏离其评分者后来所指的含义。第 5 章说明了如何把细则写下来,而这样做本身就有其价值:书面的评分细则可供第二位读者据以核对某个裁决,也可供后来的裁决与之比较。对照人工评分的检查见第 6 章。
**工作准则:**为每个条目给评判者提供书面参考答案,并且在信任建立在这些裁决之上的比率之前,像细读其 PASS 裁决一样细读其 FAIL 裁决。不要把与自己标注的一致算作校准。在评判者经过人工评分的检查之前,应把它的比率报告为评判者的比率,而不是助手质量的度量。
通过率及其区间
通过率是一个计数除以一个分母,而分母决定了这个比率的含义。同样的比率出现在大得多的集合上,是另一种论断;而在只有二十项的集合上,只要一项发生变化,比率就会大幅变动。这里的分母是集合中的每一项,正确的拒答也算作通过。
pass rate = passed items / items
区间回答的是与通过率不同的问题:如果集合是以同样方式抽取的另外二十项,比率会变动多少?所用的方法是百分位自助法(percentile bootstrap)。自助法本身出自 Efron:他 1979 年发表在 Annals of Statistics 上的论文 Bootstrap Methods: Another Look at the Jackknife 把重抽样阐述为估计统计量变动幅度的一种方法。百分位自助法是建立在重抽样之上的最简单的区间,即重抽样所得比率的中间 95%。该过程从集合中有放回地抽取各项,不考虑类别,因此一次抽取可以包含任意的类别组合。它计算每次抽取的通过率,并将抽取重复许多次。这个区间显示了二十项所能说明的程度有多大。
先计算基线的通过率及其区间,再按类别拆分通过率。
1"""The baseline's pass rate over the twenty chapter items, judged by the reference-guided judge, with its 95% interval.23Every item is in the denominator, and a correct decline counts as a pass. The interval resamples items."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.answers import chapter_items7from lib.judging import passed, verdicts8from lib.stats import bootstrap_mean, interval_pct, pct910items = chapter_items()11found = verdicts("judge-answers-baseline", [item.id for item in items]) # keyed in sorted id order12scores = passed(found) # 1.0 for PASS, 0.0 for FAIL, in the same order13print(f"items: {len(items)}")14print(f"pass rate: {pct(int(scores.sum()), len(items))}")15lo, hi = bootstrap_mean(scores) # level 0.95 for one comparison; seed SEED, 2,000 resamples16print(f"95% interval of the pass rate: {interval_pct(lo, hi)}")1718kind_of = {item.id: item.kind for item in items}19for kind in ("answerable", "unanswerable", "version-dependent", "ambiguous"):20 ids = [item_id for item_id in found if kind_of[item_id] == kind]21 passes = sum(1 for item_id in ids if found[item_id] == "PASS")22 print(f"pass rate, {kind}: {pct(passes, len(ids))}")
python pass-rate-interval.pyitems: 20
pass rate: 85.0% (17 of 20)
95% interval of the pass rate: [70.0%, 100.0%]
pass rate, answerable: 90.0% (9 of 10)
pass rate, unanswerable: 100.0% (4 of 4)
pass rate, version-dependent: 75.0% (3 of 4)
pass rate, ambiguous: 50.0% (1 of 2)
基线的通过率为 85.0% (17 of 20),其 95% 区间为 [70.0%, 100.0%]。区间在比率两侧的距离相同,上端位于任何比率都不能超过的上限处。它之所以到达上限,是因为超过 2.5% 的抽取不含任何失败项,所以中间 95% 的上端就是这个上限。这意味着该区间无法排除一个从不失败的助手,而这正是单个数字所掩盖的。
可回答类的通过率为 90.0% (9 of 10),不可回答类为 100.0% (4 of 4),依赖版本类为 75.0% (3 of 4),有歧义类为 50.0% (1 of 2)。三次失败落在三个不同的类别中。可回答类的那次失败是 q-0005,已在上文解读。依赖版本类的那次失败是对 q-0004 的拒答,也已在上文解读。有歧义类的那次失败是 q-0019,评判器在基线的回复下判其失败,下一节中的示例会说明这一点。
只有可回答类拥有不止寥寥几项:共 10 项。其余每个类别至多只有 4 项,因此它们对应的行只是关于少数几项的记录,不是结果。
区间只涵盖由抽取各项所带来的波动。它没有说明这二十项是否代表了真实用户提出的问题,任何区间都无法说明这一点。第 3 章讨论一个集合必须包含什么,其上的得分才有意义,以及为什么这个集合是依据文档编写的,而不是从用户中抽样得到的。
工作准则: 报告每个通过率时,都要同时给出其分母、其区间,以及各类别的比率。只有少数几项的类别应视为记录,而非结果,并说明它依据的是多少项。
两个提示词
团队提出的提示词是需要度量的改动。P1 是基线提示词。P2 保留相同的基本规则,只改变回答的形式:它要求模型先引用其答案所依据的那句话,再给出答案,并在每一行附上引文出处。问题在于这句引文是否有帮助。在不同的条目上得到的两个通过率,度量的多半是条目本身,因此比较必须让同一批条目经过两个提示词。这样每个条目会得到两个判定,差值逐条目计算。这就是配对比较。
两个提示词在同样的二十个条目上运行,输出报告差值及其区间。
1"""Compares the baseline prompt (P1) with prompt-b (P2) on the twenty chapter items: each pass rate, the paired difference, its interval and n.23The word 'better' is used only when the whole interval is above zero, and 'worse' only when it is below zero. A difference4whose interval includes zero is reported as no difference detected."""5import lib # first: fixes the thread counts and seeds before anything else loads67from lib.answers import chapter_items8from lib.judging import paired9from lib.stats import interval_points, pct1011result = paired("judge-answers-baseline", "judge-answers-prompt-b", [item.id for item in chapter_items()])12first, second = result.first, result.second # 1.0 or 0.0 per item, in id order13print(f"P1 (baseline) pass rate: {pct(int(first.sum()), result.n)}")14print(f"P2 (prompt-b) pass rate: {pct(int(second.sum()), result.n)}")15print(f"items that P2 passes and P1 fails: {int(((second == 1) & (first == 0)).sum())}")16print(f"items that P1 passes and P2 fails: {int(((second == 0) & (first == 1)).sum())}")17print(f"paired items (n): {result.n}")18print(f"difference, P2 minus P1: {100 * result.difference:+.1f} points")19print(f"95% interval of the difference: {interval_points(result.diff_lo, result.diff_hi)}")20if result.diff_lo > 0:21 reading = f"P2 is better at n = {result.n}"22elif result.diff_hi < 0:23 reading = f"P2 is worse at n = {result.n}"24else:25 reading = f"no difference detected at n = {result.n}"26print(f"reading: {reading}")27# The two prompts tie: the same number of items pass under each, so the pass rates and the reading say the same thing.28assert int(first.sum()) == int(second.sum()), "the two prompts must tie: the same pass count under each"
python compare-two-prompts.pyP1 (baseline) pass rate: 85.0% (17 of 20)
P2 (prompt-b) pass rate: 85.0% (17 of 20)
items that P2 passes and P1 fails: 1
items that P1 passes and P2 fails: 1
paired items (n): 20
difference, P2 minus P1: +0.0 points
95% interval of the difference: [-15.0, +15.0] points
reading: no difference detected at n = 20
在这组条目上,两个提示词打成平手。P1 通过的条目比例为 85.0% (17 of 20),P2 为 85.0% (17 of 20)。
配对视角显示了平手之下的情形。P2 通过而 P1 未通过的条目有 1 个,P1 通过而 P2 未通过的条目有 1 个。这些是评判器的判定,并不是关于提示词的事实。第一个条目是 q-0019,评判器对它给出的 PASS 违反了评判器自己的规则 3。差值(P2 减 P1)为 +0.0 points,其区间为 [-15.0, +15.0] points。区间包含零,因此在这一规模下,该示例报告未检测到差异。
下面读取评判器在两个提示词下对 q-0019 的判定,并在旁边印出该 PASS 所违反的规则。
1"""Shows one judge verdict that breaks the judge's own rule: on q-0019, the judge passes P2's reply, which answers only one of the2two readings in the reference. The rule it breaks is rule 3 of the judge's prompt, printed below from that prompt.34The hand verdict is a person's reading, written into this file. It is printed beside the judge's verdict and is the only5number here that is not a recorded one; the count of passes with it is taken from the judge's own verdicts."""6import lib # first: fixes the thread counts and seeds before anything else loads78from lib.answers import Assistant, chapter_items9from lib.judging import verdicts10from lib.prompts import P1_SHA256, P1_SYSTEM, P2_SHA256, P2_SYSTEM, REF_SYSTEM11from lib.replay import ReplayModel1213ITEM_ID = "q-0019"14HAND = {"q-0019": "FAIL"} # read by hand against rule 3: the reply answers reading 1 and not reading 21516item = next(item for item in chapter_items() if item.id == ITEM_ID)17print(f"question {item.id} (version {item.version}, {item.kind}, planted: {item.planted_why}): {item.question}")18print(f"reference answer: {item.reference_answer}")19print(f"P2 reply: {Assistant('assistant-prompt-b', P2_SYSTEM, P2_SHA256).reply(item).text}")2021raw = ReplayModel("judge-answers-prompt-b", None).reply(item.id, 0) # the judge's recorded reply to P2's answer22print(f"judge verdict on P2's reply: {verdicts('judge-answers-prompt-b', [item.id])[item.id]}")23reason = [line.strip() for line in raw.splitlines() if line.strip()][1:] # the line after the verdict24print(f"judge reason: {reason[0] if reason else 'none'}")25print(f"judge verdict on P1's reply to the same item: {verdicts('judge-answers-baseline', [item.id])[item.id]}")26print(f"hand verdict on P2's reply, under rule 3: {HAND[item.id]}")27rule_3 = next(line for line in REF_SYSTEM.splitlines() if line.startswith("3. "))28print(f"rule it breaks: {rule_3}")2930ids = [item.id for item in chapter_items()]31judged = verdicts("judge-answers-prompt-b", ids)32passes = sum(verdict == "PASS" for verdict in judged.values())33by_hand = passes - sum(1 for item_id in HAND if judged[item_id] == "PASS" and HAND[item_id] == "FAIL")34print(f"P2 passes, as judged: {passes} of {len(ids)}")35print(f"P2 passes, with {ITEM_ID} read by hand: {by_hand} of {len(ids)}")
python judge-gets-one-wrong.pyquestion q-0019 (version 2.0, ambiguous, planted: ambiguous-target): If I edit the file and then run check, will it fail?
reference answer: Reading 1: if you edited a generated file (a target), check prints DRIFT and the path for it and exits with status 1, because its hash no longer matches the lock.
Reading 2: if you edited a template, check prints nothing and exits with status 0, since a template change is not drift.
P2 reply: Quote: "Run `sablefile check` after a sync. If no output appears, no generated file has drifted since the last sync. To see what drift looks like, open a generated file, change one line, and run the check again. The check prints `DRIFT` and the path of the file." [sec:quickstart-check]
Answer: Yes, if you edit the file and then run check, it will print `DRIFT` and the path of the file, indicating a drift [sec:quickstart-check].
judge verdict on P2's reply: PASS
judge reason: The response correctly identifies that editing a generated file and running check will result in a DRIFT message and path output, aligning with Reading 1 of the reference. It does not contradict or omit any part of the reference.
judge verdict on P1's reply to the same item: FAIL
hand verdict on P2's reply, under rule 3: FAIL
rule it breaks: 3. If the reference has several readings, the response must answer every reading. A partial answer is incorrect.
P2 passes, as judged: 17 of 20
P2 passes, with q-0019 read by hand: 16 of 20
应把这份输出读作评判器判定的记录。q-0019 的参考答案有两种解读,而 P2 的回答只回答了第一种。评判器提示词的规则 3 规定,部分回答判为不通过,评判器也确实判定 P1 对同一条目的回答不通过,那个回答同样只回答了第一种解读。对 P2 的回答,它却判为通过,其理由称该回答涵盖了参考答案、没有遗漏。经人工逐条阅读,P2 通过的条目是 16 of 20 个,而不是 17 of 20 个。这个平手只差一次评判器的错误,就不再是平手。因此这里的通过率是评判器给出的通过率,这也是第 6 章要对照公开的人工评分来检验评判器的原因。
**工作准则:**在相同的条目上对提示词做配对比较,并报告差值及其区间。只有整个区间都在零以上,才可称某个提示词更好;只有整个区间都在零以下,才可称其更差。区间包含零时,应说明在这一规模下未检测到差异,这与说两个提示词相等并不是一回事。
二十个问题无法说明的事
最后这个示例仍用同样的两个提示词,来看这二十个条目能就它们说明什么。结果表明,二十个条目对这两个提示词几乎说明不了什么。
1"""Shows that twenty items cannot tell the two prompts apart: the interval of the paired difference includes zero.23The assert is the chapter's claim, checked against the numbers; the example prints no verdict word."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.answers import chapter_items7from lib.judging import paired8from lib.stats import interval_points910result = paired("judge-answers-baseline", "judge-answers-prompt-b", [item.id for item in chapter_items()])11print(f"paired items (n): {result.n}")12differing = int((result.first != result.second).sum())13print(f"items where the two verdicts differ: {differing}")14print(f"difference, P2 minus P1: {100 * result.difference:+.1f} points")15print(f"95% interval of the difference: {interval_points(result.diff_lo, result.diff_hi)}")16print(f"interval width: {100 * (result.diff_hi - result.diff_lo):.1f} points")17# Zero inside the interval is the claim: no difference that twenty items can show.18assert result.diff_lo <= 0.0 <= result.diff_hi, "the interval of the difference must include zero"19# Two items carry the whole comparison: the pass rates differ only where these two verdicts differ.20assert differing == 2, "exactly two items must have differing verdicts"
python twenty-cannot-tell.pypaired items (n): 20
items where the two verdicts differ: 2
difference, P2 minus P1: +0.0 points
95% interval of the difference: [-15.0, +15.0] points
interval width: 30.0 points
两个提示词得到不同判定的条目只有 2 个,因此二者之差只能以粗糙的步长变化,每次移动一个条目。差值的区间宽度为 30.0 points。代码中有一条 assert 要求该区间包含零,所以一旦这些数字不再支持这一论断,论断就会以显眼的方式失败。这样宽的区间只能检测到很大的变化,而对用户仍然重要的较小变化,完全可能藏在区间之内。
在生产环境中,同一种错误以三种形式出现:相信一个无法承载所赋予它的论断的数字。某团队在最先编写的二十个条目上选取数字较高的提示词,报告中又略去了区间,于是没有人看出这个数字的依据有多薄。某个评判器因其判定看上去合理而被信任,却没有人对照参考答案去读它给出的 FAIL 理由。一组已记录的回复,在助手所见的各个部分已经改变之后仍被保留,测量的其实是旧系统,而它的数字看上去依然正常。第一种情形,当报告在每个比率旁附有区间时,你就能察觉。第二种情形,当有人在每次运行时对照参考答案去读评判器的 FAIL 理由时,你就能察觉。第三种情形,只有当记录同时对所给文档做了哈希,才会显现。本书的清单对提示词和测试集做了哈希,因此提示词的改动在重放时会被拒绝,但对各部分的改动却不会被发现。
后面有两章会迈出下一步。第 3 章构建本章所缺少的更大的集合,第 8 章则算出一个差异需要多少条目才能显现。
本章做出了这些决定:
- 每个问题都标明其版本,其答案只来自该版本的文档。
- 本章的集合为二十个条目,逐条通读,并标出埋入的条目。
- 每份记录按模型、提示词和文档各录制一次,并在每次比较时重放。
- 引用与拒答先在代码中检查,然后才由评判器阅读回复。
- 评判器依据书面参考答案评分,在校准对其核验之前,它的比率只算作评判器自己的比率。
- 通过率须附带其分母、其区间,以及各类条目各自的比率。
- 提示词在相同的条目上比较,只有整个区间完全位于零的一侧,才能判定其中一个更好或更差。
工作准则: 在每一项比较旁都报告区间。二十个条目能找出有问题的答案,却无法为两个提示词排出高下。
本章,在本版次中
版次 2026.11.3示例于 2026年11月21日 通过本章 11个示例 个示例中有 11 个通过
译自英文版图书,版次 2026.11.3。