阅读回复,找出该测量什么

阅读约需 37分钟 分钟,6节中含2个示例。

版次 2026.11.3 示例于 2026年11月21日 通过 54个示例中有54个通过 变化内容

AI 翻译。由人工以英文核验。 我们如何制作图书

译自英文版图书,版次 2026.11.3。 阅读原文

读完本章,你可以阅读一组助手的回复,把每条回复的问题归入某一种失败模式,并说明其中哪些失败是检查能够发现的。

第 1 章结尾处提到一个局限:二十个条目可以发现错误的回答,却无法给两个提示词排出高低。本书中的助手依据 sablefile 文档的三个版本(1.0、1.1 和 2.0)作答,针对某一版本的提问,可能得到一个对另一版本才正确的回答。通过率把所有这类失败一视同仁地计数,却不指明其中任何一种。因此,只用通过率做测量的团队,会针对自己预料到的失败编写检查,而没有预料到的失败就无人检查。先阅读回复,正是发现后一类失败的办法。第 1 章的区间说明一个分数在多大程度上可信,本章则要问每个变体错在哪里,这是分数回答不了的。

先阅读、后计数,在本书之外也有依据。第 1 章引用了 Shankar 及其同事的 Who Validates the Validators?(2024),说明人们在评判输出的同时会不断修改自己的评判标准。该论文主要介绍的是一个名为 EvalGen 的系统,本章所依据的是其中对九位从业者的用户研究。研究记录的标准漂移,发生在人们评判输出的过程中。至于对流水线的改动可能使标准发生漂移,那是作者的讨论,并非该研究的发现。这只是一项研究,所以本章把它当作一个提醒,而不是本书所测得的结果。

本章使用 80 个开发条目,即本书在开发测量方法期间所用的那部分测试集。本章只阅读开发条目,因此这里没有任何东西是针对留出条目调整的,留出条目上的拒答次数见第 10 章。阅读产生三样东西:对每条回复的一则记录,这些记录归并而成的缺陷计数,以及针对每种缺陷的一个候选检查。这里的每条回复,都是为每个变体各录制一次、再由示例重放的记录,因此每次运行所读到的内容都相同。对回复的引文来自录制的输出,参考答案则印在各自条目的旁边,所以正文所引的任何一行,都能在示例的输出中找到。这些记录存放在一个数据文件中,由统计示例读取。

在整章中,一条回复若给出了参考答案,或者因参考答案指出文档并未说明而拒绝作答,就算作正确。它的引用另行检查,依靠第 1 章的引用检查和第 4 章中的各项检查。除此之外的情形都是失败,本章其余部分会说明属于哪一种。同样的步骤也适用于你自己的重放文件:把其中的回复按类别分组,对照参考答案逐条阅读,按主要缺陷给每条编码,再为每种模式写一个候选检查。本章对 sablefile 的回复做的正是这些,所以你可以在能够打开的数据上跟着每一步操作。

本章不挑选检查,也不编写评判者的评分细则,更不估计每种模式在实际使用中出现的频率。这些内容见后续各章,而每一章都要用到这里所命名的模式。

回复的样貌

最近一次修改见于版次 2026.11.3

三个版本的助手回答了同一批开发集条目,本章的每一项比较,都是把一个版本的回复与另一个版本的回复并排对照。基线是第 1 章所测量的配置。它只从问题所询问的那个版本中检索各节,因此它的上下文只包含一个版本的文档。未过滤版本去掉了基线的版本过滤。它的上下文可以包含三个版本的各节,每一节都标明所属版本,因此模型既能看出版本不符,也可能照抄错误版本的文字。第二模型版本沿用基线的提示词,但把回答交给一个较小的模型,即 8 位量化构建、参数量为 6 亿(0.6 billion)的 Qwen3。提示词相同,所以与基线的差异来自模型,而不是提示词。

每个开发集条目都有一个类别,类别决定了正确的回复是什么样子。可回答条目是文档能够回答的条目。不可回答条目询问的是文档没有说明的内容,正确的回复是用助手被给定的措辞拒答:“The docs don't say.”版本相关条目的答案随所询问的版本而变化,因此同一个问题问到两个版本,可能有两个正确答案。歧义条目可以有两种理解,文档对每一种理解都有回答。有些条目是植入的,专为捕捉某一种特定的错误而写。输出为每个植入条目给出其原因,原因是以下之一:错误前提、近似术语、版本互换、过时答案或歧义目标。一条回复在植入条目上失败,就表明助手犯了该条目所针对的那种错误。

阅读一个条目分三步。先看问题及其版本,因为回复只有对其所指明的版本才算正确。再读参考答案,即作者对正确回复应当说什么的陈述。然后对照参考答案,逐条阅读三个回复。读完这些之后,读者才可以对回复分组或计数。

下面的示例是本章的起点。它打印每个开发集条目及其参考答案和各版本的回复,按类别分组并按 ID 排序,因此同一问题的三条回复排在相邻的行上。请顺着一个条目的各行横向阅读,而不要顺着某一个版本的列纵向阅读。这些版本在同一个问题上的失败方式各不相同,值得看的正是这种差异。每条回复都截取前 120 个字符,其所在行会说明该回复是否拒答以及引用了什么。若回复把拒答措辞夹在其他文字之中,则标记为短语而不是拒答,这样读者可以看出解析器把界线划在哪里。

read-replies-dev.py Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0 通过
"""Prints the dev items, grouped by kind and sorted by ID, with each item's reference answer and the baseline,unfiltered and second-model replies.The reference is printed in full under its item, since the reading compares each reply with it. Each reply is one line:whether it declined, its citations, and its first 120 characters. The reader's job is to set these side by side andsee which failure each reply shows (the codes are in data/failure-codes.json)."""import lib  # first: fixes the thread counts and seeds before anything else loadsfrom lib.contract import DECLINE, parse_replyfrom lib.prompts import P1_SHA256from lib.replay import ReplayModelfrom lib.testset import load_itemsRECORDINGS = {  # the three variants, in the order the chapter discusses them    "baseline": "assistant-baseline",    "unfiltered": "assistant-unfiltered",    "second-model": "assistant-second-model",}KINDS = ("answerable", "unanswerable", "version-dependent", "ambiguous")  # plan section 3.1 orderSHOWN = 120  # characters of each reply printed; the full text is in the recording# The prompt hash is checked here, so a recording made from another prompt fails to load.models = {name: ReplayModel(recording, P1_SHA256) for name, recording in RECORDINGS.items()}items = load_items(split="dev")  # dev only: the held-out items are not read in this chapter (plan section 3.4)for kind in KINDS:    group = [item for item in items if item.kind == kind]  # load_items already sorts by ID    print(f"## {kind}: {len(group)} dev items")    for item in group:        planted = f"planted ({item.planted_why})" if item.planted else "not planted"        print(f"{item.id} | version {item.version} | {planted} | {item.question}")        reference = item.reference_answer or "none: the item is unanswerable, so the right reply declines"        print(f"  reference: {reference}")        for name in RECORDINGS:            reply = parse_reply(models[name].reply(item.id, 0))            if reply.declined:                mark = "declined"            elif DECLINE in reply.text:                # The phrase is there, but with other text: the parser does not call that a decline.                mark = "decline phrase with other text"            else:                mark = f"cites {' '.join(reply.citations)}" if reply.citations else "no citation"            text = reply.text.replace("\n", " ")  # a reply may hold line breaks; one line keeps the output readable            more = "..." if len(text) > SHOWN else ""            print(f"  {name} ({mark}): {text[:SHOWN]}{more}")
输出python read-replies-dev.py
## answerable: 40 dev items
q-0005 | version 2.0 | planted (near-miss-term) | Does sablefile 2.0 still read sablefile.lock in my project root?
  reference: Only as a fallback. If the new file is missing and an old sablefile.lock is present, a 2.0 command reads the old file but warns you about it. Once the new file exists, 2.0 reads only that one, and sablefile never deletes the old file.
  baseline (cites install-upgrade): No, sablefile 2.0 does not read `sablefile.lock` in the project root. Version 2.0 moves the lock file to a new place. Th...
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0006 | version 2.0 | not planted | Is there a way to make check exit with status 0 even when a file has drifted in 2.0?
  reference: Yes. Pass --no-fail to check. It still prints each DRIFT line, but it exits with status 0, so a script can report drift without failing. The flag changes nothing except the exit status.
  baseline (cites no-fail-flag): Yes, there is a way to make `check` exit with status 0 even when a file has drifted in 2.0. Use the `--no-fail` flag wit...
  unfiltered (cites no-fail-flag): Yes, there is a way to make `check` exit with status 0 even when a file has drifted in version 2.0. Use the `--no-fail` ...
  second-model (cites no-fail-flag): The `--no-fail` flag is used to make the check exit with status 0 even when files have drifted. [sec:no-fail-flag]
q-0007 | version 1.1 | not planted | In 1.1, will an ignore entry of docs/**/*.tmp catch a file two folders down, like docs/x/y/a.tmp?
  reference: Yes. Version 1.1 adds the double-star pattern, which matches zero or more folders. So docs/**/*.tmp matches docs/x/y/a.tmp as well as files directly in docs.
  baseline (cites ignore-recursive): Yes, an ignore entry of `docs/**/*.tmp` will catch a file two folders down, like `docs/x/y/a.tmp`. [sec:ignore-recursive...
  unfiltered (cites ignore-recursive): Yes, in version 1.1, an ignore entry of `docs/**/*.tmp` will catch a file two folders down, like `docs/x/y/a.tmp`. [sec:...
  second-model (no citation): The docs section does not allow for an ignore entry of docs/**/*.tmp to catch a file two folders down, as per the rules....
q-0008 | version 2.0 | not planted | In 2.0, is there a way to re-include something that an earlier ignore line had excluded?
  reference: Yes, with a leading exclamation mark. A line that starts with ! negates an earlier match, so a later line can restore a file that an earlier line ignored.
  baseline (cites ignore-syntax): Yes, in 2.0, you can re-include something that an earlier ignore line had excluded by using a leading `!` to negate an e...
  unfiltered (cites ignore-syntax): Yes, in version 2.0, you can re-include something that an earlier ignore line had excluded by using a leading `!` to neg...
  second-model (declined): The docs don't say.
q-0009 | version 1.0 | not planted | What does one line of the 1.0 lock file hold?
  reference: The SHA-256 hash of a generated file's content, then two spaces, then that file's path. The 1.0 lock has no mode field.
  baseline (cites lockfile): One line of the 1.0 lock file holds the SHA-256 hash of the file's content, two spaces, then the path. [sec:lockfile]
  unfiltered (cites lockfile): One line of the 1.0 lock file holds the SHA-256 hash of the file's content, then the file's mode, then the path. [sec:lo...
  second-model (cites lockfile): The lock file holds the SHA-256 hash of the file's content, two spaces, and the path. [sec:lockfile]
q-0010 | version 2.0 | not planted | Will syncing the same content twice in 2.0 produce a different lock file each time?
  reference: No. The 2.0 lock file is JSON with no timestamps, so the same content always gives back an identical file.
  baseline (cites lockfile): No, syncing the same content twice in 2.0 will not produce a different lock file each time. The lock file has no timesta...
  unfiltered (cites lockfile): No, syncing the same content twice in 2.0 will produce an identical lock file each time because the lock file has no tim...
  second-model (declined): The docs don't say.
q-0011 | version 1.1 | not planted | How many times does 1.1 retry a remote template fetch that keeps failing, and can a flag change that?
  reference: Five times, before the command gives up. In 1.1 that number is fixed, and no flag or setting can change it.
  baseline (cites remote-retries): The count is fixed in 1.1, and no flag or setting changes it. [sec:remote-retries]
  unfiltered (cites remote-retries): In version 1.1, sablefile retries a remote template fetch up to 5 times before giving up. The count is fixed and no flag...
  second-model (cites remote-retries): The count is fixed in 1.1, and no flag or setting changes it. [sec:remote-retries]
q-0012 | version 2.0 | not planted | I want sync to retry a slow remote template only twice in 2.0. What do I pass?
  reference: Pass --retries 2 to sync. The flag sets how many times a failed fetch is retried, in place of the default of five, and it only affects the command it is given to.
  baseline (cites retries-flag): Pass `--retries 2` to sync. [sec:retries-flag]
  unfiltered (cites retries-flag): Pass `--retries 2` to sync. [sec:retries-flag]
  second-model (cites retries-flag): `--retries 2` to pass. [sec:retries-flag]
q-0013 | version 1.0 | not planted | In 1.0, which table in sablefile.toml holds the values a placeholder can name?
  reference: The [values] table. Each name in it can be used as a placeholder, and a placeholder name in a template must match a key in that table.
  baseline (cites config-values): The `[values]` table holds the values a placeholder can name. [sec:config-values]
  unfiltered (cites config-values): The `[values]` table in sablefile.toml holds the values a placeholder can name. [sec:config-values]
  second-model (cites config-values): The values table holds the values a placeholder can name. [sec:config-values]
q-0014 | version 1.1 | not planted | What does sablefile init put into a new sablefile.toml in 1.1?
  reference: A [values] table with one entry, project, set to the folder's name, and no [[file]] tables. init writes no other file.
  baseline (cites init-command): `sablefile init` creates `sablefile.toml` in the folder where you run it, or in the folder you name. The new file has a ...
  unfiltered (cites init-command): `sablefile init` creates `sablefile.toml` in the folder where you run it, or in the folder you name. The new file has a ...
  second-model (declined): The docs don't say.
q-0051 | version 2.0 | planted (near-miss-term) | My 1.x ignore file uses shell globs, so does *.log only match files in the top folder in 2.0?
  reference: No. Shell glob rules no longer apply in 2.0. Entries use gitignore syntax, so *.log matches at any depth, and a trailing slash limits an entry to directories.
  baseline (cites ignore-syntax): In 2.0, entries use gitignore syntax. A line such as `*.log` matches at any depth. Therefore, `*.log` matches files in a...
  unfiltered (cites ignore-syntax): No, in 2.0, `*.log` matches at any depth, not just in the top folder. [sec:ignore-syntax]
  second-model (declined): The docs don't say.
q-0052 | version 2.0 | planted (near-miss-term) | I have {{project}} all through my templates from 1.x. Is that still valid in 2.0?
  reference: No. In 2.0 each placeholder is written as a dollar sign and braces, ${project}. The double-brace form is an error: rendering fails, sync reports the file and line, and it exits with status 1.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0053 | version 2.0 | planted (stale-answer) | I have an old sablefile.lock from 1.1. When I first run sync --apply under 2.0, what happens to it?
  reference: 2.0 writes a new .sablefile/lock.json holding each file's hash and, where the old file recorded one, its mode. The old file is not deleted, so you remove it yourself. A warning is printed when 2.0 reads the old file.
  baseline (cites lock-migration): When a 2.0 command finds an older `sablefile.lock` and no `.sablefile/lock.json`, it reads the old file and prints a war...
  unfiltered (cites lock-migration): When a 2.0 command finds an older `sablefile.lock` and no `.sablefile/lock.json`, it reads the old file and prints a war...
  second-model (cites lock-migration): The old `sablefile.lock` file from version 1.1 is not written to the new lock file, so it remains unchanged. [sec:lock-m...
q-0054 | version 2.0 | planted (stale-answer) | Does the 2.0 lock keep each generated file's mode, and what happens to modes from an old lock?
  reference: Yes. 2.0 records each generated file's mode in .sablefile/lock.json. When a 2.0 command migrates an older lock, it keeps each mode that the old file has.
  baseline (cites lock-mode lock-migration): Yes, the 2.0 lock keeps each generated file's mode, the Unix permission bits such as `0644`, in `.sablefile/lock.json` [...
  unfiltered (cites lock-mode lock-migration): Yes, the 2.0 lock keeps each generated file's mode, the Unix permission bits such as `0644`, in `.sablefile/lock.json` [...
  second-model (cites lock-mode lock-migration): The 2.0 lock keeps each generated file's mode, as specified in [sec:lock-mode]. When a lock file is migrated from `sable...
q-0055 | version 2.0 | planted (stale-answer) | How do I make sync retry a remote template fetch fewer times in 2.0?
  reference: Pass --retries N with the number you want, for example --retries 1. The flag sets the retry count for sync and diff; without it the default is five.
  baseline (cites retries-flag): Use the `--retries N` flag to set the number of times `sync` retries a failed fetch of a remote template, where N is a w...
  unfiltered (cites retries-flag): To make `sync` retry a remote template fetch fewer times in version 2.0, use the `--retries N` flag where N is a whole n...
  second-model (cites retries-flag): The `--retries` flag sets how many times `sync` and `diff` retry a failed fetch of a remote template, in place of the de...
q-0056 | version 1.0 | not planted | Does sablefile check need a network connection to run in 1.0?
  reference: No. check does not fetch remote templates or render templates, so it works without a network.
  baseline (cites check-command): No, sablefile check does not need a network connection to run. [sec:check-command]
  unfiltered (cites remote-fetch-failure): Yes, sablefile checks need a network connection to run in 1.0, as it requires fetching remote templates, and if a remote...
  second-model (declined): The docs don't say.
q-0057 | version 2.0 | not planted | Will check pick up a change that someone made on a remote template's server?
  reference: No. check never fetches remote templates, so only the next sync's output and the output of diff reflect that change.
  baseline (cites remote-retries): No, `check` does not see changes on a remote template's server. [sec:remote-retries]
  unfiltered (cites remote-retries): No, `check` does not fetch anything and never retries, so it will not pick up a change that someone made on a remote tem...
  second-model (cites remote-templates): Check does not see changes on a remote template's server. A change on the server affects what the next sync writes and w...
q-0058 | version 2.0 | not planted | If I run check --no-fail while drift is present, what does its printed output look like?
  reference: It still prints a DRIFT line and the path for each drifted file. The flag only changes the exit status to 0.
  baseline (cites quickstart-check): Run `sablefile check` after a sync. If no output appears, no generated file has drifted since the last sync. To see what...
  unfiltered (declined): The docs don't say.
  second-model (no citation): The check prints `DRIFT` and the path, and exits with status 1. A check with no drift exits with status 0. Pass `--no-fa...
q-0059 | version 1.1 | not planted | What is the mode that sablefile 1.1 stores for each generated file, and which command stores it?
  reference: The Unix permission bits of the file, such as 0644. sync saves it for every file it writes.
  baseline (cites lock-mode lock-mode): The mode that sablefile 1.1 stores for each generated file is the Unix permission bits, such as `0644` [sec:lock-mode]. ...
  unfiltered (cites lock-mode): The mode that sablefile 1.1 stores for each generated file is the Unix permission bits, such as `0644`. The `sync` comma...
  second-model (cites lock-mode lock-mode): The mode that sablefile 1.1 stores for each generated file is described in [sec:lock-mode]. The command that stores it i...
q-0060 | version 2.0 | not planted | What top-level keys does the 2.0 lock file JSON contain?
  reference: A format key, set to the string 2.0, and a files key that maps each generated path to its SHA-256 hash and mode.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (no citation): The 2.0 lock file JSON contains keys: `lock.json`.
q-0061 | version 1.0 | not planted | What is the usual way to preview sync's changes in 1.0 before writing?
  reference: Run sablefile diff. It shows the differences between each rendered template and its target on disk, without writing any file or changing the lock.
  baseline (cites diff-command): Run `sablefile diff` to preview sync's changes before writing. [sec:diff-command]
  unfiltered (cites sync-default): To see the changes before they are written, run `sablefile diff`, which shows the same differences without writing anyth...
  second-model (cites diff-command): The usual way to preview sync's changes in 1.0 before writing is to use `sablefile diff`. [sec:diff-command]
q-0062 | version 2.0 | not planted | Can I point sablefile init at a folder other than the current one in 2.0?
  reference: Yes. Give the folder as an argument, such as init path/to/folder. The folder must already exist, and a relative path is read from the folder where you run the command.
  baseline (cites init-directory): Yes, you can point `sablefile init` at a folder other than the current one. To do this, provide the path to the folder a...
  unfiltered (cites init-directory): Yes, you can point `sablefile init` at a folder other than the current one in 2.0. To do this, give the folder as an arg...
  second-model (declined): The docs don't say.
q-0063 | version 1.1 | not planted | If a target sits in folders that do not exist yet, what does sync do in 1.1?
  reference: It creates the missing folders before it writes the file, so no setup is needed. sync never removes a folder, even one that ends up empty.
  baseline (declined): The docs don't say.
  unfiltered (cites sync-create-dirs): If a target sits in folders that do not exist yet, sync creates the folder before it writes the file in version 1.1. [se...
  second-model (no citation): The lockfile is `sablefile.lock` in the project root. It holds one line for each generated file: the SHA-256 hash of the...
q-0064 | version 2.0 | not planted | After I delete a target from sablefile.toml, does sync --apply in 2.0 remove its now-empty folder?
  reference: No. A sync run never deletes a directory, which means any empty ones you leave behind remain.
  baseline (cites sync-create-dirs): No, sync --apply does not remove empty folders. sync never removes a folder, so empty folders you leave behind stay wher...
  unfiltered (cites sync-create-dirs): No, sync --apply in 2.0 does not remove its now-empty folder. sync never removes a folder, so empty folders you leave be...
  second-model (declined): The docs don't say.
q-0065 | version 1.0 | not planted | Where can template files sit inside a 1.0 project?
  reference: Anywhere in the project. The template key in each [[file]] table gives the path from the project folder, and a template is read only when a command renders it.
  baseline (cites templates-location): Template files can sit in any folder of the project. [sec:templates-location]
  unfiltered (cites templates-location): Template files can sit in any folder of the project. [sec:templates-location]
  second-model (no citation): Template files are ordinary text files, and they can sit in any folder of the project. The `template` key in a `[[file]]...
q-0066 | version 2.0 | not planted | Does sablefile run any code that is inside a template?
  reference: No. Rendering only substitutes placeholders. It executes nothing in the template, and the same template with the same values always gives the same output.
  baseline (cites template-render): No, sablefile does not run any code that is inside a template. Rendering does not run code in the template. [sec:templat...
  unfiltered (cites template-render): No, sablefile does not run any code that is inside a template. Rendering does not run code in the template. [sec:templat...
  second-model (declined): The docs don't say.
q-0067 | version 1.1 | not planted | Can a [[file]] table in 1.1 take a template that lives at an https address?
  reference: Yes. The template key can name an https URL. sablefile fetches it when a command renders the template, and the fetched text is not copied into the project.
  baseline (cites remote-templates): Yes, a [[file]] table in 1.1 can take a template that lives at an https address. [sec:remote-templates]
  unfiltered (cites config-files-table): Yes, a [[file]] table in 1.1 can take a template that lives at an https address. [sec:config-files-table]
  second-model (declined): The docs don't say.
q-0068 | version 2.0 | not planted | What happens when a remote fetch in 2.0 still fails after every retry?
  reference: The command halts, reporting the failing address in its error message, and ends with status 1. Targets it never wrote retain their previous lock records.
  baseline (cites remote-fetch-failure): If a remote template still cannot be fetched after the last retry, the command stops with an error that names the URL, a...
  unfiltered (cites remote-fetch-failure): If a remote template still cannot be fetched after the last retry, the command stops with an error that names the URL, a...
  second-model (declined): The docs don't say.
q-0069 | version 1.0 | not planted | What exit status does sync give in 1.0 when a template file is missing?
  reference: Status 1. sync cannot finish in that case, and its error message names the file that caused the failure.
  baseline (cites sync-exit-status): Sync exits with status 1 when it cannot finish, for example when a template file is missing or a remote template cannot ...
  unfiltered (cites sync-exit-status): sync exits with status 1 when a template file is missing. [sec:sync-exit-status]
  second-model (cites sync-exit-status): The exit status for sync in 1.0 when a template file is missing is 1. [sec:sync-exit-status]
q-0070 | version 1.1 | not planted | If sync in 1.1 exits with status 0, does that mean the generated text is what I want?
  reference: Not necessarily. Status 0 only shows that sync finished, so you should open the generated files and check them.
  baseline (cites sync-exit-status): No, if sync exits with status 0, it only means that sync completed successfully, not that the generated text is what you...
  unfiltered (cites sync-exit-status): No, if sync in 1.1 exits with status 0, it does not mean the generated text is what you want. A status of 0 shows that s...
  second-model (declined): The docs don't say.
q-0071 | version 1.0 | not planted | Where do I list paths for sablefile 1.0 to leave alone, and what happens to them?
  reference: In the ignore file .sablefileignore in the project root, with one entry per line. Sync leaves an ignored target unwritten, check and diff both pass over it, and no lock record is made for it.
  baseline (cites ignore-file): You list paths for sablefile 1.0 to leave alone in the ignore file, `.sablefileignore`, which sits in the project root. ...
  unfiltered (cites ignore-file): List paths for sablefile 1.0 to leave alone in the `.sablefileignore` file, which sits in the project root. Put one entr...
  second-model (no citation): The ignore file, `.sablefileignore`, sits in the project root and lists paths that sablefile must leave alone. A project...
q-0072 | version 2.0 | not planted | With --apply, does 2.0's sync write a target that my ignore file lists?
  reference: No. An ignored target is never written, and it gets no lock record either.
  baseline (cites ignore-file): No, with `--apply`, 2.0's sync does not write a target that is listed in the ignore file. [sec:ignore-file]
  unfiltered (cites ignore-file): No, with --apply, 2.0's sync does not write a target that is listed in the ignore file. [sec:ignore-file]
  second-model (declined): The docs don't say.
q-0073 | version 2.0 | not planted | Which lines in a 2.0 ignore file are skipped as comments?
  reference: Lines that begin with a hash sign. Blank lines are also ignored.
  baseline (cites ignore-syntax): Lines that start with `#` are comments. [sec:ignore-syntax]
  unfiltered (cites ignore-syntax): Lines that start with `#` are comments. [sec:ignore-syntax]
  second-model (declined): The docs don't say.
q-0074 | version 1.1 | not planted | A check in 1.1 reports drift for a file I did not mean to change. What should I do first?
  reference: Start with diff, which shows how the file differs from its template. Keep wanted changes by moving them into the template before running sync; discard unwanted ones by reverting the file and running check again.
  baseline (cites troubleshoot-drift): Look at the change before you act. Run `sablefile diff` to see how the file differs from its template. [sec:troubleshoot...
  unfiltered (cites troubleshoot-drift): Look at the change before you act. [sec:troubleshoot-drift]
  second-model (cites check-command): Run `sablefile diff` to see how the file differs from its template. [sec:check-command]
q-0075 | version 2.0 | not planted | After installing the 2.0 archive, how do I confirm which version the shell is running?
  reference: Run sablefile --version. The output has the form sablefile 2.0, and that number should match the docs you are reading.
  baseline (cites install-check-version): Run `sablefile --version` to see the installed version. [sec:install-check-version]
  unfiltered (cites install-check-version): Run `sablefile --version` to see the installed version. The output has the form `sablefile 2.0`. Check that the number m...
  second-model (declined): The docs don't say.
q-0076 | version 1.0 | not planted | How do I remove sablefile 1.0 without losing my project's files?
  reference: Run python3 -m pip uninstall sablefile. That removes the command and its package, and leaves the project's own files alone, including the config, templates, outputs and lock.
  baseline (cites install-uninstall): To remove sablefile 1.0, run `python3 -m pip uninstall sablefile`. This removes the `sablefile` command and its package....
  unfiltered (cites install-uninstall): Run `python3 -m pip uninstall sablefile`. This removes the `sablefile` command and its package. It does not change your ...
  second-model (declined): The docs don't say.
q-0077 | version 2.0 | not planted | I installed sablefile but my shell says the command is not found. What should I check in 2.0?
  reference: Ask pip where it put the package, using pip show sablefile from that very same shell. If a location comes back yet the command fails, the scripts folder may be missing from your path.
  baseline (cites troubleshoot-command-not-found): If the shell reports that `sablefile` is not found, the command is not installed in the environment that is active. Run ...
  unfiltered (cites troubleshoot-command-not-found): If the shell reports that `sablefile` is not found, the command is not installed in the environment that is active. Run ...
  second-model (declined): The docs don't say.
q-0078 | version 1.1 | not planted | Does upgrading sablefile to a newer 1.x archive change any file in my project?
  reference: No. The upgrade writes no project file. Read the changelog in the newer version's folder first, since it lists each section whose behaviour changed.
  baseline (cites install-upgrade): No, installing a new version of sablefile does not write any project file. [sec:install-upgrade]
  unfiltered (cites install-upgrade): No, installing a new version does not write any project file. [sec:install-upgrade]
  second-model (declined): The docs don't say.
q-0079 | version 2.0 | not planted | After a 2.0 sync --apply, how do I confirm the template placeholder was filled in?
  reference: Open the target, such as README.md, and look for the project name in the spot where the placeholder stood.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0080 | version 1.0 | not planted | What is the difference between a placeholder and a value in sablefile 1.0?
  reference: A placeholder is a name in a template that sync replaces. A value is the string in the [values] table that the placeholder's name points to.
  baseline (cites glossary-terms): A placeholder is a name in a template that sync replaces with a value, written as `{{project}}` in this version. A value...
  unfiltered (decline phrase with other text): The sections do not define or differentiate between a placeholder and a value in sablefile 1.0. [sec:template-render] de...
  second-model (no citation): The difference between a placeholder and a value in sablefile 1.0 is that a placeholder is a name in a template that syn...
## unanswerable: 13 dev items
q-0015 | version 1.0 | planted (false-premise) | In sablefile 1.0, what does the --dry-run flag on sync print?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0016 | version 2.0 | not planted | Does sablefile 2.0 run on Windows?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0017 | version 2.0 | not planted | How fast is sablefile check on a project with several thousand generated files?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0018 | version 1.1 | not planted | Can sablefile 1.1 fetch a remote template from a private URL that needs a login token?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (cites remote-retries): Yes, sablefile 1.1 can fetch a remote template from a private URL that needs a login token, as long as the token is incl...
  second-model (declined): The docs don't say.
q-0034 | version 1.1 | planted (false-premise) | In sablefile 1.1, what does sync --apply write to the lock file?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (cites sync-lock-update): The lock file is how check finds drift, so run sync after each change you intend to keep. Commit the lock file with the ...
q-0035 | version 1.0 | planted (false-premise) | Does check in sablefile 1.0 have a --no-fail option?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0036 | version 1.1 | planted (false-premise) | In sablefile 1.1, what does the --quiet flag on sync do?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (cites sync-default): The --quiet flag on sync is used to suppress the display of the diff output. It does not change the lock file or the syn...
q-0037 | version 1.0 | not planted | In sablefile 1.0, what happens when a template uses a placeholder whose name is not in [values]?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0038 | version 2.0 | not planted | Is there a maximum number of templates a 2.0 project can have?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0039 | version 1.1 | not planted | What exit status does sablefile diff return when it finds differences in 1.1?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (cites check-exit-status): When check finds a drifted file, it prints `DRIFT` and the path, and exits with status 1. [sec:check-exit-status]
  second-model (declined): The docs don't say.
q-0040 | version 2.0 | not planted | If a file listed in the lock is missing from disk, does check report it in 2.0?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0041 | version 2.0 | not planted | Is there a limit on how long a value in [values] can be?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0042 | version 1.0 | not planted | Can sablefile 1.0 set the output colour through a color key in sablefile.toml?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
## version-dependent: 17 dev items
q-0001 | version 1.1 | not planted | In sablefile 1.1, if I run sync with no flags, are the target files written straight away?
  reference: Yes. In 1.1 a plain sync writes each target and records it in the lock file. To preview the changes first, run diff, which shows them without writing anything.
  baseline (cites sync-default): Yes, if you run `sablefile sync` with no flags, the target files are written straight away. [sec:sync-default]
  unfiltered (cites sync-default): Yes, in sablefile 1.1, if you run `sync` with no flags, the target files are written straight away. This is because `sab...
  second-model (decline phrase with other text): The docs don't say. The docs don't say.
q-0002 | version 2.0 | not planted | In sablefile 2.0, if I run sync with no flags, are the target files written?
  reference: No. In 2.0 a plain sync prints a plan that lists what would be created or updated, and it writes nothing. Adding --apply makes sync write the targets and update the lock file.
  baseline (cites sync-default): No, the target files are not written when running `sablefile sync` with no flags. Instead, sync prints a plan and writes...
  unfiltered (cites apply-flag): No, in sablefile 2.0, if you run `sync` with no flags, the target files are not written. The `--apply` flag is required ...
  second-model (declined): The docs don't say.
q-0003 | version 1.0 | planted (version-swap) | In sablefile 1.0, is ${project} the right way to insert the project name into a template?
  reference: No. In 1.0 a placeholder is a name in double braces, written {{project}}. The text ${project} is not a placeholder in 1.0, so it is copied to the output unchanged.
  baseline (cites placeholders): No, `${project}` is not the right way to insert the project name into a template in sablefile. The correct way is to use...
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0004 | version 2.0 | not planted | In sablefile 2.0, is ${project} the right way to insert the project name into a template?
  reference: Yes. In 2.0 a placeholder is a name with a dollar sign and braces around it, so the form ${project} is replaced by the project value. The old double-brace form is an error.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0021 | version 1.0 | not planted | In sablefile 1.0, does check give a nonzero exit status when a generated file has drifted?
  reference: No. In 1.0 check prints DRIFT and the path for a drifted file but exits with status 0, so a script has to read the printed lines to see drift.
  baseline (cites check-exit-status): No, check does not give a nonzero exit status when a generated file has drifted. The check command prints "DRIFT" and th...
  unfiltered (cites quickstart-check): Yes, in sablefile 1.0, `check` gives a nonzero exit status when a generated file has drifted. This is because, without t...
  second-model (cites check-exit-status): The check exits with status 0 even when files have drifted, so a script cannot detect drift from the exit status in this...
q-0022 | version 2.0 | not planted | In sablefile 2.0, does check give a nonzero exit status when a generated file has drifted?
  reference: Yes. check exits with status 1 when a file has drifted. Adding --no-fail changes that to status 0 while still printing the DRIFT lines.
  baseline (cites check-exit-status): Yes, check gives a nonzero exit status when a generated file has drifted. [sec:check-exit-status]
  unfiltered (cites no-fail-flag no-fail-flag): No, in sablefile 2.0, `check` does not give a nonzero exit status when a generated file has drifted if the `--no-fail` f...
  second-model (cites check-exit-status): The check exits with a nonzero exit status when a generated file has drifted. [sec:check-exit-status]
q-0023 | version 1.0 | not planted | Does an ignore entry of *.log in sablefile 1.0 skip a target at docs/build.log?
  reference: No. In 1.0 each entry is a shell glob matched against the whole target path, and shell globs do not match across folders. The entry only covers files at the top level of the project.
  baseline (cites ignore-syntax): Yes, an ignore entry of `*.log` in sablefile 1.0 skips a target at `docs/build.log`. [sec:ignore-syntax]
  unfiltered (cites ignore-syntax): Yes, an ignore entry of `*.log` in sablefile 1.0 skips a target at `docs/build.log`. [sec:ignore-syntax]
  second-model (cites ignore-syntax): The docs section does not skip the target at docs/build.log. The ignore entry of *.log in sablefile 1.0 does not skip th...
q-0024 | version 2.0 | not planted | Does an ignore entry of *.log in sablefile 2.0 skip a target at docs/build.log?
  reference: Yes. In 2.0 entries use gitignore syntax, so a line such as *.log matches at any depth, including docs/build.log.
  baseline (cites ignore-syntax): Yes, an ignore entry of `*.log` in sablefile 2.0 skips a target at `docs/build.log`. [sec:ignore-syntax]
  unfiltered (cites ignore-syntax): Yes, an ignore entry of `*.log` in sablefile 2.0 skips a target at `docs/build.log`. [sec:ignore-syntax]
  second-model (declined): The docs don't say.
q-0025 | version 1.0 | planted (version-swap) | Does the lock file in sablefile 1.0 record each generated file's mode?
  reference: No. In 1.0 each lock line holds only the SHA-256 hash and the path, so no mode is recorded.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0026 | version 1.1 | not planted | Does the lock file in sablefile 1.1 record each generated file's mode?
  reference: Yes. Each lock line holds the SHA-256 hash, then the file's mode, then the path. check compares only the hash, so a change of mode alone is not reported as drift.
  baseline (cites lock-mode): Yes, the lock file in sablefile 1.1 records each generated file's mode. [sec:lock-mode]
  unfiltered (cites lock-mode): Yes, the lock file in sablefile 1.1 records each generated file's mode. [sec:lock-mode]
  second-model (cites lock-mode): The lock file in sablefile 1.1 records each generated file's mode. [sec:lock-mode]
q-0027 | version 1.0 | not planted | How many times does sablefile 1.0 retry a remote template fetch that keeps failing?
  reference: Three retries, after which the command gives up. No flag or setting changes that count in 1.0.
  baseline (cites remote-retries): Three times. [sec:remote-retries]
  unfiltered (cites remote-retries): Sablefile 1.0 retries a remote template fetch up to 3 times before giving up. [sec:remote-retries]
  second-model (cites remote-retries): The count is fixed in 1.0, and no flag or setting changes it. [sec:remote-retries]
q-0028 | version 1.1 | not planted | How many times does sablefile 1.1 retry a remote template fetch that keeps failing?
  reference: Five retries, after which the command gives up. In 1.1 no flag or setting can change that count.
  baseline (cites remote-retries): Sablefile 1.1 retries a remote template fetch up to 5 times before giving up. [sec:remote-retries]
  unfiltered (cites remote-retries): Sablefile 1.1 retries a remote template fetch up to 5 times before giving up. [sec:remote-retries]
  second-model (cites remote-retries): The count is fixed in 1.1, and no flag or setting changes it. [sec:remote-retries]
q-0029 | version 1.0 | planted (version-swap) | In sablefile 1.0, does sync --apply create the missing folder for a target?
  reference: The 1.0 docs describe no --apply flag. Sync creates a target's missing folder by itself, before it writes the file, so a target such as docs/guide/index.md needs no setup.
  baseline (cites sync-create-dirs): Yes, sync --apply creates the missing folder for a target. [sec:sync-create-dirs]
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0030 | version 2.0 | not planted | In sablefile 2.0, does sync --apply create the missing folder for a target?
  reference: Yes, but only because --apply is given. In 2.0 a plain sync creates no folder, since it writes nothing, and with --apply the folder is made before the target is written.
  baseline (cites sync-create-dirs): Yes, sync --apply creates the missing folder for a target. [sec:sync-create-dirs]
  unfiltered (cites apply-flag): Yes, sync --apply creates the missing folder for a target. [sec:apply-flag]
  second-model (cites sync-create-dirs): [sec:sync-create-dirs] Folders for targets   Yes, with `--apply`, sync creates the missing folders before writing the fi...
q-0031 | version 1.1 | planted (version-swap) | In sablefile 1.1, where is .sablefile/lock.json kept?
  reference: The 1.1 docs describe no .sablefile folder. The lock file in 1.1 is sablefile.lock in the project root, with one line for each generated file.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0032 | version 2.0 | not planted | In sablefile 2.0, where is the lock file kept?
  reference: In .sablefile/lock.json, a JSON file with a format string and one entry per generated file. A leftover sablefile.lock from 1.x is read with a warning and migrated when a command writes files.
  baseline (cites lock-migration): In sablefile 2.0, the lock file is kept in `.sablefile/lock.json`. [sec:lock-migration]
  unfiltered (cites lock-migration): In sablefile 2.0, the lock file is kept in `.sablefile/lock.json`. [sec:lock-migration]
  second-model (cites lock-mode): The lock file is kept in `.sablefile/lock.json`. [sec:lock-mode]
q-0033 | version 2.0 | not planted | In sablefile 2.0, what does sync print for a target whose content already matches its template?
  reference: The word unchanged, as that target's plan line. The plan step itself writes nothing.
  baseline (declined): The docs don't say.
  unfiltered (cites apply-flag): For a target whose content already matches its template, sync prints "unchanged". [sec:apply-flag]
  second-model (declined): The docs don't say.
## ambiguous: 10 dev items
q-0019 | version 2.0 | planted (ambiguous-target) | If I edit the file and then run check, will it fail?
  reference: Reading 1: if you edited a generated file (a target), check prints DRIFT and the path for it and exits with status 1, because its hash no longer matches the lock.
Reading 2: if you edited a template, check prints nothing and exits with status 0, since a template change is not drift.
  baseline (cites quickstart-check check-exit-status): Yes, if you edit the file and then run `sablefile check`, it will report `DRIFT` and the path of the file, indicating a ...
  unfiltered (cites check-exit-status no-fail-flag): Yes, if you edit the file and then run `sablefile check`, it will detect the drift and print `DRIFT` and the path of the...
  second-model (cites check-exit-status): If you edit the file and then run `sablefile check`, it will fail because the check will detect a drift and print a `DRI...
q-0020 | version 2.0 | not planted | Does sync change the lock file?
  reference: Reading 1: a plain sync, with no flag, writes nothing, so the lock file is left alone.
Reading 2: sync --apply writes the targets and updates .sablefile/lock.json with the hash and mode of each file it wrote.
  baseline (cites sync-lock-update): Yes, sync changes the lock file when the `--apply` flag is used. After sync writes a target, it records that file's SHA-...
  unfiltered (cites diff-command): No, sync does not change the lock file. [sec:diff-command]
  second-model (cites sync-lock-update): The lock file is how check finds drift, so run sync after each change you intend to keep. [sec:sync-lock-update]
q-0043 | version 2.0 | planted (ambiguous-target) | Is it drifted now that I changed the template?
  reference: Reading 1: if 'it' means the target, it has not drifted. Nothing has edited the target since the last sync, so check prints nothing for it.
Reading 2: if 'it' means a target that you hand-edited after the last sync, it has drifted, and check prints DRIFT and exits with status 1.
  baseline (cites drift-meaning): No, it is not drifted now that you changed the template. A change to a template is not drift, because the generated file...
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0044 | version 2.0 | not planted | Does sync create a folder for a target?
  reference: Reading 1: a plain sync writes nothing, so it creates no folder.
Reading 2: sync --apply creates the target's missing folder before it writes the file.
  baseline (cites sync-create-dirs): Yes, sync creates a folder for a target if the folder does not exist, but only when the `--apply` flag is used. [sec:syn...
  unfiltered (declined): The docs don't say.
  second-model (cites sync-create-dirs): Yes, sync creates a folder for a target. [sec:sync-create-dirs]
q-0045 | version 2.0 | not planted | Does --retries change how often the command retries?
  reference: Reading 1: for sync and diff, yes. They use the count you give, in place of the default of five.
Reading 2: for check, no. check never fetches a remote template, so it never retries.
  baseline (cites retries-flag): Yes, the `--retries` flag changes how often the command retries a failed fetch of a remote template. [sec:retries-flag]
  unfiltered (cites retries-flag remote-retries): Yes, the `--retries` flag changes how often the command retries a failed fetch of a remote template. It sets the number ...
  second-model (cites retries-flag): The `--retries` flag changes how often the command retries a failed fetch of a remote template, in place of the default ...
q-0046 | version 2.0 | not planted | If I delete sablefile.lock, will check stop reporting drift?
  reference: Reading 1: if the old file was the only lock, deleting it leaves check with nothing to read, so check prints an error message instead of DRIFT lines.
Reading 2: once .sablefile/lock.json exists, 2.0 reads only that file, so deleting sablefile.lock changes nothing.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0047 | version 2.0 | not planted | When does a 2.0 project's old lock file get migrated?
  reference: Reading 1: a command that only reads, such as check, reads the old file and prints a warning, but writes nothing, so both files stay as they are.
Reading 2: a command that writes files, such as sync --apply, also writes .sablefile/lock.json and keeps each entry's mode where the old file has one.
  baseline (cites lock-migration): When a 2.0 command finds an older `sablefile.lock` and no `.sablefile/lock.json`, it reads the old file and prints a war...
  unfiltered (cites lock-migration): When a 2.0 command finds an older `sablefile.lock` and no `.sablefile/lock.json`, it reads the old file and prints a war...
  second-model (no citation): The 2.0 project's old lock file is migrated when a 2.0 command finds an older `sablefile.lock` and no `.sablefile/lock.j...
q-0048 | version 1.0 | not planted | In 1.0, does check report the file if I change it?
  reference: Reading 1: if 'the file' is a template, no. check does not render templates, so it prints nothing for a template change.
Reading 2: if 'the file' is a generated target you edited, yes. check prints DRIFT and the path for it.
  baseline (declined): The docs don't say.
  unfiltered (cites check-command): Yes, in 1.0, `check` reports the file if it is changed. [sec:check-command]
  second-model (cites check-command): The check command does not report the file if it changes. [sec:check-command]
q-0049 | version 2.0 | not planted | Will init overwrite my config?
  reference: Reading 1: if sablefile.toml already exists, init stops with an error and leaves it alone.
Reading 2: if you mean another file in the folder, such as a README, init writes no other file, so it does not touch that file.
  baseline (cites init-command): No, init will not overwrite your config. It stops with an error if `sablefile.toml` already exists, so it never overwrit...
  unfiltered (declined): The docs don't say.
  second-model (cites init-command): init stops with an error if `sablefile.toml` already exists, so it never overwrites your configuration. [sec:init-comman...
q-0050 | version 1.1 | not planted | Does the 1.1 lock file care about a file's mode?
  reference: Reading 1: yes, the lock records each generated file's mode alongside its hash.
Reading 2: no, for comparison. check compares only the hash, so a mode-only change is not reported as drift.
  baseline (cites lock-mode): Yes, the 1.1 lock file cares about a file's mode. [sec:lock-mode]
  unfiltered (cites lock-mode): Yes, the 1.1 lock file cares about a file's mode. The mode field is described in lock-mode, and sync records the mode of...
  second-model (cites lock-mode): The 1.1 lock file does not care about a file's mode. The mode field is described in lock-mode. [sec:lock-mode]

以条目 q-0005 为例。它询问 sablefile 2.0 是否仍然读取项目根目录中的 sablefile.lock,它是一个植入的近似术语条目,也就是所用名称与文档中的名称相近,用来捕捉粗心的回复。基线回答否,并引用了一个关于升级的节。参考答案说,2.0 会回退到该文件,并给出警告,因此基线的“否”是错的。未过滤版本和第二模型版本都拒答了,而文档确实回答了这个问题,所以它们的拒答同样是错的。

q-0005 的三条回复以两种不同的方式失败。对可回答条目拒答,与自信地给出错误答案,是不同的失败,能捕捉其中一种的检查并不能捕捉另一种。通过率会把三条回复都算作失败,从而丢掉这一差异。数据文件中对这条回复的注记以“says 2.0 never reads sablefile.lock; reference says it falls back to it with a warning.”开头。每条注记用一句话说明回复做了什么以及参考答案怎么说,这样随后的分组就建立在读者可以核对的内容之上。同一个计数可能来自不同的缺陷,两条同样没有通过的回复也可能是不同的失败。这就是先看回复、后看计数的原因。

这一比较有其局限,而这些局限界定了计数所能支持的结论。开发集条目是为探查助手而写的,植入条目也不是用户所问问题的随机样本。因此,计数能说明在这些条目上哪个版本以何种方式失败,却不能说明用户遇到每种失败的频率。要得到后者,需要一个按用户提问方式抽取的问题样本。

**工作准则:**在计数之前,先阅读每个版本的回复,按类别分组,并写下每条回复的问题所在。计数可以说明一个版本失败了,只有回复才能说明它是如何失败的。

这部分尚未翻译,因此以英文显示。

Coding the failures

Reading becomes counting once each reply has a note and a mode. The drafting model wrote one note per reply, as the one on q-0005 does, and then grouped the notes. Each reply got one mode, its main defect, and a reply with no defect got ok. The codes were made by the AI model that drafted this book, as one coder; no second coder checked them, and nobody measured agreement on the codes. The 80 dev items give 240 replies, 80 from each variant. The codes sit in a data file beside the examples, one entry per reply.

The counts carry the label "coded by the drafting model (one coder; not human labels)", and the label means what it says. This chapter has no agreement figure, and no one else has reproduced the counts. Chapter 1 made the same point: agreement with your own labels doesn't calibrate a judge. Chapter 6 takes that up.

The tally example turns the notes into counts, and it checks the codes first. Each dev item has exactly one code per variant, no code names a mode the file doesn't define, and the file carries the label. Then it prints one row per mode and one column per variant, and one labelled line for each count, which are the numbers the prose below quotes.

tally-failure-modes.py Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0 通过
"""Counts the drafting model's failure codes over the dev items, per variant, and prints one table and one labelled lineper count.The point rests on the table's reading of each mode: the codes were made by the drafting model, as one coder, and are nothuman labels. The codes live in data/failure-codes.json; the checks below stop a reply being counted twice or left out.The labelled lines are the numbers the chapter's prose quotes."""import lib  # first: fixes the thread counts and seeds before anything else loadsimport jsonfrom collections import Counterfrom pathlib import Pathfrom lib.testset import load_itemsCODES = Path(__file__).resolve().parent / "data" / "failure-codes.json"VARIANTS = ("baseline", "unfiltered", "second-model")book = json.loads(CODES.read_text(encoding="utf-8"))assert book["labelled"] == "coded by the drafting model (one coder; not human labels)", "the codes are the model's, not human labels"  # the label the prose usesassert book["variants"] == list(VARIANTS), "the variants in the file are the three the chapter names"dev_ids = {item.id for item in load_items(split="dev")}n = len(dev_ids)codes = book["codes"]assert {code["item"] for code in codes} <= dev_ids, "a code names an item that is not in the dev split"pairs = [(code["item"], code["variant"]) for code in codes]# One code per dev item and variant: a pair repeated would be counted twice, a missing one would go unseen.assert len(pairs) == len(set(pairs)) == n * len(VARIANTS), "expected one code per dev item and variant"assert all(code["mode"] in book["modes"] for code in codes), "a code uses a mode the file does not define"counts = Counter((code["variant"], code["mode"]) for code in codes)print(f"dev items: {n}")print(f"replies coded: {len(codes)}")print(f"labelled: {book['labelled']}")print(f"{'mode':<22}" + "".join(f"{variant:>14}" for variant in VARIANTS))for mode in book["modes"]:  # the file's order: ok first, then the failure modes    total = sum(counts[(variant, mode)] for variant in VARIANTS)    if total == 0:  # a mode no variant shows is printed as zero of N, so the reader sees it was looked for        print(f"{mode}: 0 of {n}")        continue    print(f"{mode:<22}" + "".join(f"{counts[(variant, mode)]:>14}" for variant in VARIANTS))totals = {variant: sum(counts[(variant, mode)] for mode in book["modes"]) for variant in VARIANTS}assert all(total == n for total in totals.values()), "each variant's counts add up to the dev items"print(f"{'all modes':<22}" + "".join(f"{totals[variant]:>14}" for variant in VARIANTS))print()for variant in VARIANTS:  # one labelled line per count, the form the chapter's prose binds to    for mode in book["modes"]:        print(f"{mode}, {variant}: {counts[(variant, mode)]}")    print(f"defective, {variant}: {n - counts[(variant, 'ok')]}")
输出python tally-failure-modes.py
dev items: 80
replies coded: 240
labelled: coded by the drafting model (one coder; not human labels)
mode                        baseline    unfiltered  second-model
ok                                56            49            24
over-declining                    10            15            31
wrong-version-answer               2             3             0
wrong-answer                       3             5             9
missed-refusal                     0             2             2
incomplete-answer                  8             4             7
citation-problem                   1             2             6
malformed-reply                    0             0             1
all modes                         80            80            80

ok, baseline: 56
over-declining, baseline: 10
wrong-version-answer, baseline: 2
wrong-answer, baseline: 3
missed-refusal, baseline: 0
incomplete-answer, baseline: 8
citation-problem, baseline: 1
malformed-reply, baseline: 0
defective, baseline: 24
ok, unfiltered: 49
over-declining, unfiltered: 15
wrong-version-answer, unfiltered: 3
wrong-answer, unfiltered: 5
missed-refusal, unfiltered: 2
incomplete-answer, unfiltered: 4
citation-problem, unfiltered: 2
malformed-reply, unfiltered: 0
defective, unfiltered: 31
ok, second-model: 24
over-declining, second-model: 31
wrong-version-answer, second-model: 0
wrong-answer, second-model: 9
missed-refusal, second-model: 2
incomplete-answer, second-model: 7
citation-problem, second-model: 6
malformed-reply, second-model: 1
defective, second-model: 56

Read the ok row, then the rows under it. The baseline has the most replies with no defect, 56, and the second-model variant has the fewest, 24. Over-declining is the largest defect in each variant. In the second-model variant it is more than half of the defects: 31 of 56.

The baseline doesn't escape the worse defects. It gives 2 wrong-version answers, on q-0023 and q-0029, and the unfiltered variant gives the most, 3. Missed refusals come only from the variants, and the baseline has none. Every mode appears at least once. A mode that no variant showed would print as zero, so a reader could see it was looked for.

Working rule: Give each reply one note and one mode, and print one labelled line per count, so that every number the prose gives can be checked against the output.

过度拒答

过度拒答的回复,是拒绝了文档本可回答的条目。这是读者最先遇到的缺陷,因为拒答看上去很谨慎,而它让用户失望的程度并不亚于给出错误答案。这里用到的术语,即拒答用语、无法回答的条目和各变体,已在回复的样子中定义。统计范围涵盖文档能够回答的所有条目,包括取决于版本的条目,因此就基线而言,这个数字大于第 10 章中的误拒率,后者只统计可回答的条目。条目 q-0004 就是取决于版本的情形,第 1 章对此有完整的说明。三个变体都用同样的措辞拒答该条目,所以基线也是如此。

统计拒答有一个陷阱,它藏在解析器里。条目 q-0080 是可回答的,而未过滤变体对它的回复先用两句话说明各节没有定义二者的区别,最后以拒答用语收尾。文档能够回答这个问题,所以这条回复属于过度拒答。解析器只有在拒答用语构成整条回复时才计为拒答,因此把这条回复读成了引用某一节的回答。读者去搜索这句用语,能够找到它,而基于解析器构建的检查却会漏掉它。

候选检查是:对于参考答案指明了支撑节的条目,回复不得拒答,无论措辞如何。用代码实现检查构建这类检查。对于文档没有涵盖的条目,拒答是否正确属于判断问题,第 10 章会讨论。

工作准则: 按回复的实际作为来统计拒答,而不是看用语是否单独成句;并且把过度拒答与错误回答分开统计,因为二者的检查方式不同。

版本错误的回答

版本错误的回答,是把另一个版本的行为当作所问版本的行为,这样的回答可能行文流畅、引用齐全,却依然是错的。在 q-0029 中,基线回答答复为是,称在 1.0 中sync --apply会创建缺失的文件夹,并引用了介绍创建文件夹的那一节。回复附带的说明指出,这是 2.0 的行为,而 1.0 根本没有--apply这个标志。

未过滤的变体更常出现这种情况。在 q-0021 中,问题是在 1.0 中check遇到漂移时是否返回非零退出状态,基线回答与第二模型变体都答复不会,而未过滤变体却答复会。说明指出,这条未过滤的回复针对一个 1.0 的问题给出了 2.0 的规则。它的上下文包含所有版本,因此 2.0 的规则与 1.0 的规则并列其中。

引用并不能解决这个问题。在 q-0023 中,基线回答对一个 2.0 的结果答复为是,引用的却是 1.0 的一节。若只检查每条引用是否都指向所问版本的某一节,这条回复可以通过检查。只有把其中的论断与 1.0 的参考文档对照阅读,才能发现错误。

工作准则: 在两处检查每条回复的版本。其引用必须属于所问的版本,其论断必须与该版本的参考文档相符。只做引用检查,会放过一个对错误版本而言正确的回答。

这部分尚未翻译,因此以英文显示。

Missed refusals

A missed refusal answers an unanswerable item with content the docs don't hold. It is the mode a reader is most likely to believe, because the reply reads like any other answer. The terms used here are defined in what the replies look like. The dev items show it only in the variants, 2 times in the unfiltered variant and 2 in the second-model variant, and each is short enough to read whole.

In q-0018 the unfiltered variant says a login token can be sent with a remote template fetch, and cites a section about retries. In q-0034 the second-model variant gives a tip about the lock file, cited to a section that doesn't cover the question. In q-0036 it says --quiet suppresses the diff output, citing a section that never mentions the flag. In q-0039 the unfiltered variant answers about check when the question asks about diff. The baseline declines every one of them.

A code check can require a decline on every unanswerable item, and it would catch each of them. On an answerable item the same fault goes unseen: a reply that cites a real section silent on its claim would pass, because it answers, as an answer is expected there. Reading the claim against the cited section is the work of chapter 9 and of chapter 10.

Working rule: Require a decline on every unanswerable item, and read every answer to one against the section it cites before you count it as right.

从失败到检查

失败模式一旦有了候选检查就有了用处。候选检查由四部分组成:失败模式、检查所读取的输入、它所检验的论断,以及它看不到的东西。没有写明局限的候选检查,说明还没有想透。候选检查也还不是检查。代码中的检查构建程序能够运行的检查,而判断类的工作交给评判器。下面的列表为对失败编码中的七种失败模式各给出一个候选检查,并指明能说明它的案例。

  • **过度拒答。**对于参考信息指明了支撑章节的条目,回复不得拒答,无论措辞如何。该检查无法区分两种情形:一是条目超出文档范围时的正确拒答,二是错误拒答,因此这一判断交给第 10 章。
  • **错版本回答。**每处引用都必须属于所问的版本,且论断必须与该版本的参考答案一致。前一半是代码中的检查中构建的代码检查。后一半能发现 q-0023,属于第 9 章的版本正确性。它看不到参考答案未涉及的论断。
  • **错误回答。**论断必须与所问版本的参考答案一致,对有歧义的条目,回复必须回答每一种理解,如一个评判器所述。带有书面评分细则的评判器把每个论断与其参考答案相比较,如第 5 章所示。它的测试案例是 q-0005:基线回复说 2.0 从不读取 sablefile.lock,而参考答案说它会回退到该文件。对于有歧义的条目,评分细则必须把两种理解都写入参考答案,q-0048 就需要如此:未过滤的变体回答“是”,第二模型变体回答“否”,各自对其中一种理解而言都是错的。评判器本身也可能出错,第 6 章对此做了度量。
  • **漏拒。**对无法回答的条目,每条回复要么拒答,要么由读者对照其所指明的章节核查它的引用。前者是代码检查。后者是第 9 章的工作。代码检查不读取章节文本,因此对可回答的条目,若回复引用了一个真实存在、但对其论断只字未提的章节,这类回复会通过检查。
  • **回答不完整。**参考答案列出一个正确回答所需的每个数字、步骤和理解,代码检查可以在回复中逐项查找。本书没有构建这样的检查。在 q-0011 中,基线回复说重试次数是固定的,没有任何标志能改变它,却始终没有给出这个数字。未过滤的变体给出了该数字,因此针对参考答案中数字的检查会判基线的回复不通过。它的局限:数字可以出现在回复里,却并不是答案。
  • **引用问题。**每处引用都必须是该条目的支撑章节之一。在 q-0057 中,基线说 check 看不到远程服务器上的变更,这一点是对的,但它引用了条目未列为支撑的章节,因此该引用不在名单之内。对照该名单的代码检查能发现它。它的局限:名单中的章节仍可能不支撑该论断,这需要读者来判断,如代码中的检查所解释。
  • **格式错误的回复。**对照提示词所要求的格式,检查回复的形状,可以发现顺序出错的回复。第二模型对 q-0030 的回复把引用放在最前面,并且句子混乱,尽管它的答案是对的。它的局限:形状对答案不置一词,因此回复可能通过形状检查却仍然是错的。该检查由代码中的检查里的格式检查构建。

列表中的每个候选检查都指明了它本该发现的那条回复,这是对它的第一项测试。如果候选检查没有标记出它所针对的那条回复,它就还不是该模式的检查。

先构建代码检查。它们几秒内即可运行,无需调用模型,因此每次变更都可以运行。评判器对每条回复都要调用一次模型,并且需要自己的校准,因此留给参考答案和引用无法定论的模式。错误回答需要一个评判器来判断论断。错版本回答只有在对照参考答案阅读论断的那部分才需要评判器,漏拒只有在对照所引章节阅读论断的那部分才需要。

在生产环境中,失败模式不会带着标签出现。用户报告了一条错误回复,首先要问的是它属于哪种模式,因为针对错误模式所写的检查会放过真正失败的回复。当没有任何模式适用时,重新阅读失败的回复,对它们编码,增加一种模式并为它写检查。这就是本章在计数之前先阅读并编码回复的原因。

这份列表并非封闭:留出条目和之后的任何变体都会暴露这些条目所没有的模式,每一种都需要自己的说明和自己的候选检查。

本章做出了以下决定:

  • 编码由一位编码者完成:起草模型。这些编码已如此标注,也没有度量一致性。
  • 开发条目是这里唯一阅读过的条目;留出条目的拒答计数见第 10 章。
  • 基线的回复与各变体一起计数,因此它的 2 条错版本回答照此报告。

**工作准则:**在选择检查之前,先按主要缺陷对回复编码,并让每个检查只针对一种模式。依据你所预期的失败构建的检查,只能检验这些失败,而对你没有预料到的失败则放行。

本章,在本版次中

版次 2026.11.3示例于 2026年11月21日 通过本章 2个示例 个示例中有 2 个通过

译自英文版图书,版次 2026.11.3。

引用本节

格式

每种格式均引用英文原书,本译本以其为准。

每种格式都会注明该版次。其链接会打开当前版次,节标题处会说明该节文本最后一次修改的时间。访问日期和检索日期即为您打开此页的当天。

本页内容

第 2 章,共 11 章: 阅读回复,找出该测量什么

  1. 回复的样貌
  2. Coding the failures
  3. 过度拒答
  4. 版本错误的回答
  5. Missed refusals
  6. 从失败到检查

返回顶部全部章节