답변을 읽고 무엇을 측정할지 찾는다

읽는 데 약 31분분 걸린다, 6절에 2개 예제이(가) 있음.

2026.11.3판 예제 통과 2026년 11월 21 54개 예제 중 54개 통과 변경 사항

AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식

영어판 2026.11.3판을 번역한 것이다. 원문 읽기

이 장을 마치면 어시스턴트가 내놓은 답변 묶음을 읽고, 각 답변의 문제를 실패 유형으로 분류하고, 그 실패 가운데 어떤 것을 검사가 잡아낼 수 있는지 말할 수 있다.

1장은 한계 하나로 끝났다. 항목 스무 개로는 틀린 답을 찾을 수 있어도 두 프롬프트의 순위를 매길 수는 없다. 이 책의 어시스턴트는 sablefile 문서의 세 버전인 1.0, 1.1, 2.0을 근거로 답하며, 한 버전에 대한 질문에 다른 버전에서만 맞는 답이 나올 수 있다. 통과율은 이런 실패를 모두 같은 방식으로 세고, 어느 것의 이름도 붙이지 않는다. 그래서 통과율만으로 측정하는 팀은 자신이 예상하는 실패에 대해서만 검사를 작성하게 되고, 예상하지 못한 실패는 검사되지 않은 채 남는다. 답변을 먼저 읽는 것이 두 번째 종류의 실패를 찾는 방법이다. 1장의 구간은 점수를 어디까지 믿을 수 있는지 알려 준다. 이 장은 각 변형이 무엇을 틀리는지 묻는데, 점수로는 알 수 없는 내용이다.

세기 전에 읽는 방식은 이 책 밖에서도 뒷받침을 받는다. 1장은 사람들이 출력을 채점하는 동안 채점 기준을 다듬는다는 점을 들어 Shankar와 동료들의 Who Validates the Validators? (2024)를 인용했다. 이 논문은 주로 EvalGen이라는 시스템을 다루며, 이 장이 의존하는 부분은 실무자 아홉 명을 대상으로 한 사용자 연구다. 연구가 기록한 기준의 변동은 사람들이 출력을 채점하는 동안 일어났다. 파이프라인을 바꾸면 기준이 변동할 수 있다는 것은 저자들의 논의이지 연구의 발견이 아니다. 연구가 하나뿐이므로 이 장은 이를 책이 측정하는 결과가 아니라 주의할 점으로 다룬다.

이 장은 80개의 개발 항목을 사용한다. 개발 항목은 이 책이 측정 방법을 개발하는 동안 다루는 테스트 세트의 분할이다. 이 장은 개발 항목만 읽으므로 여기서 하는 일은 보류된 항목에 맞춰 조정되지 않으며, 보류된 항목에서 답변을 거절한 횟수는 10장에 나온다. 읽기의 결과물은 세 가지다. 각 답변에 대한 메모, 그 메모가 속하는 결함의 집계, 그리고 결함마다 하나씩 정하는 검사 후보다. 여기 나오는 답변은 모두 변형마다 한 번씩 기록해 두었다가 예제가 재생하는 것이므로, 읽는 내용은 실행할 때마다 같다. 답변을 인용한 부분은 기록된 출력에서 가져왔고 기준 답변은 해당 항목 옆에 출력되므로, 본문이 인용하는 모든 줄은 예제의 출력에서 찾을 수 있다. 메모는 집계 예제가 읽는 데이터 파일에 들어 있다.

이 장 전체에서 답변은 기준 답변을 내놓았을 때, 또는 문서에 나와 있지 않다는 것이 기준이어서 답변을 거절했을 때 옳은 것으로 친다. 인용은 따로 검사하며, 1장의 인용 검사와 4장의 검사가 이를 맡는다. 그 밖의 경우는 모두 실패이고, 어떤 종류인지는 장의 나머지 부분에서 다룬다. 같은 단계를 직접 만든 재생 파일에도 적용할 수 있다. 답변을 종류별로 묶고, 각 답변을 기준과 대조해 읽고, 주된 결함에 따라 각각 분류 코드를 매기고, 유형마다 검사 후보를 하나씩 적는다. 이 장은 sablefile 답변으로 그 과정을 수행하므로, 열어 볼 수 있는 데이터로 각 단계를 따라갈 수 있다.

이 장은 검사를 고르거나 심사자의 평가 기준표를 작성하지 않으며, 각 유형이 실제 사용에서 얼마나 자주 일어나는지도 추정하지 않는다. 그 내용은 이후 장에 나오며, 각 장은 여기서 이름 붙인 유형에 의존한다.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

What the replies look like

마지막으로 변경된 판: 2026.11.3판

Three variants of the assistant answered the same dev items, and every comparison in this chapter sets one variant's replies beside another's. The baseline is the setup chapter 1 measured. It retrieves sections only from the version the question asks about, so its context holds one version's docs. The unfiltered variant drops the baseline's version filter. Its context can hold sections from all three versions, each marked with its version, so the model can see a mismatch and can also copy the wrong version's text. The second-model variant keeps the baseline's prompt and hands the answer to a smaller model, Qwen3 with 0.6 billion parameters in an 8-bit build. The prompt is the same, so a difference from the baseline comes from the model, not the prompt.

Each dev item has a kind, and the kind says what a right reply looks like. An answerable item is one the docs answer. An unanswerable item asks for something the docs don't say, and the right reply is a decline in the words the assistant was given: "The docs don't say." A version-dependent item has an answer that changes with the version asked about, so the same question asked of two versions can have two right answers. An ambiguous item can be read two ways, and the docs answer each reading. Some items are planted, written to catch one specific mistake. The output gives each planted item its reason, which is one of a false premise, a near-miss term, a version swap, a stale answer or an ambiguous target. A reply that fails a planted item shows whether the assistant made the mistake that item was written for.

Reading one item takes three steps. Start with the question and its version, since a reply is right only for the version it names. Then read the reference answer, which is the author's statement of what a right reply says. Then read the three replies against it, one at a time. Only after that does a reader group the replies or count them.

The example that follows is where the chapter starts. It prints every dev item with its reference answer and each variant's reply, grouped by kind and sorted by ID, so the three replies to one question sit on adjacent lines. Read across an item's lines, not down one variant's column. The variants fail in different ways on the same question, and that difference is what to see. Each reply is cut to its first 120 characters, and its line says whether the reply declined and what it cited. A reply that holds the decline phrase inside other text is marked as a phrase, not a decline, so you can see where the parser draws the line.

read-replies-dev.py Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0 통과
"""Prints the dev items, grouped by kind and sorted by ID, with each item's reference answer and the baseline,unfiltered and second-model replies.The reference is printed in full under its item, since the reading compares each reply with it. Each reply is one line:whether it declined, its citations, and its first 120 characters. The reader's job is to set these side by side andsee which failure each reply shows (the codes are in data/failure-codes.json)."""import lib  # first: fixes the thread counts and seeds before anything else loadsfrom lib.contract import DECLINE, parse_replyfrom lib.prompts import P1_SHA256from lib.replay import ReplayModelfrom lib.testset import load_itemsRECORDINGS = {  # the three variants, in the order the chapter discusses them    "baseline": "assistant-baseline",    "unfiltered": "assistant-unfiltered",    "second-model": "assistant-second-model",}KINDS = ("answerable", "unanswerable", "version-dependent", "ambiguous")  # plan section 3.1 orderSHOWN = 120  # characters of each reply printed; the full text is in the recording# The prompt hash is checked here, so a recording made from another prompt fails to load.models = {name: ReplayModel(recording, P1_SHA256) for name, recording in RECORDINGS.items()}items = load_items(split="dev")  # dev only: the held-out items are not read in this chapter (plan section 3.4)for kind in KINDS:    group = [item for item in items if item.kind == kind]  # load_items already sorts by ID    print(f"## {kind}: {len(group)} dev items")    for item in group:        planted = f"planted ({item.planted_why})" if item.planted else "not planted"        print(f"{item.id} | version {item.version} | {planted} | {item.question}")        reference = item.reference_answer or "none: the item is unanswerable, so the right reply declines"        print(f"  reference: {reference}")        for name in RECORDINGS:            reply = parse_reply(models[name].reply(item.id, 0))            if reply.declined:                mark = "declined"            elif DECLINE in reply.text:                # The phrase is there, but with other text: the parser does not call that a decline.                mark = "decline phrase with other text"            else:                mark = f"cites {' '.join(reply.citations)}" if reply.citations else "no citation"            text = reply.text.replace("\n", " ")  # a reply may hold line breaks; one line keeps the output readable            more = "..." if len(text) > SHOWN else ""            print(f"  {name} ({mark}): {text[:SHOWN]}{more}")
출력python read-replies-dev.py
## answerable: 40 dev items
q-0005 | version 2.0 | planted (near-miss-term) | Does sablefile 2.0 still read sablefile.lock in my project root?
  reference: Only as a fallback. If the new file is missing and an old sablefile.lock is present, a 2.0 command reads the old file but warns you about it. Once the new file exists, 2.0 reads only that one, and sablefile never deletes the old file.
  baseline (cites install-upgrade): No, sablefile 2.0 does not read `sablefile.lock` in the project root. Version 2.0 moves the lock file to a new place. Th...
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0006 | version 2.0 | not planted | Is there a way to make check exit with status 0 even when a file has drifted in 2.0?
  reference: Yes. Pass --no-fail to check. It still prints each DRIFT line, but it exits with status 0, so a script can report drift without failing. The flag changes nothing except the exit status.
  baseline (cites no-fail-flag): Yes, there is a way to make `check` exit with status 0 even when a file has drifted in 2.0. Use the `--no-fail` flag wit...
  unfiltered (cites no-fail-flag): Yes, there is a way to make `check` exit with status 0 even when a file has drifted in version 2.0. Use the `--no-fail` ...
  second-model (cites no-fail-flag): The `--no-fail` flag is used to make the check exit with status 0 even when files have drifted. [sec:no-fail-flag]
q-0007 | version 1.1 | not planted | In 1.1, will an ignore entry of docs/**/*.tmp catch a file two folders down, like docs/x/y/a.tmp?
  reference: Yes. Version 1.1 adds the double-star pattern, which matches zero or more folders. So docs/**/*.tmp matches docs/x/y/a.tmp as well as files directly in docs.
  baseline (cites ignore-recursive): Yes, an ignore entry of `docs/**/*.tmp` will catch a file two folders down, like `docs/x/y/a.tmp`. [sec:ignore-recursive...
  unfiltered (cites ignore-recursive): Yes, in version 1.1, an ignore entry of `docs/**/*.tmp` will catch a file two folders down, like `docs/x/y/a.tmp`. [sec:...
  second-model (no citation): The docs section does not allow for an ignore entry of docs/**/*.tmp to catch a file two folders down, as per the rules....
q-0008 | version 2.0 | not planted | In 2.0, is there a way to re-include something that an earlier ignore line had excluded?
  reference: Yes, with a leading exclamation mark. A line that starts with ! negates an earlier match, so a later line can restore a file that an earlier line ignored.
  baseline (cites ignore-syntax): Yes, in 2.0, you can re-include something that an earlier ignore line had excluded by using a leading `!` to negate an e...
  unfiltered (cites ignore-syntax): Yes, in version 2.0, you can re-include something that an earlier ignore line had excluded by using a leading `!` to neg...
  second-model (declined): The docs don't say.
q-0009 | version 1.0 | not planted | What does one line of the 1.0 lock file hold?
  reference: The SHA-256 hash of a generated file's content, then two spaces, then that file's path. The 1.0 lock has no mode field.
  baseline (cites lockfile): One line of the 1.0 lock file holds the SHA-256 hash of the file's content, two spaces, then the path. [sec:lockfile]
  unfiltered (cites lockfile): One line of the 1.0 lock file holds the SHA-256 hash of the file's content, then the file's mode, then the path. [sec:lo...
  second-model (cites lockfile): The lock file holds the SHA-256 hash of the file's content, two spaces, and the path. [sec:lockfile]
q-0010 | version 2.0 | not planted | Will syncing the same content twice in 2.0 produce a different lock file each time?
  reference: No. The 2.0 lock file is JSON with no timestamps, so the same content always gives back an identical file.
  baseline (cites lockfile): No, syncing the same content twice in 2.0 will not produce a different lock file each time. The lock file has no timesta...
  unfiltered (cites lockfile): No, syncing the same content twice in 2.0 will produce an identical lock file each time because the lock file has no tim...
  second-model (declined): The docs don't say.
q-0011 | version 1.1 | not planted | How many times does 1.1 retry a remote template fetch that keeps failing, and can a flag change that?
  reference: Five times, before the command gives up. In 1.1 that number is fixed, and no flag or setting can change it.
  baseline (cites remote-retries): The count is fixed in 1.1, and no flag or setting changes it. [sec:remote-retries]
  unfiltered (cites remote-retries): In version 1.1, sablefile retries a remote template fetch up to 5 times before giving up. The count is fixed and no flag...
  second-model (cites remote-retries): The count is fixed in 1.1, and no flag or setting changes it. [sec:remote-retries]
q-0012 | version 2.0 | not planted | I want sync to retry a slow remote template only twice in 2.0. What do I pass?
  reference: Pass --retries 2 to sync. The flag sets how many times a failed fetch is retried, in place of the default of five, and it only affects the command it is given to.
  baseline (cites retries-flag): Pass `--retries 2` to sync. [sec:retries-flag]
  unfiltered (cites retries-flag): Pass `--retries 2` to sync. [sec:retries-flag]
  second-model (cites retries-flag): `--retries 2` to pass. [sec:retries-flag]
q-0013 | version 1.0 | not planted | In 1.0, which table in sablefile.toml holds the values a placeholder can name?
  reference: The [values] table. Each name in it can be used as a placeholder, and a placeholder name in a template must match a key in that table.
  baseline (cites config-values): The `[values]` table holds the values a placeholder can name. [sec:config-values]
  unfiltered (cites config-values): The `[values]` table in sablefile.toml holds the values a placeholder can name. [sec:config-values]
  second-model (cites config-values): The values table holds the values a placeholder can name. [sec:config-values]
q-0014 | version 1.1 | not planted | What does sablefile init put into a new sablefile.toml in 1.1?
  reference: A [values] table with one entry, project, set to the folder's name, and no [[file]] tables. init writes no other file.
  baseline (cites init-command): `sablefile init` creates `sablefile.toml` in the folder where you run it, or in the folder you name. The new file has a ...
  unfiltered (cites init-command): `sablefile init` creates `sablefile.toml` in the folder where you run it, or in the folder you name. The new file has a ...
  second-model (declined): The docs don't say.
q-0051 | version 2.0 | planted (near-miss-term) | My 1.x ignore file uses shell globs, so does *.log only match files in the top folder in 2.0?
  reference: No. Shell glob rules no longer apply in 2.0. Entries use gitignore syntax, so *.log matches at any depth, and a trailing slash limits an entry to directories.
  baseline (cites ignore-syntax): In 2.0, entries use gitignore syntax. A line such as `*.log` matches at any depth. Therefore, `*.log` matches files in a...
  unfiltered (cites ignore-syntax): No, in 2.0, `*.log` matches at any depth, not just in the top folder. [sec:ignore-syntax]
  second-model (declined): The docs don't say.
q-0052 | version 2.0 | planted (near-miss-term) | I have {{project}} all through my templates from 1.x. Is that still valid in 2.0?
  reference: No. In 2.0 each placeholder is written as a dollar sign and braces, ${project}. The double-brace form is an error: rendering fails, sync reports the file and line, and it exits with status 1.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0053 | version 2.0 | planted (stale-answer) | I have an old sablefile.lock from 1.1. When I first run sync --apply under 2.0, what happens to it?
  reference: 2.0 writes a new .sablefile/lock.json holding each file's hash and, where the old file recorded one, its mode. The old file is not deleted, so you remove it yourself. A warning is printed when 2.0 reads the old file.
  baseline (cites lock-migration): When a 2.0 command finds an older `sablefile.lock` and no `.sablefile/lock.json`, it reads the old file and prints a war...
  unfiltered (cites lock-migration): When a 2.0 command finds an older `sablefile.lock` and no `.sablefile/lock.json`, it reads the old file and prints a war...
  second-model (cites lock-migration): The old `sablefile.lock` file from version 1.1 is not written to the new lock file, so it remains unchanged. [sec:lock-m...
q-0054 | version 2.0 | planted (stale-answer) | Does the 2.0 lock keep each generated file's mode, and what happens to modes from an old lock?
  reference: Yes. 2.0 records each generated file's mode in .sablefile/lock.json. When a 2.0 command migrates an older lock, it keeps each mode that the old file has.
  baseline (cites lock-mode lock-migration): Yes, the 2.0 lock keeps each generated file's mode, the Unix permission bits such as `0644`, in `.sablefile/lock.json` [...
  unfiltered (cites lock-mode lock-migration): Yes, the 2.0 lock keeps each generated file's mode, the Unix permission bits such as `0644`, in `.sablefile/lock.json` [...
  second-model (cites lock-mode lock-migration): The 2.0 lock keeps each generated file's mode, as specified in [sec:lock-mode]. When a lock file is migrated from `sable...
q-0055 | version 2.0 | planted (stale-answer) | How do I make sync retry a remote template fetch fewer times in 2.0?
  reference: Pass --retries N with the number you want, for example --retries 1. The flag sets the retry count for sync and diff; without it the default is five.
  baseline (cites retries-flag): Use the `--retries N` flag to set the number of times `sync` retries a failed fetch of a remote template, where N is a w...
  unfiltered (cites retries-flag): To make `sync` retry a remote template fetch fewer times in version 2.0, use the `--retries N` flag where N is a whole n...
  second-model (cites retries-flag): The `--retries` flag sets how many times `sync` and `diff` retry a failed fetch of a remote template, in place of the de...
q-0056 | version 1.0 | not planted | Does sablefile check need a network connection to run in 1.0?
  reference: No. check does not fetch remote templates or render templates, so it works without a network.
  baseline (cites check-command): No, sablefile check does not need a network connection to run. [sec:check-command]
  unfiltered (cites remote-fetch-failure): Yes, sablefile checks need a network connection to run in 1.0, as it requires fetching remote templates, and if a remote...
  second-model (declined): The docs don't say.
q-0057 | version 2.0 | not planted | Will check pick up a change that someone made on a remote template's server?
  reference: No. check never fetches remote templates, so only the next sync's output and the output of diff reflect that change.
  baseline (cites remote-retries): No, `check` does not see changes on a remote template's server. [sec:remote-retries]
  unfiltered (cites remote-retries): No, `check` does not fetch anything and never retries, so it will not pick up a change that someone made on a remote tem...
  second-model (cites remote-templates): Check does not see changes on a remote template's server. A change on the server affects what the next sync writes and w...
q-0058 | version 2.0 | not planted | If I run check --no-fail while drift is present, what does its printed output look like?
  reference: It still prints a DRIFT line and the path for each drifted file. The flag only changes the exit status to 0.
  baseline (cites quickstart-check): Run `sablefile check` after a sync. If no output appears, no generated file has drifted since the last sync. To see what...
  unfiltered (declined): The docs don't say.
  second-model (no citation): The check prints `DRIFT` and the path, and exits with status 1. A check with no drift exits with status 0. Pass `--no-fa...
q-0059 | version 1.1 | not planted | What is the mode that sablefile 1.1 stores for each generated file, and which command stores it?
  reference: The Unix permission bits of the file, such as 0644. sync saves it for every file it writes.
  baseline (cites lock-mode lock-mode): The mode that sablefile 1.1 stores for each generated file is the Unix permission bits, such as `0644` [sec:lock-mode]. ...
  unfiltered (cites lock-mode): The mode that sablefile 1.1 stores for each generated file is the Unix permission bits, such as `0644`. The `sync` comma...
  second-model (cites lock-mode lock-mode): The mode that sablefile 1.1 stores for each generated file is described in [sec:lock-mode]. The command that stores it i...
q-0060 | version 2.0 | not planted | What top-level keys does the 2.0 lock file JSON contain?
  reference: A format key, set to the string 2.0, and a files key that maps each generated path to its SHA-256 hash and mode.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (no citation): The 2.0 lock file JSON contains keys: `lock.json`.
q-0061 | version 1.0 | not planted | What is the usual way to preview sync's changes in 1.0 before writing?
  reference: Run sablefile diff. It shows the differences between each rendered template and its target on disk, without writing any file or changing the lock.
  baseline (cites diff-command): Run `sablefile diff` to preview sync's changes before writing. [sec:diff-command]
  unfiltered (cites sync-default): To see the changes before they are written, run `sablefile diff`, which shows the same differences without writing anyth...
  second-model (cites diff-command): The usual way to preview sync's changes in 1.0 before writing is to use `sablefile diff`. [sec:diff-command]
q-0062 | version 2.0 | not planted | Can I point sablefile init at a folder other than the current one in 2.0?
  reference: Yes. Give the folder as an argument, such as init path/to/folder. The folder must already exist, and a relative path is read from the folder where you run the command.
  baseline (cites init-directory): Yes, you can point `sablefile init` at a folder other than the current one. To do this, provide the path to the folder a...
  unfiltered (cites init-directory): Yes, you can point `sablefile init` at a folder other than the current one in 2.0. To do this, give the folder as an arg...
  second-model (declined): The docs don't say.
q-0063 | version 1.1 | not planted | If a target sits in folders that do not exist yet, what does sync do in 1.1?
  reference: It creates the missing folders before it writes the file, so no setup is needed. sync never removes a folder, even one that ends up empty.
  baseline (declined): The docs don't say.
  unfiltered (cites sync-create-dirs): If a target sits in folders that do not exist yet, sync creates the folder before it writes the file in version 1.1. [se...
  second-model (no citation): The lockfile is `sablefile.lock` in the project root. It holds one line for each generated file: the SHA-256 hash of the...
q-0064 | version 2.0 | not planted | After I delete a target from sablefile.toml, does sync --apply in 2.0 remove its now-empty folder?
  reference: No. A sync run never deletes a directory, which means any empty ones you leave behind remain.
  baseline (cites sync-create-dirs): No, sync --apply does not remove empty folders. sync never removes a folder, so empty folders you leave behind stay wher...
  unfiltered (cites sync-create-dirs): No, sync --apply in 2.0 does not remove its now-empty folder. sync never removes a folder, so empty folders you leave be...
  second-model (declined): The docs don't say.
q-0065 | version 1.0 | not planted | Where can template files sit inside a 1.0 project?
  reference: Anywhere in the project. The template key in each [[file]] table gives the path from the project folder, and a template is read only when a command renders it.
  baseline (cites templates-location): Template files can sit in any folder of the project. [sec:templates-location]
  unfiltered (cites templates-location): Template files can sit in any folder of the project. [sec:templates-location]
  second-model (no citation): Template files are ordinary text files, and they can sit in any folder of the project. The `template` key in a `[[file]]...
q-0066 | version 2.0 | not planted | Does sablefile run any code that is inside a template?
  reference: No. Rendering only substitutes placeholders. It executes nothing in the template, and the same template with the same values always gives the same output.
  baseline (cites template-render): No, sablefile does not run any code that is inside a template. Rendering does not run code in the template. [sec:templat...
  unfiltered (cites template-render): No, sablefile does not run any code that is inside a template. Rendering does not run code in the template. [sec:templat...
  second-model (declined): The docs don't say.
q-0067 | version 1.1 | not planted | Can a [[file]] table in 1.1 take a template that lives at an https address?
  reference: Yes. The template key can name an https URL. sablefile fetches it when a command renders the template, and the fetched text is not copied into the project.
  baseline (cites remote-templates): Yes, a [[file]] table in 1.1 can take a template that lives at an https address. [sec:remote-templates]
  unfiltered (cites config-files-table): Yes, a [[file]] table in 1.1 can take a template that lives at an https address. [sec:config-files-table]
  second-model (declined): The docs don't say.
q-0068 | version 2.0 | not planted | What happens when a remote fetch in 2.0 still fails after every retry?
  reference: The command halts, reporting the failing address in its error message, and ends with status 1. Targets it never wrote retain their previous lock records.
  baseline (cites remote-fetch-failure): If a remote template still cannot be fetched after the last retry, the command stops with an error that names the URL, a...
  unfiltered (cites remote-fetch-failure): If a remote template still cannot be fetched after the last retry, the command stops with an error that names the URL, a...
  second-model (declined): The docs don't say.
q-0069 | version 1.0 | not planted | What exit status does sync give in 1.0 when a template file is missing?
  reference: Status 1. sync cannot finish in that case, and its error message names the file that caused the failure.
  baseline (cites sync-exit-status): Sync exits with status 1 when it cannot finish, for example when a template file is missing or a remote template cannot ...
  unfiltered (cites sync-exit-status): sync exits with status 1 when a template file is missing. [sec:sync-exit-status]
  second-model (cites sync-exit-status): The exit status for sync in 1.0 when a template file is missing is 1. [sec:sync-exit-status]
q-0070 | version 1.1 | not planted | If sync in 1.1 exits with status 0, does that mean the generated text is what I want?
  reference: Not necessarily. Status 0 only shows that sync finished, so you should open the generated files and check them.
  baseline (cites sync-exit-status): No, if sync exits with status 0, it only means that sync completed successfully, not that the generated text is what you...
  unfiltered (cites sync-exit-status): No, if sync in 1.1 exits with status 0, it does not mean the generated text is what you want. A status of 0 shows that s...
  second-model (declined): The docs don't say.
q-0071 | version 1.0 | not planted | Where do I list paths for sablefile 1.0 to leave alone, and what happens to them?
  reference: In the ignore file .sablefileignore in the project root, with one entry per line. Sync leaves an ignored target unwritten, check and diff both pass over it, and no lock record is made for it.
  baseline (cites ignore-file): You list paths for sablefile 1.0 to leave alone in the ignore file, `.sablefileignore`, which sits in the project root. ...
  unfiltered (cites ignore-file): List paths for sablefile 1.0 to leave alone in the `.sablefileignore` file, which sits in the project root. Put one entr...
  second-model (no citation): The ignore file, `.sablefileignore`, sits in the project root and lists paths that sablefile must leave alone. A project...
q-0072 | version 2.0 | not planted | With --apply, does 2.0's sync write a target that my ignore file lists?
  reference: No. An ignored target is never written, and it gets no lock record either.
  baseline (cites ignore-file): No, with `--apply`, 2.0's sync does not write a target that is listed in the ignore file. [sec:ignore-file]
  unfiltered (cites ignore-file): No, with --apply, 2.0's sync does not write a target that is listed in the ignore file. [sec:ignore-file]
  second-model (declined): The docs don't say.
q-0073 | version 2.0 | not planted | Which lines in a 2.0 ignore file are skipped as comments?
  reference: Lines that begin with a hash sign. Blank lines are also ignored.
  baseline (cites ignore-syntax): Lines that start with `#` are comments. [sec:ignore-syntax]
  unfiltered (cites ignore-syntax): Lines that start with `#` are comments. [sec:ignore-syntax]
  second-model (declined): The docs don't say.
q-0074 | version 1.1 | not planted | A check in 1.1 reports drift for a file I did not mean to change. What should I do first?
  reference: Start with diff, which shows how the file differs from its template. Keep wanted changes by moving them into the template before running sync; discard unwanted ones by reverting the file and running check again.
  baseline (cites troubleshoot-drift): Look at the change before you act. Run `sablefile diff` to see how the file differs from its template. [sec:troubleshoot...
  unfiltered (cites troubleshoot-drift): Look at the change before you act. [sec:troubleshoot-drift]
  second-model (cites check-command): Run `sablefile diff` to see how the file differs from its template. [sec:check-command]
q-0075 | version 2.0 | not planted | After installing the 2.0 archive, how do I confirm which version the shell is running?
  reference: Run sablefile --version. The output has the form sablefile 2.0, and that number should match the docs you are reading.
  baseline (cites install-check-version): Run `sablefile --version` to see the installed version. [sec:install-check-version]
  unfiltered (cites install-check-version): Run `sablefile --version` to see the installed version. The output has the form `sablefile 2.0`. Check that the number m...
  second-model (declined): The docs don't say.
q-0076 | version 1.0 | not planted | How do I remove sablefile 1.0 without losing my project's files?
  reference: Run python3 -m pip uninstall sablefile. That removes the command and its package, and leaves the project's own files alone, including the config, templates, outputs and lock.
  baseline (cites install-uninstall): To remove sablefile 1.0, run `python3 -m pip uninstall sablefile`. This removes the `sablefile` command and its package....
  unfiltered (cites install-uninstall): Run `python3 -m pip uninstall sablefile`. This removes the `sablefile` command and its package. It does not change your ...
  second-model (declined): The docs don't say.
q-0077 | version 2.0 | not planted | I installed sablefile but my shell says the command is not found. What should I check in 2.0?
  reference: Ask pip where it put the package, using pip show sablefile from that very same shell. If a location comes back yet the command fails, the scripts folder may be missing from your path.
  baseline (cites troubleshoot-command-not-found): If the shell reports that `sablefile` is not found, the command is not installed in the environment that is active. Run ...
  unfiltered (cites troubleshoot-command-not-found): If the shell reports that `sablefile` is not found, the command is not installed in the environment that is active. Run ...
  second-model (declined): The docs don't say.
q-0078 | version 1.1 | not planted | Does upgrading sablefile to a newer 1.x archive change any file in my project?
  reference: No. The upgrade writes no project file. Read the changelog in the newer version's folder first, since it lists each section whose behaviour changed.
  baseline (cites install-upgrade): No, installing a new version of sablefile does not write any project file. [sec:install-upgrade]
  unfiltered (cites install-upgrade): No, installing a new version does not write any project file. [sec:install-upgrade]
  second-model (declined): The docs don't say.
q-0079 | version 2.0 | not planted | After a 2.0 sync --apply, how do I confirm the template placeholder was filled in?
  reference: Open the target, such as README.md, and look for the project name in the spot where the placeholder stood.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0080 | version 1.0 | not planted | What is the difference between a placeholder and a value in sablefile 1.0?
  reference: A placeholder is a name in a template that sync replaces. A value is the string in the [values] table that the placeholder's name points to.
  baseline (cites glossary-terms): A placeholder is a name in a template that sync replaces with a value, written as `{{project}}` in this version. A value...
  unfiltered (decline phrase with other text): The sections do not define or differentiate between a placeholder and a value in sablefile 1.0. [sec:template-render] de...
  second-model (no citation): The difference between a placeholder and a value in sablefile 1.0 is that a placeholder is a name in a template that syn...
## unanswerable: 13 dev items
q-0015 | version 1.0 | planted (false-premise) | In sablefile 1.0, what does the --dry-run flag on sync print?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0016 | version 2.0 | not planted | Does sablefile 2.0 run on Windows?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0017 | version 2.0 | not planted | How fast is sablefile check on a project with several thousand generated files?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0018 | version 1.1 | not planted | Can sablefile 1.1 fetch a remote template from a private URL that needs a login token?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (cites remote-retries): Yes, sablefile 1.1 can fetch a remote template from a private URL that needs a login token, as long as the token is incl...
  second-model (declined): The docs don't say.
q-0034 | version 1.1 | planted (false-premise) | In sablefile 1.1, what does sync --apply write to the lock file?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (cites sync-lock-update): The lock file is how check finds drift, so run sync after each change you intend to keep. Commit the lock file with the ...
q-0035 | version 1.0 | planted (false-premise) | Does check in sablefile 1.0 have a --no-fail option?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0036 | version 1.1 | planted (false-premise) | In sablefile 1.1, what does the --quiet flag on sync do?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (cites sync-default): The --quiet flag on sync is used to suppress the display of the diff output. It does not change the lock file or the syn...
q-0037 | version 1.0 | not planted | In sablefile 1.0, what happens when a template uses a placeholder whose name is not in [values]?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0038 | version 2.0 | not planted | Is there a maximum number of templates a 2.0 project can have?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0039 | version 1.1 | not planted | What exit status does sablefile diff return when it finds differences in 1.1?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (cites check-exit-status): When check finds a drifted file, it prints `DRIFT` and the path, and exits with status 1. [sec:check-exit-status]
  second-model (declined): The docs don't say.
q-0040 | version 2.0 | not planted | If a file listed in the lock is missing from disk, does check report it in 2.0?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0041 | version 2.0 | not planted | Is there a limit on how long a value in [values] can be?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0042 | version 1.0 | not planted | Can sablefile 1.0 set the output colour through a color key in sablefile.toml?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
## version-dependent: 17 dev items
q-0001 | version 1.1 | not planted | In sablefile 1.1, if I run sync with no flags, are the target files written straight away?
  reference: Yes. In 1.1 a plain sync writes each target and records it in the lock file. To preview the changes first, run diff, which shows them without writing anything.
  baseline (cites sync-default): Yes, if you run `sablefile sync` with no flags, the target files are written straight away. [sec:sync-default]
  unfiltered (cites sync-default): Yes, in sablefile 1.1, if you run `sync` with no flags, the target files are written straight away. This is because `sab...
  second-model (decline phrase with other text): The docs don't say. The docs don't say.
q-0002 | version 2.0 | not planted | In sablefile 2.0, if I run sync with no flags, are the target files written?
  reference: No. In 2.0 a plain sync prints a plan that lists what would be created or updated, and it writes nothing. Adding --apply makes sync write the targets and update the lock file.
  baseline (cites sync-default): No, the target files are not written when running `sablefile sync` with no flags. Instead, sync prints a plan and writes...
  unfiltered (cites apply-flag): No, in sablefile 2.0, if you run `sync` with no flags, the target files are not written. The `--apply` flag is required ...
  second-model (declined): The docs don't say.
q-0003 | version 1.0 | planted (version-swap) | In sablefile 1.0, is ${project} the right way to insert the project name into a template?
  reference: No. In 1.0 a placeholder is a name in double braces, written {{project}}. The text ${project} is not a placeholder in 1.0, so it is copied to the output unchanged.
  baseline (cites placeholders): No, `${project}` is not the right way to insert the project name into a template in sablefile. The correct way is to use...
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0004 | version 2.0 | not planted | In sablefile 2.0, is ${project} the right way to insert the project name into a template?
  reference: Yes. In 2.0 a placeholder is a name with a dollar sign and braces around it, so the form ${project} is replaced by the project value. The old double-brace form is an error.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0021 | version 1.0 | not planted | In sablefile 1.0, does check give a nonzero exit status when a generated file has drifted?
  reference: No. In 1.0 check prints DRIFT and the path for a drifted file but exits with status 0, so a script has to read the printed lines to see drift.
  baseline (cites check-exit-status): No, check does not give a nonzero exit status when a generated file has drifted. The check command prints "DRIFT" and th...
  unfiltered (cites quickstart-check): Yes, in sablefile 1.0, `check` gives a nonzero exit status when a generated file has drifted. This is because, without t...
  second-model (cites check-exit-status): The check exits with status 0 even when files have drifted, so a script cannot detect drift from the exit status in this...
q-0022 | version 2.0 | not planted | In sablefile 2.0, does check give a nonzero exit status when a generated file has drifted?
  reference: Yes. check exits with status 1 when a file has drifted. Adding --no-fail changes that to status 0 while still printing the DRIFT lines.
  baseline (cites check-exit-status): Yes, check gives a nonzero exit status when a generated file has drifted. [sec:check-exit-status]
  unfiltered (cites no-fail-flag no-fail-flag): No, in sablefile 2.0, `check` does not give a nonzero exit status when a generated file has drifted if the `--no-fail` f...
  second-model (cites check-exit-status): The check exits with a nonzero exit status when a generated file has drifted. [sec:check-exit-status]
q-0023 | version 1.0 | not planted | Does an ignore entry of *.log in sablefile 1.0 skip a target at docs/build.log?
  reference: No. In 1.0 each entry is a shell glob matched against the whole target path, and shell globs do not match across folders. The entry only covers files at the top level of the project.
  baseline (cites ignore-syntax): Yes, an ignore entry of `*.log` in sablefile 1.0 skips a target at `docs/build.log`. [sec:ignore-syntax]
  unfiltered (cites ignore-syntax): Yes, an ignore entry of `*.log` in sablefile 1.0 skips a target at `docs/build.log`. [sec:ignore-syntax]
  second-model (cites ignore-syntax): The docs section does not skip the target at docs/build.log. The ignore entry of *.log in sablefile 1.0 does not skip th...
q-0024 | version 2.0 | not planted | Does an ignore entry of *.log in sablefile 2.0 skip a target at docs/build.log?
  reference: Yes. In 2.0 entries use gitignore syntax, so a line such as *.log matches at any depth, including docs/build.log.
  baseline (cites ignore-syntax): Yes, an ignore entry of `*.log` in sablefile 2.0 skips a target at `docs/build.log`. [sec:ignore-syntax]
  unfiltered (cites ignore-syntax): Yes, an ignore entry of `*.log` in sablefile 2.0 skips a target at `docs/build.log`. [sec:ignore-syntax]
  second-model (declined): The docs don't say.
q-0025 | version 1.0 | planted (version-swap) | Does the lock file in sablefile 1.0 record each generated file's mode?
  reference: No. In 1.0 each lock line holds only the SHA-256 hash and the path, so no mode is recorded.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0026 | version 1.1 | not planted | Does the lock file in sablefile 1.1 record each generated file's mode?
  reference: Yes. Each lock line holds the SHA-256 hash, then the file's mode, then the path. check compares only the hash, so a change of mode alone is not reported as drift.
  baseline (cites lock-mode): Yes, the lock file in sablefile 1.1 records each generated file's mode. [sec:lock-mode]
  unfiltered (cites lock-mode): Yes, the lock file in sablefile 1.1 records each generated file's mode. [sec:lock-mode]
  second-model (cites lock-mode): The lock file in sablefile 1.1 records each generated file's mode. [sec:lock-mode]
q-0027 | version 1.0 | not planted | How many times does sablefile 1.0 retry a remote template fetch that keeps failing?
  reference: Three retries, after which the command gives up. No flag or setting changes that count in 1.0.
  baseline (cites remote-retries): Three times. [sec:remote-retries]
  unfiltered (cites remote-retries): Sablefile 1.0 retries a remote template fetch up to 3 times before giving up. [sec:remote-retries]
  second-model (cites remote-retries): The count is fixed in 1.0, and no flag or setting changes it. [sec:remote-retries]
q-0028 | version 1.1 | not planted | How many times does sablefile 1.1 retry a remote template fetch that keeps failing?
  reference: Five retries, after which the command gives up. In 1.1 no flag or setting can change that count.
  baseline (cites remote-retries): Sablefile 1.1 retries a remote template fetch up to 5 times before giving up. [sec:remote-retries]
  unfiltered (cites remote-retries): Sablefile 1.1 retries a remote template fetch up to 5 times before giving up. [sec:remote-retries]
  second-model (cites remote-retries): The count is fixed in 1.1, and no flag or setting changes it. [sec:remote-retries]
q-0029 | version 1.0 | planted (version-swap) | In sablefile 1.0, does sync --apply create the missing folder for a target?
  reference: The 1.0 docs describe no --apply flag. Sync creates a target's missing folder by itself, before it writes the file, so a target such as docs/guide/index.md needs no setup.
  baseline (cites sync-create-dirs): Yes, sync --apply creates the missing folder for a target. [sec:sync-create-dirs]
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0030 | version 2.0 | not planted | In sablefile 2.0, does sync --apply create the missing folder for a target?
  reference: Yes, but only because --apply is given. In 2.0 a plain sync creates no folder, since it writes nothing, and with --apply the folder is made before the target is written.
  baseline (cites sync-create-dirs): Yes, sync --apply creates the missing folder for a target. [sec:sync-create-dirs]
  unfiltered (cites apply-flag): Yes, sync --apply creates the missing folder for a target. [sec:apply-flag]
  second-model (cites sync-create-dirs): [sec:sync-create-dirs] Folders for targets   Yes, with `--apply`, sync creates the missing folders before writing the fi...
q-0031 | version 1.1 | planted (version-swap) | In sablefile 1.1, where is .sablefile/lock.json kept?
  reference: The 1.1 docs describe no .sablefile folder. The lock file in 1.1 is sablefile.lock in the project root, with one line for each generated file.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0032 | version 2.0 | not planted | In sablefile 2.0, where is the lock file kept?
  reference: In .sablefile/lock.json, a JSON file with a format string and one entry per generated file. A leftover sablefile.lock from 1.x is read with a warning and migrated when a command writes files.
  baseline (cites lock-migration): In sablefile 2.0, the lock file is kept in `.sablefile/lock.json`. [sec:lock-migration]
  unfiltered (cites lock-migration): In sablefile 2.0, the lock file is kept in `.sablefile/lock.json`. [sec:lock-migration]
  second-model (cites lock-mode): The lock file is kept in `.sablefile/lock.json`. [sec:lock-mode]
q-0033 | version 2.0 | not planted | In sablefile 2.0, what does sync print for a target whose content already matches its template?
  reference: The word unchanged, as that target's plan line. The plan step itself writes nothing.
  baseline (declined): The docs don't say.
  unfiltered (cites apply-flag): For a target whose content already matches its template, sync prints "unchanged". [sec:apply-flag]
  second-model (declined): The docs don't say.
## ambiguous: 10 dev items
q-0019 | version 2.0 | planted (ambiguous-target) | If I edit the file and then run check, will it fail?
  reference: Reading 1: if you edited a generated file (a target), check prints DRIFT and the path for it and exits with status 1, because its hash no longer matches the lock.
Reading 2: if you edited a template, check prints nothing and exits with status 0, since a template change is not drift.
  baseline (cites quickstart-check check-exit-status): Yes, if you edit the file and then run `sablefile check`, it will report `DRIFT` and the path of the file, indicating a ...
  unfiltered (cites check-exit-status no-fail-flag): Yes, if you edit the file and then run `sablefile check`, it will detect the drift and print `DRIFT` and the path of the...
  second-model (cites check-exit-status): If you edit the file and then run `sablefile check`, it will fail because the check will detect a drift and print a `DRI...
q-0020 | version 2.0 | not planted | Does sync change the lock file?
  reference: Reading 1: a plain sync, with no flag, writes nothing, so the lock file is left alone.
Reading 2: sync --apply writes the targets and updates .sablefile/lock.json with the hash and mode of each file it wrote.
  baseline (cites sync-lock-update): Yes, sync changes the lock file when the `--apply` flag is used. After sync writes a target, it records that file's SHA-...
  unfiltered (cites diff-command): No, sync does not change the lock file. [sec:diff-command]
  second-model (cites sync-lock-update): The lock file is how check finds drift, so run sync after each change you intend to keep. [sec:sync-lock-update]
q-0043 | version 2.0 | planted (ambiguous-target) | Is it drifted now that I changed the template?
  reference: Reading 1: if 'it' means the target, it has not drifted. Nothing has edited the target since the last sync, so check prints nothing for it.
Reading 2: if 'it' means a target that you hand-edited after the last sync, it has drifted, and check prints DRIFT and exits with status 1.
  baseline (cites drift-meaning): No, it is not drifted now that you changed the template. A change to a template is not drift, because the generated file...
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0044 | version 2.0 | not planted | Does sync create a folder for a target?
  reference: Reading 1: a plain sync writes nothing, so it creates no folder.
Reading 2: sync --apply creates the target's missing folder before it writes the file.
  baseline (cites sync-create-dirs): Yes, sync creates a folder for a target if the folder does not exist, but only when the `--apply` flag is used. [sec:syn...
  unfiltered (declined): The docs don't say.
  second-model (cites sync-create-dirs): Yes, sync creates a folder for a target. [sec:sync-create-dirs]
q-0045 | version 2.0 | not planted | Does --retries change how often the command retries?
  reference: Reading 1: for sync and diff, yes. They use the count you give, in place of the default of five.
Reading 2: for check, no. check never fetches a remote template, so it never retries.
  baseline (cites retries-flag): Yes, the `--retries` flag changes how often the command retries a failed fetch of a remote template. [sec:retries-flag]
  unfiltered (cites retries-flag remote-retries): Yes, the `--retries` flag changes how often the command retries a failed fetch of a remote template. It sets the number ...
  second-model (cites retries-flag): The `--retries` flag changes how often the command retries a failed fetch of a remote template, in place of the default ...
q-0046 | version 2.0 | not planted | If I delete sablefile.lock, will check stop reporting drift?
  reference: Reading 1: if the old file was the only lock, deleting it leaves check with nothing to read, so check prints an error message instead of DRIFT lines.
Reading 2: once .sablefile/lock.json exists, 2.0 reads only that file, so deleting sablefile.lock changes nothing.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0047 | version 2.0 | not planted | When does a 2.0 project's old lock file get migrated?
  reference: Reading 1: a command that only reads, such as check, reads the old file and prints a warning, but writes nothing, so both files stay as they are.
Reading 2: a command that writes files, such as sync --apply, also writes .sablefile/lock.json and keeps each entry's mode where the old file has one.
  baseline (cites lock-migration): When a 2.0 command finds an older `sablefile.lock` and no `.sablefile/lock.json`, it reads the old file and prints a war...
  unfiltered (cites lock-migration): When a 2.0 command finds an older `sablefile.lock` and no `.sablefile/lock.json`, it reads the old file and prints a war...
  second-model (no citation): The 2.0 project's old lock file is migrated when a 2.0 command finds an older `sablefile.lock` and no `.sablefile/lock.j...
q-0048 | version 1.0 | not planted | In 1.0, does check report the file if I change it?
  reference: Reading 1: if 'the file' is a template, no. check does not render templates, so it prints nothing for a template change.
Reading 2: if 'the file' is a generated target you edited, yes. check prints DRIFT and the path for it.
  baseline (declined): The docs don't say.
  unfiltered (cites check-command): Yes, in 1.0, `check` reports the file if it is changed. [sec:check-command]
  second-model (cites check-command): The check command does not report the file if it changes. [sec:check-command]
q-0049 | version 2.0 | not planted | Will init overwrite my config?
  reference: Reading 1: if sablefile.toml already exists, init stops with an error and leaves it alone.
Reading 2: if you mean another file in the folder, such as a README, init writes no other file, so it does not touch that file.
  baseline (cites init-command): No, init will not overwrite your config. It stops with an error if `sablefile.toml` already exists, so it never overwrit...
  unfiltered (declined): The docs don't say.
  second-model (cites init-command): init stops with an error if `sablefile.toml` already exists, so it never overwrites your configuration. [sec:init-comman...
q-0050 | version 1.1 | not planted | Does the 1.1 lock file care about a file's mode?
  reference: Reading 1: yes, the lock records each generated file's mode alongside its hash.
Reading 2: no, for comparison. check compares only the hash, so a mode-only change is not reported as drift.
  baseline (cites lock-mode): Yes, the 1.1 lock file cares about a file's mode. [sec:lock-mode]
  unfiltered (cites lock-mode): Yes, the 1.1 lock file cares about a file's mode. The mode field is described in lock-mode, and sync records the mode of...
  second-model (cites lock-mode): The 1.1 lock file does not care about a file's mode. The mode field is described in lock-mode. [sec:lock-mode]

Take item q-0005. It asks whether sablefile 2.0 still reads sablefile.lock in the project root, and it is a planted near-miss term, a name close to one the docs use, which catches a careless reply. The baseline says no and cites a section about upgrading. The reference answer says that 2.0 falls back to the file, with a warning, so the baseline's no is wrong. The unfiltered and second-model variants both decline, and the docs do answer the question, so their declines are wrong as well.

The three replies to q-0005 fail in two different ways. A decline on an answerable item and a confident wrong answer are different failures, and a check that catches one doesn't catch the other. A pass rate would count all three replies as failures and lose that difference. The note on this reply in the data file begins "says 2.0 never reads sablefile.lock; reference says it falls back to it with a warning." Each note says what the reply did and what the reference says, in one sentence, so the grouping that follows rests on something a reader can check. A count can come from different defects, and two replies that both fail a pass rate can be different failures. That is why the replies come before the count.

The comparison has limits, and they bound what the counts can claim. The dev items were written to probe the assistant, and the planted items are not a random sample of what users ask. So the counts say which variant fails in which way on these items. They don't say how often a user would meet each failure. A sample of questions drawn the way users ask them would be needed for that.

Working rule: Before you count anything, read replies from every variant, grouped by kind, and write down what is wrong with each one. A count can say that a variant fails. Only the replies say how.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Coding the failures

Reading becomes counting once each reply has a note and a mode. The drafting model wrote one note per reply, as the one on q-0005 does, and then grouped the notes. Each reply got one mode, its main defect, and a reply with no defect got ok. The codes were made by the AI model that drafted this book, as one coder; no second coder checked them, and nobody measured agreement on the codes. The 80 dev items give 240 replies, 80 from each variant. The codes sit in a data file beside the examples, one entry per reply.

The counts carry the label "coded by the drafting model (one coder; not human labels)", and the label means what it says. This chapter has no agreement figure, and no one else has reproduced the counts. Chapter 1 made the same point: agreement with your own labels doesn't calibrate a judge. Chapter 6 takes that up.

The tally example turns the notes into counts, and it checks the codes first. Each dev item has exactly one code per variant, no code names a mode the file doesn't define, and the file carries the label. Then it prints one row per mode and one column per variant, and one labelled line for each count, which are the numbers the prose below quotes.

tally-failure-modes.py Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0 통과
"""Counts the drafting model's failure codes over the dev items, per variant, and prints one table and one labelled lineper count.The point rests on the table's reading of each mode: the codes were made by the drafting model, as one coder, and are nothuman labels. The codes live in data/failure-codes.json; the checks below stop a reply being counted twice or left out.The labelled lines are the numbers the chapter's prose quotes."""import lib  # first: fixes the thread counts and seeds before anything else loadsimport jsonfrom collections import Counterfrom pathlib import Pathfrom lib.testset import load_itemsCODES = Path(__file__).resolve().parent / "data" / "failure-codes.json"VARIANTS = ("baseline", "unfiltered", "second-model")book = json.loads(CODES.read_text(encoding="utf-8"))assert book["labelled"] == "coded by the drafting model (one coder; not human labels)", "the codes are the model's, not human labels"  # the label the prose usesassert book["variants"] == list(VARIANTS), "the variants in the file are the three the chapter names"dev_ids = {item.id for item in load_items(split="dev")}n = len(dev_ids)codes = book["codes"]assert {code["item"] for code in codes} <= dev_ids, "a code names an item that is not in the dev split"pairs = [(code["item"], code["variant"]) for code in codes]# One code per dev item and variant: a pair repeated would be counted twice, a missing one would go unseen.assert len(pairs) == len(set(pairs)) == n * len(VARIANTS), "expected one code per dev item and variant"assert all(code["mode"] in book["modes"] for code in codes), "a code uses a mode the file does not define"counts = Counter((code["variant"], code["mode"]) for code in codes)print(f"dev items: {n}")print(f"replies coded: {len(codes)}")print(f"labelled: {book['labelled']}")print(f"{'mode':<22}" + "".join(f"{variant:>14}" for variant in VARIANTS))for mode in book["modes"]:  # the file's order: ok first, then the failure modes    total = sum(counts[(variant, mode)] for variant in VARIANTS)    if total == 0:  # a mode no variant shows is printed as zero of N, so the reader sees it was looked for        print(f"{mode}: 0 of {n}")        continue    print(f"{mode:<22}" + "".join(f"{counts[(variant, mode)]:>14}" for variant in VARIANTS))totals = {variant: sum(counts[(variant, mode)] for mode in book["modes"]) for variant in VARIANTS}assert all(total == n for total in totals.values()), "each variant's counts add up to the dev items"print(f"{'all modes':<22}" + "".join(f"{totals[variant]:>14}" for variant in VARIANTS))print()for variant in VARIANTS:  # one labelled line per count, the form the chapter's prose binds to    for mode in book["modes"]:        print(f"{mode}, {variant}: {counts[(variant, mode)]}")    print(f"defective, {variant}: {n - counts[(variant, 'ok')]}")
출력python tally-failure-modes.py
dev items: 80
replies coded: 240
labelled: coded by the drafting model (one coder; not human labels)
mode                        baseline    unfiltered  second-model
ok                                56            49            24
over-declining                    10            15            31
wrong-version-answer               2             3             0
wrong-answer                       3             5             9
missed-refusal                     0             2             2
incomplete-answer                  8             4             7
citation-problem                   1             2             6
malformed-reply                    0             0             1
all modes                         80            80            80

ok, baseline: 56
over-declining, baseline: 10
wrong-version-answer, baseline: 2
wrong-answer, baseline: 3
missed-refusal, baseline: 0
incomplete-answer, baseline: 8
citation-problem, baseline: 1
malformed-reply, baseline: 0
defective, baseline: 24
ok, unfiltered: 49
over-declining, unfiltered: 15
wrong-version-answer, unfiltered: 3
wrong-answer, unfiltered: 5
missed-refusal, unfiltered: 2
incomplete-answer, unfiltered: 4
citation-problem, unfiltered: 2
malformed-reply, unfiltered: 0
defective, unfiltered: 31
ok, second-model: 24
over-declining, second-model: 31
wrong-version-answer, second-model: 0
wrong-answer, second-model: 9
missed-refusal, second-model: 2
incomplete-answer, second-model: 7
citation-problem, second-model: 6
malformed-reply, second-model: 1
defective, second-model: 56

Read the ok row, then the rows under it. The baseline has the most replies with no defect, 56, and the second-model variant has the fewest, 24. Over-declining is the largest defect in each variant. In the second-model variant it is more than half of the defects: 31 of 56.

The baseline doesn't escape the worse defects. It gives 2 wrong-version answers, on q-0023 and q-0029, and the unfiltered variant gives the most, 3. Missed refusals come only from the variants, and the baseline has none. Every mode appears at least once. A mode that no variant showed would print as zero, so a reader could see it was looked for.

Working rule: Give each reply one note and one mode, and print one labelled line per count, so that every number the prose gives can be checked against the output.

과잉 거절

과잉 거절 응답은 문서가 답하는 항목을 거절하는 응답이다. 거절은 신중해 보이기 때문에 독자가 가장 먼저 마주치는 결함이며, 틀린 답과 마찬가지로 사용자의 기대를 저버린다. 여기서 쓰는 용어인 거절 문구, 답할 수 없는 항목, 변형은 응답의 모습에서 정의한다. 집계 대상은 문서가 답하는 모든 항목이며 버전에 따라 답이 달라지는 항목도 포함한다. 그래서 기준선의 경우 답할 수 있는 항목만 세는 10장의 거짓 거절률보다 수치가 크다. 항목 q-0004는 1장에서 자세히 설명하는, 버전에 따라 답이 달라지는 사례다. 세 변형 모두 같은 문구로 이 항목을 거절하므로 기준선도 마찬가지로 거절한다.

거절을 세는 일에는 함정이 있고, 그 함정은 파서에 있다. 항목 q-0080은 답할 수 있는 항목인데, 필터를 적용하지 않은 변형은 이 항목에 대해 절들이 그 차이를 정의하지 않는다고 말하는 두 문장을 쓴 뒤 거절 문구로 끝낸다. 문서가 이 질문에 답하므로 이 응답은 과잉 거절이다. 그러나 파서는 문구가 응답 전체일 때만 거절로 세므로, 이 응답을 절을 인용한 답으로 읽는다. 문구를 검색하는 독자는 그 문구를 찾아내지만, 파서에 기반한 검사는 이 응답을 놓친다.

검사 후보는 참조 답안이 근거 절을 명시하는 항목에 대한 응답이 어떤 말로도 거절해서는 안 된다는 것이다. 코드로 작성하는 검사에서 이런 종류의 검사를 만든다. 문서가 다루지 않는 항목에서 거절이 옳은지는 판단의 문제이며, 10장에서 다룬다.

작업 규칙: 문구가 단독으로 서 있는지가 아니라 응답이 실제로 무엇을 하는지를 기준으로 거절을 세고, 과잉 거절은 틀린 답과 따로 센다. 두 결함은 검사 방법이 다르기 때문이다.

잘못된 버전의 답변

잘못된 버전의 답변은 질문이 묻는 버전이 아니라 다른 버전의 동작을 알려 주는 답변이며, 유창하고 출처까지 달려 있으면서도 틀릴 수 있다. q-0029에서 기준 변형은 1.0에서 sync --apply가 없는 폴더를 만드느냐는 질문에 그렇다고 답하고, 폴더 생성을 다루는 절을 인용한다. 이 답변에 달린 메모는 그것이 2.0의 동작이며 1.0에는 --apply 플래그 자체가 없다고 지적한다.

필터를 적용하지 않은 변형은 이런 오류를 더 자주 낸다. q-0021에서 이 변형은 1.0에서 드리프트가 있을 때 check가 0이 아닌 종료 상태를 반환하느냐는 질문에 그렇다고 답하는데, 기준 변형과 두 번째 모델 변형은 둘 다 반환하지 않는다고 답한다. 메모는 필터를 적용하지 않은 답변이 1.0 질문에 2.0의 규칙을 제시했다고 적는다. 이 변형의 컨텍스트에는 모든 버전이 들어 있으므로 2.0의 규칙이 1.0의 규칙 옆에 놓여 있다.

인용만으로는 정오가 가려지지 않는다. q-0023에서 기준 변형은 2.0의 결과에 대해 그렇다고 답하면서 1.0의 절을 인용한다. 모든 인용이 질문한 버전의 절을 가리키는지 확인하는 검사는 이 답변을 통과시킨다. 주장을 1.0 레퍼런스와 대조해 읽어야만 오류가 드러난다.

작업 규칙: 각 답변의 버전을 두 곳에서 확인한다. 인용은 질문한 버전에 속해야 하고, 주장은 그 버전의 레퍼런스와 일치해야 한다. 인용 검사만으로는 틀린 버전에 대해 옳게 답한 답변을 통과시키게 된다.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Missed refusals

A missed refusal answers an unanswerable item with content the docs don't hold. It is the mode a reader is most likely to believe, because the reply reads like any other answer. The terms used here are defined in what the replies look like. The dev items show it only in the variants, 2 times in the unfiltered variant and 2 in the second-model variant, and each is short enough to read whole.

In q-0018 the unfiltered variant says a login token can be sent with a remote template fetch, and cites a section about retries. In q-0034 the second-model variant gives a tip about the lock file, cited to a section that doesn't cover the question. In q-0036 it says --quiet suppresses the diff output, citing a section that never mentions the flag. In q-0039 the unfiltered variant answers about check when the question asks about diff. The baseline declines every one of them.

A code check can require a decline on every unanswerable item, and it would catch each of them. On an answerable item the same fault goes unseen: a reply that cites a real section silent on its claim would pass, because it answers, as an answer is expected there. Reading the claim against the cited section is the work of chapter 9 and of chapter 10.

Working rule: Require a decline on every unanswerable item, and read every answer to one against the section it cites before you count it as right.

실패에서 검사로

실패 유형은 검사 후보를 갖출 때 비로소 쓸모가 있다. 후보는 네 부분으로 이루어진다. 유형, 검사가 읽는 입력, 검사가 시험하는 주장, 그리고 검사가 볼 수 없는 것이다. 한계를 밝히지 않은 후보는 충분히 따져 본 것이 아니다. 또한 후보는 아직 검사가 아니다. 코드로 하는 검사에서는 프로그램이 실행할 수 있는 검사를 만들고, 판단이 필요한 것은 판정기에 넘긴다. 아래 목록은 실패 부호화에서 다룬 일곱 가지 실패 유형 각각에 후보를 주고, 그 유형을 보여 주는 사례를 짚는다.

  • 과도한 거절. 참조 답이 뒷받침 절을 지목하는 항목에 대한 응답은 어떤 말로도 거절해서는 안 된다. 이 검사는 문서가 다루지 않는 항목에 대한 올바른 거절과 잘못된 거절을 구별하지 못하므로, 그 판단은 10장으로 넘어간다.
  • 잘못된 버전의 답. 모든 인용은 질문한 버전에 속해야 하고, 주장은 그 버전의 참조 답과 일치해야 한다. 앞의 절반은 코드로 하는 검사의 코드 검사이다. 뒤의 절반은 q-0023을 잡아내며, 9장의 버전 정확성에 해당한다. 이 검사는 참조 답이 다루지 않는 주장은 볼 수 없다.
  • 틀린 답. 주장은 질문한 버전의 참조 답과 일치해야 하며, 모호한 항목에 대한 응답은 판정기 하나에서 정한 대로 각 해석에 답해야 한다. 서면 평가 기준을 가진 판정기가 각 주장을 참조 답과 비교하며, 이는 5장과 같다. 시험 사례는 q-0005이다. 기준선은 2.0이 sablefile.lock을 전혀 읽지 않는다고 답하지만, 참조 답은 2.0이 그 파일로 되돌아가 읽는다고 한다. 모호한 항목에서는 평가 기준이 참조 답에 두 해석을 모두 담아야 하며, q-0048이 그 예이다. 필터링하지 않은 변형은 그렇다고 답하고 두 번째 모델 변형은 아니라고 답하는데, 각각 한 해석에서 틀린다. 판정기도 틀릴 수 있으며, 이는 6장에서 측정한다.
  • 놓친 거절. 답할 수 없는 항목에 대한 모든 응답은 거절하거나, 읽는 사람이 그 인용을 지목된 절과 대조해 확인한다. 앞의 것은 코드 검사이다. 뒤의 것은 9장의 작업이다. 코드 검사는 절의 본문을 읽지 않으므로, 답할 수 있는 항목에서는 실제 절을 인용했지만 그 절이 주장에 대해 아무 말도 하지 않는 응답을 통과시킨다.
  • 불완전한 답. 참조 답은 옳은 답에 필요한 수치, 단계, 해석을 하나씩 나열하며, 코드 검사는 응답에서 그 각각을 찾아볼 수 있다. 이 책은 그런 검사를 만들지 않는다. q-0011에서 기준선은 재시도 횟수가 고정이며 이를 바꾸는 플래그는 없다고 말하지만 그 수치는 끝내 제시하지 않는다. 필터링하지 않은 변형은 수치를 제시하므로, 참조 답의 수치를 찾는 검사는 기준선의 응답을 불합격시킬 것이다. 이 검사의 한계는, 수치가 응답에 나타나도 그것이 답이 아닐 수 있다는 점이다.
  • 인용 문제. 모든 인용은 그 항목의 뒷받침 절 가운데 하나여야 한다. q-0057에서 기준선은 check가 원격 서버의 변경을 보지 못한다고 옳게 말하지만, 항목이 뒷받침 절로 열거하지 않은 절을 인용하므로 그 인용은 목록에 없다. 그 목록과 대조하는 코드 검사가 이를 잡는다. 이 검사의 한계는, 목록에 있는 절도 주장을 뒷받침하지 못할 수 있다는 점이며, 이는 코드로 하는 검사에서 설명하듯 읽는 사람이 필요하다.
  • 형식이 잘못된 응답. 프롬프트가 요구하는 형식에 대해 응답의 형태를 확인하는 검사는 순서가 틀리게 나온 응답을 잡아낸다. q-0030에 대한 두 번째 모델의 응답은 답은 옳지만 인용을 맨 앞에 두고 문장을 뒤틀어 놓는다. 이 검사의 한계는, 형태는 답에 대해 아무것도 말해 주지 않으므로 응답이 형태 검사를 통과하고도 틀릴 수 있다는 점이다. 코드로 하는 검사의 형식 검사가 이를 만든다.

목록의 각 후보는 자신이 잡아냈을 응답을 명시하며, 그것이 첫 번째 시험이다. 후보가 자신이 겨냥한 응답을 표시하지 못하면 그것은 아직 그 유형의 검사가 아니다.

코드 검사를 먼저 만든다. 코드 검사는 몇 초 만에 실행되고 모델 호출이 필요 없으므로 모든 변경마다 실행할 수 있다. 판정기는 응답마다 모델 호출이 들고 자체 보정이 필요하므로, 참조 답과 인용만으로는 결론이 나지 않는 유형에 쓴다. 틀린 답에는 주장을 판정할 판정기가 필요하다. 잘못된 버전의 답에는 주장을 참조 답과 대조해 읽는 부분에만, 놓친 거절에는 주장을 인용된 절과 대조해 읽는 부분에만 필요하다.

실제 운영에서는 유형에 꼬리표가 붙어 도착하지 않는다. 사용자가 잘못된 응답을 신고하면 첫 질문은 그것이 어느 유형이냐는 것이다. 잘못된 유형을 위해 쓴 검사는 실패한 응답을 통과시키기 때문이다. 맞는 유형이 없으면 실패한 응답을 다시 읽고 부호를 붙이고, 유형을 추가하고, 그 검사를 쓴다. 이 장이 무엇이든 세기 전에 먼저 응답을 읽고 부호화하는 이유가 이것이다.

이 목록은 닫혀 있지 않다. 보류 항목과 이후의 변형은 이 항목들에 없는 유형을 보여 줄 것이고, 각각에 고유한 메모와 후보가 필요할 것이다.

이 장에서 내린 결정은 다음과 같다.

  • 부호는 한 명의 부호화자가 붙였다. 초안 작성 모델이다. 그렇게 표시해 두며, 일치도는 측정하지 않는다.
  • 여기서 읽은 항목은 개발 항목뿐이다. 보류 항목의 거절 집계는 10장에 나온다.
  • 기준선의 응답도 변형과 함께 집계하므로, 기준선의 잘못된 버전의 답 2건도 그대로 보고한다.

작업 규칙: 검사를 고르기 전에 응답을 주된 결함에 따라 부호화하고, 각 검사는 한 가지 유형을 위해 쓴다. 예상한 실패로 만든 검사는 그 실패를 시험할 뿐이며, 예상하지 못한 실패는 통과시킨다.

이 판의 이 장

2026.11.3판예제 통과 2026년 11월 21이 장에서 2개 예제개 중 2개가 통과한다

영어판 2026.11.3판을 번역한 것이다.

이 절 인용

형식

각 형식은 이 번역이 따르는 영문 원서를 출처로 표기한다.

모든 형식은 이 판을 명시한다. 링크는 현재 판을 열며, 여기서 절의 제목은 본문이 마지막으로 바뀐 시점을 알려 준다. 접근 날짜와 검색 날짜는 이 자료를 여는 날짜다.

이 페이지의 내용

11장 중 2장: 답변을 읽고 무엇을 측정할지 찾는다

  1. What the replies look like
  2. Coding the failures
  3. 과잉 거절
  4. 잘못된 버전의 답변
  5. Missed refusals
  6. 실패에서 검사로

맨 위로 돌아가기전체 장