為你找到的失敗寫程式檢查
閱讀約需 43 分鐘 分鐘,9 節 中共有 5 個範例。
讀完本章,你能寫出格式與引用檢查,讓它們對每一則回覆執行而不花任何成本;能說明這些檢查各自看得到、看不到第 2 章中的哪些失敗;也能指出兩種標準的重疊分數在哪些情況下,把錯誤的答案排在正確的答案之上(評分高於正確的答案)。
第 2 章為 240 則開發集回覆編碼,三個變體各 80 則,並依每則回覆的主要缺陷,把它們歸入七種失敗模式。其中有些模式會在回覆的文字中留下痕跡。引用提示從未提供的那一節,就是一例;回答文件並未涵蓋的問題,也是一例。程式只要把回覆與它所回答的項目對照著讀,就能找出這兩種情況。另一些模式則不留下程式讀得出的痕跡。一個答案可能對某個版本是正確的,對另一個版本卻是錯的,而它的引用本身並無錯誤,回覆的文字中也沒有任何線索指出它指的是哪個版本。
如果團隊只為看得見的痕跡寫檢查,回報的就會是一次乾淨的執行結果,卻漏掉了它從未去找的失敗。本章依檢查能看見什麼來分類,建構能以程式執行的那幾種,並說明每一種遺漏了什麼。本章所有數字背後的測試集,包含 80 個開發集項目與 40 個保留集項目。本章只讀開發集項目。保留集項目只在一個範例中以計數出現,其拒答率則在第 10 章給出。
程式檢查能看見什麼
程式檢查會讀兩樣東西:回覆的文字,以及該題項本身的資料,其中記有參考答案與支持該答案的各節清單。它在每次執行時給出相同的結果,在新一批回覆上重跑也不需任何成本。這些是它的長處。它的限制在於只讀形式與歸屬,不讀語意。它能判斷回覆是否附有引用、所引用的 ID 是否出現在提示中、回覆是否恰好就是拒答句。它無法判斷被引用的那一節所說的內容,是否與答案所說的相符。
以下是第 2 章所列的各種模式,以及程式檢查對每一種能看見什麼。
- 過度拒答。 計算拒答句出現次數的檢查,只能看見使用該句的拒答。換成其他說法的拒答,會被當成答案放行。q-0080 就是這樣通過詞句檢查的,如第 2 章所示。
- 版本錯誤的答案。 檢查可以看出每一則引用是否屬於所詢問的版本,卻看不出其主張是否與該版本的參考答案相符。
- 錯誤的答案。 程式看不見。主張是否正確,需要把主張與參考答案對照閱讀。
- 漏掉的拒答。 檢查可以要求每個無法回答的題項都必須出現拒答句,而
answered-declined-expected會在對這類題項的任何作答上觸發,無論該作答引用了什麼。它依賴題項的標籤,因此文件並未涵蓋、測試集卻標為可回答的題項,會從中溜過。 - 不完整的答案。 在本章中,程式看不見。回覆是否涵蓋整個問題,要由評審依評分準則的規則判斷。
- 引用問題。 可分為兩部分來看:引用了提示之外的內容,以及引用了題項未列出的小節。第二部分需要由人來讀,如支持度一節所說明。
- 格式不良的回覆。 部分可見。空白的回覆、附帶文字的拒答、未附引用的答案,以及各部分次序顛倒的回覆,都找得出來。下一節的次序檢查負責找出最後一類。
實用原則: 只有在回覆與題項本身已含有檢查所需的事實時,才為某種失敗撰寫程式檢查。若某種失敗的證據就是主張本身,就交給附有書面評分準則的評審,如第 5 章所做的那樣,程式檢查則只保留給該失敗中可以直接讀出的部分。
此部分尚未翻譯,因此以英文呈現。
Format checks
The format check asks whether a reply has the shape the prompt asks for, before anyone reads what it says. The function format_problems in lib/contract.py returns a sorted list of fixed names, one for each defect it finds. There are five. empty is a reply with no text. declined-with-text is a reply that contains the decline sentence along with other text. no-citation is an answer that cites nothing. unknown-section is a citation to a section the prompt didn't give. answered-declined-expected is an answer to an item that has to be declined. A second function, order_problems, reads the order of a reply's lines and returns two names: citation-first, for a reply whose first line opens with a citation, and stray-line, for a line that ends no sentence once its citations are taken out.
The format check reads the reply, the list of sections the prompt gave, and whether the item is unanswerable. The order check reads only the reply's lines. Neither reads what the answer says, so a correct answer and a wrong one with the same shape pass alike.
Run the format check on the baseline's reply to each item, and count each name, with the dev items and the held-out items kept apart.
1"""Runs format_problems over the baseline replies to every item, and counts each problem by its fixed name.23Dev and held-out items are counted apart, and held-out items appear only as counts. An item can have more than one4problem, so the counts by name can add up to more than the number of items with a problem."""5import lib # first: fixes the thread counts and seeds before anything else loads67from collections import Counter89from lib.answers import Assistant10from lib.contract import PROBLEMS, format_problems11from lib.prompts import P1_SHA256, P1_SYSTEM12from lib.stats import pct13from lib.testset import load_items1415assistant = Assistant("assistant-baseline", P1_SYSTEM, P1_SHA256)16assert len(load_items()) == 120 # the baseline recording holds one reply per item, so every item is counted1718for split in ("dev", "held-out"):19 items = load_items(split=split)20 counts = Counter()21 with_problem = 022 for item in items:23 problems = format_problems(24 assistant.reply(item),25 context_ids=assistant.context_ids(item),26 expect_decline=item.kind == "unanswerable", # an unanswerable item must be declined27 )28 with_problem += bool(problems)29 counts.update(problems)30 print(f"{split} items: {len(items)}")31 print(f"{split} items with a format problem: {pct(with_problem, len(items))}")32 for name in PROBLEMS: # the fixed names, in the order lib.contract lists them33 print(f"{split} problem {name}: {counts[name]}")
python format-problems-baseline.pydev items: 80
dev items with a format problem: 0.0% (0 of 80)
dev problem answered-declined-expected: 0
dev problem declined-with-text: 0
dev problem empty: 0
dev problem no-citation: 0
dev problem unknown-section: 0
held-out items: 40
held-out items with a format problem: 0.0% (0 of 40)
held-out problem answered-declined-expected: 0
held-out problem declined-with-text: 0
held-out problem empty: 0
held-out problem no-citation: 0
held-out problem unknown-section: 0
The baseline has no format problem on the dev items, 0.0% (0 of 80), or on the held-out items, 0.0% (0 of 40). A zero counts only if the check could have fired, and the next example shows that these checks do fire, on other recordings.
1"""Runs the format check and the order check over the unfiltered and second-model replies to the dev items. The baseline2has no problem of either kind (format-problems-baseline), so these two recordings are where the checks fire.34Each name is also run once on a fixture reply built to break it, and the example stops if a name does not fire. A zero5on a real split then means the check found nothing, not that it could not fire. Held-out items are not read here."""6import lib # first: fixes the thread counts and seeds before anything else loads78from lib.answers import Assistant9from lib.contract import ORDER_PROBLEMS, PROBLEMS, format_problems, order_problems, parse_reply10from lib.prompts import P1_SHA256, P1_SYSTEM11from lib.testset import load_items1213VARIANTS = (("unfiltered", "assistant-unfiltered"), ("second-model", "assistant-second-model"))14FIXTURE_IDS = frozenset({"example-start"}) # the one section a fixture may cite15FIXTURES = ( # name, reply text, whether the item must be declined16 ("empty", "", False),17 ("declined-with-text", "The docs don't say. [sec:example-start]", False),18 ("no-citation", "Run svc start to start the service.", False),19 ("unknown-section", "Run svc start to start the service. [sec:made-up]", False),20 ("answered-declined-expected", "Run svc start to start the service. [sec:example-start]", True),21 ("citation-first", "[sec:example-start] Start the service.\nRun svc start.", False),22 ("stray-line", "Run `svc start` [sec:example-start]\nService", False),23)2425items = load_items(split="dev")26for label, recording in VARIANTS:27 assistant = Assistant(recording, P1_SYSTEM, P1_SHA256)28 found = {name: [] for name in (*PROBLEMS, *ORDER_PROBLEMS)}29 for item in items:30 reply = assistant.reply(item)31 for name in format_problems(reply, context_ids=assistant.context_ids(item), expect_decline=item.kind == "unanswerable"):32 found[name].append(item.id)33 for name in order_problems(reply.text):34 found[name].append(item.id)35 for name, ids in found.items():36 shown = f"{len(ids)} of {len(items)}" + (f" ({', '.join(ids)})" if ids else "")37 print(f"{label} {name}: {shown}")3839for name, raw, must_decline in FIXTURES:40 reply = parse_reply(raw)41 fired = name in format_problems(reply, context_ids=FIXTURE_IDS, expect_decline=must_decline) or name in order_problems(reply.text)42 assert fired, f"the fixture for {name} does not fire the check"43 print(f"fixture {name}: fired")
python format-problems-variants.pyunfiltered answered-declined-expected: 2 of 80 (q-0018, q-0039)
unfiltered declined-with-text: 1 of 80 (q-0080)
unfiltered empty: 0 of 80
unfiltered no-citation: 0 of 80
unfiltered unknown-section: 1 of 80 (q-0019)
unfiltered citation-first: 0 of 80
unfiltered stray-line: 0 of 80
second-model answered-declined-expected: 2 of 80 (q-0034, q-0036)
second-model declined-with-text: 1 of 80 (q-0001)
second-model empty: 0 of 80
second-model no-citation: 9 of 80 (q-0001, q-0007, q-0047, q-0058, q-0060, q-0063, q-0065, q-0071, q-0080)
second-model unknown-section: 0 of 80
second-model citation-first: 1 of 80 (q-0030)
second-model stray-line: 2 of 80 (q-0007, q-0030)
fixture empty: fired
fixture declined-with-text: fired
fixture no-citation: fired
fixture unknown-section: fired
fixture answered-declined-expected: fired
fixture citation-first: fired
fixture stray-line: fired
The unfiltered variant answers q-0018 and q-0039, which are unanswerable, so answered-declined-expected counts them: 2 of 80 (q-0018, q-0039). The second model's replies carry more problems. Their no-citation count is 9 of 80 (q-0001, q-0007, q-0047, q-0058, q-0060, q-0063, q-0065, q-0071, q-0080).
The order check flags the second model's reply to q-0030. That reply opens with its citation, so it is citation-first: 1 of 80 (q-0030). Its first line also holds a fragment of a heading, which is a stray-line. A reply whose last line ends no sentence gets that name too, and one of the two is q-0007, whose text is cut off in the middle of a repeated sentence: 2 of 80 (q-0007, q-0030).
The format check passes that q-0030 reply. format_problems counts citations and doesn't read where they fall, so a reply can cite only sections the prompt gave and still have its parts out of order. That gap is what the order check is for. The example also runs one fixture reply for each name, and it stops if a name does not fire, so each zero above means the check looked and found nothing.
A check that never fires on the baseline still earns its place, because it costs nothing and catches the next prompt that breaks a shape.
Working rule: Run the format check and the order check on every reply before a judge reads it, and report each name's count on each split. A zero counts as a result only when the check could have fired, so keep the fixture run with the report. When a count rises, read the replies behind it before you change the check, because a name says which shape broke and not why it broke.
此部分尚未翻譯,因此以英文呈現。
Citation checks
Chapter 1 wrote the first citation check: every ID a reply cites must be one the prompt gave. That check says whether a citation is inside the prompt. It doesn't say whether the citation is the right one for the item. This section asks two more questions of each cited ID. Does it exist in the docs for the version the item asks about? And is it a section the item lists as supporting the answer?
citation_outcome gives each cited ID one of four outcomes. supported means the ID is on the item's list for the asked version. unlisted means a section of the asked version that the item doesn't list. wrong-version means a section that exists in another version only. missing means no version has a section with the cited ID. Two rules apply to all four. A reply carries IDs and not versions, so an ID that appears in two versions can't be wrong-version, and it's judged against the version asked. And unlisted is never called wrong. The item's list was written by hand, and a section it doesn't list may still support the answer.
The check runs over the dev items for three recordings: the baseline; the unfiltered variant, which uses the same prompt but retrieves from every version and shows the versions to the model; and prompt-b, chapter 1's quote-first prompt P2 (two prompts), which asks the model to quote first the sentence its answer relies on and cites on each line. Its 117 citations come from the same items, so its shares are per citation. Each recording's citations are sorted the same way.
1"""Sorts each section ID that a reply cites into one outcome, for three recordings, over the dev items.23The outcomes are supported (listed by the item for its asked version), unlisted (a section of the asked version that the4item does not list), wrong-version (a section of another version only) and missing (a section of no version). The5check reads IDs, not sentences: whether a listed section supports the sentence is the judge's job, in grounded-answers.67An ID that exists in two versions cannot be told apart by the reply, because the citation carries no version. It is8judged against the asked version, so an ID shared by both versions is never wrong-version."""9import lib # first: fixes the thread counts and seeds before anything else loads1011from collections import Counter1213from lib.contract import CITATION_OUTCOMES, citation_outcomes, parse_reply, section_ids_by_version14from lib.prompts import P1_SHA256, P2_SHA25615from lib.replay import ReplayModel16from lib.stats import pct17from lib.testset import load_items1819RECORDINGS = ( # label, recording, the sha256 of the prompt it was made from20 ("baseline", "assistant-baseline", P1_SHA256),21 ("unfiltered", "assistant-unfiltered", P1_SHA256), # same prompt; retrieval from every version, versions shown22 ("prompt-b", "assistant-prompt-b", P2_SHA256),23)2425ids = section_ids_by_version()26items = load_items(split="dev") # dev only: held-out items are not read here (plan section 3.4)2728for label, recording, prompt_sha256 in RECORDINGS:29 model = ReplayModel(recording, prompt_sha256)30 counts = Counter()31 for item in items:32 reply = parse_reply(model.reply(item.id, 0))33 # The item's list for its asked version. An unanswerable item may also list another version: not counted here.34 listed = frozenset(s.section_id for s in item.supporting_sections if s.version == item.version)35 counts.update(citation_outcomes(reply, item.version, listed, ids))36 total = sum(counts.values())37 print(f"{label} citations on dev items: {total}")38 for outcome in CITATION_OUTCOMES:39 print(f"{label} {outcome}: {pct(counts[outcome], total)}")
python check-citation-support.pybaseline citations on dev items: 60
baseline supported: 93.3% (56 of 60)
baseline unlisted: 6.7% (4 of 60)
baseline wrong-version: 0.0% (0 of 60)
baseline missing: 0.0% (0 of 60)
unfiltered citations on dev items: 59
unfiltered supported: 84.7% (50 of 59)
unfiltered unlisted: 15.3% (9 of 59)
unfiltered wrong-version: 0.0% (0 of 59)
unfiltered missing: 0.0% (0 of 59)
prompt-b citations on dev items: 117
prompt-b supported: 93.2% (109 of 117)
prompt-b unlisted: 6.8% (8 of 117)
prompt-b wrong-version: 0.0% (0 of 117)
prompt-b missing: 0.0% (0 of 117)
Most citations in every recording point at a section the item lists. The listed share is 93.3% (56 of 60) for the baseline, 84.7% (50 of 59) for the unfiltered variant, and 93.2% (109 of 117) for prompt-b.
存在性與版本
在這些記錄中,版本檢查沒有找到任何問題。依上文的定義,錯誤版本引用是指只引用了另一個版本的某一節。基準版本的錯誤版本比例為 0.0% (0 of 60)。未過濾變體為 0.0% (0 of 59)。prompt-b 為 0.0% (0 of 117)。缺漏這項結果同樣沒有出現在這三份記錄中。這只是針對引用所做的陳述,並不表示助理對每個問題都會依所問的版本作答。
未過濾變體最值得留意,因為它的脈絡包含所有版本的各節。若有任何一份記錄引用了錯誤的版本,就會是這一份。但它一次也沒有。真正要緊的錯誤出在主張之中。第 2 章把基準版本對 q-0023 的回覆編碼為錯誤版本的答案。它給出的是 2.0 中忽略項目的結果,針對 1.0 中的忽略項目,而它引用的是 1.0 的某一節。引用落在所問的版本內,因此這項檢查會放行。必須拿論述去對照 1.0 的參考資料,錯誤才會顯現。
實用原則: 將錯誤版本與缺漏這兩種結果各自單獨報告為比率,且絕不讓其中的零被當成答對。對每個取決於版本的項目,都要拿論述去對照所問版本的參考資料。有依據的答案把這種對照做成了另一個獨立的比率。
支持度
這項檢查的作用主要體現在「未列出」(unlisted)這個結果上,而它需要人工判讀。依本檢查的規則,引用所問版本中某個未列出的節並不算錯誤,因為題目的清單是人工撰寫的,可能並不完整。基準版本對 q-0057 的回答就是一例:它引用了 remote-retries,而該題目並未列出這一節。程式把這筆引用歸為未列出,第 2 章則把該回答編碼為引用問題,因此要判定這筆引用是否有問題,必須由人來判讀。第 2 章也把這個候選條件表述為一條規則:每一筆引用都必須是題目所列出的。本章放寬了這條規則,因為清單出自人工撰寫,最終做判斷的是讀者,而不是清單。
未列出的比例,基準版本為 6.7% (4 of 60),未過濾版本為 15.3% (9 of 59),prompt-b 為 6.8% (8 of 117)。未過濾版本的未列出比例最高。它的每一筆未列出引用,對讀者而言都是一個待判斷的問題:這一節是否支持它被引用來支撐的那個句子?
supported 的結果只代表該 ID 在清單上。清單上的節仍可能無法支持它被引用來支撐的句子,因為這項檢查讀的是 ID,而不是句子。列出的節是否支持其句子,屬於判斷,本檢查並不做這項判斷。有依據的回答沿用此處定義的四種結果作為它的引用準確度,該指標同樣只讀 ID,並另外加上回答層級的忠實度評判,評判的寫法如附評分標準的評判所述。
實用原則: 只有在引用的 ID 位於題目清單上時,才將該引用計為受支持,切勿將其視為正確。在稱某筆未列出的引用為問題之前,先對照其所引的節閱讀。將未列出與錯誤版本、缺漏分開報告,因為較完整的清單會使部分未列出的引用轉為受支持。
精確比對與模糊比對
本書的大多數字串檢查採用兩種比較方式之一。精確比對判斷兩個字串是否相等,適用於內容固定的文字。第 1 章的拒答檢查就是一例:回覆只有在整段文字都等於拒答語句時,才算拒答,而該語句正是解析後的回覆中 declined 欄位所存的內容。節 ID、旗標名稱與檔案名稱也同樣是固定的。精確比對只給出是或否,不會回傳「差一點就相符」的結果。
模糊比對則為兩個字串的相似程度打分數。Python 標準函式庫中的 difflib 即提供此功能。其 SequenceMatcher 類別以 ratio() 給出分數,文件將該分數定義為相符元素數量的兩倍,除以兩個序列中元素的總數。關於這個分數,有三點對檢查而言很重要。
- 分數取決於引數的順序。文件提醒,
ratio()呼叫的結果可能隨引數順序而異,因此檢查時每次都必須把參考字串放在同一個位置。以範例中的長字串組為例,參考字串在前時分數為 0.480,回覆在前時為 0.861。 - 分數計算的是相符的元素,而非語意。旗標中只差一個字母的錯字,與正確旗標比對時得分為 0.933,但該旗標其實是錯的。回覆即使有一項事實出錯,仍可能維持很高的分數。
- 對長字串,分數可能略過其中一部分。對於長度達 200 個項目以上的序列,文件指出,若某個項目的重複次數超過序列的百分之一,便會被視為雜訊(junk),除非關閉
autojunk。對字串而言,項目就是字元,因此 200 個字元以上的回覆可能受到影響。此啟發式規則是依第二個引數建立的,也就是參考字串在前時的回覆。在長字串組上,預設分數為 0.480,使用autojunk=False時則為 0.861。
1"""Compares three pairs of sablefile answers with an exact match and with difflib's ratio(), a score from 0 to 1.23Each pair is a reference and a reply written for this example, not a recorded reply: two file names, a flag with one4wrong letter, and a long answer that leaves out its folder sentence. ratio() is printed with the reference first and with5the reply first, because the documentation says the argument order can change the score. With the reference first, the6long reply is the second sequence and is 200 characters or more, so the autojunk heuristic applies to it; the long answer7is also scored with autojunk turned off."""8import lib # first: fixes the thread counts and seeds before anything else loads910from difflib import SequenceMatcher1112FOLDERS = " Folders are created only with --apply, because a plain sync writes nothing."13PLAN = " The plan lines start with create, update or unchanged, and the lock file is written only by the apply run."14LONG_REFERENCE = (15 "With no flag, sync prints the plan and writes nothing. --apply writes the files and updates the lock file, "16 ".sablefile/lock.json." + FOLDERS + PLAN17)18LONG_REPLY = LONG_REFERENCE.replace(FOLDERS, "") # the answer leaves out the folder sentence19PAIRS = (20 ("file names", ".sablefile/lock.json", "sablefile.lock"),21 ("flag", "--apply", "--appply"),22 ("long answer", LONG_REFERENCE, LONG_REPLY),23)2425for name, reference, reply in PAIRS:26 print(f"{name}, exact match: {reply == reference}")27 print(f"{name}, ratio: {SequenceMatcher(None, reference, reply).ratio():.3f}")28 print(f"{name}, ratio with the reply first: {SequenceMatcher(None, reply, reference).ratio():.3f}")29# autojunk=False: the default treats the reply's common characters as junk once it is 200 characters or longer.30print(f"long answer, ratio with autojunk off: {SequenceMatcher(None, LONG_REFERENCE, LONG_REPLY, autojunk=False).ratio():.3f}")
python fuzzy-match.pyfile names, exact match: False
file names, ratio: 0.765
file names, ratio with the reply first: 0.765
flag, exact match: False
flag, ratio: 0.933
flag, ratio with the reply first: 0.933
long answer, exact match: False
long answer, ratio: 0.480
long answer, ratio with the reply first: 0.861
long answer, ratio with autojunk off: 0.861
這三組字串是為此範例而寫的,因此分數呈現的是各項特性的作用,而不是回覆出現這些情形的頻率。
實用原則: 只要預期的文字是固定的,就使用精確比對。模糊比對的分數只用來排列回覆以便閱讀,不要當作通過或不通過的判準,除非已有紀錄下來的執行結果,顯示針對本書的答案,門檻應設在何處。呼叫 ratio() 時,每次都先傳入參考字串,並在檢查函式的 docstring 中說明哪一個字串是哪一個。
BLEU 與 ROUGE 在哪些地方會誤導
有兩種衍生自其他任務的詞彙重疊分數,可以用來檢查生成的文字。BLEU 源自機器翻譯,ROUGE 源自摘要任務。兩者的計算成本都很低。
Papineni 等人於 2002 年提出 BLEU,作為評分機器翻譯的快速且低成本的方法。它採用修正後的 n-gram 精確率(Papineni and colleagues, 2002)。Reiter 回顧了 BLEU 與人類判斷的吻合程度,發現現有證據支持將它用於機器翻譯系統的診斷性評估;但不支持在機器翻譯之外使用 BLEU,也不支持用它評估個別文本或進行假設檢定(Reiter, 2018)。對系統整體成立的分數,並不因此對單一答案也成立。
Lin 將 ROUGE-N 定義為 n-gram 召回率:候選文字涵蓋了參考答案中多大比例的詞語片段(Lin, 2004)。該論文談的是摘要。另有一項研究,把借自機器翻譯的指標用在對話回應上,而不是文件答案上。結果發現,在一般聊天資料集上,這些指標與人類判斷的相關性非常弱;在技術性資料集(Ubuntu 求助資料)上則毫無相關(Liu and colleagues, 2016)。這只是針對單一任務的一項研究。Sablefile 的答案屬於技術性內容,因此這項結果是一個警示,而不是對這個助理的實際量測。
以下三個關於 sablefile 2.0 的問題,最能清楚呈現這個問題。每一題都有一個撰寫好的參考答案、一個抄了參考答案大部分用詞的錯誤答案,以及一個意思相同但只用了少數相同用詞的正確答案。這些答案是為了這個範例而撰寫的,並非取自實際錄製的內容,因此它們只能證明這種失敗可能發生,無法說明發生的頻率。BLEU 式分數取 1 至 4 元 n-gram 的截斷精確率、其幾何平均數,再乘上簡短懲罰。ROUGE-1 F1 則計算單一詞彙。兩條公式都已寫在範例的程式碼中。本書沒有固定任何 BLEU 或 ROUGE 套件的版本,而且範例回報的是結合了召回率與精確率的 F1,並非 Lin 所定義的召回率。
1"""Scores a wrong answer that shares most of its words with the reference above a right answer that shares few.23Each pair in data/overlap-pairs.json is a question on sablefile 2.0 with its reference, a wrong answer and a right answer.4All three pairs are hand-written for this example, not recorded replies. The scores are the textbook formulas, written5here in plain Python: ROUGE-1 F1 (unigram overlap, F1 of precision and recall) and a BLEU-style score (clipped 1- to64-gram precision, geometric mean, brevity penalty), with no smoothing. No BLEU or ROUGE package is pinned, and none is7used. The reference facts are in examples/data/docs/2.0 (sync.md, quickstart.md, lockfile.md)."""8import lib # first: fixes the thread counts and seeds before anything else loads910import json11import math12import re13from collections import Counter14from pathlib import Path1516PAIRS = json.loads((Path(__file__).resolve().parent / "data" / "overlap-pairs.json").read_text(encoding="utf-8"))171819def words(text: str) -> list[str]:20 """Lowercase runs of letters and digits: `--apply` is the word apply, and `lock.json` is two words."""21 return re.findall(r"[a-z0-9]+", text.lower())222324def rouge_1_f1(candidate: str, reference: str) -> float:25 """ROUGE-1 F1: shared words, counted with repeats up to the smaller count, over both lengths."""26 cand, ref = words(candidate), words(reference)27 overlap = sum((Counter(cand) & Counter(ref)).values())28 if overlap == 0:29 return 0.030 precision, recall = overlap / len(cand), overlap / len(ref)31 return 2 * precision * recall / (precision + recall)323334def bleu_sentence(candidate: str, reference: str, max_n: int = 4) -> float:35 """Sentence BLEU-style score: clipped n-gram precision for n = 1 to 4, geometric mean, brevity penalty.3637 Without smoothing, a single n-gram order with no match gives 0, as the textbook formula does.38 """39 cand, ref = words(candidate), words(reference)40 precisions = []41 for n in range(1, max_n + 1):42 cand_grams = Counter(tuple(cand[i : i + n]) for i in range(len(cand) - n + 1))43 ref_grams = Counter(tuple(ref[i : i + n]) for i in range(len(ref) - n + 1))44 clipped = sum(min(count, ref_grams[gram]) for gram, count in cand_grams.items()) # each n-gram up to the reference's count45 if clipped == 0:46 return 0.047 precisions.append(clipped / sum(cand_grams.values()))48 # Short answers are penalised by the brevity penalty: a candidate shorter than the reference scores less.49 brevity = (1.0 if len(cand) > len(ref) else math.exp(1 - len(ref) / len(cand)))50 return brevity * math.exp(sum(math.log(p) for p in precisions) / max_n)515253for number, pair in enumerate(PAIRS, start=1):54 print(f"pair {number} (hand-written), question: {pair['question']}")55 for label in ("wrong", "right"):56 print(f"pair {number}, {label} answer, ROUGE-1 F1: {rouge_1_f1(pair[label], pair['reference']):.3f}")57 print(f"pair {number}, {label} answer, BLEU: {bleu_sentence(pair[label], pair['reference']):.3f}")58# The claim the prose makes: the wrong answer overlaps the reference more than the right one, in every pair.59higher = sum(rouge_1_f1(p["wrong"], p["reference"]) > rouge_1_f1(p["right"], p["reference"]) for p in PAIRS)60print(f"pairs where the wrong answer has the higher ROUGE-1 F1: {higher} of {len(PAIRS)}")
python overlap-misleads.pypair 1 (hand-written), question: In version 2.0, what does a plain sablefile sync do to your files?
pair 1, wrong answer, ROUGE-1 F1: 0.815
pair 1, wrong answer, BLEU: 0.540
pair 1, right answer, ROUGE-1 F1: 0.167
pair 1, right answer, BLEU: 0.000
pair 2 (hand-written), question: In version 2.0, when does sync create a target's missing folder?
pair 2, wrong answer, ROUGE-1 F1: 0.720
pair 2, wrong answer, BLEU: 0.484
pair 2, right answer, ROUGE-1 F1: 0.211
pair 2, right answer, BLEU: 0.000
pair 3 (hand-written), question: Where does version 2.0 keep its lock file?
pair 3, wrong answer, ROUGE-1 F1: 0.952
pair 3, wrong answer, BLEU: 0.905
pair 3, right answer, ROUGE-1 F1: 0.286
pair 3, right answer, BLEU: 0.000
pairs where the wrong answer has the higher ROUGE-1 F1: 3 of 3
在 3 of 3 組配對中,錯誤答案的 ROUGE-1 F1 高於正確答案。以第一組為例,錯誤答案得 0.815,正確答案得 0.167。
閱讀 BLEU 這幾行時要小心。程式碼寫出的公式沒有平滑處理,因此只要四種 n-gram 階數中有任何一種完全沒有匹配,分數就是零。每個正確答案與其參考答案共有單一詞彙,卻沒有任何兩個詞連續相同的片段,所以它的二元詞精確率為零,BLEU 也為零。這是公式本身的行為,並不能說明答案好不好。
ROUGE-1 F1 計算單一詞彙,並忽略詞序。一個重複了參考答案大部分用詞的錯誤答案,無論其餘用詞錯在哪裡,都能拿回大部分的分數。第三組就是一例,其錯誤答案得 0.952。該答案寫的是 1.1 版的檔案名稱,而參考答案寫的是 .sablefile/lock.json。錯誤答案的每個詞都出現在參考答案中,它唯一缺少的參考詞是 json。檔案名稱正是答案中最關鍵的部分,而這個分數無法把這一部分和其他部分區分開來。
**實用原則:**對於只有一種正確形式的答案,不要把 BLEU 或 ROUGE 當作及格門檻。若要回報這兩種分數,請寫明所用的公式,並說明是否經過平滑處理。重疊度高但主張仍有疑問的答案,每一則都要閱讀,因為上述配對顯示,重疊度會獎勵照抄參考答案用詞的錯誤答案。請依第 5 章所述,以評審來檢驗答案的主張。
程式碼層面的拒答
第 1 章已經有一項針對拒答的程式碼檢查。它依據確切的拒答句,計算兩種錯誤:助理拒答了可回答的題目,以及助理回答了無法回答的題目。本章的格式檢查另外加入一種情形,即拒答句與其他文字出現在同一則回覆中,並且對兩個資料切分的每一題都執行無法回答題目的檢查,而第 1 章只對二十題執行。如格式檢查所示,基準線在兩個資料切分上的格式計數皆為零。在第 1 章的二十題上,拒答檢查兩種錯誤都沒有發現。本章不在開發題上重新執行這項檢查。第 10 章會給出基準線在這些題目上的錯誤拒答率。
這兩項檢查讀取的都是回覆的文字,因此兩者都有一項隨之而來的限制。以其他措辭表達的拒答並不是那個句子,所以會被當作回答放行,第 2 章中的 q-0080 即是一例。拒答率同樣不包含與版本相關的題目,而 q-0004 這一題顯示了這樣做的代價,第 1 章對此有所說明。基準線對這一題選擇拒答,而 2.0 版文件在其佔位符一節中提供了答案。拒答率不會把這種情形算作錯誤拒答。有依據的回答會建立能夠處理這種情形的版本正確率。
在文件未涵蓋的題目上,拒答是否正確屬於判斷,而不是程式碼檢查。文件所述是否足以作答,則是文件未說明時的拒答所要處理的問題。
實用原則: 在每個資料切分上計算兩種拒答錯誤,並對照各版本對應的文件,以人工逐一檢視與版本相關的拒答,因為拒答率並未納入這些題目。
執行檢查,以及檢查能確定什麼
在正式環境中,每則回覆都會經過檢查,因此提示詞或模型的任何變動,都會在同一份報告中讓格式計數、拒答率與未列名比例一併出現變化。若這些數字全都持平,使用者卻回報答案有誤,就表示這些檢查看不見該種失敗。請依 第 2 章 所述,依主要缺陷為新的回覆編碼,並針對相符的失敗模式撰寫檢查。
本章所確立的內容範圍有限。程式碼檢查能讓回覆的形式、引用的成員資格與拒答語句保持正確,並且能在每則回覆上以零成本做到這一點。回覆的意義仍需要由讀者或評審來判斷。因此,請依主要缺陷為每則回覆編碼,讓每項檢查只負責它看得見的缺陷,並把它看不見的部分交給讀者或評審,這樣乾淨的計數就不會被誤當成乾淨的答案。
本章做出了以下決定:
- 格式檢查是一道帶有五個名稱的防護關卡,在任何評審之前執行。
- 格式檢查只計算引用的數量,並不讀取引用的順序,因此第二個模型對 q-0030 的回覆能通過這項檢查。順序檢查則會標記該回覆。
- 引用的 ID 若在該題目的清單上,這則引用即為
supported;引用周圍的句子正確與否,並不影響此判定。 - 引用的版本檢查在這三份記錄中找不到版本錯誤的引用,本章並不把這一點當作對答案版本的檢查。
- 重疊分數來自該範例自身的公式,任何分數都不作為及格標準。
- 模糊比對的特性以 sablefile 的答案來呈現,任何比率都不作為及格標準。
- 保留題目只在格式檢查中以計數的形式出現,由 第 10 章 解讀。
本章,於本版次中
版次 2026.11.3範例於 2026年11 月 21 通過測試本章 5 個範例 個範例中有 5 個通過
譯自英文書,版次 2026.11.3。