檢查答案是否依據所引用的文件,並符合所詢問的版本
閱讀約需 44 分鐘 分鐘,8 節 中共有 6 個範例。
讀完本章,你能計算:答案的引用是否支持其所述內容,答案中的每項主張是否都有依據,且依據來自助理取得的各節;答案是否符合讀者所詢問的版本;以及檢索過程是否找到了正確的節。
正確的答案仍可能是不好的答案。本書中的助理依據 sablefile 的文件回答關於 sablefile 的問題,只要回覆與問題所詢問版本的參考答案相符,就算通過。回覆可能通過這項測試,卻引用了一個並未支持該句內容的節,加入了任何節都沒有提供的細節,或是從文件的另一個版本取得事實。通過率看不出這些問題,而讀者若對照主張旁的引用,就能看出,這也正是要求助理附上引用的原因。
除了通過率之外,另有四項檢查,各有其自己的計數與分母。引用準確度檢視每個引用是否指向該題項列為支持來源的節。忠實度檢視每項主張是否出自助理拿到的各節。版本正確性檢視答案是否符合所詢問的版本。結果與路徑對照檢視通過的答案是否確實取得了對應的節,以及未通過的答案是否也取得了該節(即檢索是否到達正確的節)。引用準確度與忠實度不計入拒答,因為拒答並未提出可供檢查的主張。版本正確性與檢索表則和通過率一樣保留拒答,並各自印出其中包含的拒答數量。
此處「有依據」的意思
當讀者依循答案中的引用,能在引用所指的位置、於他們所問的版本中找到該項主張,這個答案就是有依據的。要做到這一點,必須同時成立三件事。引用必須存在於該版本中,並且支持它所附著的那個句子。答案中的每一項主張都必須能從助理所讀的各節推得。而且答案必須適用於該版本,而不只是適用於另一個使用相同字句的版本。
設想有位讀者詢問遠端擷取在放棄之前會等待多久。假設回覆給出一個看似合理的答案,並引用了關於擷取遠端範本的那一節,但該節從未說明等待時間有多長。讀者會把這則回覆視為有依據,因為引用看起來正確。可是打開該節的讀者找不到任何內容能支持那個數字。若這個數字恰好與參考答案相符,通過率便會把這則回覆算作正確。忠實度是透過評審來閱讀各節內容的檢查;引用檢查只讀取 ID,並且明白標示這一點。
這幾種失誤各不相同,每一種都需要各自的計數。回覆可以有支持性的引用,卻仍陳述該節未提及的細節,因此引用準確度無法顯示忠實度。回覆可以忠於某一版本的某一節,卻不適用於另一個版本,因此忠實度無法顯示版本正確性。回覆也可以正確且引用得當,但檢索步驟卻從未取得它所倚賴的那一節。比對結果與路徑的目的,正是要找出這種情況。
實用原則: 讓每項檢查在各自的分母上有各自的計數,並把通過視為閱讀的開始,而不是結束。
引用正確率
引用是對文件的一項主張:這個版本中的這一節,支持它所附屬的那個句子。和任何主張一樣,引用也可能出錯。該節可能確實存在,卻說著別的內容;可能屬於另一個版本;也可能回覆引用了該題項未列出的節。入門指南涵蓋句子層級的檢查,見檢查被引用的題項支持什麼,而對照脈絡檢查引用則涵蓋模型當時取得的脈絡。本書的檢查較為簡單,第 4 章的引用檢查給出了它的四種結果。它將每個被引用的 ID,與該題項針對所問版本列出的支持節清單比對。它從不閱讀該節的內容,因此對句子本身不作任何判斷。
程式碼從同一批回覆印出兩項計數。引用率計算已作答題項中,每一則受支持的引用。題項率則是:已作答題項只要至少有一則引用受支持,就計為一次,因此一則良好的引用即可支撐整個題項。若引用的是所問版本中的某一節,而該題項並未列出,則印為 unlisted(未列出)。該節很可能是正確的,所以檢查不稱它為錯誤,也不會打開該節去查證。閱讀未列出的引用屬於人工判斷:若該節確實支持該句子,表示該題項的支持節清單不完整,測試集的負責人應將它補上;若不支持,則該引用是錯的,這則回覆應視為引用失敗。
請在兩個切分上都執行,並且先看題項那一行,再看引用那一行。
1"""Counts the baseline's answered items that cite a supporting section of the asked version, dev and held-out apart.23An item counts once, and only if one of its citations is supported; its other citations sit in the citation lines. A decline4is not an answer, so it is left out of every count. The check matches each cited ID against the item's listed supporting5section IDs for its asked version. It never reads the section's text, so it does not check what the sentence says.6The first line of each split is the point: the item rate."""7import lib # first: fixes the thread counts and seeds before anything else loads89from lib.contract import citation_outcomes, parse_reply, section_ids_by_version10from lib.prompts import P1_SHA25611from lib.replay import ReplayModel12from lib.stats import pct13from lib.testset import load_items1415ids = section_ids_by_version() # every version's section IDs, read from the docs, for telling wrong-version from missing16model = ReplayModel("assistant-baseline", P1_SHA256) # the baseline's replies, sample 0, made from prompt P1171819def supporting_ids(item):20 """The item's supporting section IDs in its asked version. Another version's listed sections are not counted here."""21 return frozenset(s.section_id for s in item.supporting_sections if s.version == item.version)222324for split in ("dev", "held-out"):25 declined = 026 results = {} # item id -> its citation outcomes, for each answered item27 for item in load_items(split=split):28 reply = parse_reply(model.reply(item.id, 0))29 if reply.declined:30 declined += 131 else:32 results[item.id] = citation_outcomes(reply, item.version, supporting_ids(item), ids)33 # An answer to an unanswerable item lists no section of its asked version, so it can never count as supported here.34 outcomes = [outcome for found in results.values() for outcome in found] # every citation, in item order35 with_support = [item_id for item_id, found in results.items() if "supported" in found]36 no_support = sorted(set(results) - set(with_support))37 print(f"{split} declined items, left out: {declined}")38 print(f"{split} answered items: {len(results)}")39 print(f"{split} answered items citing a supporting section of the asked version: {pct(len(with_support), len(results))}")40 print(f"{split} answered items with no supported citation: {len(no_support)}")41 print(" " + (", ".join(no_support) or "none"))42 print(f"{split} citations: {len(outcomes)}")43 print(f"{split} citations that are supported: {pct(outcomes.count('supported'), len(outcomes))}")44 print(f"{split} citations that are unlisted: {pct(outcomes.count('unlisted'), len(outcomes))}")45 print(f"{split} citations that are wrong-version or missing: "46 f"{pct(outcomes.count('wrong-version') + outcomes.count('missing'), len(outcomes))}")
python citation-accuracy-counts.pydev declined items, left out: 23
dev answered items: 57
dev answered items citing a supporting section of the asked version: 94.7% (54 of 57)
dev answered items with no supported citation: 3
q-0005, q-0057, q-0058
dev citations: 60
dev citations that are supported: 93.3% (56 of 60)
dev citations that are unlisted: 6.7% (4 of 60)
dev citations that are wrong-version or missing: 0.0% (0 of 60)
held-out declined items, left out: 15
held-out answered items: 25
held-out answered items citing a supporting section of the asked version: 96.0% (24 of 25)
held-out answered items with no supported citation: 1
q-0090
held-out citations: 33
held-out citations that are supported: 93.9% (31 of 33)
held-out citations that are unlisted: 6.1% (2 of 33)
held-out citations that are wrong-version or missing: 0.0% (0 of 33)
在開發集上,題項率為 94.7% (54 of 57)。
在開發集上,有 3 個已作答題項沒有受支持的引用,輸出會列出這些題項的名稱。在保留集上,這個數量是 1。
開發集的引用率為 93.3% (56 of 60)。開發集上未列出的引用有 6.7% (4 of 60) 則,輸出會回報它們,但不計為錯誤。
在保留集上,題項率為 96.0% (24 of 25),引用率為 93.9% (31 of 33)。保留集只有 25 個已作答題項,因此一個題項就足以讓其比率大幅變動。兩個切分之間若出現差異,必須先有信賴區間,才能認定該差異是真實的,區間與樣本大小說明如何建立區間。
版本錯誤或缺漏的引用,是那些因檢查能夠指名的原因而出錯的引用。在開發集上有 0.0% (0 of 60) 則,在保留集上有 0.0% (0 of 33) 則。因此,引用率之所以低於 100%,是由未列出的引用造成的,而這些必須由人來閱讀。
兩項比率可能因一個簡單的原因而分歧:一個題項若有一則受支持的引用與一則未列出的引用,會計入題項率,卻會拉低引用率。同時閱讀兩者,才能看出缺失是出在整個題項,還是出在其餘部分站得住腳的題項內的單一引用。
這項檢查背後的想法,在本書之外也已有人研究。Gao 等人(2023)發表於 EMNLP 2023,亦見於 arXiv,他們建立了一套針對生成回答中引用的基準,並報告許多強大模型的回答並未完全受其所引段落支持。他們的設定與本書不同,因此不沿用其中任何數字;引用該論文,只是為了其想法。
**實用原則:**同時閱讀題項率與引用率,並且在修改提示詞之前,先閱讀每一則未列出的引用。未列出的引用是對文件提出的問題,而不是應計入的錯誤。
忠實度
引用準確率不需要模型就能執行,但它的計數只說明有多少個引用指向清單上的某一節,並未說明每個句子是否成立。忠實度是接下來的問題,它能抓出完全沒有引用的主張。
引用準確率只檢查帶有引用的句子。回覆中未附引用的主張不在檢查範圍內,被引用章節只支持一部分的主張同樣會漏網。忠實度問的是更廣的問題:答案中的每一項主張,是否都能從助理看到的各節推得出來?
Ragas(Es 等人,2023,同樣是預印本)把忠實度計為答案中受脈絡支持的陳述所占的比例。它先把答案拆成陳述,再逐一檢查。本書則改為每個答案只問一個問題。lib/prompts.py 中的忠實度提示詞把各節與答案交給評審。評審逐項主張閱讀答案,只有在每項主張都成立時,才會在第一行寫下 VERDICT: SUPPORTED。否則它會寫下 VERDICT: UNSUPPORTED,再附上一句理由。
它與引用準確率的差異出現在答案的中段。假設一則回覆有兩個句子:第一句附有引用,指向支持它的某一節;第二句沒有引用,因為它加入了該節未提及的細節。引用準確率會評分第一句,對第二句則沒有任何說法。忠實度會讀完整個答案,因此光憑第二句就能把這則回覆判為無依據。兩種檢查都不能取代對方,所以兩者都保留。
這位評審的判定從未與人工判定核對過,因此這裡所有的忠實度比率都是未校準的。評審同時也是助理自己的模型 Qwen3-4B-Instruct-2507,所以它評分的是該模型自己寫的答案。自我偏好說明了這為什麼重要,以及本書為何無法測試這一點。校準評審需要對這些回覆的人工評分,本書並沒有。校準評審描述了方法,也就是把評審與人工評分相比較。由於這些比率未經校準,遷移假設便沒有可以帶過去的東西,何時停手中的限制也完全適用於它們。
1"""The faithfulness rate of assistant-baseline's answered items, as judged in judge-faithfulness-baseline, dev and held-out apart.23An answered item is one the baseline did not decline and whose reply is not empty; a decline is left out. The judge writes4SUPPORTED or UNSUPPORTED on its first line. A reply that is neither is printed by item ID and left out of the denominator,5never counted as UNSUPPORTED. The judge is uncalibrated: this book has no human ratings for it, so the rates are not6corrected for its errors and are not a calibrated measure. When every readable verdict is the same, the percentile7bootstrap cannot vary, so the example says so and prints the exact binomial bound for n of n instead."""8import lib # first: fixes the thread counts and seeds before anything else loads910from lib.contract import parse_reply11from lib.judging import parse_support_verdict12from lib.prompts import FAITH_SHA256, P1_SHA25613from lib.replay import ReplayModel14from lib.stats import bootstrap_mean, interval_pct, pct15from lib.testset import load_items1617answers = ReplayModel("assistant-baseline", P1_SHA256) # the baseline's replies, sample 0, made from prompt P118judge = ReplayModel("judge-faithfulness-baseline", FAITH_SHA256) # the judge's replies, made from the faithfulness prompt1920print("judge: judge-4b, uncalibrated (no human ratings in this book)")21for split in ("dev", "held-out"):22 answered = []23 for item in load_items(split=split):24 reply = parse_reply(answers.reply(item.id, 0))25 if not reply.declined and reply.text:26 answered.append(item.id)27 readable = {} # item id -> 1.0 for SUPPORTED, 0.0 for UNSUPPORTED28 for item_id in answered:29 raw = judge.reply(item_id, 0)30 verdict = parse_support_verdict(raw)31 if verdict is None:32 line = next((ln.strip().strip("*`").strip() for ln in raw.splitlines() if ln.strip()), "")33 print(f'{split} unreadable verdict, left out: {item_id} ("{line}")')34 else:35 readable[item_id] = 1.0 if verdict == "SUPPORTED" else 0.036 scores = list(readable.values())37 lo, hi = bootstrap_mean(scores) # level 0.95; seed SEED, 2,000 resamples38 print(f"{split} answered items: {len(answered)}")39 print(f"{split} unreadable verdicts: {len(answered) - len(readable)}")40 print(f"{split} faithfulness rate: {pct(int(sum(scores)), len(scores))}")41 print(f"{split} 95% interval of the faithfulness rate: {interval_pct(lo, hi)}")42 if len(set(scores)) == 1: # every readable verdict is the same, so resampling cannot vary43 print(f"{split} bootstrap interval is degenerate: all {len(scores)} readable verdicts are the same")44 if sum(scores) == len(scores):45 print(f"{split} exact 95% lower bound for {len(scores)} of {len(scores)}: {100 * 0.025 ** (1 / len(scores)):.1f}%")46 if split == "dev":47 unfaithful = sorted(item_id for item_id, score in readable.items() if score == 0.0)48 print(f"dev unfaithful items, first five: {', '.join(unfaithful[:5])}")
python faithfulness-rate.pyjudge: judge-4b, uncalibrated (no human ratings in this book)
dev answered items: 57
dev unreadable verdicts: 0
dev faithfulness rate: 94.7% (54 of 57)
dev 95% interval of the faithfulness rate: [87.7%, 100.0%]
dev unfaithful items, first five: q-0013, q-0029, q-0055
held-out answered items: 25
held-out unreadable verdicts: 0
held-out faithfulness rate: 100.0% (25 of 25)
held-out 95% interval of the faithfulness rate: [100.0%, 100.0%]
held-out bootstrap interval is degenerate: all 25 readable verdicts are the same
held-out exact 95% lower bound for 25 of 25: 86.3%
此範例取基準線的已回答項目,排除拒答,再逐項詢問評審。它只對能讀懂的判定印出比率。既非 SUPPORTED 也非 UNSUPPORTED 的回覆,會逐項印出並排除在分母之外,絕不會被算作失敗,如判定優先所說明。
dev 上未校準的忠實度比率為 94.7% (54 of 57),其區間為 [87.7%, 100.0%]。該區間的上端落在比率尺度的上界,因為自助法無法重抽樣到上界以外。區間仍然很寬,這正是對 57 個已回答項目的誠實解讀。
held-out 上未校準的忠實度比率為 100.0% (25 of 25)。它印出的區間是 [100.0%, 100.0%],但那不算區間。輸出指出自助法已退化:all 25 readable verdicts are the same。對全部通過的項目重抽樣,只能得出相同的比率,因此自助法無法說明比率落在何處。
精確的 95% 下界為 86.3%。若真實比率低於此值,held-out 回覆全部通過的機率會低於 2.5%。這個下界等於 0.025 的 1/n 次方。它顯示的是評審放行了這些回覆,而這個分割太小,無法顯示比率落在何處。請不要把 held-out 的數字解讀為助理忠實的發現。
兩種檢查各自印出自己的清單。引用的輸出列出沒有任何受支持引用的項目,忠實度的輸出則列出最先出現的不忠實 dev 項目。請並排閱讀這兩份清單,因為本章並不假設它們列出的是相同的項目。
**實用原則:**在每個忠實度比率旁標示「未校準」,並且在相信該比率之前,先讀過最先出現的不忠實項目。忠實度檢查顯示的是評審認為哪裡的主張缺乏依據,並不表示評審是對的。
此部分尚未翻譯,因此以英文呈現。
Version-correctness
sablefile has three versions of its docs, 1.0, 1.1 and 2.0, and some questions have different answers in each. A reply can be grounded in a section of 1.1 and still be wrong for a reader on 2.0. The book calls these version-dependent items. Each has a reference answer for the version the reader asked about, and its check asks whether the reply matches that reference. There are 17 of them on dev and 8 on held-out.
Version-dependent items count in the pass rate with every other item, and they also get their own count, beside it. A reply to one of them is judged against the reference for the asked version, so a correct answer for another version fails. That miss would look like a weak assistant if it were only mixed into the general rate. The judged-correct rate is the count for version-dependent items alone, so a version miss shows on its own line, and a team can see whether a change to retrieval moved it. Declines stay in that denominator, as they do in the pass rate, and each split prints how many it holds.
A wrong-version citation is a separate count. It is a citation to a section that exists only in another version. A reply can be right for the asked version and still cite another version's section, and a reader who copies that citation lands in the wrong docs.
This check sees only what the ID tells it. A citation carries an ID, not a version. Most IDs repeat across versions, so a section ID that appears in 1.0, 1.1 and 2.0 looks the same in each. An ID that appears in every version can never be flagged as wrong-version, whichever version the reader asked about. The check can flag a citation only when the asked version lacks its ID and another version has it. The index matters too. A citation to another version is possible only when that version's sections are in the index the reply was built from, so the count has to be read again when the index changes, and a zero from an old index says nothing about a new one.
Two variants are read here. The baseline retrieves from the index of the asked version only. The unfiltered variant retrieves from the indexes of every version together, and shows each section's version in its header, so the model can see where a section comes from. The unfiltered variant is the one that could cite another version, since it is shown them.
1"""Reads the 25 version-dependent items for two variants: whether the judge passed the answer for the asked version, and whether a2citation names a section that exists only in another version.34The judge's verdict for each variant is read from its own recording, and an unreadable verdict is printed and left out of the5pass rate, never counted as a fail. A declined reply has a verdict like any other reply, so it stays in the denominator; its6count is printed beside each rate. A wrong-version citation is counted per item, over every item, so a zero shows as 0 of 25.7The last lines list the items where the two variants disagree, to read item by item."""8import lib # first: fixes the thread counts and seeds before anything else loads910from lib.calibration import readable_verdicts11from lib.contract import citation_outcomes, parse_reply, section_ids_by_version12from lib.prompts import P1_SHA256, REF_SHA25613from lib.replay import ReplayModel14from lib.stats import pct15from lib.testset import load_items1617ids = section_ids_by_version() # every version's section IDs, so a section of another version is known as one18# label, the assistant's recording, the judge's recording of that assistant's replies19VARIANTS = (("baseline", "assistant-baseline", "judge-answers-baseline"), ("unfiltered", "assistant-unfiltered", "judge-answers-unfiltered"))20items = [item for item in load_items() if item.kind == "version-dependent"]21item_ids = [item.id for item in items]22listed = {item.id: frozenset(s.section_id for s in item.supporting_sections if s.version == item.version) for item in items}23verdict, wrong = {}, {} # label -> {item id: PASS or FAIL, readable only}; label -> item ids with a wrong-version citation24for label, assistant, judge in VARIANTS:25 verdict[label], unreadable = readable_verdicts(judge, item_ids, REF_SHA256)26 for item_id, first in unreadable:27 print(f'{label} unreadable verdict, left out: {item_id} ("{first}")')28 replies = ReplayModel(assistant, P1_SHA256)29 wrong[label] = {item.id for item in items30 if "wrong-version" in citation_outcomes(parse_reply(replies.reply(item.id, 0)), item.version, listed[item.id], ids)}31 for split in ("dev", "held-out"):32 split_items = [item for item in items if item.split == split]33 readable = [item.id for item in split_items if item.id in verdict[label]]34 passed = sum(verdict[label][item_id] == "PASS" for item_id in readable)35 print(f"{label} {split} version-dependent items: {len(split_items)}")36 print(f"{label} {split} declined replies, counted in the denominator: "37 f"{sum(parse_reply(replies.reply(item.id, 0)).declined for item in split_items)}")38 print(f"{label} {split} judged correct for the asked version: {pct(passed, len(readable))}")39 print(f"{label} {split} citing a wrong-version section: "40 f"{pct(sum(item.id in wrong[label] for item in split_items), len(split_items))}")41 print(f"{label} all version-dependent items citing a wrong-version section: {pct(len(wrong[label]), len(items))}")4243both = [item_id for item_id in item_ids if item_id in verdict["baseline"] and item_id in verdict["unfiltered"]]44print(f"items with a readable verdict in both variants: {len(both)}")45differ = [i for i in both if verdict["baseline"][i] != verdict["unfiltered"][i]]46print(f"items where the verdicts differ: {len(differ)}")47print(" " + (", ".join(differ) or "none"))48flag_differ = [i for i in item_ids if (i in wrong["baseline"]) != (i in wrong["unfiltered"])]49print(f"items where the wrong-version flag differs: {len(flag_differ)}")50print(" " + (", ".join(flag_differ) or "none"))
python version-correct-answers.pybaseline dev version-dependent items: 17
baseline dev declined replies, counted in the denominator: 4
baseline dev judged correct for the asked version: 64.7% (11 of 17)
baseline dev citing a wrong-version section: 0.0% (0 of 17)
baseline held-out version-dependent items: 8
baseline held-out declined replies, counted in the denominator: 2
baseline held-out judged correct for the asked version: 75.0% (6 of 8)
baseline held-out citing a wrong-version section: 0.0% (0 of 8)
baseline all version-dependent items citing a wrong-version section: 0.0% (0 of 25)
unfiltered dev version-dependent items: 17
unfiltered dev declined replies, counted in the denominator: 5
unfiltered dev judged correct for the asked version: 52.9% (9 of 17)
unfiltered dev citing a wrong-version section: 0.0% (0 of 17)
unfiltered held-out version-dependent items: 8
unfiltered held-out declined replies, counted in the denominator: 2
unfiltered held-out judged correct for the asked version: 75.0% (6 of 8)
unfiltered held-out citing a wrong-version section: 0.0% (0 of 8)
unfiltered all version-dependent items citing a wrong-version section: 0.0% (0 of 25)
items with a readable verdict in both variants: 25
items where the verdicts differ: 6
q-0003, q-0021, q-0022, q-0033, q-0082, q-0085
items where the wrong-version flag differs: 0
none
Across every version-dependent item, the wrong-version share is 0.0% (0 of 25) for the baseline and 0.0% (0 of 25) for the unfiltered variant. A zero says only that the check flagged no ID. The next example shows how few IDs it could have flagged.
1"""The two counts behind the version check, and the version-dependent items where the variants' verdicts differ, by split.23A wrong-version citation is one whose ID exists only in another version, so an ID that several versions share cannot be4attributed to one of them. This prints how many section IDs are found in one version only, then how many citations to the5version-dependent items name a shared ID, for each variant. The last lines list the items where the verdicts differ, by split."""6import lib # first: fixes the thread counts and seeds before anything else loads78from collections import Counter910from lib.calibration import readable_verdicts11from lib.contract import parse_reply, section_ids_by_version12from lib.prompts import P1_SHA256, REF_SHA25613from lib.replay import ReplayModel14from lib.stats import pct15from lib.testset import load_items1617ids = section_ids_by_version()18shared = Counter(i for found in ids.values() for i in found) # how many versions each section ID appears in19print("section IDs found in one version only, by version: "20 + ", ".join(f"{v} {sum(shared[i] == 1 for i in found)}" for v, found in ids.items()))21items = [item for item in load_items() if item.kind == "version-dependent"]22split_of = {item.id: item.split for item in items}23verdict = {} # label -> {item id: PASS or FAIL}, readable only24for label, assistant, judge in (("baseline", "assistant-baseline", "judge-answers-baseline"),25 ("unfiltered", "assistant-unfiltered", "judge-answers-unfiltered")):26 replies = ReplayModel(assistant, P1_SHA256)27 cited = [cid for item in items for cid in parse_reply(replies.reply(item.id, 0)).citations]28 print(f"{label} citations to a section ID found in more than one version: "29 f"{pct(sum(shared[cid] > 1 for cid in cited), len(cited))}")30 verdict[label], _ = readable_verdicts(judge, list(split_of), REF_SHA256)31differ = [i for i in verdict["baseline"] if i in verdict["unfiltered"] and verdict["baseline"][i] != verdict["unfiltered"][i]]32for split in ("dev", "held-out"):33 part = [i for i in differ if split_of[i] == split]34 print(f"{split} items where the verdicts differ: {len(part)}")35 print(" " + (", ".join(part) or "none"))
python version-check-detail.pysection IDs found in one version only, by version: 1.0 0, 1.1 0, 2.0 4
baseline citations to a section ID found in more than one version: 87.0% (20 of 23)
unfiltered citations to a section ID found in more than one version: 63.2% (12 of 19)
dev items where the verdicts differ: 4
q-0003, q-0021, q-0022, q-0033
held-out items where the verdicts differ: 2
q-0082, q-0085
The section IDs found in only one version, counted by version, are 1.0 0, 1.1 0, 2.0 4. An ID found in more than one version can still be flagged, but only for a reader on a version that lacks it. For the baseline, the share of citations to version-dependent items that name a shared ID is 87.0% (20 of 23). For the unfiltered variant it is 63.2% (12 of 19).
The judged-correct rates differ, though each is read on a small split. On dev, the baseline is judged correct on 64.7% (11 of 17).
The unfiltered variant is judged correct on 52.9% (9 of 17). On these items, the unfiltered variant passes fewer.
On held-out, the baseline is judged correct on 75.0% (6 of 8). The unfiltered variant has the same rate, but it passes different items. The verdicts go in opposite directions on q-0082 and q-0085, and the other held-out items agree.
Across both variants, the verdicts differ on 6 items, out of 25 with a readable verdict in each. On dev, 4 of them differ, and on held-out, 2. The output lists them by split.
No interval is computed for these differences, and a dev split of this size is small. So this chapter reports the counts and does not call either variant better or worse. The counts say three things: on these items the unfiltered variant passed fewer, no citation was flagged as wrong-version on either variant, and the differing verdicts are the place to read first.
Working rule: Read each citation's ID with the version of the answer beside it. A zero from this check means no flagged ID, not no wrong-version citation. Keep version-dependent items in their own count beside the pass rate, so a version miss shows up on its own line.
結果與路徑
結果是指一則回覆是否通過。路徑是指回覆之前的各個步驟所觸及的內容。在本書中,路徑很短:一次檢索,接著一則回覆。有兩種組合需要同時計入兩項數據。一則通過的回覆,若其佐證的那一節從未送達模型,答案便來自別處,也許是模型自身的知識,即使判定結果正確,這仍然是一個警訊。一則失敗的回覆,若相關的那一節已被檢索到,這正是其他各項檢查所要解釋的情形。
含有多個步驟的執行,路徑也較長。Agent Harnesses透過路徑來計算一項任務,想了解這部分,請閱讀該書。本章評分的是助理僅有的那一個步驟。
檢索步驟
檢索是在所問版本的各節上執行 BM25。它是確定性的:同一個問題每次都得到相同的前幾節。因此這個步驟不需要錄製。程式從索引為每個項目計算出前幾節,並將結果與評審對基線回覆所給的判定相對照。
所問版本中沒有支持節的開發項目,不列入表中。這些是無法回答的項目,因為它們所問的版本沒有任何一節可供檢索找到。其數量為 13。
1"""Sets the baseline's judge verdict against whether a supporting section reached the top k, for each dev item that has one.23Retrieval is BM25 over the asked version, so it is the same on every run and needs no recording; k is the baseline's, read from4its manifest. The two diagonal cells are the expected pairs. The two off-diagonal cells are the ones to read: a pass without the5right section retrieved, and a fail with it retrieved. Unanswerable items are left out, since they have no section of their asked6version to retrieve; their count is printed. An unreadable verdict is printed and left out, never counted as a fail. A declined reply is counted in its cell like any7other reply; the count of declines is printed. The FAIL cell is split into declines and answers in fail-with-section."""8import lib # first: fixes the thread counts and seeds before anything else loads910from lib.answers import Assistant11from lib.calibration import readable_verdicts12from lib.contract import parse_reply13from lib.prompts import P1_SHA256, P1_SYSTEM, REF_SHA25614from lib.replay import ReplayModel15from lib.testset import load_items1617RECORDING = "assistant-baseline"18k = ReplayModel(RECORDING, P1_SHA256).manifest["k"] # the top k the baseline's prompt was built from, read from its manifest19assistant = Assistant(RECORDING, P1_SYSTEM, P1_SHA256, k=k) # its retriever only; the recording is not read here20replies = ReplayModel(RECORDING, P1_SHA256) # the baseline's replies, sample 0, made from prompt P1, for the decline check2122dev = load_items(split="dev")23has_support = [item for item in dev if any(s.version == item.version for s in item.supporting_sections)]24print(f"top k: {k}")25print(f"dev items: {len(dev)}")26print(f"dev items with no supporting section of the asked version, left out: {len(dev) - len(has_support)}")2728verdicts, unreadable = readable_verdicts("judge-answers-baseline", [item.id for item in has_support], REF_SHA256)29for item_id, first in unreadable:30 print(f'unreadable verdict, left out: {item_id} ("{first}")')3132# Keyed by (retrieved, passed): the four cells of the 2 by 2 table, each a list of item ids in sorted order.33cells = {(True, True): [], (True, False): [], (False, True): [], (False, False): []}34declined = set() # item ids whose reply is a decline; they sit in their cells like any other reply35for item in has_support:36 if item.id not in verdicts:37 continue38 listed = {s.section_id for s in item.supporting_sections if s.version == item.version}39 top = {section.section_id for section in assistant.sections(item)} # the top k sections, by BM25 rank40 cells[(bool(listed & top), verdicts[item.id] == "PASS")].append(item.id)41 if parse_reply(replies.reply(item.id, 0)).declined:42 declined.add(item.id)4344print(f"items in the table: {sum(len(ids) for ids in cells.values())}")45print(f"declined items in the table, counted in their cells: {len(declined)}")46print(f"right section in the top {k}, PASS: {len(cells[(True, True)])}")47print(f"right section in the top {k}, FAIL: {len(cells[(True, False)])}")48print(f"right section not in the top {k}, PASS: {len(cells[(False, True)])}")49print(f"right section not in the top {k}, FAIL: {len(cells[(False, False)])}")50print("passed without the right section retrieved: " + (", ".join(cells[(False, True)]) or "none"))51print("failed with the right section retrieved: " + (", ".join(cells[(True, False)]) or "none"))
python retrieval-path.pytop k: 3
dev items: 80
dev items with no supporting section of the asked version, left out: 13
unreadable verdict, left out: q-0043 ("VERDICT: PARTIAL")
items in the table: 66
declined items in the table, counted in their cells: 10
right section in the top 3, PASS: 49
right section in the top 3, FAIL: 10
right section not in the top 3, PASS: 0
right section not in the top 3, FAIL: 7
passed without the right section retrieved: none
failed with the right section retrieved: q-0019, q-0023, q-0025, q-0029, q-0033, q-0045, q-0046, q-0048, q-0063, q-0074
前 k 的 k 值為 3,取自基線的 manifest。表中共有 66 個項目,其中 10 個是拒答。拒答與其他回覆一樣落在各自的儲存格中,因為評審已對它給出判定。每個儲存格將一種判定與支持節是否進入前 k 配成一對。
| 判定 | 支持節在前 k 內 | 前 k 內沒有支持節 |
|---|---|---|
| PASS | 49 | 0 |
| FAIL | 10 | 7 |
先讀 PASS 那一列。支持節在前 k 內的通過儲存格有 49 個通過,沒有支持節的通過儲存格有 0 個。這與通過來自文件的說法一致,但並不能證明它們確實來自文件,因為這張表計算的是兩件事同時出現的頻率,而不是哪一件導致了另一件。通過的計數也可用來檢查參考答案。沒有檢索到任何節卻通過,可能來自重述常見事實的回覆,也可能來自讀者猜得到的參考答案,表本身無法分辨。閱讀這些項目則可以。
FAIL 那一列則是其他檢查登場之處。在支持節位於前 k 內的失敗中,有 5 個是拒答,其餘是作答。文件沒有說明時的拒答說明了當文件確實含有答案時,拒答代表什麼。下面的範例將這個儲存格拆開來看。
1"""The dev items the baseline failed although a listed section of the asked version was in its top k, split into declines and answers.23For each answer it prints whether the reply cites a listed section, and the faithfulness judge's verdict on it. A decline has no4faithfulness verdict, because the judge reads only answers. An unreadable verdict is printed and left out, never counted as a fail."""5import lib # first: fixes the thread counts and seeds before anything else loads67import re89from lib.answers import Assistant10from lib.calibration import readable_verdicts11from lib.contract import citation_outcomes, parse_reply, section_ids_by_version12from lib.prompts import FAITH_SHA256, P1_SHA256, P1_SYSTEM, REF_SHA25613from lib.replay import ReplayModel14from lib.testset import load_items1516RECORDING = "assistant-baseline"17k = ReplayModel(RECORDING, P1_SHA256).manifest["k"] # the baseline's top k, as in retrieval-path18assistant = Assistant(RECORDING, P1_SYSTEM, P1_SHA256, k=k) # its retriever only19replies = ReplayModel(RECORDING, P1_SHA256) # the baseline's replies, sample 0, made from prompt P120faith = ReplayModel("judge-faithfulness-baseline", FAITH_SHA256)21ids = section_ids_by_version()22VERDICT = re.compile(r"^VERDICT:\s*(SUPPORTED|UNSUPPORTED)$")232425def listed(item):26 """The item's supporting section IDs in its asked version."""27 return frozenset(s.section_id for s in item.supporting_sections if s.version == item.version)282930dev = load_items(split="dev")31verdicts, unreadable = readable_verdicts("judge-answers-baseline", [item.id for item in dev], REF_SHA256)32for item_id, first in unreadable:33 print(f'unreadable verdict, left out: {item_id} ("{first}")')34fails = [item for item in dev if verdicts.get(item.id) == "FAIL" and listed(item) & {s.section_id for s in assistant.sections(item)}]35declined = [item for item in fails if parse_reply(replies.reply(item.id, 0)).declined]36answered = [item for item in fails if item not in declined]37print(f"right section in the top {k}, FAIL, declined: {len(declined)}")38print(f"right section in the top {k}, FAIL, answered: {len(answered)}")39print(f"answered FAIL with the section, citing a listed section: "40 f"{sum('supported' in citation_outcomes(parse_reply(replies.reply(item.id, 0)), item.version, listed(item), ids) for item in answered)}")41judged = {} # item id -> SUPPORTED, UNSUPPORTED or unreadable, for each answered fail42for item in answered:43 line = next((ln.strip().strip("*`").strip() for ln in faith.reply(item.id, 0).splitlines() if ln.strip()), "")44 match = VERDICT.match(line)45 judged[item.id] = match.group(1) if match else "unreadable"46print(f"answered FAIL with the section, faithfulness SUPPORTED: {sum(v == 'SUPPORTED' for v in judged.values())}")47print(f"answered FAIL with the section, faithfulness UNSUPPORTED: {sum(v == 'UNSUPPORTED' for v in judged.values())}")48print(" " + (", ".join(i for i, v in judged.items() if v == "UNSUPPORTED") or "none"))49print(f"answered FAIL with the section, faithfulness unreadable: {sum(v == 'unreadable' for v in judged.values())}")
python fail-with-section.pyunreadable verdict, left out: q-0043 ("VERDICT: PARTIAL")
right section in the top 3, FAIL, declined: 5
right section in the top 3, FAIL, answered: 5
answered FAIL with the section, citing a listed section: 5
answered FAIL with the section, faithfulness SUPPORTED: 4
answered FAIL with the section, faithfulness UNSUPPORTED: 1
q-0029
answered FAIL with the section, faithfulness unreadable: 0
在作答的失敗中,有 5 個引用了項目所列的節。忠實度評審判定其中 4 個通過,並將 1 個標為缺乏支持,也就是 q-0029。這裡的數字無法說明其餘的失敗是源於錯誤的拒答、版本落差,還是答案評審的失誤。閱讀回覆則可以。
有一個判定是 PARTIAL,而不是 PASS 或 FAIL。評審對 q-0043 的判定是 PARTIAL,所以表中不列入它。無法讀取的判定絕不會被算作失敗。
實用原則: 把檢索當作通過的關卡,也當作失敗的線索。沒有支持節的通過需要解釋,有支持節的失敗則指向回覆本身。
Agent Harnesses 從這裡接續
當一項任務需要多個步驟時,每一步都可能呼叫工具,並改變下一步所看到的內容。最終結果可能通過,但中間某一步其實出了錯,因此計數必須涵蓋整條路徑。本書評分的是一次檢索與一次回覆,所以每個項目的路徑計數只有是或否兩種結果。如何為這類執行過程評分,請見 Agent Harnesses 中關於以路徑計數任務的一節,本章不再重述。
實用原則: 在本書中,路徑就是一次檢索與一次回覆,因此對每個項目,應將檢索計數與判定結果對照著看。
此部分尚未翻譯,因此以英文呈現。
Retrieval quality belongs to another book
How well retrieval orders the sections, and how much of the context it fills, is a question about ranking, and it belongs to A Primer on Search Relevance. Its choosing a ranking metric section covers recall at k, mean reciprocal rank and nDCG, and how to choose among them. Measuring the context covers the context a model is given. Read both before you pick a metric for this assistant.
The retrieval-path count asks one question of each item: is a supporting section of the asked version in the top k? It returns yes or no, so it is not a ranking metric. It cannot say how far down the list that section sits, and that is what a ranking metric is for. Its k is read from the baseline's manifest and is not chosen in this chapter, so a change to k moves the table, and the table has to be read again.
Taken together, the four checks say that a pass rate on this assistant hides three different ways of being wrong. The reply can cite the wrong thing, state what no section says, or answer for the wrong version. Each one needs its own count, its own denominator and its own word for what it found. The decisions that follow from the results are these:
- Each reply is checked for citations, faithfulness and version, and each check keeps its own count.
- Every faithfulness rate is printed as uncalibrated until someone checks the judge against people.
- Version-dependent items keep their own count beside the pass rate, and their own version check.
- Retrieval is a gate on passes and a lead on fails. A pass whose section was not retrieved is read before it is counted as grounded.
Working rule: Before you call a reply grounded, read its citation, its claims and its version, and read the retrieval that came before it.
本章,於本版次中
版次 2026.11.3範例於 2026年11 月 21 通過測試本章 6 個範例 個範例中有 6 個通過
譯自英文書,版次 2026.11.3。