检查答案是否基于所引用的文档及所问的版本
阅读约需 35分钟 分钟,8节中含6个示例。
学完本章,你可以统计:答案的引用是否支持它所说的内容,答案中的每个论断是否都基于提供给助手的各节,答案是否适用于读者所问的版本,以及检索在此过程中是否找到了正确的那一节。
正确的答案仍然可能是质量不佳的答案。本书中的助手依据 sablefile 的文档回答有关它的问题,而当一条回复与问题所问版本的参考答案一致时,即视为通过。一条回复可以通过这项检验,却仍然引用了与其句子内容不符的一节,添加了任何一节都没有给出的细节,或者从另一个版本的文档中取用事实。通过率发现不了这些问题。而读者只要核对论断旁边的引用,就能发现,这正是要求助手提供引用的原因。
在通过率之外还有四项检查,每一项都在各自的分母上有自己的计数。引用准确率考察每条引用所指的节是否是该条目列为支持依据的节。忠实度考察每个论断是否出现在向助手展示过的各节之中。版本正确性考察答案是否适用于所问的版本。结果与路径对照考察通过的回复是否拿到了对应的那一节,以及未通过的回复是否也拿到了。引用准确率和忠实度不计拒答,因为拒答没有可供核对的论断。版本正确性和检索表则与通过率一样保留拒答,并各自列出其中包含的拒答数量。
这里所说的“有依据”是什么意思
如果读者顺着答案中的引用去查,能在引用所指的位置、在他们所问的那个版本中找到对应的论断,这个答案就是有依据的。这需要三点同时成立。其一,引用在该版本中确实存在,并且支持它所附着的那句话。其二,答案中的每一个论断都必须能从助手读过的各节中推出。其三,答案必须对该版本正确,而不是仅仅对另一个措辞相同的版本正确。
设想一位读者问:远程获取在放弃之前会等待多久。假设回复给出了一个听上去合理的答案,并引用了关于获取远程模板的那一节,而那一节从未说明等待时间是多少。读者会把这条回复当作有依据的,引用也看起来正确。可是读者打开那一节,却找不到任何内容来支持那个数字。如果这个数字恰好与参考答案一致,通过率会把这条回复算作正确。忠实性是通过评判模型去阅读各节的检查;引用检查只读取标识符,并且明确说明了这一点。
这几种失败各不相同,每一种都需要单独计数。一条回复可以有支持性的引用,却仍然陈述了该节没有提到的细节,所以引用准确率并不能反映忠实性。一条回复可以忠实于某一版本的某一节,却对另一个版本是错的,所以忠实性并不能反映版本正确性。还有一条回复可以既正确又引用得当,而检索步骤却从未触及它所依赖的那一节。“结果对照路径”正是为发现这种情形而设。
工作准则: 让每项检查在各自的分母上单独计数,并把通过视为阅读的开始,而不是阅读的终点。
引用准确率
引用是对文档的一项断言:这一节在这个版本中,支持它所附着的那句话。与任何断言一样,引用也可能出错。该节可能存在,但所述内容与该句话不同;可能属于另一个版本;也可能回复引用了该条目未列出的一节。Primer 介绍了句子层面的检查,即检查被引用条目支持什么,对照上下文检查引用则讨论模型所获得的上下文。本书的检查更为简单,第 4 章的引用检查给出了它的四种结果:把每个被引用的 ID 与该条目针对所问版本列出的支持节清单相匹配。它从不阅读该节的内容,因此对那句话本身不作任何判断。
代码根据同一批回复输出两个计数。引用率统计已作答条目中每一条受支持的引用。条目率则在已作答条目至少有一条引用受支持时,将该条目计一次,因此一条良好的引用即可支撑整个条目。若引用的是所问版本中的某一节,而该条目并未列出它,则输出为未列出。那一节很可能是对的,所以检查不称其为错误,也不会打开该节去核实。阅读未列出的引用需要人工判断:如果该节确实支持那句话,说明条目的支持节清单不完整,应由测试集的负责人补充;如果不支持,则引用有误,应将该回复视为引用失败。
请在两个数据划分上分别运行,并先看条目行,再看引用行。
1"""Counts the baseline's answered items that cite a supporting section of the asked version, dev and held-out apart.23An item counts once, and only if one of its citations is supported; its other citations sit in the citation lines. A decline4is not an answer, so it is left out of every count. The check matches each cited ID against the item's listed supporting5section IDs for its asked version. It never reads the section's text, so it does not check what the sentence says.6The first line of each split is the point: the item rate."""7import lib # first: fixes the thread counts and seeds before anything else loads89from lib.contract import citation_outcomes, parse_reply, section_ids_by_version10from lib.prompts import P1_SHA25611from lib.replay import ReplayModel12from lib.stats import pct13from lib.testset import load_items1415ids = section_ids_by_version() # every version's section IDs, read from the docs, for telling wrong-version from missing16model = ReplayModel("assistant-baseline", P1_SHA256) # the baseline's replies, sample 0, made from prompt P1171819def supporting_ids(item):20 """The item's supporting section IDs in its asked version. Another version's listed sections are not counted here."""21 return frozenset(s.section_id for s in item.supporting_sections if s.version == item.version)222324for split in ("dev", "held-out"):25 declined = 026 results = {} # item id -> its citation outcomes, for each answered item27 for item in load_items(split=split):28 reply = parse_reply(model.reply(item.id, 0))29 if reply.declined:30 declined += 131 else:32 results[item.id] = citation_outcomes(reply, item.version, supporting_ids(item), ids)33 # An answer to an unanswerable item lists no section of its asked version, so it can never count as supported here.34 outcomes = [outcome for found in results.values() for outcome in found] # every citation, in item order35 with_support = [item_id for item_id, found in results.items() if "supported" in found]36 no_support = sorted(set(results) - set(with_support))37 print(f"{split} declined items, left out: {declined}")38 print(f"{split} answered items: {len(results)}")39 print(f"{split} answered items citing a supporting section of the asked version: {pct(len(with_support), len(results))}")40 print(f"{split} answered items with no supported citation: {len(no_support)}")41 print(" " + (", ".join(no_support) or "none"))42 print(f"{split} citations: {len(outcomes)}")43 print(f"{split} citations that are supported: {pct(outcomes.count('supported'), len(outcomes))}")44 print(f"{split} citations that are unlisted: {pct(outcomes.count('unlisted'), len(outcomes))}")45 print(f"{split} citations that are wrong-version or missing: "46 f"{pct(outcomes.count('wrong-version') + outcomes.count('missing'), len(outcomes))}")
python citation-accuracy-counts.pydev declined items, left out: 23
dev answered items: 57
dev answered items citing a supporting section of the asked version: 94.7% (54 of 57)
dev answered items with no supported citation: 3
q-0005, q-0057, q-0058
dev citations: 60
dev citations that are supported: 93.3% (56 of 60)
dev citations that are unlisted: 6.7% (4 of 60)
dev citations that are wrong-version or missing: 0.0% (0 of 60)
held-out declined items, left out: 15
held-out answered items: 25
held-out answered items citing a supporting section of the asked version: 96.0% (24 of 25)
held-out answered items with no supported citation: 1
q-0090
held-out citations: 33
held-out citations that are supported: 93.9% (31 of 33)
held-out citations that are unlisted: 6.1% (2 of 33)
held-out citations that are wrong-version or missing: 0.0% (0 of 33)
在开发集上,条目率为 94.7% (54 of 57)。
在开发集上,有 3 个已作答条目没有受支持的引用,输出中列出了它们的名称。在保留集上,这一数量为 1。
开发集上的引用率为 93.3% (56 of 60)。开发集上未列出的引用为 6.7% (4 of 60),输出对其予以报告,但不计为错误。
在保留集上,条目率为 96.0% (24 of 25),引用率为 93.9% (31 of 33)。保留集只有 25 个已作答条目,因此一个条目就会使其比率大幅变动。两个划分之间的差异,在有人称其为真实差异之前,需要先给出区间,区间与样本量说明了如何构建区间。
版本错误或缺失的引用,是那些会因检查能够指明的原因而出错的引用。在开发集上为 0.0% (0 of 60),在保留集上为 0.0% (0 of 33)。因此,引用率达不到 100%,原因在于未列出的引用,而这些引用需要人来阅读。
两个比率出现分歧,原因很简单。一个条目若有一条受支持的引用和一条未列出的引用,会计入条目率,同时拉低引用率。两者对照阅读,就能看出未命中集中在整个条目上,还是出在本来站得住脚的条目内部的个别引用上。
这项检查背后的思路在本书之外也有人研究过。Gao 及其同事(2023)发表于 EMNLP 2023,论文见 arXiv,他们构建了一个针对生成答案中引用的基准,并报告称,强模型给出的许多答案并未完全得到所引段落的支持。他们的设置与本书不同,因此不沿用他们的任何数字,引用该论文只是为了借用这一思路。
工作准则: 同时阅读条目率和引用率,并且在修改提示词之前,先阅读每一条未列出的引用。未列出的引用是对文档提出的问题,而不是应当计数的错误。
忠实性
引用准确率的计算不需要模型,但它的计数只能说明有多少条引用指向了列出的某一节,并不能说明每个句子是否成立。忠实性是接下来的问题,也只有它能发现完全没有引用的论断。
引用准确率只检查带有引用的句子。回答中没有附引用的论断会躲过这项检查,被引用的某一节仅部分支持的论断同样会躲过。忠实性问的是更宽泛的问题:回答中的每一条论断,是否都能从助手所看到的各节中推出?
Ragas(Es 等人,2023,同样是一篇预印本)把忠实性计为回答中被上下文支持的陈述所占的比例。它先把回答拆成若干陈述,再逐条核对。本书改为对每个回答只提一个问题。lib/prompts.py 中的忠实性提示词把各节和回答交给评判模型。评判模型逐条论断地阅读回答,只有每条论断都成立时,才在第一行写下 VERDICT: SUPPORTED。否则它写下 VERDICT: UNSUPPORTED,再附上一句理由。
它与引用准确率的差别出现在回答的中间部分。假设一个回答有两句话。第一句带有引用,所引用的一节支持它;第二句没有引用,因为它添加了该节没有提到的细节。引用准确率给第一句打分,对第二句则只字不提。忠实性读的是整个回答,因此仅凭第二句就可以把这个回答判为不受支持。两种检查都不能取代对方,所以两者都保留。
没有人把这个评判模型的判定与人工判定做过对照,因此这里的每一个忠实率都是未校准的。评判模型还是助手自己所用的模型 Qwen3-4B-Instruct-2507,所以它评判的是该模型自己写出的回答。自我偏好解释了这为何重要,以及本书为何无法检验这一点。校准评判模型需要对这些回答做人工评分,而本书没有。校准评判模型介绍了具体方法,即把评判模型与人工评分作比较。由于这些比率未经校准,迁移假设便没有可供迁移的内容,何时止步中的各项限制对它们完全适用。
1"""The faithfulness rate of assistant-baseline's answered items, as judged in judge-faithfulness-baseline, dev and held-out apart.23An answered item is one the baseline did not decline and whose reply is not empty; a decline is left out. The judge writes4SUPPORTED or UNSUPPORTED on its first line. A reply that is neither is printed by item ID and left out of the denominator,5never counted as UNSUPPORTED. The judge is uncalibrated: this book has no human ratings for it, so the rates are not6corrected for its errors and are not a calibrated measure. When every readable verdict is the same, the percentile7bootstrap cannot vary, so the example says so and prints the exact binomial bound for n of n instead."""8import lib # first: fixes the thread counts and seeds before anything else loads910from lib.contract import parse_reply11from lib.judging import parse_support_verdict12from lib.prompts import FAITH_SHA256, P1_SHA25613from lib.replay import ReplayModel14from lib.stats import bootstrap_mean, interval_pct, pct15from lib.testset import load_items1617answers = ReplayModel("assistant-baseline", P1_SHA256) # the baseline's replies, sample 0, made from prompt P118judge = ReplayModel("judge-faithfulness-baseline", FAITH_SHA256) # the judge's replies, made from the faithfulness prompt1920print("judge: judge-4b, uncalibrated (no human ratings in this book)")21for split in ("dev", "held-out"):22 answered = []23 for item in load_items(split=split):24 reply = parse_reply(answers.reply(item.id, 0))25 if not reply.declined and reply.text:26 answered.append(item.id)27 readable = {} # item id -> 1.0 for SUPPORTED, 0.0 for UNSUPPORTED28 for item_id in answered:29 raw = judge.reply(item_id, 0)30 verdict = parse_support_verdict(raw)31 if verdict is None:32 line = next((ln.strip().strip("*`").strip() for ln in raw.splitlines() if ln.strip()), "")33 print(f'{split} unreadable verdict, left out: {item_id} ("{line}")')34 else:35 readable[item_id] = 1.0 if verdict == "SUPPORTED" else 0.036 scores = list(readable.values())37 lo, hi = bootstrap_mean(scores) # level 0.95; seed SEED, 2,000 resamples38 print(f"{split} answered items: {len(answered)}")39 print(f"{split} unreadable verdicts: {len(answered) - len(readable)}")40 print(f"{split} faithfulness rate: {pct(int(sum(scores)), len(scores))}")41 print(f"{split} 95% interval of the faithfulness rate: {interval_pct(lo, hi)}")42 if len(set(scores)) == 1: # every readable verdict is the same, so resampling cannot vary43 print(f"{split} bootstrap interval is degenerate: all {len(scores)} readable verdicts are the same")44 if sum(scores) == len(scores):45 print(f"{split} exact 95% lower bound for {len(scores)} of {len(scores)}: {100 * 0.025 ** (1 / len(scores)):.1f}%")46 if split == "dev":47 unfaithful = sorted(item_id for item_id, score in readable.items() if score == 0.0)48 print(f"dev unfaithful items, first five: {', '.join(unfaithful[:5])}")
python faithfulness-rate.pyjudge: judge-4b, uncalibrated (no human ratings in this book)
dev answered items: 57
dev unreadable verdicts: 0
dev faithfulness rate: 94.7% (54 of 57)
dev 95% interval of the faithfulness rate: [87.7%, 100.0%]
dev unfaithful items, first five: q-0013, q-0029, q-0055
held-out answered items: 25
held-out unreadable verdicts: 0
held-out faithfulness rate: 100.0% (25 of 25)
held-out 95% interval of the faithfulness rate: [100.0%, 100.0%]
held-out bootstrap interval is degenerate: all 25 readable verdicts are the same
held-out exact 95% lower bound for 25 of 25: 86.3%
该示例取基线的已作答条目,剔除拒答,然后就每一条询问评判模型。它只对能够读取的判定打印比率。既不是 SUPPORTED 也不是 UNSUPPORTED 的回复,按条目逐一打印,并从分母中剔除。它绝不会被计为失败,理由见判定优先。
开发集上未校准的忠实率为 94.7% (54 of 57)。其区间为 [87.7%, 100.0%]。该区间的上端落在比率尺度的上边界上,因为自助法无法重采样到边界之外。这个区间仍然很宽,这正是对 57 个已作答条目的如实解读。
留出集上未校准的忠实率为 100.0% (25 of 25)。打印出的区间为 [100.0%, 100.0%],但这并不是一个区间。输出指出自助法已退化:all 25 readable verdicts are the same。对全部通过的条目做重采样,只能得到相同的比率,所以自助法无法说明该比率落在何处。
精确的 95% 下界为 86.3%。若真实比率低于此值,留出集中的回复全部通过的概率不到 2.5%。这个下界等于 0.025 的 n 分之一次方。它表明评判模型放行了这些回复,但这个划分太小,看不出该比率落在何处。不要把留出集上的这个数字解读为“助手是忠实的”这一结论。
两项检查各自打印自己的清单。引用检查的输出列出没有任何受支持引用的条目,忠实性检查的输出列出最先出现的不忠实的开发集条目。请把两份清单并排阅读,因为本章并不假定它们列出的是相同的条目。
工作准则: 在每个忠实率旁边都标注“未校准”,并在相信该比率之前先阅读最先出现的不忠实条目。忠实性检查显示的是评判模型认为哪里的论断不受支持,并不能证明评判模型是对的。
版本正确性
sablefile 的文档有 1.0、1.1 和 2.0 三个版本,有些问题在各版本中的答案并不相同。一条回复可能以 1.1 的某一节为依据,对使用 2.0 的读者来说却仍是错的。本书把这类题目称为版本相关题。每道版本相关题都有一份针对读者所问版本的参考答案,其检查项判断回复是否与该参考答案一致。开发集上有 17 道,保留集上有 8 道。
版本相关题与其他题目一样计入通过率,同时也有各自的计数,列在通过率旁边。对这类题目的回复,是依据所问版本的参考答案来评判的,因此,对另一个版本正确的答案会判为不通过。如果只把这类失误混入总体通过率,看起来就像是助手能力偏弱。判定正确率就是仅针对版本相关题的计数,版本方面的失误因此单独成行,团队可以据此看出检索方面的改动是否带来了变化。拒答与在通过率中一样,仍留在该分母之内,每个划分也都会打印其中包含的数量。
错误版本引用是另一项单独的计数。它指的是引用了只存在于另一个版本中的某一节。一条回复可能对所问版本是正确的,却仍引用了另一个版本的某一节,而照着这条引用去查的读者会落到错误的文档里。
这项检查只能看到 ID 所提供的信息。引用带有的是 ID,而不是版本。多数 ID 会在各版本之间重复,所以一个同时出现在 1.0、1.1 和 2.0 中的节 ID,在每个版本里看起来都一样。出现在所有版本中的 ID,无论读者问的是哪个版本,都不可能被标记为错误版本。只有当所问版本没有该 ID、而另一个版本有该 ID 时,检查才能标记这条引用。索引同样重要。只有当另一个版本的各节位于生成回复所用的索引中时,才可能出现对该版本的引用,所以索引变化后必须重新解读这项计数,旧索引得出的零并不能说明新索引的情况。
这里考察两个变体。基线只从所问版本的索引中检索。未过滤变体则同时从所有版本的索引中检索,并在每一节的标题中显示该节所属的版本,这样模型就能看出某一节来自哪里。可能引用另一个版本的是未过滤变体,因为它能看到那些版本的节。
1"""Reads the 25 version-dependent items for two variants: whether the judge passed the answer for the asked version, and whether a2citation names a section that exists only in another version.34The judge's verdict for each variant is read from its own recording, and an unreadable verdict is printed and left out of the5pass rate, never counted as a fail. A declined reply has a verdict like any other reply, so it stays in the denominator; its6count is printed beside each rate. A wrong-version citation is counted per item, over every item, so a zero shows as 0 of 25.7The last lines list the items where the two variants disagree, to read item by item."""8import lib # first: fixes the thread counts and seeds before anything else loads910from lib.calibration import readable_verdicts11from lib.contract import citation_outcomes, parse_reply, section_ids_by_version12from lib.prompts import P1_SHA256, REF_SHA25613from lib.replay import ReplayModel14from lib.stats import pct15from lib.testset import load_items1617ids = section_ids_by_version() # every version's section IDs, so a section of another version is known as one18# label, the assistant's recording, the judge's recording of that assistant's replies19VARIANTS = (("baseline", "assistant-baseline", "judge-answers-baseline"), ("unfiltered", "assistant-unfiltered", "judge-answers-unfiltered"))20items = [item for item in load_items() if item.kind == "version-dependent"]21item_ids = [item.id for item in items]22listed = {item.id: frozenset(s.section_id for s in item.supporting_sections if s.version == item.version) for item in items}23verdict, wrong = {}, {} # label -> {item id: PASS or FAIL, readable only}; label -> item ids with a wrong-version citation24for label, assistant, judge in VARIANTS:25 verdict[label], unreadable = readable_verdicts(judge, item_ids, REF_SHA256)26 for item_id, first in unreadable:27 print(f'{label} unreadable verdict, left out: {item_id} ("{first}")')28 replies = ReplayModel(assistant, P1_SHA256)29 wrong[label] = {item.id for item in items30 if "wrong-version" in citation_outcomes(parse_reply(replies.reply(item.id, 0)), item.version, listed[item.id], ids)}31 for split in ("dev", "held-out"):32 split_items = [item for item in items if item.split == split]33 readable = [item.id for item in split_items if item.id in verdict[label]]34 passed = sum(verdict[label][item_id] == "PASS" for item_id in readable)35 print(f"{label} {split} version-dependent items: {len(split_items)}")36 print(f"{label} {split} declined replies, counted in the denominator: "37 f"{sum(parse_reply(replies.reply(item.id, 0)).declined for item in split_items)}")38 print(f"{label} {split} judged correct for the asked version: {pct(passed, len(readable))}")39 print(f"{label} {split} citing a wrong-version section: "40 f"{pct(sum(item.id in wrong[label] for item in split_items), len(split_items))}")41 print(f"{label} all version-dependent items citing a wrong-version section: {pct(len(wrong[label]), len(items))}")4243both = [item_id for item_id in item_ids if item_id in verdict["baseline"] and item_id in verdict["unfiltered"]]44print(f"items with a readable verdict in both variants: {len(both)}")45differ = [i for i in both if verdict["baseline"][i] != verdict["unfiltered"][i]]46print(f"items where the verdicts differ: {len(differ)}")47print(" " + (", ".join(differ) or "none"))48flag_differ = [i for i in item_ids if (i in wrong["baseline"]) != (i in wrong["unfiltered"])]49print(f"items where the wrong-version flag differs: {len(flag_differ)}")50print(" " + (", ".join(flag_differ) or "none"))
python version-correct-answers.pybaseline dev version-dependent items: 17
baseline dev declined replies, counted in the denominator: 4
baseline dev judged correct for the asked version: 64.7% (11 of 17)
baseline dev citing a wrong-version section: 0.0% (0 of 17)
baseline held-out version-dependent items: 8
baseline held-out declined replies, counted in the denominator: 2
baseline held-out judged correct for the asked version: 75.0% (6 of 8)
baseline held-out citing a wrong-version section: 0.0% (0 of 8)
baseline all version-dependent items citing a wrong-version section: 0.0% (0 of 25)
unfiltered dev version-dependent items: 17
unfiltered dev declined replies, counted in the denominator: 5
unfiltered dev judged correct for the asked version: 52.9% (9 of 17)
unfiltered dev citing a wrong-version section: 0.0% (0 of 17)
unfiltered held-out version-dependent items: 8
unfiltered held-out declined replies, counted in the denominator: 2
unfiltered held-out judged correct for the asked version: 75.0% (6 of 8)
unfiltered held-out citing a wrong-version section: 0.0% (0 of 8)
unfiltered all version-dependent items citing a wrong-version section: 0.0% (0 of 25)
items with a readable verdict in both variants: 25
items where the verdicts differ: 6
q-0003, q-0021, q-0022, q-0033, q-0082, q-0085
items where the wrong-version flag differs: 0
none
在所有版本相关题上,基线的错误版本占比为 0.0% (0 of 25),未过滤变体为 0.0% (0 of 25)。零只说明检查没有标记任何 ID。下一个示例展示了它本来能够标记的 ID 少到什么程度。
1"""The two counts behind the version check, and the version-dependent items where the variants' verdicts differ, by split.23A wrong-version citation is one whose ID exists only in another version, so an ID that several versions share cannot be4attributed to one of them. This prints how many section IDs are found in one version only, then how many citations to the5version-dependent items name a shared ID, for each variant. The last lines list the items where the verdicts differ, by split."""6import lib # first: fixes the thread counts and seeds before anything else loads78from collections import Counter910from lib.calibration import readable_verdicts11from lib.contract import parse_reply, section_ids_by_version12from lib.prompts import P1_SHA256, REF_SHA25613from lib.replay import ReplayModel14from lib.stats import pct15from lib.testset import load_items1617ids = section_ids_by_version()18shared = Counter(i for found in ids.values() for i in found) # how many versions each section ID appears in19print("section IDs found in one version only, by version: "20 + ", ".join(f"{v} {sum(shared[i] == 1 for i in found)}" for v, found in ids.items()))21items = [item for item in load_items() if item.kind == "version-dependent"]22split_of = {item.id: item.split for item in items}23verdict = {} # label -> {item id: PASS or FAIL}, readable only24for label, assistant, judge in (("baseline", "assistant-baseline", "judge-answers-baseline"),25 ("unfiltered", "assistant-unfiltered", "judge-answers-unfiltered")):26 replies = ReplayModel(assistant, P1_SHA256)27 cited = [cid for item in items for cid in parse_reply(replies.reply(item.id, 0)).citations]28 print(f"{label} citations to a section ID found in more than one version: "29 f"{pct(sum(shared[cid] > 1 for cid in cited), len(cited))}")30 verdict[label], _ = readable_verdicts(judge, list(split_of), REF_SHA256)31differ = [i for i in verdict["baseline"] if i in verdict["unfiltered"] and verdict["baseline"][i] != verdict["unfiltered"][i]]32for split in ("dev", "held-out"):33 part = [i for i in differ if split_of[i] == split]34 print(f"{split} items where the verdicts differ: {len(part)}")35 print(" " + (", ".join(part) or "none"))
python version-check-detail.pysection IDs found in one version only, by version: 1.0 0, 1.1 0, 2.0 4
baseline citations to a section ID found in more than one version: 87.0% (20 of 23)
unfiltered citations to a section ID found in more than one version: 63.2% (12 of 19)
dev items where the verdicts differ: 4
q-0003, q-0021, q-0022, q-0033
held-out items where the verdicts differ: 2
q-0082, q-0085
仅出现在一个版本中的节 ID,按版本统计为 1.0 0, 1.1 0, 2.0 4。出现在多个版本中的 ID 仍然可能被标记,但只限于使用缺少该 ID 的版本的读者。对基线而言,版本相关题的引用中指向共有 ID 的占比为 87.0% (20 of 23)。对未过滤变体,这一占比为 63.2% (12 of 19)。
两者的判定正确率有差异,不过每个比率都是在较小的划分上得出的。在开发集上,基线被判定为正确的是 64.7% (11 of 17)。
未过滤变体被判定为正确的是 52.9% (9 of 17)。在这些题目上,未过滤变体通过的更少。
在保留集上,基线被判定为正确的是 75.0% (6 of 8)。未过滤变体的正确率与之相同,但通过的题目不同。q-0082 和 q-0085 两题上,两者的判定方向相反,保留集的其余题目则判定一致。
在两个变体之间,判定不同的题目有 6 道,这是在两者都有可读判定的 25 道题中得出的。其中开发集上有 4 道不同,保留集上有 2 道。输出按划分列出这些题目。
这些差异没有计算区间,而且这样规模的开发集很小。因此本章只报告计数,不断言哪个变体更好或更差。这些计数说明了三件事:在这些题目上,未过滤变体通过的更少;两个变体都没有任何引用被标记为错误版本;判定不同的题目是应当首先阅读的地方。
工作准则: 读每条引用的 ID 时,要把该回复的答案所针对的版本放在旁边对照。这项检查得出零,意思是没有被标记的 ID,而不是没有错误版本引用。把版本相关题放在通过率旁边单独计数,这样版本方面的失误就会单独成行。
结果与路径
结果指一次回复是否通过。路径指回复之前的各个步骤所到达的内容。在本书中,路径很短:一次检索,然后一次回复。有两种组合需要同时参考这两项计数。一次通过,但支撑它的节从未送达模型,其答案来自别处,也许是模型自身的知识,即使判定正确,这也是一个警示。一次失败,而该节已被检索到,这正是其他各项检查要去解释的情形。
含有多个步骤的运行,路径更长。Agent Harnesses 通过路径来统计一项任务,想了解这一点应去读那本书。本章只为助手拥有的那一个步骤打分。
检索步骤
检索是在所问版本的各节上做 BM25。它是确定性的:同一个问题每次得到同样的前几节。因此这一步无需录制。代码根据索引为每个条目计算排在前面的各节,再把它们与评判者对基线回复给出的裁定对照。
开发集中没有所问版本支持节的条目,不列入该表。这些是无法回答的条目,因为其所问版本不存在可供检索找到的节。它们的数量是 13。
1"""Sets the baseline's judge verdict against whether a supporting section reached the top k, for each dev item that has one.23Retrieval is BM25 over the asked version, so it is the same on every run and needs no recording; k is the baseline's, read from4its manifest. The two diagonal cells are the expected pairs. The two off-diagonal cells are the ones to read: a pass without the5right section retrieved, and a fail with it retrieved. Unanswerable items are left out, since they have no section of their asked6version to retrieve; their count is printed. An unreadable verdict is printed and left out, never counted as a fail. A declined reply is counted in its cell like any7other reply; the count of declines is printed. The FAIL cell is split into declines and answers in fail-with-section."""8import lib # first: fixes the thread counts and seeds before anything else loads910from lib.answers import Assistant11from lib.calibration import readable_verdicts12from lib.contract import parse_reply13from lib.prompts import P1_SHA256, P1_SYSTEM, REF_SHA25614from lib.replay import ReplayModel15from lib.testset import load_items1617RECORDING = "assistant-baseline"18k = ReplayModel(RECORDING, P1_SHA256).manifest["k"] # the top k the baseline's prompt was built from, read from its manifest19assistant = Assistant(RECORDING, P1_SYSTEM, P1_SHA256, k=k) # its retriever only; the recording is not read here20replies = ReplayModel(RECORDING, P1_SHA256) # the baseline's replies, sample 0, made from prompt P1, for the decline check2122dev = load_items(split="dev")23has_support = [item for item in dev if any(s.version == item.version for s in item.supporting_sections)]24print(f"top k: {k}")25print(f"dev items: {len(dev)}")26print(f"dev items with no supporting section of the asked version, left out: {len(dev) - len(has_support)}")2728verdicts, unreadable = readable_verdicts("judge-answers-baseline", [item.id for item in has_support], REF_SHA256)29for item_id, first in unreadable:30 print(f'unreadable verdict, left out: {item_id} ("{first}")')3132# Keyed by (retrieved, passed): the four cells of the 2 by 2 table, each a list of item ids in sorted order.33cells = {(True, True): [], (True, False): [], (False, True): [], (False, False): []}34declined = set() # item ids whose reply is a decline; they sit in their cells like any other reply35for item in has_support:36 if item.id not in verdicts:37 continue38 listed = {s.section_id for s in item.supporting_sections if s.version == item.version}39 top = {section.section_id for section in assistant.sections(item)} # the top k sections, by BM25 rank40 cells[(bool(listed & top), verdicts[item.id] == "PASS")].append(item.id)41 if parse_reply(replies.reply(item.id, 0)).declined:42 declined.add(item.id)4344print(f"items in the table: {sum(len(ids) for ids in cells.values())}")45print(f"declined items in the table, counted in their cells: {len(declined)}")46print(f"right section in the top {k}, PASS: {len(cells[(True, True)])}")47print(f"right section in the top {k}, FAIL: {len(cells[(True, False)])}")48print(f"right section not in the top {k}, PASS: {len(cells[(False, True)])}")49print(f"right section not in the top {k}, FAIL: {len(cells[(False, False)])}")50print("passed without the right section retrieved: " + (", ".join(cells[(False, True)]) or "none"))51print("failed with the right section retrieved: " + (", ".join(cells[(True, False)]) or "none"))
python retrieval-path.pytop k: 3
dev items: 80
dev items with no supporting section of the asked version, left out: 13
unreadable verdict, left out: q-0043 ("VERDICT: PARTIAL")
items in the table: 66
declined items in the table, counted in their cells: 10
right section in the top 3, PASS: 49
right section in the top 3, FAIL: 10
right section not in the top 3, PASS: 0
right section not in the top 3, FAIL: 7
passed without the right section retrieved: none
failed with the right section retrieved: q-0019, q-0023, q-0025, q-0029, q-0033, q-0045, q-0046, q-0048, q-0063, q-0074
top k 为 3,取自基线的清单。该表共有 66 个条目,其中 10 个是拒答。拒答与其他回复一样落在各自的单元格中,因为评判者对它给出了裁定。每个单元格把一种裁定与支持节是否进入 top k 配成一对。
| 裁定 | 支持节在 top k 中 | 支持节不在 top k 中 |
|---|---|---|
| PASS | 49 | 0 |
| FAIL | 10 | 7 |
先看 PASS 一行。支持节在 top k 中的通过单元格有 49 个通过,支持节不在 top k 中的通过单元格有 0 个。这与通过来自文档的说法相符。但它并不能证明通过确实来自文档,因为该表统计的是两件事同时出现的频率,而不是哪一件导致了另一件。通过的数量也是对参考答案的一项检查。一个没有检索到任何节的通过,可能来自重复常见事实的回复,也可能来自读者能够猜到的参考答案,而该表无法分辨是哪一种。阅读这些条目则可以。
FAIL 一行是其他检查介入的地方。在支持节位于 top k 中的失败里,有 5 个是拒答,其余是作答。文档未说明时的拒答讲的是:当文档中有答案时,拒答意味着什么。下面的示例把该单元格拆开来看。
1"""The dev items the baseline failed although a listed section of the asked version was in its top k, split into declines and answers.23For each answer it prints whether the reply cites a listed section, and the faithfulness judge's verdict on it. A decline has no4faithfulness verdict, because the judge reads only answers. An unreadable verdict is printed and left out, never counted as a fail."""5import lib # first: fixes the thread counts and seeds before anything else loads67import re89from lib.answers import Assistant10from lib.calibration import readable_verdicts11from lib.contract import citation_outcomes, parse_reply, section_ids_by_version12from lib.prompts import FAITH_SHA256, P1_SHA256, P1_SYSTEM, REF_SHA25613from lib.replay import ReplayModel14from lib.testset import load_items1516RECORDING = "assistant-baseline"17k = ReplayModel(RECORDING, P1_SHA256).manifest["k"] # the baseline's top k, as in retrieval-path18assistant = Assistant(RECORDING, P1_SYSTEM, P1_SHA256, k=k) # its retriever only19replies = ReplayModel(RECORDING, P1_SHA256) # the baseline's replies, sample 0, made from prompt P120faith = ReplayModel("judge-faithfulness-baseline", FAITH_SHA256)21ids = section_ids_by_version()22VERDICT = re.compile(r"^VERDICT:\s*(SUPPORTED|UNSUPPORTED)$")232425def listed(item):26 """The item's supporting section IDs in its asked version."""27 return frozenset(s.section_id for s in item.supporting_sections if s.version == item.version)282930dev = load_items(split="dev")31verdicts, unreadable = readable_verdicts("judge-answers-baseline", [item.id for item in dev], REF_SHA256)32for item_id, first in unreadable:33 print(f'unreadable verdict, left out: {item_id} ("{first}")')34fails = [item for item in dev if verdicts.get(item.id) == "FAIL" and listed(item) & {s.section_id for s in assistant.sections(item)}]35declined = [item for item in fails if parse_reply(replies.reply(item.id, 0)).declined]36answered = [item for item in fails if item not in declined]37print(f"right section in the top {k}, FAIL, declined: {len(declined)}")38print(f"right section in the top {k}, FAIL, answered: {len(answered)}")39print(f"answered FAIL with the section, citing a listed section: "40 f"{sum('supported' in citation_outcomes(parse_reply(replies.reply(item.id, 0)), item.version, listed(item), ids) for item in answered)}")41judged = {} # item id -> SUPPORTED, UNSUPPORTED or unreadable, for each answered fail42for item in answered:43 line = next((ln.strip().strip("*`").strip() for ln in faith.reply(item.id, 0).splitlines() if ln.strip()), "")44 match = VERDICT.match(line)45 judged[item.id] = match.group(1) if match else "unreadable"46print(f"answered FAIL with the section, faithfulness SUPPORTED: {sum(v == 'SUPPORTED' for v in judged.values())}")47print(f"answered FAIL with the section, faithfulness UNSUPPORTED: {sum(v == 'UNSUPPORTED' for v in judged.values())}")48print(" " + (", ".join(i for i, v in judged.items() if v == "UNSUPPORTED") or "none"))49print(f"answered FAIL with the section, faithfulness unreadable: {sum(v == 'unreadable' for v in judged.values())}")
python fail-with-section.pyunreadable verdict, left out: q-0043 ("VERDICT: PARTIAL")
right section in the top 3, FAIL, declined: 5
right section in the top 3, FAIL, answered: 5
answered FAIL with the section, citing a listed section: 5
answered FAIL with the section, faithfulness SUPPORTED: 4
answered FAIL with the section, faithfulness UNSUPPORTED: 1
q-0029
answered FAIL with the section, faithfulness unreadable: 0
在作答的失败中,有 5 个引用了条目所列的某一节。忠实性评判者判定其中 4 个通过,并把 1 个标为无依据,即 q-0029。这里的计数并不能说明,其余部分是由错误的拒答、版本错配还是答案评判者的错误造成的。阅读回复则可以。
有一个裁定读作 PARTIAL,而不是 PASS 或 FAIL。评判者对 q-0043 的裁定是 PARTIAL,因此该表将其略去。无法读取的裁定绝不计为失败。
工作准则: 把检索当作通过的关卡,同时当作失败的线索。没有支持节的通过需要解释,有支持节的失败则指向回复本身。
从哪里接续讲智能体测试框架
当一项任务需要多个步骤时,每一步都可能调用工具,并改变下一步所看到的内容。最终结果可能通过,而中间某一步却出了错,所以计数必须覆盖整条路径。本书评分的对象只有一次检索和一次回复,因此每个条目的路径计数只是一个“是”或“否”。如何为这类运行过程评分,参见Agent Harnesses中关于按路径计数任务的一节,本章不再重复。
工作准则: 在本书中,路径就是一次检索加一次回复,因此应将每个条目的检索计数与其判定结果对照着读。
这部分尚未翻译,因此以英文显示。
Retrieval quality belongs to another book
How well retrieval orders the sections, and how much of the context it fills, is a question about ranking, and it belongs to A Primer on Search Relevance. Its choosing a ranking metric section covers recall at k, mean reciprocal rank and nDCG, and how to choose among them. Measuring the context covers the context a model is given. Read both before you pick a metric for this assistant.
The retrieval-path count asks one question of each item: is a supporting section of the asked version in the top k? It returns yes or no, so it is not a ranking metric. It cannot say how far down the list that section sits, and that is what a ranking metric is for. Its k is read from the baseline's manifest and is not chosen in this chapter, so a change to k moves the table, and the table has to be read again.
Taken together, the four checks say that a pass rate on this assistant hides three different ways of being wrong. The reply can cite the wrong thing, state what no section says, or answer for the wrong version. Each one needs its own count, its own denominator and its own word for what it found. The decisions that follow from the results are these:
- Each reply is checked for citations, faithfulness and version, and each check keeps its own count.
- Every faithfulness rate is printed as uncalibrated until someone checks the judge against people.
- Version-dependent items keep their own count beside the pass rate, and their own version check.
- Retrieval is a gate on passes and a lead on fails. A pass whose section was not retrieved is read before it is counted as grounded.
Working rule: Before you call a reply grounded, read its citation, its claims and its version, and read the retrieval that came before it.
本章,在本版次中
版次 2026.11.3示例于 2026年11月21日 通过本章 6个示例 个示例中有 6 个通过
译自英文版图书,版次 2026.11.3。