编写评判者能够执行的评分细则
阅读约需 35分钟 分钟,6节中含3个示例。
读完本章,你可以编写一份评分细则,使评判模型以同样的方式评判每条回复;可以用代码读取评判结果,凡是无法读成结论的内容一律不计入;还可以判断成对比较或打分量表在何时值得付出相应的成本。
第 4 章止步于代码力所不及之处。检查可以保证回复的形式规范,也可以指出回复引用了哪一节,却读不出回复究竟主张了什么,因此需要有一个读者把回复的主张与参考答案对照起来。本章构建的就是这样一位读者,即评判者:它是一个模型,阅读一条回复,将其与书面的参考答案比较,然后写出这条回复是否正确。助手回答有关命令行工具 sablefile 的问题,第 1 章最先用到了这位评判者。评判者是 Qwen3-4B-Instruct-2507,固定在其 GGUF 转换版本的提交 a06e946 上,与回答问题的是同一个模型,因此本书并不用更强的模型来检验某个模型。评判者的回复只记录了一次,本章的每个示例读取的都是这些记录,所以其中的数字不会因运行次数不同而变化。如果一个团队报告的通过率出自一位从未被检验过的评判者,那么它报告的其实是关于评判者的数字;若不做本章所述的检验,读者仅凭页面上的数字无法分辨这两者。
评判者依据指令工作,而指令就是评分细则。评分细则由两段文本构成。系统文本陈述任务、规则以及回复必须采用的形式。模板则用标记把请求、参考答案和回复包裹起来,使评判者能够区分它们。评分细则是测量的一部分,而不是写在测量之前的说明,因为评判者拿到的正是其中的措辞。本章展示本书如何编写评分细则、如何读取每一个评判结果,以及团队必须在哪些地方做出选择。
评审者被要求做什么
评审者的回复会交给代码处理,代码把回复的第一行当作裁决来读取。因此,评分细则所要求的输出形式本身就是评分细则的一部分。评审者若以其他形式写出裁决,就没有给代码留下可读的内容,代码在这种情况下如何处理,见下文关于裁决的一节。
本书共有五份评分细则。答案评审者把一条关于 sablefile 的回复与该问题的书面参考答案相对照。校准评分细则在没有参考答案的情况下检查正确性,所用的数据行来自 HelpSteer2,这是一个公开数据集,收录人工对其他模型针对通用提示所作回复的评分,第 6 章对其有所介绍。其数据集卡片于 2026 年 11 月 20 日查阅。评分细则 B 是校准任务的另一种措辞。成对评分细则在两条回复中选出更好的一条。忠实性评分细则检查一个答案是否得到所给定各节的支持。每份评分细则都写成文本并带有哈希值,第一个示例打印出全部五份评分细则的名称和哈希值。对于文档未涉及的条目,参考答案为单词 none,而评分细则会告知评审者:只有当回复恰为拒答语句时,才算通过。
1"""The five rubrics the book's judges use, each with the sha256 its recording's manifest holds, and how many lines its system2text has. The hash is the one lib.prompts.sha256_text gives for the system text and the user template together: each3recording's manifest was made from that pair, not from the system text alone, so the check compares the pair. The first4line of each rubric is printed to show which rubric it is; the full texts are in lib/prompts.py."""5import lib # first: fixes the thread counts and seeds before anything else loads67from lib.prompts import (8 FAITH_SYSTEM, FAITH_TEMPLATE, HS2_SYSTEM, HS2_TEMPLATE, HS2B_SYSTEM, HS2B_TEMPLATE, PAIR_SYSTEM, PAIR_TEMPLATE,9 REF_SYSTEM, REF_TEMPLATE, sha256_text,10)11from lib.replay import ReplayModel1213# (the rubric's name, the recording whose manifest holds its hash, its system text, its user template)14RUBRICS = (15 ("answer judge, reference-guided", "judge-answers-baseline", REF_SYSTEM, REF_TEMPLATE),16 ("HelpSteer2 calibration rubric, reference-free", "judge-helpsteer2", HS2_SYSTEM, HS2_TEMPLATE),17 ("HelpSteer2 rubric B, second wording", "judge-helpsteer2-rubric-b", HS2B_SYSTEM, HS2B_TEMPLATE),18 ("pairwise rubric", "judge-hs2-pairs-ab", PAIR_SYSTEM, PAIR_TEMPLATE),19 ("faithfulness rubric", "judge-faithfulness-baseline", FAITH_SYSTEM, FAITH_TEMPLATE),20)2122for name, recording, system, template in RUBRICS:23 digest = sha256_text(system, template)24 # ReplayModel raises ValueError when the recording's manifest hash is not this one, so a match is checked, not assumed.25 ReplayModel(recording, digest)26 print(f"{name}: {len(system.splitlines())} lines")27 print(f" sha256 of system text and template: {digest}")28 print(f" first line: {system.splitlines()[0]}")
python rubric-text.pyanswer judge, reference-guided: 10 lines
sha256 of system text and template: 4c409dcd59d0926646c3f8f5683da5de589e1586007dcfcf79d71a6b49e7e227
first line: You check an AI assistant's answer about the sablefile tool against the written reference answer for the question.
HelpSteer2 calibration rubric, reference-free: 20 lines
sha256 of system text and template: 56d617ac94b9fa5c4f0b2ce1eee17b7e0ce42c7e6b6c617edcb55a7b48d472e6
first line: You check the correctness of an AI assistant's response to a user's request.
HelpSteer2 rubric B, second wording: 17 lines
sha256 of system text and template: 7f512725080e8e78437603f7445a0480caaa0d2a344aa51d58d5475cba2ac384
first line: You grade whether an AI assistant's reply to a user is right.
pairwise rubric: 13 lines
sha256 of system text and template: 24a735e79373f7fa0b8bf1210c3e516964d93164508c666b8d333b93c27de992
first line: You compare two AI assistant responses to the same request and say which one is better.
faithfulness rubric: 15 lines
sha256 of system text and template: a584b8d56f63c212497fa67602f2fbd2add41823ca5895ba2bad541389970555
first line: You check whether an answer about the sablefile tool is supported by the documentation sections it was given.
输出给出了每份系统文本的第一行,以及该文本连同其模板的哈希值。哈希值同时涵盖两者,是因为评审者同时拿到了两者,任何一方的改动都会改变评审者所看到的内容。运行时会把每个哈希值与用该评分细则所做的记录逐一核对,若二者不符便报错停止,因此记录不可能在并非产生它的评分细则下被重放。读者在本书中看到某个数字时,可以按名称和哈希值找到它背后的评分细则,也能把该评分细则与仅改动一个词的版本区分开来。
模板中的标记向评审者标明请求、参考答案和回复各自从何处开始、到何处结束,这正是它们的用途。至于这些标记能否防止引用了评分细则自身指令的回复被误当作指令来读,本书并未检验。
工作准则: 把每份评分细则写成团队可以阅读的文本,并对其计算哈希值,与它所产生的运行记录一并保存。以第一行为其命名,使读者能分辨某个数字出自哪份评分细则。只能通过另建一份带有自己记录的评分细则来改变评分细则的文本,绝不能去编辑已产生页面上现有数字的那份评分细则。
有参考答案还是无参考答案
有参考答案的评判器会拿到针对该问题写好的答案,并据此核对回复。无参考答案的评判器只拿到请求和回复,凭自身所知判断回复是否正确。参考答案是一个固定点。评判器不必了解 sablefile,因为参考答案说明了什么是正确的。这个固定点同时也是一种限制:评判器的正确程度不可能超过参考答案,而评分细则必须说明,对于措辞不同、补充了细节或只覆盖问题一部分的回复应如何处理。
答案评分细则的第一条规则接受这样的回复:它陈述了参考答案所陈述的内容。回复可以使用不同的措辞,也可以补充与参考答案不矛盾的细节。回复若说出参考答案不支持的内容,则为不正确。第三条规则处理有多种解读的参考答案:回复必须回答每一种解读,只回答一部分则为不正确。另外两条规则把该用例的拒答带入评判器。如果参考答案是单词 "none",则文档并未回答该问题,只有当回复恰好是拒答语句 "The docs don't say." 时,回复才算正确。而一条既给出了答案、又说出拒答语句的回复,则为不正确。第 10 章衡量了这条规则所控制的两类错误,评判器对每一条回复都适用同一规则。
校准评分细则不使用参考答案,因此它必须在没有参考答案可比的情况下,说明什么算错误。它要求评判器只判断正确性,并指明评判器不得评分的内容:长度、语气、风格和格式。它还说明,没有作出任何错误断言的回复不算不正确,即使它不完整或要求提供更多信息。这句话是一项决定。HelpSteer2 对多个属性评分,正确性只是其中之一。评分细则只提到正确性,因此这一问题由评分细则确定,而不是留给评判器自行决定。第 6 章衡量了评判器这种更窄的回答与人工评分的一致程度。
当参考答案有误时,评判器会把正确的回复判为不通过,而判定结果对此毫无提示。团队会把不通过率的上升解读为助手变弱,而问题其实出在参考答案上。在解读通过率之前,应先对照参考答案阅读一批不通过的样本。
**工作准则:**只要数据集有参考答案,就使用参考答案,并像答案评分细则那样,把参考答案为空时的规则写进评分细则。只有在不存在参考答案时,才使用无参考答案的评分细则。在其文本中,要写明评判器不得因哪些方面而扣分,例如长度、语气或不完整,这样做出该选择的是评分细则,而不是评判器。
通过或不通过,还是量表
通过或不通过的评分细则只给评判者两种答案,量表则给出多种答案。HelpSteer2 中的人工评分采用 0 到 4 的量表,量表看上去比一个是或否包含更多信息。它的代价在于各档之间的界线:评判者要分清相邻的两档,团队要说明每一档的含义,再检验评判者所给的档次是否与人工评分一致。“通过或不通过”对每条回复只要求做出一个判定,本书中的每一个比率,都是这类判定的比率。
答案评分细则和两份校准评分细则给出的都是通过或不通过。本章的检索曾寻找一项在二元人工标注上比较量表与通过或不通过的研究,没有找到。因此这一选择是本书的设计,而不是来自某个文献的发现,读者应当把它当作一种设计来权衡。
本书提出的问题是一条回复是否正确,这个问题只有两种答案。一条回复可能几乎正确,却在某个事实或某条命令上出错。量表会给它一个中间档,团队要先把这一档转成是或否,才能读出任何比率。通过率是评判者判为通过的条目数占集合中全部条目数的比例,正确的拒答也算通过,详见第 10 章。量表则需要另一种汇总方式,例如均值,而均值会掩盖哪些回复落在哪一档。以拒答条目为例:一条回复要么拒答,要么不拒答。量表会让评判者给一条先拒答、又附上猜测的回复打中间档,而这正是第 10 章算作错误的那种回复,团队随后还得决定量表的哪一侧算通过。
工作准则: 团队需要对正确性做出是或否的判定时,使用通过或不通过,并且只报告这类判定的比率。只有当每一档都对应团队会据以行动的含义,并且每一档都有人工评分可用来检验评判者时,才使用量表。要在正文中说明团队选了哪种形式及其理由,因为这一选择决定了团队报告的每一个比率。第 6 章说明如何将评判者与人工评分进行比较。
一个答案还是两个
成对判断向评判者展示针对同一请求的两条回复,并询问哪一条更好。它适合那些必须在两个版本之间做选择、而又难以设定绝对评分的团队。成对评分标准给评判者两个字母,A 或 B,没有平局选项。它的代价在于位置。每条回复在提示中都有一个位置,第一或第二;有研究报告称,顺序可以决定最终的选择。这些研究测量的是其他评判者在各自设置下的表现,本书不把它们的比率当作自己的结果。第 7 章回顾了这些研究,并在本书的评判者上测量了这种倾向。
成对评判者没有需要定义的量表等级,它只需要说出两条回复中哪一条更好,这恰好是在两个版本之间做选择的团队所提出的问题。它仍然需要对照人类偏好进行检验,第 7 章对其一致配对就是这样检验的。代价体现在位置上,因此本节对每一对都评判两次。
成对示例使用来自 HelpSteer2 的人类偏好配对。每一对都评判两次:一次将第一条回复显示为 A,一次将第二条回复显示为 A。随后把每个裁决映射回它所选中的那条回复,这样就可以比较两种顺序。如果在两种顺序下获胜的是同一条回复,则该配对是一致的。如果在两种顺序下获胜的是同一个字母,则该配对是顺序改变的,这意味着位置决定了裁决;示例的输出把这类配对称为 flipped。平局和无法解析是另外两种情况。
1"""How the pairwise judge's verdict on each HelpSteer2 preference pair depends on which response is shown as A. Each pair is2judged twice, once as ab (response_a shown as A) and once as ba (response_b shown as A), and both verdicts are mapped back3to the original responses. A flipped pair picked the same letter both times, so its verdict followed the position: the4flipped lines, on dev and on held-out, are the ones the point rests on.56Only the pair IDs of the examples are printed; the counts are the rates the chapter quotes."""7import lib # first: fixes the thread counts and seeds before anything else loads89from lib.judging import KINDS, both_orders, human_pairs10from lib.stats import pct1112AB = "judge-hs2-pairs-ab"13BA = "judge-hs2-pairs-ba"1415pairs = human_pairs() # all 150 pairs, sorted by id, each with its split16orders = both_orders(AB, BA, list(pairs)) # the two recordings, read for every pair, sorted by id1718for split in ("dev", "held-out"):19 in_split = [p for p in pairs if pairs[p].split == split]20 for kind in KINDS:21 # the rate's denominator is the split's pairs, so a kind with no pairs still prints as 0 of that count22 n = sum(1 for o in orders if o.kind == kind and pairs[o.pair_id].split == split)23 print(f"{kind}, {split}: {pct(n, len(in_split))}")2425for kind in KINDS:26 picked = [o.pair_id for o in orders if o.kind == kind][:3] # the first three in id order27 print(f"examples, {kind}: {', '.join(picked) if picked else 'none'}")
python pairwise-both-orders.pyconsistent, dev: 80.0% (60 of 75)
flipped, dev: 20.0% (15 of 75)
tie, dev: 0.0% (0 of 75)
unreadable, dev: 0.0% (0 of 75)
consistent, held-out: 74.7% (56 of 75)
flipped, held-out: 25.3% (19 of 75)
tie, held-out: 0.0% (0 of 75)
unreadable, held-out: 0.0% (0 of 75)
examples, consistent: hs2-pref-0001, hs2-pref-0002, hs2-pref-0005
examples, flipped: hs2-pref-0003, hs2-pref-0004, hs2-pref-0012
examples, tie: none
examples, unreadable: none
需要记录的是顺序改变的数量,而不是获胜者的数量。在 dev 划分上,裁决随顺序改变的配对数为:
20.0% (15 of 75)。
按 id 顺序排在最前面的顺序改变配对是 hs2-pref-0003, hs2-pref-0004, hs2-pref-0012。平局是第三种情况,在 dev 上其数量为零:
0.0% (0 of 75)。
解析器除了 A 和 B 之外,也能读取 TIE 一行,因此回复若给出 TIE,会被计为平局,而不是无法解析的裁决。不过成对评分标准只要求回复 A 或 B,dev 上也没有任何回复给出 TIE。这就是平局数量为零的原因。
顺序改变的配对,是指裁决跟随位置而变化的配对,因此它并不能说明哪一条回复更好。只看一种顺序的读者会把这一对计为某条回复的胜利。哪个位置更占优势,是第 7 章的主题。本章只说明为什么只用一种顺序是不够的。
工作准则: 对每一对都按两种顺序评判,在计数之前先把每个裁决映射回对应的回复,并在报告获胜者数量的同时报告顺序改变的数量。把顺序改变的配对计为无偏好,不要让它决定任何比较。如果某个数据集上顺序改变的数量很高,说明这次比较测量的是位置,而在单一顺序下增大样本量并不能解决这个问题。
这部分尚未翻译,因此以英文显示。
Verdict first
The rubric asks for the verdict on the first line and one short sentence of reason on the next. The code reads only the first non-empty line. The check is short: one line has to hold the verdict, and a reply that does not begin with a verdict is not read as one. The reason comes after the verdict, so the reader and the code can both see the pick before the explanation.
parse_verdict returns PASS or FAIL when the first non-empty line is exactly that verdict, and it ignores asterisks or backticks around the line. Anything else returns nothing, and that is an unreadable verdict. It is not a FAIL. A FAIL means the judge judged the answer wrong, and an unreadable verdict means the judge gave no judgment at all. The code has two readers of verdicts, and they treat an unreadable verdict differently. verdicts in lib/judging.py stops with an error, so a run that reads through it cannot count the reply as a FAIL. readable_verdicts in lib/calibration.py prints each unreadable reply and leaves it out of the verdicts, and so out of the denominator. Most examples that report a rate read through it. A rate that leaves replies out prints their count beside it.
Fixed replies go through the parser, one line of output each, which prints the verdict, or the label refused for a reply with no readable verdict.
1"""What lib.contract.parse_verdict accepts and refuses: the verdict on a reply's first non-empty line, PASS or FAIL in capitals,2after optional emphasis marks. Each fixed reply below prints one line, the verdict or 'refused'. A refusal is not a FAIL:3the judge has not given a verdict, and the evaluation counts that as an error, not a failed answer."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.contract import parse_verdict78CASES = (9 ("verdict first, PASS", "VERDICT: PASS\nReason: The answer matches the reference."),10 ("verdict first, FAIL", "VERDICT: FAIL\nReason: The answer gives a different version number."),11 ("emphasis around the line", "**VERDICT: FAIL**\nReason: The answer gives a different version number."),12 ("backticks around the line", "`VERDICT: PASS`\nReason: The answer matches the reference."),13 ("blank lines before the verdict", "\n\nVERDICT: PASS\nReason: The answer matches the reference."),14 ("no space after the colon", "VERDICT:PASS\nReason: The answer matches the reference."),15 ("two verdict lines, first one read", "VERDICT: FAIL\nVERDICT: PASS\nReason: Two verdicts."),16 ("lower case", "VERDICT: pass\nReason: The answer matches the reference."),17 ("title case", "Verdict: PASS\nReason: The answer matches the reference."),18 ("PARTIAL", "VERDICT: PARTIAL\nReason: Half of the answer matches."),19 ("verdict after a reason", "Reason: The answer matches the reference.\nVERDICT: PASS"),20 ("full stop after the verdict", "VERDICT: PASS.\nReason: The answer matches the reference."),21 ("words after the verdict", "VERDICT: PASS, mostly correct\nReason: The answer matches the reference."),22 ("bullet before the verdict", "- VERDICT: PASS\nReason: The answer matches the reference."),23 ("pairwise letter", "VERDICT: A\nReason: Response A is better."),24 ("bare word", "PASS\nReason: The answer matches the reference."),25 ("empty reply", ""),26 ("blank lines only", " \n\n "),27)2829for label, raw in CASES:30 verdict = parse_verdict(raw)31 print(f"{label}: {verdict if verdict is not None else 'refused'}")
python parse-verdicts.pyverdict first, PASS: PASS
verdict first, FAIL: FAIL
emphasis around the line: FAIL
backticks around the line: PASS
blank lines before the verdict: PASS
no space after the colon: PASS
two verdict lines, first one read: FAIL
lower case: refused
title case: refused
PARTIAL: refused
verdict after a reason: refused
full stop after the verdict: refused
words after the verdict: refused
bullet before the verdict: refused
pairwise letter: refused
bare word: refused
empty reply: refused
blank lines only: refused
The first lines of the output show the forms the parser accepts. A verdict on the first line is read, and so is one with asterisks or backticks around it, blank lines above it, or no space after the colon. When two verdict lines appear, the parser reads the first, so a reply that says FAIL and then PASS is read as FAIL.
The unreadable lines matter most for a rubric. Lower case, title case and a PARTIAL verdict are all unreadable. So are a verdict placed after the reason, a full stop or extra words after PASS, a bullet in front of the verdict, a pairwise letter sent to a pass-or-fail rubric, a bare PASS with no label, an empty reply, and a reply of blank lines. Each unreadable form is a reply written in a shape the rubric did not ask for. A parser that guessed at what the judge meant would be reading the judge's intent, not its verdict.
The sentence to keep from this example is that a reply the parser cannot read is not a failed answer. It is a reply with no verdict, and no rate in this book turns it into a PASS or a FAIL. The book's advice is to print the unreadable count on each split. A split with more unreadable replies than the other is one the judge formats differently, which is worth finding out before the rate is read, since it may say more about the judge than about the assistant.
Working rule: Read the first non-empty line only. Never map an unreadable verdict to PASS or FAIL. Say in each rate what the run does with an unreadable verdict, whether it stops or leaves the reply out, and print the count beside any rate that leaves replies out. Check the parser on the replies the judge actually wrote, since a rubric that works on a sample can still lose verdicts to a format it did not ask for.
在 dev 上调整措辞
评分细则的措辞是一项设置,团队总希望反复调整,直到评判器与其标注的一致程度更高。风险在于,团队在调整措辞时所依据的数字,随后又被它拿来报告。本书把各条目分成两个划分。Dev 用于调整措辞。留出集在措辞确定之后运行,且每章所报告的每一项比较只运行一次。一个已被用来选择措辞的留出划分,不再能检验该措辞。若在留出条目上调整评分细则,所报告的一致程度衡量的就是调整本身,这样的一致程度也就无法说明评判器的任何情况。
在任何调整之前,团队先用评分细则自己的语言,为数据集中的每一类条目写下规则:什么算通过,什么算不通过,拒绝回答值多少。然后逐条阅读评分细则判错的 dev 回复,只有当某条回复显示出规则未涵盖的缺口时,才修改措辞。为迁就一条回复而做的改动,就是对评分细则的改动,需要为它单独做一次录制。
第 1 章中的答案评分细则已被冻结。它的录制中保存着生成该录制的措辞的哈希。改动一个词,哈希就会改变,回放便会拒绝这份录制。因此,改动意味着整个答案评判器的运行要在改动后的措辞下,对每一条回复重新录制。本书不调整这一评分细则;想要调整它的读者,应预期要重新录制它的全部回复。
评分细则 B是校准任务的第二种措辞,它录制于 HelpSteer2。校准评分细则指出,不含错误断言的回复属于“并非不正确”。评分细则 B 则称这样的回复为“对”,并且在校准评分细则使用“正确”和“不正确”的地方,使用“对”和“错”。评分细则 B 还省去了校准评分细则中的两点,即回复不得误导读者,以及不得奖励自信的语气,并且把格式从评判器不得评分的事项中删去。它的指令段落和回复格式如下。该评分细则还有两个虚构的示例,本章略去不录。它们属于参与哈希计算的文本。
You grade whether an AI assistant's reply to a user is right.
A reply is right when every fact it states is true, every calculation or result in it is correct, and any code or reasoning in it works. Mark it wrong when any of those fail, or when it answers a different question from the one asked. A reply that makes no false claim is right, even when it is short, asks a question back, or leaves something out. Do not grade style, length or tone.
Reply in this form: on the first line, write exactly VERDICT: PASS if the reply is right, or VERDICT: FAIL if it is wrong. Then, on the next line, one short sentence of reason.
该示例会把评分细则 B 的哈希与其余四个并排打印,读者由此可以区分两种校准措辞(即校准评分细则与评分细则 B)。本章只展示文本和哈希,评分细则 B 与人工评分的一致程度如何,由第 6 章报告。
本章做出了以下决定:
- 评判器只读取第一个非空行。其他任何回复都属于无法读取的判定,绝不算作 FAIL,读取判定的两条代码路径已在上文说明。
- 每份评分细则都与其录制一同计算哈希,改动过的评分细则就是一份新的评分细则,有新的录制。
- 每一项比率都按通过与不通过计算。不使用量表,因为本书并非在每一步都有人工评分可用来核对量表。
- 成对评判器按两种顺序各运行一次,并在其胜出计数旁报告顺序改变后结果不同的配对。
工作准则: 在留出集上运行任何内容之前,先把措辞写下来,并且只在 dev 上调整。措辞必须改动时,就把它作为另一份评分细则,配以它自己的录制和哈希。每个数字都要连同产生它的措辞的哈希一起报告,这样两种措辞就不会被当作同一种来读。
本章,在本版次中
版次 2026.11.3示例于 2026年11月21日 通过本章 3个示例 个示例中有 3 个通过
译自英文版图书,版次 2026.11.3。