撰寫評審能夠一致套用的評分規準
閱讀約需 50 分鐘 分鐘,6 節 中共有 3 個範例。
讀完本章,你能寫出一份評分規準,讓評審模型以相同的方式套用在每一則回覆上;能用程式讀取評審的裁決,而且程式絕不計入任何無法判讀為裁決的內容;也能判斷成對比較或量表在何時值得付出其成本。
第 4 章結束於程式碼力有未逮之處。檢查可以確保回覆的形式無誤,也能顯示它引用了哪一節,卻無法讀懂回覆的主張,因此必須有東西拿這項主張去對照參考答案。本章建構的就是這位讀者,也就是評審:一個模型,它讀取回覆,將回覆與寫好的參考答案比對,並寫出該回覆是否正確。助理回答有關命令列工具 sablefile 的問題,第 1 章最先用到這位評審。評審是 Qwen3-4B-Instruct-2507,固定在其 GGUF 轉換版本的 commit a06e946,與回答問題的是同一個模型,因此本書並未以更強的模型來檢驗模型。評審的回覆只錄製一次,本章的每個範例都讀取這些錄製內容,所以其中的數字不會因每次執行而改變。團隊若從一位從未被檢驗過的評審報告通過率,所報告的其實是關於評審本身的數字;而讀者若沒有本章所述的檢查,光看頁面也無從分辨這兩者。
評審依據其指示運作,而指示就是評分規準。評分規準由兩段文字構成。系統文字說明任務、規則,以及回覆必須採用的形式。範本則以標記包住請求、參考答案與回覆,讓評審能夠區分它們。評分規準是量測的一部分,而不只是置於量測之前的備註,因為評審拿到的正是其中的措辭。本章說明本書如何撰寫評分規準、如何讀取每一項裁決,以及團隊必須在何處做出選擇。
此部分尚未翻譯,因此以英文呈現。
What a judge is asked
The judge's reply goes to code, and the code reads the first line of it as the verdict. The form the rubric asks for is therefore part of the rubric. A judge that writes its verdict in another shape has given the code nothing to read, and the section on verdicts below says what the code does then.
The book has five rubrics. The answer judge checks a reply about sablefile against the written reference answer for its question. The calibration rubric checks correctness with no reference, on rows from HelpSteer2, a public dataset of human ratings of other models' replies to general prompts, which chapter 6 describes. Its dataset card was read on 20 November 2026. Rubric B is a second wording of the calibration task. The pairwise rubric chooses the better of two replies. The faithfulness rubric checks whether an answer is supported by the sections it was given. Each rubric is written as text and carries a hash, and the first example prints the name and the hash of all five. For an item the docs do not cover, the reference is the word none, and the rubric is what tells the judge that a reply passes only when it is the decline sentence.
1"""The five rubrics the book's judges use, each with the sha256 its recording's manifest holds, and how many lines its system2text has. The hash is the one lib.prompts.sha256_text gives for the system text and the user template together: each3recording's manifest was made from that pair, not from the system text alone, so the check compares the pair. The first4line of each rubric is printed to show which rubric it is; the full texts are in lib/prompts.py."""5import lib # first: fixes the thread counts and seeds before anything else loads67from lib.prompts import (8 FAITH_SYSTEM, FAITH_TEMPLATE, HS2_SYSTEM, HS2_TEMPLATE, HS2B_SYSTEM, HS2B_TEMPLATE, PAIR_SYSTEM, PAIR_TEMPLATE,9 REF_SYSTEM, REF_TEMPLATE, sha256_text,10)11from lib.replay import ReplayModel1213# (the rubric's name, the recording whose manifest holds its hash, its system text, its user template)14RUBRICS = (15 ("answer judge, reference-guided", "judge-answers-baseline", REF_SYSTEM, REF_TEMPLATE),16 ("HelpSteer2 calibration rubric, reference-free", "judge-helpsteer2", HS2_SYSTEM, HS2_TEMPLATE),17 ("HelpSteer2 rubric B, second wording", "judge-helpsteer2-rubric-b", HS2B_SYSTEM, HS2B_TEMPLATE),18 ("pairwise rubric", "judge-hs2-pairs-ab", PAIR_SYSTEM, PAIR_TEMPLATE),19 ("faithfulness rubric", "judge-faithfulness-baseline", FAITH_SYSTEM, FAITH_TEMPLATE),20)2122for name, recording, system, template in RUBRICS:23 digest = sha256_text(system, template)24 # ReplayModel raises ValueError when the recording's manifest hash is not this one, so a match is checked, not assumed.25 ReplayModel(recording, digest)26 print(f"{name}: {len(system.splitlines())} lines")27 print(f" sha256 of system text and template: {digest}")28 print(f" first line: {system.splitlines()[0]}")
python rubric-text.pyanswer judge, reference-guided: 10 lines
sha256 of system text and template: 4c409dcd59d0926646c3f8f5683da5de589e1586007dcfcf79d71a6b49e7e227
first line: You check an AI assistant's answer about the sablefile tool against the written reference answer for the question.
HelpSteer2 calibration rubric, reference-free: 20 lines
sha256 of system text and template: 56d617ac94b9fa5c4f0b2ce1eee17b7e0ce42c7e6b6c617edcb55a7b48d472e6
first line: You check the correctness of an AI assistant's response to a user's request.
HelpSteer2 rubric B, second wording: 17 lines
sha256 of system text and template: 7f512725080e8e78437603f7445a0480caaa0d2a344aa51d58d5475cba2ac384
first line: You grade whether an AI assistant's reply to a user is right.
pairwise rubric: 13 lines
sha256 of system text and template: 24a735e79373f7fa0b8bf1210c3e516964d93164508c666b8d333b93c27de992
first line: You compare two AI assistant responses to the same request and say which one is better.
faithfulness rubric: 15 lines
sha256 of system text and template: a584b8d56f63c212497fa67602f2fbd2add41823ca5895ba2bad541389970555
first line: You check whether an answer about the sablefile tool is supported by the documentation sections it was given.
The output gives the first line of each system text and a hash of that text together with its template. The hash covers both because the judge is given both, and a change to either changes what the judge sees. The run reads each hash back against the recording that was made with that rubric, and it stops with an error if they differ, so a recording cannot be replayed under a rubric it was not made with. A reader who sees a number in this book can find the rubric behind it by name and by hash, and can tell that rubric apart from one with a single word changed.
The template's markers show the judge where the request, the reference and the reply each start and stop. That is what they are for. The book does not test whether they keep a reply that quotes the rubric's own instructions from being read as instructions.
Working rule: Write each rubric as text that a team can read, and hash it with the run it produces. Name it by its first line, so a reader can tell which rubric a number came from. Change a rubric's text only by making another rubric with its own recording, and never by editing the rubric that produced the numbers already on the page.
有參考答案或無參考答案
有參考答案的評審會拿到針對該問題寫好的答案,並將回覆與該答案對照檢查。無參考答案的評審只拿到請求與回覆,憑自己所知判斷回覆是否正確。參考答案是一個固定基準。評審不必認識 sablefile,因為參考答案已說明什麼才是正確的。這個固定基準同時也是限制:評審不可能比參考答案更正確,而評分準則必須說明,遇到用詞不同、補充細節或只涵蓋問題一部分的回覆,該如何處理。
答案評分準則的第一條規則,接受陳述內容與參考答案一致的回覆。回覆可以使用不同的措辭,也可以補充不與參考答案相抵觸的細節。回覆若說出參考答案無法支持的內容,即屬不正確。第三條規則處理有多種解讀的參考答案:回覆必須回答每一種解讀,只回答一部分即屬不正確。另有兩條規則,把該案例的拒答帶進評審的判斷。若參考答案是「none」這個詞,表示文件並未回答該問題,此時回覆唯有恰好是拒答句「The docs don't say.」才算正確。而回覆若給出答案,同時又說出這句拒答句,則屬不正確。第 10 章衡量這條規則所控制的兩種錯誤,評審對每一則回覆都套用同一條規則。
校準評分準則不使用參考答案,因此必須在沒有可比對之參考的情況下,說明什麼算是錯誤。它要求評審只判斷正確性,並指明評審不得評分的項目:長度、語氣、風格與格式。它還規定,沒有任何錯誤主張的回覆不算不正確,即使它並不完整,或要求提供更多資訊。這句話是一項決定。HelpSteer2 對多個屬性評分,正確性只是其中之一。評分準則只點名正確性,因此這個問題由評分準則設定,而不是留給評審自行決定。第 6 章衡量評審這個較窄的答案,與人工評分一致到什麼程度。
參考答案若有誤,評審就會判正確的回覆為不通過,而判定結果不會顯示任何跡象。團隊會把不通過比率的上升解讀為助理變弱,但實際上問題出在參考答案。在解讀比率之前,應先對照參考答案,閱讀一份不通過案例的樣本。
實用原則: 只要資料集有參考答案就使用,並像答案評分準則那樣,在評分準則中寫明參考答案為空時的規則。唯有在不存在參考答案時,才使用無參考答案的評分準則。在其文字中,要指明評審不得因哪些項目而扣減回覆的評價,例如長度、語氣或不完整,讓這項選擇由評分準則來做,而不是由評審來做。
此部分尚未翻譯,因此以英文呈現。
Pass, fail, or a scale
A pass or fail rubric gives the judge two answers. A scale gives the judge several answers. The human ratings in HelpSteer2 are on a 0 to 4 scale, and a scale can look like more information than a yes or no. Its cost is in the boundaries. The judge has to tell one point from the next, and the team has to say what each point means and then check that the judge's points agree with the human ones. A pass or fail asks for one decision per reply, and every rate in this book is a rate of such decisions.
The answer rubric and both calibration rubrics give pass or fail. The search for this chapter looked for a study that compares a scale with pass and fail on binary human labels, and it found none. So this choice is the book's design, not a finding from a source, and a reader should weigh it as a design.
The question this book asks is whether a reply is right, and that question has two answers. A reply can be almost right and still wrong on one fact or one command. A scale would give it a middle grade, and the team would then have to turn that grade into a yes or no before it could read any rate. The pass rate is the share of items the judge passes, over every item in the set, and a correct decline counts as a pass, as chapter 10 explains. A scale would need a different summary, such as a mean, and a mean hides which replies sit at which point. Take the decline item. A reply either declines or it does not. A scale would let a judge give a middle grade to a reply that declines and then adds a guess, which is the reply that chapter 10 counts as an error, and the team would then have to decide which side of the scale counts as passing.
Working rule: Use pass and fail when the team needs a yes or no on correctness, and report only rates of such decisions. Use a scale only when each step means something the team will act on, and when human ratings exist at each step to check the judge against. Say in the text which form the team chose and why, because that choice sets every rate the team reports. Chapter 6 shows how the judge is compared with human ratings.
一個答案還是兩個答案
成對判斷(pairwise judgment)會讓評審看到針對同一個請求的兩則回覆,並要求它指出哪一則較好。當團隊必須在兩個版本之間做選擇,而絕對評分又難以訂定時,這種方式很合適。成對評分準則只給評審 A 或 B 兩個字母,不設平手。它的代價在於位置。每則回覆在提示中都有一個位置,不是第一就是第二,而評審相關的研究指出,順序可能決定最後的選擇。這些研究量測的是其他評審在它們自己的設定下的表現,本書不把它們的比率當作自己的數據。第 7 章會回顧這些研究,並在這個評審身上量測其偏向。
成對評審沒有量表等級需要定義,它只需要說出兩則回覆何者較好,而這正是團隊在兩個版本之間做選擇時所問的問題。它仍然必須對照人類偏好加以檢驗,就像第 7 章對其一致的配對所做的那樣。代價是以位置來支付的,這也是本節把每一組配對都判斷兩次的原因。
成對範例使用來自 HelpSteer2 的人類偏好配對。每一組配對都判斷兩次:一次將第一則回應顯示為 A,另一次將第二則回應顯示為 A。接著把每個判決對應回它所選的回應,如此就能比較兩種順序。若同一則回應在兩種順序下都獲勝,這組配對即為一致。若同一個字母在兩種順序下都獲勝,這組配對即為順序改變,意思是位置決定了判決;範例的輸出將這類配對稱為「flipped」。平手與無法解讀則是另外兩種類型。
1"""How the pairwise judge's verdict on each HelpSteer2 preference pair depends on which response is shown as A. Each pair is2judged twice, once as ab (response_a shown as A) and once as ba (response_b shown as A), and both verdicts are mapped back3to the original responses. A flipped pair picked the same letter both times, so its verdict followed the position: the4flipped lines, on dev and on held-out, are the ones the point rests on.56Only the pair IDs of the examples are printed; the counts are the rates the chapter quotes."""7import lib # first: fixes the thread counts and seeds before anything else loads89from lib.judging import KINDS, both_orders, human_pairs10from lib.stats import pct1112AB = "judge-hs2-pairs-ab"13BA = "judge-hs2-pairs-ba"1415pairs = human_pairs() # all 150 pairs, sorted by id, each with its split16orders = both_orders(AB, BA, list(pairs)) # the two recordings, read for every pair, sorted by id1718for split in ("dev", "held-out"):19 in_split = [p for p in pairs if pairs[p].split == split]20 for kind in KINDS:21 # the rate's denominator is the split's pairs, so a kind with no pairs still prints as 0 of that count22 n = sum(1 for o in orders if o.kind == kind and pairs[o.pair_id].split == split)23 print(f"{kind}, {split}: {pct(n, len(in_split))}")2425for kind in KINDS:26 picked = [o.pair_id for o in orders if o.kind == kind][:3] # the first three in id order27 print(f"examples, {kind}: {', '.join(picked) if picked else 'none'}")
python pairwise-both-orders.pyconsistent, dev: 80.0% (60 of 75)
flipped, dev: 20.0% (15 of 75)
tie, dev: 0.0% (0 of 75)
unreadable, dev: 0.0% (0 of 75)
consistent, held-out: 74.7% (56 of 75)
flipped, held-out: 25.3% (19 of 75)
tie, held-out: 0.0% (0 of 75)
unreadable, held-out: 0.0% (0 of 75)
examples, consistent: hs2-pref-0001, hs2-pref-0002, hs2-pref-0005
examples, flipped: hs2-pref-0003, hs2-pref-0004, hs2-pref-0012
examples, tie: none
examples, unreadable: none
要保留記錄的是順序改變的數量,而不是勝者的數量。在開發集(dev split)上,判決隨順序而改變的配對數為:
20.0% (15 of 75)。
依 id 順序排列,最先出現的順序改變配對是 hs2-pref-0003, hs2-pref-0004, hs2-pref-0012。平手是第三種類型,在開發集上其數量為零:
0.0% (0 of 75)。
剖析器除了 A 與 B 之外也會讀取 TIE 這一行,因此若回覆寫的是 TIE,會被計為平手,而不是無法解讀的判決。不過成對評分準則只要求回答 A 或 B,開發集上也沒有任何回覆給出 TIE,所以平手的數量為零。
順序改變的配對,是判決跟著位置走的配對,因此它無法說明哪一則回覆較好。只看單一順序的讀者,會把這組配對算作其中一則回應的勝利。哪個位置會獲勝,是第 7 章的主題。本章只說明為什麼單一順序並不足夠。
實用原則: 每一組配對都以兩種順序判斷,計數之前先把每個判決對應回它的回應,並在勝者數量旁一併報告順序改變的數量。將順序改變的配對視為沒有偏好,不要讓它決定任何比較結果。若某個資料集上順序改變的數量偏高,代表這項比較量到的是位置,而在單一順序下增加樣本數並無法解決這個問題。
此部分尚未翻譯,因此以英文呈現。
Verdict first
The rubric asks for the verdict on the first line and one short sentence of reason on the next. The code reads only the first non-empty line. The check is short: one line has to hold the verdict, and a reply that does not begin with a verdict is not read as one. The reason comes after the verdict, so the reader and the code can both see the pick before the explanation.
parse_verdict returns PASS or FAIL when the first non-empty line is exactly that verdict, and it ignores asterisks or backticks around the line. Anything else returns nothing, and that is an unreadable verdict. It is not a FAIL. A FAIL means the judge judged the answer wrong, and an unreadable verdict means the judge gave no judgment at all. The code has two readers of verdicts, and they treat an unreadable verdict differently. verdicts in lib/judging.py stops with an error, so a run that reads through it cannot count the reply as a FAIL. readable_verdicts in lib/calibration.py prints each unreadable reply and leaves it out of the verdicts, and so out of the denominator. Most examples that report a rate read through it. A rate that leaves replies out prints their count beside it.
Fixed replies go through the parser, one line of output each, which prints the verdict, or the label refused for a reply with no readable verdict.
1"""What lib.contract.parse_verdict accepts and refuses: the verdict on a reply's first non-empty line, PASS or FAIL in capitals,2after optional emphasis marks. Each fixed reply below prints one line, the verdict or 'refused'. A refusal is not a FAIL:3the judge has not given a verdict, and the evaluation counts that as an error, not a failed answer."""4import lib # first: fixes the thread counts and seeds before anything else loads56from lib.contract import parse_verdict78CASES = (9 ("verdict first, PASS", "VERDICT: PASS\nReason: The answer matches the reference."),10 ("verdict first, FAIL", "VERDICT: FAIL\nReason: The answer gives a different version number."),11 ("emphasis around the line", "**VERDICT: FAIL**\nReason: The answer gives a different version number."),12 ("backticks around the line", "`VERDICT: PASS`\nReason: The answer matches the reference."),13 ("blank lines before the verdict", "\n\nVERDICT: PASS\nReason: The answer matches the reference."),14 ("no space after the colon", "VERDICT:PASS\nReason: The answer matches the reference."),15 ("two verdict lines, first one read", "VERDICT: FAIL\nVERDICT: PASS\nReason: Two verdicts."),16 ("lower case", "VERDICT: pass\nReason: The answer matches the reference."),17 ("title case", "Verdict: PASS\nReason: The answer matches the reference."),18 ("PARTIAL", "VERDICT: PARTIAL\nReason: Half of the answer matches."),19 ("verdict after a reason", "Reason: The answer matches the reference.\nVERDICT: PASS"),20 ("full stop after the verdict", "VERDICT: PASS.\nReason: The answer matches the reference."),21 ("words after the verdict", "VERDICT: PASS, mostly correct\nReason: The answer matches the reference."),22 ("bullet before the verdict", "- VERDICT: PASS\nReason: The answer matches the reference."),23 ("pairwise letter", "VERDICT: A\nReason: Response A is better."),24 ("bare word", "PASS\nReason: The answer matches the reference."),25 ("empty reply", ""),26 ("blank lines only", " \n\n "),27)2829for label, raw in CASES:30 verdict = parse_verdict(raw)31 print(f"{label}: {verdict if verdict is not None else 'refused'}")
python parse-verdicts.pyverdict first, PASS: PASS
verdict first, FAIL: FAIL
emphasis around the line: FAIL
backticks around the line: PASS
blank lines before the verdict: PASS
no space after the colon: PASS
two verdict lines, first one read: FAIL
lower case: refused
title case: refused
PARTIAL: refused
verdict after a reason: refused
full stop after the verdict: refused
words after the verdict: refused
bullet before the verdict: refused
pairwise letter: refused
bare word: refused
empty reply: refused
blank lines only: refused
The first lines of the output show the forms the parser accepts. A verdict on the first line is read, and so is one with asterisks or backticks around it, blank lines above it, or no space after the colon. When two verdict lines appear, the parser reads the first, so a reply that says FAIL and then PASS is read as FAIL.
The unreadable lines matter most for a rubric. Lower case, title case and a PARTIAL verdict are all unreadable. So are a verdict placed after the reason, a full stop or extra words after PASS, a bullet in front of the verdict, a pairwise letter sent to a pass-or-fail rubric, a bare PASS with no label, an empty reply, and a reply of blank lines. Each unreadable form is a reply written in a shape the rubric did not ask for. A parser that guessed at what the judge meant would be reading the judge's intent, not its verdict.
The sentence to keep from this example is that a reply the parser cannot read is not a failed answer. It is a reply with no verdict, and no rate in this book turns it into a PASS or a FAIL. The book's advice is to print the unreadable count on each split. A split with more unreadable replies than the other is one the judge formats differently, which is worth finding out before the rate is read, since it may say more about the judge than about the assistant.
Working rule: Read the first non-empty line only. Never map an unreadable verdict to PASS or FAIL. Say in each rate what the run does with an unreadable verdict, whether it stops or leaves the reply out, and print the count beside any rate that leaves replies out. Check the parser on the replies the judge actually wrote, since a rubric that works on a sample can still lose verdicts to a format it did not ask for.
此部分尚未翻譯,因此以英文呈現。
Tuning the wording on dev
The wording of a rubric is a setting, and a team will want to tune it until the judge agrees better with its labels. The risk is tuning on the numbers the team then reports. The book divides its items into two splits. Dev is where wording is tuned. Held-out is run once, for each comparison a chapter reports, after the wording is fixed. A held-out split that has been used to choose the wording no longer tests it. Tuning a rubric on held-out items turns the reported agreement into a measure of the tuning, and then the agreement says nothing about the judge.
Before any tuning, the team writes the rule for each kind of item the set holds, in the rubric's own words: what passes, what fails, and what a decline is worth. Then it reads the dev replies the rubric gets wrong, one at a time, and changes the wording only when a reply shows a gap the rule does not cover. A change made to fit one reply is a change to the rubric, and it needs a recording of its own.
The answer rubric from chapter 1 is frozen. Its recording holds the hash of the wording that made it. Changing one word changes the hash, and the replay refuses the recording. A change therefore means the whole answer judge run is recorded again, for every reply, under the changed wording. This book does not tune that rubric, and a reader who wants to tune it should expect to re-record all of its replies.
Rubric B is a second wording of the calibration task, and it is recorded on HelpSteer2. The calibration rubric says that a reply with no false claim is "not incorrect". Rubric B says that such a reply is "right", and it uses "right" and "wrong" where the calibration rubric uses "correct" and "incorrect". Rubric B also leaves out two points from the calibration rubric, that a reply must not mislead and that confidence must not be rewarded, and it drops formatting from the things the judge must not grade. Its instruction paragraph and its reply form read as follows. The rubric also has two made-up examples, which this chapter leaves out. They are part of the text that is hashed.
You grade whether an AI assistant's reply to a user is right.
A reply is right when every fact it states is true, every calculation or result in it is correct, and any code or reasoning in it works. Mark it wrong when any of those fail, or when it answers a different question from the one asked. A reply that makes no false claim is right, even when it is short, asks a question back, or leaves something out. Do not grade style, length or tone.
Reply in this form: on the first line, write exactly VERDICT: PASS if the reply is right, or VERDICT: FAIL if it is wrong. Then, on the next line, one short sentence of reason.
The example prints the hash of rubric B beside the other four, so a reader can tell the two calibration wordings apart. This chapter shows the text and the hash only, and chapter 6 reports how well rubric B agrees with the human ratings.
The chapter made these decisions:
- The judge reads the first non-empty line only. Any other reply is an unreadable verdict, never a FAIL, and the two code paths that read verdicts are described above.
- Every rubric is hashed with its recording, and a changed rubric is a new rubric with a new recording.
- Every rate is pass and fail. A scale is not used, because the book has no human ratings at each step to check one against.
- A pairwise judge is run in both orders, and its order-changed pairs are reported beside its winner counts.
Working rule: Write the wording down before you run anything on held-out, and tune it on dev only. When the wording has to change, make it another rubric with its own recording and hash. Report each number with the hash of the wording that made it, so two wordings cannot be read as one.
本章,於本版次中
版次 2026.11.3範例於 2026年11 月 21 通過測試本章 3 個範例 個範例中有 3 個通過
譯自英文書,版次 2026.11.3。