판정자가 적용할 수 있는 루브릭 쓰기

읽는 데 약 33분분 걸린다, 6절에 3개 예제이(가) 있음.

2026.11.3판 예제 통과 2026년 11월 21 54개 예제 중 54개 통과 변경 사항

AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식

영어판 2026.11.3판을 번역한 것이다. 원문 읽기

이 장을 마치면 판정자 모델이 각 응답에 똑같이 적용할 수 있는 루브릭을 쓸 수 있고, 판정으로 읽을 수 없는 것은 하나도 세지 않는 코드로 그 판정을 읽을 수 있으며, 쌍대 비교나 척도가 그 비용만큼의 가치를 갖는 경우가 언제인지 말할 수 있다.

4장은 코드가 멈추는 지점에서 끝났다. 검사는 응답의 형식을 바르게 유지하고 응답이 어느 절을 인용하는지 보여 줄 수 있지만, 응답이 무엇을 주장하는지는 읽지 못한다. 따라서 주장을 기준 답안과 대조하여 읽는 무언가가 있어야 한다. 이 장은 그 독자인 판정자를 만든다. 판정자는 응답을 읽고, 이를 작성된 기준 답안과 비교하여, 응답이 올바른지 적는 모델이다. 어시스턴트는 명령줄 도구인 sablefile에 관한 질문에 답하며, 1장에서 이 판정자를 처음 사용했다. 판정자는 Qwen3-4B-Instruct-2507이며, GGUF 변환본의 커밋 a06e946에 고정되어 있다. 이는 질문에 답하는 모델과 같은 모델이므로, 이 책은 더 강한 모델로 검증하지 않는다. 판정자의 응답은 한 번 기록되었고, 이 장의 모든 예제는 그 기록을 읽으므로 수치는 실행할 때마다 달라지지 않는다. 검증한 적 없는 판정자가 내놓은 통과율을 보고하는 팀은 사실 판정자에 관한 수치를 보고하는 것이며, 이 장이 설명하는 검사 없이는 독자가 지면만 보고 둘을 구별할 수 없다.

판정자는 자신에게 주어진 지시에 따라 작동하며, 그 지시가 루브릭이다. 루브릭은 두 개의 텍스트로 이루어진다. 시스템 텍스트는 과제, 규칙, 그리고 응답이 갖춰야 할 형식을 서술한다. 템플릿은 요청, 기준 답안, 응답을 표지로 감싸서 판정자가 이들을 구별할 수 있게 한다. 판정자에게 주어지는 것이 바로 그 문구이므로, 루브릭은 측정에 앞서 붙는 메모가 아니라 측정의 일부이다. 이 장은 이 책이 루브릭을 어떻게 쓰는지, 각 판정을 어떻게 읽는지, 그리고 팀이 어디에서 선택을 해야 하는지를 보여 준다.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

What a judge is asked

마지막으로 변경된 판: 2026.11.3판

The judge's reply goes to code, and the code reads the first line of it as the verdict. The form the rubric asks for is therefore part of the rubric. A judge that writes its verdict in another shape has given the code nothing to read, and the section on verdicts below says what the code does then.

The book has five rubrics. The answer judge checks a reply about sablefile against the written reference answer for its question. The calibration rubric checks correctness with no reference, on rows from HelpSteer2, a public dataset of human ratings of other models' replies to general prompts, which chapter 6 describes. Its dataset card was read on 20 November 2026. Rubric B is a second wording of the calibration task. The pairwise rubric chooses the better of two replies. The faithfulness rubric checks whether an answer is supported by the sections it was given. Each rubric is written as text and carries a hash, and the first example prints the name and the hash of all five. For an item the docs do not cover, the reference is the word none, and the rubric is what tells the judge that a reply passes only when it is the decline sentence.

rubric-text.py Python 3.12.15 · numpy 2.5.5 통과
"""The five rubrics the book's judges use, each with the sha256 its recording's manifest holds, and how many lines its systemtext has. The hash is the one lib.prompts.sha256_text gives for the system text and the user template together: eachrecording's manifest was made from that pair, not from the system text alone, so the check compares the pair. The firstline of each rubric is printed to show which rubric it is; the full texts are in lib/prompts.py."""import lib  # first: fixes the thread counts and seeds before anything else loadsfrom lib.prompts import (    FAITH_SYSTEM, FAITH_TEMPLATE, HS2_SYSTEM, HS2_TEMPLATE, HS2B_SYSTEM, HS2B_TEMPLATE, PAIR_SYSTEM, PAIR_TEMPLATE,    REF_SYSTEM, REF_TEMPLATE, sha256_text,)from lib.replay import ReplayModel# (the rubric's name, the recording whose manifest holds its hash, its system text, its user template)RUBRICS = (    ("answer judge, reference-guided", "judge-answers-baseline", REF_SYSTEM, REF_TEMPLATE),    ("HelpSteer2 calibration rubric, reference-free", "judge-helpsteer2", HS2_SYSTEM, HS2_TEMPLATE),    ("HelpSteer2 rubric B, second wording", "judge-helpsteer2-rubric-b", HS2B_SYSTEM, HS2B_TEMPLATE),    ("pairwise rubric", "judge-hs2-pairs-ab", PAIR_SYSTEM, PAIR_TEMPLATE),    ("faithfulness rubric", "judge-faithfulness-baseline", FAITH_SYSTEM, FAITH_TEMPLATE),)for name, recording, system, template in RUBRICS:    digest = sha256_text(system, template)    # ReplayModel raises ValueError when the recording's manifest hash is not this one, so a match is checked, not assumed.    ReplayModel(recording, digest)    print(f"{name}: {len(system.splitlines())} lines")    print(f"  sha256 of system text and template: {digest}")    print(f"  first line: {system.splitlines()[0]}")
출력python rubric-text.py
answer judge, reference-guided: 10 lines
  sha256 of system text and template: 4c409dcd59d0926646c3f8f5683da5de589e1586007dcfcf79d71a6b49e7e227
  first line: You check an AI assistant's answer about the sablefile tool against the written reference answer for the question.
HelpSteer2 calibration rubric, reference-free: 20 lines
  sha256 of system text and template: 56d617ac94b9fa5c4f0b2ce1eee17b7e0ce42c7e6b6c617edcb55a7b48d472e6
  first line: You check the correctness of an AI assistant's response to a user's request.
HelpSteer2 rubric B, second wording: 17 lines
  sha256 of system text and template: 7f512725080e8e78437603f7445a0480caaa0d2a344aa51d58d5475cba2ac384
  first line: You grade whether an AI assistant's reply to a user is right.
pairwise rubric: 13 lines
  sha256 of system text and template: 24a735e79373f7fa0b8bf1210c3e516964d93164508c666b8d333b93c27de992
  first line: You compare two AI assistant responses to the same request and say which one is better.
faithfulness rubric: 15 lines
  sha256 of system text and template: a584b8d56f63c212497fa67602f2fbd2add41823ca5895ba2bad541389970555
  first line: You check whether an answer about the sablefile tool is supported by the documentation sections it was given.

The output gives the first line of each system text and a hash of that text together with its template. The hash covers both because the judge is given both, and a change to either changes what the judge sees. The run reads each hash back against the recording that was made with that rubric, and it stops with an error if they differ, so a recording cannot be replayed under a rubric it was not made with. A reader who sees a number in this book can find the rubric behind it by name and by hash, and can tell that rubric apart from one with a single word changed.

The template's markers show the judge where the request, the reference and the reply each start and stop. That is what they are for. The book does not test whether they keep a reply that quotes the rubric's own instructions from being read as instructions.

Working rule: Write each rubric as text that a team can read, and hash it with the run it produces. Name it by its first line, so a reader can tell which rubric a number came from. Change a rubric's text only by making another rubric with its own recording, and never by editing the rubric that produced the numbers already on the page.

참조 기반 판정과 참조 없는 판정

참조 기반 판정자는 질문에 대해 작성된 정답을 함께 받고, 응답을 그 정답에 비추어 확인한다. 참조 없는 판정자는 요청과 응답만 받고, 자신이 아는 바에 따라 응답이 옳은지 판단한다. 참조는 고정점이다. 참조가 무엇이 옳은지 알려 주므로 판정자는 sablefile을 알 필요가 없다. 그러나 같은 고정점은 한계이기도 하다. 판정자는 참조보다 더 정확할 수 없으며, 다른 표현을 쓰거나 세부 내용을 덧붙이거나 질문의 일부만 다루는 응답을 어떻게 처리할지는 평가 기준이 정해 주어야 한다.

정답 평가 기준의 첫째 규칙은 참조가 진술하는 바를 똑같이 진술하는 응답을 받아들이는 것이다. 응답은 다른 표현을 쓸 수 있고, 참조와 모순되지 않는 세부 내용을 덧붙일 수 있다. 참조가 뒷받침하지 않는 내용을 말하는 응답은 틀린 것이다. 셋째 규칙은 여러 가지로 읽히는 참조를 다룬다. 응답은 모든 해석에 답해야 하며, 부분적인 답은 틀린 것이다. 나머지 두 규칙은 사례의 거절을 판정자에게 옮겨 온다. 참조가 "none"이라는 단어이면 문서가 그 질문에 답하지 않는다는 뜻이며, 응답이 정확히 거절 문장인 "The docs don't say."일 때만 옳다. 또 답을 제시하면서 거절 문장도 함께 말하는 응답은 틀린 것이다. 10장은 이 규칙이 통제하는 두 종류의 오류를 측정하며, 판정자는 모든 응답에 같은 규칙을 적용한다.

보정 평가 기준은 참조가 없으므로, 비교할 참조 없이 무엇이 틀린 것인지를 스스로 말해야 한다. 이 기준은 판정자에게 정확성만 판단하라고 지시하고, 판정자가 평가해서는 안 되는 것으로 길이, 어조, 문체, 서식을 명시한다. 또 거짓 주장을 하지 않은 응답은 불완전하거나 추가 정보를 요청하더라도 틀린 것이 아니라고 밝힌다. 이 문장은 하나의 결정이다. HelpSteer2는 여러 속성을 평가하며 정확성은 그중 하나다. 이 기준은 정확성만을 명시하므로, 그 질문은 판정자에게 맡겨지지 않고 평가 기준 안에서 정해진다. 6장은 판정자의 더 좁은 답이 사람의 평가와 얼마나 일치하는지를 측정한다.

참조가 틀리면 판정자는 올바른 응답을 불합격 처리하고, 판정 결과에는 그 징후가 나타나지 않는다. 결함이 참조에 있는데도 팀은 불합격의 증가를 어시스턴트가 나빠진 것으로 읽는다. 불합격 비율을 해석하기 전에, 불합격 사례의 표본을 참조와 대조하며 읽어야 한다.

작업 규칙: 데이터 집합에 참조가 있으면 언제나 참조를 사용하고, 정답 평가 기준처럼 참조가 비어 있을 때의 규칙을 평가 기준에 써 넣는다. 참조 없는 평가 기준은 참조가 존재하지 않을 때만 사용한다. 그 본문에는 길이, 어조, 불완전함처럼 판정자가 응답에 불리하게 계산해서는 안 되는 것을 명시하여, 그 선택을 판정자가 아니라 평가 기준이 하게 한다.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Pass, fail, or a scale

마지막으로 변경된 판: 2026.11.3판

A pass or fail rubric gives the judge two answers. A scale gives the judge several answers. The human ratings in HelpSteer2 are on a 0 to 4 scale, and a scale can look like more information than a yes or no. Its cost is in the boundaries. The judge has to tell one point from the next, and the team has to say what each point means and then check that the judge's points agree with the human ones. A pass or fail asks for one decision per reply, and every rate in this book is a rate of such decisions.

The answer rubric and both calibration rubrics give pass or fail. The search for this chapter looked for a study that compares a scale with pass and fail on binary human labels, and it found none. So this choice is the book's design, not a finding from a source, and a reader should weigh it as a design.

The question this book asks is whether a reply is right, and that question has two answers. A reply can be almost right and still wrong on one fact or one command. A scale would give it a middle grade, and the team would then have to turn that grade into a yes or no before it could read any rate. The pass rate is the share of items the judge passes, over every item in the set, and a correct decline counts as a pass, as chapter 10 explains. A scale would need a different summary, such as a mean, and a mean hides which replies sit at which point. Take the decline item. A reply either declines or it does not. A scale would let a judge give a middle grade to a reply that declines and then adds a guess, which is the reply that chapter 10 counts as an error, and the team would then have to decide which side of the scale counts as passing.

Working rule: Use pass and fail when the team needs a yes or no on correctness, and report only rates of such decisions. Use a scale only when each step means something the team will act on, and when human ratings exist at each step to check the judge against. Say in the text which form the team chose and why, because that choice sets every rate the team reports. Chapter 6 shows how the judge is compared with human ratings.

답이 하나인가 둘인가

쌍별 판정은 같은 요청에 대한 두 응답을 판정자에게 보여 주고 어느 쪽이 더 나은지 묻는다. 절대 점수를 매기기 어려울 때 두 버전 중 하나를 골라야 하는 팀에 알맞다. 쌍별 평가 기준은 판정자에게 A 또는 B라는 두 글자만 주며, 무승부는 없다. 이 방식의 대가는 위치다. 각 응답은 프롬프트에서 첫째 또는 둘째라는 자리를 차지하며, 판정자를 다룬 연구들은 순서가 선택을 좌우할 수 있다고 보고한다. 그 연구들은 다른 판정자를 각자의 환경에서 측정한 것이므로, 이 책은 그 비율을 그대로 가져오지 않는다. 7장은 이 연구들을 검토하고 이 판정자의 쏠림을 직접 측정한다.

쌍별 판정자에게는 정의해야 할 척도 단계가 없다. 두 응답 중 어느 쪽이 더 나은지만 말하면 되며, 이는 두 버전 중 하나를 고르려는 팀이 묻는 바로 그 질문이다. 그래도 7장이 일관된 쌍에 대해 하듯이, 사람의 선호와 대조하여 검증해야 한다. 대가는 위치로 치르게 되며, 그래서 이 절에서는 모든 쌍을 두 번씩 판정한다.

쌍별 예제는 HelpSteer2의 사람 선호 쌍을 사용한다. 각 쌍은 두 번 판정한다. 한 번은 첫 번째 응답을 A로 보여 주고, 다른 한 번은 두 번째 응답을 A로 보여 준다. 그런 다음 각 판정을 그것이 고른 응답으로 되돌려 매핑하여 두 순서를 비교할 수 있게 한다. 두 순서에서 같은 응답이 이기면 그 쌍은 일관된 쌍이다. 두 순서에서 같은 글자가 이기면 순서 변경 쌍이다. 이는 위치가 판정을 결정했다는 뜻이며, 예제의 출력에서는 이를 flipped라고 부른다. 무승부와 판독 불가가 나머지 두 종류다.

pairwise-both-orders.py Python 3.12.15 · numpy 2.5.5 통과
"""How the pairwise judge's verdict on each HelpSteer2 preference pair depends on which response is shown as A. Each pair isjudged twice, once as ab (response_a shown as A) and once as ba (response_b shown as A), and both verdicts are mapped backto the original responses. A flipped pair picked the same letter both times, so its verdict followed the position: theflipped lines, on dev and on held-out, are the ones the point rests on.Only the pair IDs of the examples are printed; the counts are the rates the chapter quotes."""import lib  # first: fixes the thread counts and seeds before anything else loadsfrom lib.judging import KINDS, both_orders, human_pairsfrom lib.stats import pctAB = "judge-hs2-pairs-ab"BA = "judge-hs2-pairs-ba"pairs = human_pairs()  # all 150 pairs, sorted by id, each with its splitorders = both_orders(AB, BA, list(pairs))  # the two recordings, read for every pair, sorted by idfor split in ("dev", "held-out"):    in_split = [p for p in pairs if pairs[p].split == split]    for kind in KINDS:        # the rate's denominator is the split's pairs, so a kind with no pairs still prints as 0 of that count        n = sum(1 for o in orders if o.kind == kind and pairs[o.pair_id].split == split)        print(f"{kind}, {split}: {pct(n, len(in_split))}")for kind in KINDS:    picked = [o.pair_id for o in orders if o.kind == kind][:3]  # the first three in id order    print(f"examples, {kind}: {', '.join(picked) if picked else 'none'}")
출력python pairwise-both-orders.py
consistent, dev: 80.0% (60 of 75)
flipped, dev: 20.0% (15 of 75)
tie, dev: 0.0% (0 of 75)
unreadable, dev: 0.0% (0 of 75)
consistent, held-out: 74.7% (56 of 75)
flipped, held-out: 25.3% (19 of 75)
tie, held-out: 0.0% (0 of 75)
unreadable, held-out: 0.0% (0 of 75)
examples, consistent: hs2-pref-0001, hs2-pref-0002, hs2-pref-0005
examples, flipped: hs2-pref-0003, hs2-pref-0004, hs2-pref-0012
examples, tie: none
examples, unreadable: none

세어 둘 값은 승자 수가 아니라 순서 변경 수다. dev 분할에서 순서에 따라 판정이 바뀐 쌍의 수는 다음과 같다.

20.0% (15 of 75).

id 순서로 처음 나오는 순서 변경 쌍은 hs2-pref-0003, hs2-pref-0004, hs2-pref-0012이다. 무승부는 세 번째 종류이며, dev에서 그 수는 0이다.

0.0% (0 of 75).

파서는 A와 B뿐 아니라 TIE 줄도 읽으므로, TIE라고 답한 응답은 판독 불가가 아니라 무승부로 센다. 그러나 쌍별 평가 기준은 A 또는 B만 요구하고, dev에서는 TIE를 낸 응답이 없다. 무승부 수가 0인 이유가 이것이다.

순서 변경 쌍은 판정이 위치를 따라간 쌍이므로, 어느 응답이 더 나은지에 대해서는 아무것도 말해 주지 않는다. 한 가지 순서만 본 독자는 그 쌍을 한 응답의 승리로 셀 것이다. 어느 위치가 이기는지는 7장의 주제다. 이 장은 한 가지 순서로는 충분하지 않은 이유만 보인다.

작업 규칙: 모든 쌍을 두 순서로 판정하고, 세기 전에 각 판정을 해당 응답으로 되돌려 매핑하며, 승자 수 옆에 순서 변경 수를 함께 보고한다. 순서 변경 쌍은 선호 없음으로 세고, 비교의 결과를 좌우하게 두지 않는다. 어떤 집합에서 순서 변경 수가 높다면 그 비교는 위치를 측정하고 있는 것이며, 한 가지 순서로 표본을 늘려도 이는 해결되지 않는다.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Verdict first

The rubric asks for the verdict on the first line and one short sentence of reason on the next. The code reads only the first non-empty line. The check is short: one line has to hold the verdict, and a reply that does not begin with a verdict is not read as one. The reason comes after the verdict, so the reader and the code can both see the pick before the explanation.

parse_verdict returns PASS or FAIL when the first non-empty line is exactly that verdict, and it ignores asterisks or backticks around the line. Anything else returns nothing, and that is an unreadable verdict. It is not a FAIL. A FAIL means the judge judged the answer wrong, and an unreadable verdict means the judge gave no judgment at all. The code has two readers of verdicts, and they treat an unreadable verdict differently. verdicts in lib/judging.py stops with an error, so a run that reads through it cannot count the reply as a FAIL. readable_verdicts in lib/calibration.py prints each unreadable reply and leaves it out of the verdicts, and so out of the denominator. Most examples that report a rate read through it. A rate that leaves replies out prints their count beside it.

Fixed replies go through the parser, one line of output each, which prints the verdict, or the label refused for a reply with no readable verdict.

parse-verdicts.py Python 3.12.15 · numpy 2.5.5 통과
"""What lib.contract.parse_verdict accepts and refuses: the verdict on a reply's first non-empty line, PASS or FAIL in capitals,after optional emphasis marks. Each fixed reply below prints one line, the verdict or 'refused'. A refusal is not a FAIL:the judge has not given a verdict, and the evaluation counts that as an error, not a failed answer."""import lib  # first: fixes the thread counts and seeds before anything else loadsfrom lib.contract import parse_verdictCASES = (    ("verdict first, PASS", "VERDICT: PASS\nReason: The answer matches the reference."),    ("verdict first, FAIL", "VERDICT: FAIL\nReason: The answer gives a different version number."),    ("emphasis around the line", "**VERDICT: FAIL**\nReason: The answer gives a different version number."),    ("backticks around the line", "`VERDICT: PASS`\nReason: The answer matches the reference."),    ("blank lines before the verdict", "\n\nVERDICT: PASS\nReason: The answer matches the reference."),    ("no space after the colon", "VERDICT:PASS\nReason: The answer matches the reference."),    ("two verdict lines, first one read", "VERDICT: FAIL\nVERDICT: PASS\nReason: Two verdicts."),    ("lower case", "VERDICT: pass\nReason: The answer matches the reference."),    ("title case", "Verdict: PASS\nReason: The answer matches the reference."),    ("PARTIAL", "VERDICT: PARTIAL\nReason: Half of the answer matches."),    ("verdict after a reason", "Reason: The answer matches the reference.\nVERDICT: PASS"),    ("full stop after the verdict", "VERDICT: PASS.\nReason: The answer matches the reference."),    ("words after the verdict", "VERDICT: PASS, mostly correct\nReason: The answer matches the reference."),    ("bullet before the verdict", "- VERDICT: PASS\nReason: The answer matches the reference."),    ("pairwise letter", "VERDICT: A\nReason: Response A is better."),    ("bare word", "PASS\nReason: The answer matches the reference."),    ("empty reply", ""),    ("blank lines only", "  \n\n   "),)for label, raw in CASES:    verdict = parse_verdict(raw)    print(f"{label}: {verdict if verdict is not None else 'refused'}")
출력python parse-verdicts.py
verdict first, PASS: PASS
verdict first, FAIL: FAIL
emphasis around the line: FAIL
backticks around the line: PASS
blank lines before the verdict: PASS
no space after the colon: PASS
two verdict lines, first one read: FAIL
lower case: refused
title case: refused
PARTIAL: refused
verdict after a reason: refused
full stop after the verdict: refused
words after the verdict: refused
bullet before the verdict: refused
pairwise letter: refused
bare word: refused
empty reply: refused
blank lines only: refused

The first lines of the output show the forms the parser accepts. A verdict on the first line is read, and so is one with asterisks or backticks around it, blank lines above it, or no space after the colon. When two verdict lines appear, the parser reads the first, so a reply that says FAIL and then PASS is read as FAIL.

The unreadable lines matter most for a rubric. Lower case, title case and a PARTIAL verdict are all unreadable. So are a verdict placed after the reason, a full stop or extra words after PASS, a bullet in front of the verdict, a pairwise letter sent to a pass-or-fail rubric, a bare PASS with no label, an empty reply, and a reply of blank lines. Each unreadable form is a reply written in a shape the rubric did not ask for. A parser that guessed at what the judge meant would be reading the judge's intent, not its verdict.

The sentence to keep from this example is that a reply the parser cannot read is not a failed answer. It is a reply with no verdict, and no rate in this book turns it into a PASS or a FAIL. The book's advice is to print the unreadable count on each split. A split with more unreadable replies than the other is one the judge formats differently, which is worth finding out before the rate is read, since it may say more about the judge than about the assistant.

Working rule: Read the first non-empty line only. Never map an unreadable verdict to PASS or FAIL. Say in each rate what the run does with an unreadable verdict, whether it stops or leaves the reply out, and print the count beside any rate that leaves replies out. Check the parser on the replies the judge actually wrote, since a rubric that works on a sample can still lose verdicts to a format it did not ask for.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Tuning the wording on dev

마지막으로 변경된 판: 2026.11.3판

The wording of a rubric is a setting, and a team will want to tune it until the judge agrees better with its labels. The risk is tuning on the numbers the team then reports. The book divides its items into two splits. Dev is where wording is tuned. Held-out is run once, for each comparison a chapter reports, after the wording is fixed. A held-out split that has been used to choose the wording no longer tests it. Tuning a rubric on held-out items turns the reported agreement into a measure of the tuning, and then the agreement says nothing about the judge.

Before any tuning, the team writes the rule for each kind of item the set holds, in the rubric's own words: what passes, what fails, and what a decline is worth. Then it reads the dev replies the rubric gets wrong, one at a time, and changes the wording only when a reply shows a gap the rule does not cover. A change made to fit one reply is a change to the rubric, and it needs a recording of its own.

The answer rubric from chapter 1 is frozen. Its recording holds the hash of the wording that made it. Changing one word changes the hash, and the replay refuses the recording. A change therefore means the whole answer judge run is recorded again, for every reply, under the changed wording. This book does not tune that rubric, and a reader who wants to tune it should expect to re-record all of its replies.

Rubric B is a second wording of the calibration task, and it is recorded on HelpSteer2. The calibration rubric says that a reply with no false claim is "not incorrect". Rubric B says that such a reply is "right", and it uses "right" and "wrong" where the calibration rubric uses "correct" and "incorrect". Rubric B also leaves out two points from the calibration rubric, that a reply must not mislead and that confidence must not be rewarded, and it drops formatting from the things the judge must not grade. Its instruction paragraph and its reply form read as follows. The rubric also has two made-up examples, which this chapter leaves out. They are part of the text that is hashed.

You grade whether an AI assistant's reply to a user is right.

A reply is right when every fact it states is true, every calculation or result in it is correct, and any code or reasoning in it works. Mark it wrong when any of those fail, or when it answers a different question from the one asked. A reply that makes no false claim is right, even when it is short, asks a question back, or leaves something out. Do not grade style, length or tone.

Reply in this form: on the first line, write exactly VERDICT: PASS if the reply is right, or VERDICT: FAIL if it is wrong. Then, on the next line, one short sentence of reason.

The example prints the hash of rubric B beside the other four, so a reader can tell the two calibration wordings apart. This chapter shows the text and the hash only, and chapter 6 reports how well rubric B agrees with the human ratings.

The chapter made these decisions:

  • The judge reads the first non-empty line only. Any other reply is an unreadable verdict, never a FAIL, and the two code paths that read verdicts are described above.
  • Every rubric is hashed with its recording, and a changed rubric is a new rubric with a new recording.
  • Every rate is pass and fail. A scale is not used, because the book has no human ratings at each step to check one against.
  • A pairwise judge is run in both orders, and its order-changed pairs are reported beside its winner counts.

Working rule: Write the wording down before you run anything on held-out, and tune it on dev only. When the wording has to change, make it another rubric with its own recording and hash. Report each number with the hash of the wording that made it, so two wordings cannot be read as one.

이 판의 이 장

2026.11.3판예제 통과 2026년 11월 21이 장에서 3개 예제개 중 3개가 통과한다

영어판 2026.11.3판을 번역한 것이다.

이 절 인용

형식

각 형식은 이 번역이 따르는 영문 원서를 출처로 표기한다.

모든 형식은 이 판을 명시한다. 링크는 현재 판을 열며, 여기서 절의 제목은 본문이 마지막으로 바뀐 시점을 알려 준다. 접근 날짜와 검색 날짜는 이 자료를 여는 날짜다.

이 페이지의 내용

11장 중 5장: 판정자가 적용할 수 있는 루브릭 쓰기

  1. What a judge is asked
  2. 참조 기반 판정과 참조 없는 판정
  3. Pass, fail, or a scale
  4. 답이 하나인가 둘인가
  5. Verdict first
  6. Tuning the wording on dev

맨 위로 돌아가기전체 장