找出評審的判定在何處不再可靠

閱讀約需 37 分鐘 分鐘,4 節 中共有 3 個範例。

版次 2026.11.3 範例於 2026年11 月 21 通過測試 54 個範例中有 54 個通過 變更內容

由 AI 翻譯。經人工以英文驗證。 我們如何製作書籍

譯自英文書,版次 2026.11.3。 閱讀原文

讀完本章,你能針對本書的資料所能呈現的兩種偏差,也就是位置偏差與長度偏差,對評審進行檢驗。你能說明每項檢驗顯示了什麼、又留下什麼未決的問題,並能寫下在何種條件下你不再信任評審的某一項判定。

第 6 章以人工評分為基準,衡量評審的錯誤,並在結尾提出一個問題:成對比較的評審在兩則回覆互換位置時會有什麼表現。本章要問的是,在評審所評分的 sablefile 回覆中,還有什麼會左右判定。評審閱讀對測試問題的回覆,將它與參考答案對照,再輸出 PASS 或 FAIL,這正是第 5 章所設定的流程。通過率是團隊會對外報告的數字,而只有當判定取決於回覆本身時,這個數字才值得報告。若判定會因兩則答案互換位置而改變,或通過率隨著回覆長度上升,而較長的回覆其實並未多說什麼,那麼它衡量的就不是答案本身。建立在這種判定上的比率,看起來可能像是進步,實際上卻並非如此。

有兩項提醒適用於整章,因此在此一次說明。評審是 Qwen3-4B-Instruct-2507,本章的每一個比率都是在這個模型上量測的。本章引用的已發表數據來自其他評審,它們顯示的是評審可能發生什麼事,而不是這個評審實際上做了什麼。第二項提醒關於規模。經人工評分的成對回覆與開發集項目都是小型資料集,因此每個比率都附上其計數,每項比較也都附上區間。

本章依序討論三項限制。第一是位置:成對比較的判定是否取決於哪一則答案先被呈現。第二是長度:通過率是否隨回覆的長度而上升。第三是自我偏好:評審是否偏愛由其自身模型所撰寫的答案。本書能以自己的錄製資料檢驗前兩項,因為這些資料包含檢驗所需的內容:每一組經人工評分的成對回覆都以兩種順序判定過,而每則回覆的長度都可以計數。第三項則無法檢驗,相關的節會說明原因。最後一節把這三項限制轉化為讀者可以應用的規則,每條規則都對應一個計數。

位置

向評審詢問兩則回覆何者較佳,它的答案可能取決於它先讀到哪一則回覆。若團隊只採用一種順序,就看不出這種相依性,其報告也會連同答案一併帶入該順序的影響。

這項檢驗很容易說明。每一組配對各評審兩次,一次依某一順序呈現回覆,一次對調順序。若判定跟隨它偏好的那則回覆,即為一致。若判定跟隨的是位置,順序一變判定就跟著變,這表示評審的選擇不僅取決於回覆的內容,也取決於回覆擺在哪個位置。評審可能偏向它先讀到的回覆,也可能偏向後讀到的回覆。這項檢驗只在順序改變時才顯現這種偏向;它無法說明偏向的成因,也不需要說明,因為順序改變的判定,讀者本就不應照單全收。第 5 章建立了兩種順序與判定的做法,第 6 章則以人工評分檢查評審的選擇。

每一組經人工評分的配對都以兩種順序評審,並歸入四種類型之一:一致、順序改變、平手或無法讀取。範例的輸出將順序改變的配對稱為「flipped」(翻轉)。無法讀取的配對,是指在任一順序中缺少判定。對於順序改變的配對,要問的是哪一個位置在兩種順序中都獲勝。

position-flips.py Python 3.12.15 · numpy 2.5.5 通過
"""How the pairwise judge's verdict follows the position it was shown in, on the HelpSteer2 pairs. Each pair is judged in bothorders and sorted as consistent, flipped, tie or unreadable. The flipped pairs are split by the position that won both orders,and the consistent pairs are checked against the human preference."""import lib  # first: fixes the thread counts and seeds before anything else loadsfrom lib.bias import flip_directionfrom lib.judging import both_orders, human_pairsfrom lib.stats import bootstrap_mean, interval_pct, pctAB = "judge-hs2-pairs-ab"BA = "judge-hs2-pairs-ba"SPLITS = ("dev", "held-out")pairs = human_pairs()  # all 150 pairs, sorted by id, each with its split and the human's preferred responseorders = both_orders(AB, BA, list(pairs))  # every pair judged in both orders, sorted by idfor split in SPLITS:    in_split = [o for o in orders if pairs[o.pair_id].split == split]    for kind in ("consistent", "flipped", "tie", "unreadable"):        # the denominator is the split's pairs, so the counts add up to the split's size        print(f"{kind}, {split}: {sum(1 for o in in_split if o.kind == kind)} of {len(in_split)} pairs")for split in SPLITS:    flipped = [o for o in orders if o.kind == "flipped" and pairs[o.pair_id].split == split]    # the letter A's position won both times when flip_direction says 'first'; B's when it says 'second'    first = sum(1 for o in flipped if flip_direction(o) == "first")    print(f"flipped, first position won both orders, {split}: {pct(first, len(flipped))}")# both splits pooled, one 0 or 1 per flipped pair: 1.0 when the first position won both ordersflipped_all = [1.0 if flip_direction(o) == "first" else 0.0 for o in orders if o.kind == "flipped"]print(f"flipped, first position won both orders, both splits: {pct(int(sum(flipped_all)), len(flipped_all))}")lo, hi = bootstrap_mean(flipped_all)  # percentile bootstrap over the flipped pairs, as every interval in the bookprint(f"first-position share interval, both splits: {interval_pct(lo, hi)}")for split in SPLITS:    consistent = [o for o in orders if o.kind == "consistent" and pairs[o.pair_id].split == split]    # in a consistent pair the two orders agree, so ab is the original response the judge picked    agree = sum(1 for o in consistent if o.ab == pairs[o.pair_id].preferred)    print(f"consistent pairs agreeing with the human, {split}: {pct(agree, len(consistent))}")
輸出python position-flips.py
consistent, dev: 60 of 75 pairs
flipped, dev: 15 of 75 pairs
tie, dev: 0 of 75 pairs
unreadable, dev: 0 of 75 pairs
consistent, held-out: 56 of 75 pairs
flipped, held-out: 19 of 75 pairs
tie, held-out: 0 of 75 pairs
unreadable, held-out: 0 of 75 pairs
flipped, first position won both orders, dev: 20.0% (3 of 15)
flipped, first position won both orders, held-out: 26.3% (5 of 19)
flipped, first position won both orders, both splits: 23.5% (8 of 34)
first-position share interval, both splits: [8.8%, 38.2%]
consistent pairs agreeing with the human, dev: 81.7% (49 of 60)
consistent pairs agreeing with the human, held-out: 83.9% (47 of 56)

大多數判定在兩種順序下維持不變。在 dev 上,60 of 75 pairs 組一致,15 of 75 pairs 組隨順序改變。在 held-out 上,56 of 75 pairs 組一致,19 of 75 pairs 組隨順序改變。

平手算是可讀取的判定,因此仍留在順序改變率的分母中。此處 dev 上的平手為 0 of 75 pairs 組,無法讀取的判定為 0 of 75 pairs 組,所以每一組 dev 配對在兩種順序下都有判定。

合併兩個資料切分,順序改變率為 22.7% (34 of 150)。其分母是在兩種順序下都讀得到判定的配對,因此任一順序中判定無法讀取的配對不計入。

順序改變偏向第二個位置,而由於各切分的計數很小,應以合併後的計數為準。在兩個切分合計下,第一個位置在順序改變中所占的比例為 23.5% (8 of 34),其區間為 [8.8%, 38.2%],且整個區間都落在二分之一以下。因此就這些配對而言,合併後的計數足以把這種偏向與均勻分布區分開來。合併後的比例建立在為數不多的順序改變之上,只要少數幾組往任一方向移動,就會使它變動數個百分點。僅看 dev,該比例為 20.0% (3 of 15),而在 held-out 上為 26.3% (5 of 19)。

其他研究在其他評審身上也發現了這種偏向,而且偏向的方向並不一致。Wang 等人(2023)是一篇預印本,報告在使用不同模型的設定中,GPT-4 偏好第一個位置,ChatGPT 則偏好第二個位置。Zheng 等人(2023)發表於 NeurIPS 2023 Datasets and Benchmarks 軌,報告 GPT-4 在預設提示下,於 65.0% 的比較中,兩種順序給出相同的判定。這些都是針對其他評審的已發表設定,它們告訴讀者該量測什麼,卻無法說明對一個 4B 評審該預期什麼。

一致的配對,其判定才能視為一種選擇,因此範例將它們與人類偏好對照檢查。在 dev 上,評審在一致的配對中與人類相符的比率為:

81.7% (49 of 60)。

在 held-out 上,該比率為:

83.9% (47 of 56)。

順序改變的配對得不到這樣的檢查,因為順序改變的判定並未指出評審選擇了哪一則回覆。若團隊只報告一致配對的符合程度,量到的是資料集中爭議較小的部分。順序改變的部分才是評審的選擇有疑義之處,需要另行計數。

這種偏向對團隊如何進行比較有實際的影響。假設團隊總是把候選回覆放在第二個位置,而且從不對調。若評審偏向第二個位置,候選回覆就會得到與其答案無關的加分。總是把候選回覆放在第一位的團隊,則會得到相反的誤差。每一組配對都對調順序,並報告順序改變的情形,是把這種偏向從比較中排除,而不是寄望它並不存在。

**實用原則:**在閱讀任何成對比較的判定之前,先以兩種順序各評審一次,並在判定計數旁報告順序改變率。順序改變的配對,應視為對任一則回覆都沒有判定,而不是算作先出現或後出現的那一則獲勝。當順序改變偏向某一方時,要說明偏向哪一方、有多少組、區間為何,而不要假設其他評審也偏向同一方。

此部分尚未翻譯,因此以英文呈現。

Length

最近一次變動於第 2026.11.3 版

Length asks a different question. Position is about which reply the judge sees first; length is about the reply itself. The question is whether the judge gives a longer reply a higher score for being longer. Published studies describe that bias, so a test in this book has to look for it, and a clean result would need a test that could have shown the bias.

Zheng and colleagues (2023) reported that all the models they tested may be prone to verbosity bias, and that GPT-4 defends against it much better than the others. In one attack, a repetitive list was added to 23 answers. The failure rate was 91.3% for two of the models and 8.7% for GPT-4. Those figures come from other judges and a different attack. They are a reason to measure length here, and they say nothing about the size of the effect in this book.

The test is simpler than the attack. It takes the dev replies that the baseline assistant answered, sorts them into three bands by word count, and prints the judge's pass rate for each band. The edges that set the bands are fixed in the example. Other edges would make other bands, and the picture could change with them.

length-and-verdicts.py Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0 通過
"""The reference-guided judge's PASS rate by the length of the assistant's reply, on the dev items the baseline answered. Thereply lengths are in words, in three bands with the edges printed below. Each band's rate is printed with its count. Declinedreplies and unreadable verdicts are left out, and each is printed with its count."""import lib  # first: fixes the thread counts and seeds before anything else loadsfrom lib.bias import LENGTH_EDGES, band_outcomes, read_verdicts, reply_lengthsfrom lib.stats import pctfrom lib.testset import load_itemsASSISTANT = "assistant-baseline"  # the reply the judge read, and the one whose length is measuredJUDGE = "judge-answers-baseline"dev = [item.id for item in load_items(split="dev")]  # 80 dev items, sorted by idlengths = reply_lengths(ASSISTANT, dev)  # the answered items only: declined replies are left out hereverdicts = read_verdicts(JUDGE, dev)  # None where the judge's first line is not a verdictprint(f"band edges, in words: {LENGTH_EDGES[0]} and {LENGTH_EDGES[1]}")print(f"declined, left out: {len(dev) - len(lengths)} of {len(dev)} dev items")unreadable = [item_id for item_id in lengths if verdicts[item_id] is None]print(f"unreadable verdict, left out: {', '.join(unreadable) if unreadable else 'none'}")for band, outcomes in band_outcomes(lengths, verdicts):    print(f"{band}: {pct(int(outcomes.sum()), outcomes.size)}")
輸出python length-and-verdicts.py
band edges, in words: 15 and 30
declined, left out: 23 of 80 dev items
unreadable verdict, left out: q-0043
under 15 words: 94.1% (16 of 17)
15 to 29 words: 83.3% (15 of 18)
30 words or more: 85.7% (18 of 21)

The edges fall at 15 and 30 words. Declined replies are left out of the bands, which is 23 of 80 dev items. The judge's first line is read as its verdict. When that line is not a verdict, the verdict is unreadable, and the example prints such an item and leaves it out of the bands. It is never counted as FAIL. Here the unreadable verdict is on q-0043.

The bands are built from answered replies only, so their denominators are smaller than the set. The second-model comparison in the last section uses a different set of items, counting every dev item the judge read for both assistants and counting a correct decline as a pass under the answer rubric in Chapter 5. The two sets are not the same, and their rates should not be set side by side.

The rates do not climb with length. The middle band passed least often, and the longest band passed more often than the middle one, though less often than the shortest. A reader who fits one straight line through three bands would get a slope, and these bands do not support that line.

The shortest band passed 94.1% (16 of 17).

The middle band passed 83.3% (15 of 18).

The longest band passed 85.7% (18 of 21).

The comparison the book reports is the longest band minus the shortest. Its order is fixed by length, so it is set before anyone reads the rates. The highest-rate band minus the lowest-rate band would be chosen after the rates are seen, and it could never fall below zero. The gap is -8.4 points, interval [-28.6, +8.1] points.

The method is percentile bootstrap over items, each band resampled alone, 2000 resamples, 95% level. The two bands hold different items, so each is resampled on its own, and the bands' counts are the ones in their rates above. The band counts are small, so the interval is wide. The interval includes zero, so the result is no difference detected between the longest and the shortest band.

No difference detected does not mean that length is harmless. It says that bands of this size cannot separate a gap from zero. They cannot pick out a few points of bias, because one reply moves a band's rate by several points. The bootstrap over items that intervals and sample size describes is the same resampling. The two bands here are resampled on their own, an interval for two separate groups, which that chapter does not build.

A longer reply is also not always a padded one. A reply that covers more of the question may pass more often for that reason, and a rate that climbs with length would not, by itself, show padding. Telling the two apart needs replies in which length and content come apart, such as a complete answer and the same answer with filler added. The bands here cannot separate them, because the replies in each band differ in more than their length.

Working rule: Don't name a length effect, or its absence, from a few bands. Compare the longest band with the shortest, with the gap's interval and each band's count. When the interval includes zero, say "no difference detected" with those counts, and stop there. If length could matter to your product, build a test in which length changes and the content stays the same, and measure that.

自我偏好,本書無法檢驗的問題

長度一節探討的是評審是否因回覆篇幅較長而給予較高評價。本節要問的是,評審是否因回覆由誰撰寫而給予較高評價。本書無法回答這個問題,而本書自身的設定正說明了原因。

Panickssery 等人(2024)的預印本將自我偏好定義為:LLM 評估者傾向給自己所寫的摘要較高分,而給其他模型所寫的摘要較低分,但人類標註者認為兩組摘要的品質相當。作者稱其因果結果為初步證據,並指出相關性本身並不足以證明因果主張。任何評審都值得提出這個問題,而答案必須針對該評審實際測量。

要進行檢驗,需要一批品質相當的答案,其品質由人類評分者判定,並由評審自己的模型與另一個模型分別產出,或被標示為分別出自這兩個模型。同一個評審接著會在每種標示下讀取相同的答案,兩個通過率的差異就只會來自撰寫者。這個設計的輪廓很清楚,但本書的紀錄並未執行這樣的檢驗。

本書只有一個評審讀取兩位撰寫者的答案。基準助理與評審是同一個模型,即 Qwen3-4B-Instruct-2507。另一組答案來自較小的模型 Qwen3-0.6B,同樣由這個評審讀取。本書沒有人類對這些答案做品質相當的評分,而這個評審是衡量其品質的唯一標準,因此這項比較無法在撰寫者改變的同時保持品質不變。

在相同的開發集項目上,基準的通過率為 78.5% (62 of 79),第二個模型的通過率為 40.5% (32 of 79),差異為 -38.0 points, interval [-48.1, -26.6] points, n = 79。

直接解讀的話,第二個模型的答案通過的頻率低得多。品質較低會造成這樣的差距,評審偏袒自己的模型同樣會造成這樣的差距。本書的資料無法區分這兩種可能,因為衡量品質的唯一標準就是這個評審。所以這個差距,只是這個評審眼中兩個助理之間的差異。它既不是自我偏好的結果,也不是確認不存在自我偏好的乾淨證據。

若報告把評審描述為對自己的模型公平,就必須說明該檢驗並未在本書的紀錄上執行。對此保持沉默,讀起來像是得到了乾淨的結果,而這裡並非如此。團隊能提出的主張更為狹窄:紀錄顯示了兩個助理之間的差距,但對評審是否偏袒自己的模型毫無說明。

**實用原則:**當評審與助理是同一個模型時,請在通過率旁一併報告這一點,且不要把自我偏好報告為已經檢驗過。除非上述檢查已就位,否則不要用評審來比較它自己的模型與另一個模型;該檢查要求:品質相當的答案、由兩個模型產出或被標示為出自兩個模型、並由同一個評審讀取。第 6 章建立了這類檢查所需的人類評分。

何時不再信任判定結果

這些限制只有在讀者能當作規則來套用時才有用,而且每條規則都需要一個計數來核對。只看表格的讀者拿到的是限制,而不是限制的理由;上面各節已說明每一行的理由。限制表以帶標籤的行分組列出:順序變動率;長度差距及其方法;無法讀取的判定;同模型的數量;以及第二模型的比較。順序變動率不計入在任一順序下都無法讀取的配對,而上面的位置範例則計入其分割中的每一個配對。請從表格頂端讀起,因為每一行回答一個問題,沒有任何一行回答另一行的問題。

verdict-limits.py Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0 通過
"""The judge's limits as one table, one labelled line per limit, each computed as in position-flips and length-and-verdicts.The length line compares the longest band with the shortest, fixed in advance by length order: each band holds differentitems, so each is resampled on its own. The unreadable count is over every answer-judge recording on every item."""import lib  # first: fixes the thread counts and seeds before anything else loadsfrom lib.bias import band_outcomes, paired_passes, read_verdicts, reply_lengths, same_model_count, unreadable_countfrom lib.judging import both_orders, human_pairsfrom lib.stats import RESAMPLES, bootstrap_difference, bootstrap_mean, interval_points, pctfrom lib.testset import load_itemsJUDGE = "judge-answers-baseline"ANSWER_JUDGES = tuple(f"judge-answers-{variant}" for variant in ("baseline", "k5", "prompt-b", "second-model", "unfiltered"))items = [item.id for item in load_items()]dev = [item.id for item in load_items(split="dev")]# Order-change rate: flipped pairs over the pairs read in both orders. A tie is a readable verdict; an unreadable pair is left out.pairs = human_pairs()read_both = [o for o in both_orders("judge-hs2-pairs-ab", "judge-hs2-pairs-ba", list(pairs)) if o.kind != "unreadable"]print(f"order-change rate: {pct(sum(1 for o in read_both if o.kind == 'flipped'), len(read_both))}")# Length gap: the longest band's PASS rate minus the shortest band's, fixed by length order, on the dev items.shortest, _, longest = band_outcomes(reply_lengths("assistant-baseline", dev), read_verdicts(JUDGE, dev))lo, hi = bootstrap_difference(shortest[1], longest[1])print(f"length gap, longest band minus shortest band: {100 * (longest[1].mean() - shortest[1].mean()):+.1f} points, "      f"interval {interval_points(lo, hi)}")print(f"length gap method: percentile bootstrap over items, each band resampled alone, {RESAMPLES} resamples, 95% level")print(f"shortest band, {shortest[0]}: {pct(int(shortest[1].sum()), shortest[1].size)}")print(f"longest band, {longest[0]}: {pct(int(longest[1].sum()), longest[1].size)}")unreadable, total = unreadable_count(ANSWER_JUDGES, items)print(f"unreadable verdicts, five answer-judge recordings: {unreadable} of {total}")print(f"items per answer-judge recording: {len(items)}")worst = max((unreadable_count([name], items) for name in ANSWER_JUDGES), key=lambda found: found[0])print(f"most unreadable in one recording: {worst[0]} of {worst[1]}")same, judged = same_model_count(ANSWER_JUDGES)print("self-preference test on these recordings: not possible")print(f"same-model recordings: {same} of {judged}")# Second model beside the baseline, on the dev items both judged: a None verdict in either leaves the item out.both, base_pass, second_pass = paired_passes(JUDGE, "judge-answers-second-model", dev)print(f"baseline PASS rate, same dev items: {pct(int(base_pass.sum()), len(both))}")print(f"second-model PASS rate, same dev items: {pct(int(second_pass.sum()), len(both))}")diff_lo, diff_hi = bootstrap_mean(second_pass - base_pass)print(f"second-model minus baseline, same dev items: {100 * (second_pass - base_pass).mean():+.1f} points, "      f"interval {interval_points(diff_lo, diff_hi)}, n = {len(both)}")
輸出python verdict-limits.py
order-change rate: 22.7% (34 of 150)
length gap, longest band minus shortest band: -8.4 points, interval [-28.6, +8.1] points
length gap method: percentile bootstrap over items, each band resampled alone, 2000 resamples, 95% level
shortest band, under 15 words: 94.1% (16 of 17)
longest band, 30 words or more: 85.7% (18 of 21)
unreadable verdicts, five answer-judge recordings: 2 of 600
items per answer-judge recording: 120
most unreadable in one recording: 1 of 120
self-preference test on these recordings: not possible
same-model recordings: 4 of 5
baseline PASS rate, same dev items: 78.5% (62 of 79)
second-model PASS rate, same dev items: 40.5% (32 of 79)
second-model minus baseline, same dev items: -38.0 points, interval [-48.1, -26.6] points, n = 79

請依各自的分母來讀每個比率。各行並不共用同一個分母,分母小的比率只要一個項目改變就會大幅移動。這張表並未說明評審是否與一般人的看法一致。那是第 6 章的工作,而評審可以通過這裡的每一行,卻仍在重要的配對上與人工評分不一致。這些限制是判定在值得進行校準之前必須滿足的條件,而不是校準的替代品。

以下規則依循這些行,每一條都以其中一行為依據。

  • 順序變動。 當成對判定會隨順序而改變時,不要把它當作結果來讀。請在兩種分割上計算順序變動的次數,並把順序變動率印在判定計數旁邊。
  • 無法讀取的判定。 把它們印出來、排除在外,絕不當作 FAIL 計算。表中在五份答案評審記錄中共計 2 of 600 筆判定:基準、k5、prompt-b、第二模型與未過濾這幾個變體,各在 120 個項目上讀取。單一記錄損失最多的是 1 of 120 筆。這是經驗法則,而不是實測的臨界值:當某份記錄中每 100 筆判定有超過 1 筆無法讀取時,請先修正提示,再從該記錄讀取比率。
  • 長度。 不要只憑少數幾個區間就斷言有長度效應,也不要據此斷言沒有。長度這一行比較最長與最短的區間,並附上其區間估計。當區間包含零時,請連同計數說「未偵測到差異」,然後到此為止。
  • 同模型。 當評審與助理是同一個模型時,比率旁應註明這一事實。這裡有 4 of 5 份經評審的記錄屬於這種情形。本書無法在這些記錄上檢驗自我偏好,因此不要用評審來比較它自己的模型與其他模型,如自我偏好一節所述。
  • 品質相當。 兩個助理通過率之間的差距,是助理之間的差異,而不是評審的偏誤。第二模型這一行就是範例,也是自我偏好一節不給出任何一方結果的原因。
  • 遷移。 比率所依據的校準是在其他資料列上測得的,因此只能在遷移假設允許的範圍內沿用。

符合停止條件的判定不會被丟棄。順序變動的配對或無法讀取的判定,對該項目而言就成為沒有判定。它會被印出、計數,並從原本會納入的比率中排除,就像無法讀取的判定一樣,而該比率接著會報告它排除了多少個項目。同模型的記錄不是停止條件,而是在比率旁報告。

在實際運作中,這些限制表現為一個因錯誤原因而變動的比率。假設某團隊更改了助理的提示,通過率上升。在團隊把功勞歸給提示之前,限制表會提出三個問題。判定是否隨順序改變,或是無法讀取?回覆是否變長,而長度這一行的區間是否能把差距與零區分開來?評審是否就是撰寫回覆的模型?前兩項只要有一項為是,就表示這個上升在報告之前需要自己的檢查。第三項為是,則表示報告這個上升時要在旁邊註明這一事實。

實用原則: 當判定隨順序改變或無法讀取時,就不再信任它,並在報告中說明。當評審就是其答案受評分的那個模型時,請在比率旁報告這一點,並且不要用評審來比較它自己的模型與其他模型,如自我偏好一節所述。在發布通過率之前,請如第 6 章所做的,以人工評分檢查評審,並在比率旁印出順序變動率、無法讀取的數量與同模型的事實。通過率只有在評審與人的看法一致的範圍內,才描述這些答案,因此比率應始終附帶其校準。

本章,於本版次中

版次 2026.11.3範例於 2026年11 月 21 通過測試本章 3 個範例 個範例中有 3 個通過

譯自英文書,版次 2026.11.3。

引用本節

格式

各種格式皆引用英文原書,本譯本即依循該原書。

每種格式都會標示本版。其連結會開啟目前版次,節標題會說明該節內文最後一次變動的時間。存取日期與擷取日期即為你開啟此頁的當日。

本頁內容

第 7 章,共 11 章: 找出評審的判定在何處不再可靠

  1. 位置
  2. Length
  3. 自我偏好,本書無法檢驗的問題
  4. 何時不再信任判定結果

返回頂端所有章節