LLM 應用的檢索
閱讀約需 97 分鐘 分鐘,10 節 中共有 10 個範例。
讀完本章,你將能夠衡量 LLM 回答時所依據的檢索:以答案引用的產品來為答案評分,將每筆引用與檢索實際回傳的結果核對,衡量 LLM 讀到的產品,選擇要以什麼作為單一可檢索單元來建立索引,並從工具呼叫的紀錄中解讀代理(agent)的搜尋。
the store 的每一次搜尋,最後都會產生一份排序後的家具與居家用品清單,由購物者自行閱讀。購物助理則可以改用文字來回答。購物者寫下自己想要的東西,LLM 便推薦幾項產品,並以產品 ID 標示每一項的出處。提示詞要求 LLM 只推薦搜尋結果擺在它面前的產品,因此助理的表現不可能優於它的檢索。代理(agent)也是如此:它是自己執行搜尋、並自行決定要搜尋幾次的 LLM。
所以,本章依據商店的人工判斷,對答案、引用、上下文與代理的搜尋評分。關於 LLM 的每項發現,都只適用於這個模型、這些提示詞與 30 筆測試查詢,不適用於一般的 LLM。在回答時,LLM 以文字讀取每項產品的名稱、類別(class)、分類(category)、描述與屬性,而代理的搜尋工具呈現給它的內容更少。人工評審還看過每項產品的圖片與網頁(LLM 看到了什麼)。語意檢索依據 Snowflake/snowflake-arctic-embed-m-v1.5 的既有嵌入向量排序,BM25則是本書未做詞幹還原的基準線,即 lib.lexical。
我們在開啟任何結果之前,就規劃了 2 項比較,全部使用測試查詢,且每項比較都是語意檢索對上 BM25:
這 2 項規劃的比較共同分攤本書的 0.05,因此每一項都需要進行配對排列檢定,且 p 值須低於 0.025。本章同時報告配對 t 檢定與 97.5% 的自助法區間。測試查詢上的其他結果,一律只是描述,或標示為探索性結果。
將檢索到的產品放進提示詞
檢索增強生成(retrieval-augmented generation,RAG)是購物助理背後的設計,它將 LLM 與檢索器配對使用。每次收到請求時,檢索器會從索引中找出幾段文字,LLM 再根據這些文字寫出回答。Lewis 等人的 RAG 模型,把非參數式記憶存放在維基百科的稠密向量索引中,由預訓練的神經檢索器進行搜尋。這些作者也指出,語言模型單獨運作時有兩個問題仍是開放的研究問題:為其決策提供出處,以及更新其世界知識(Lewis et al., 2020)。商店同時面對這兩個問題:目錄會變動,而且回答應說明推薦的是哪些產品。檢索到的文字會放進提示詞(prompt),也就是模型讀取的輸入;提示詞中檢索到的文字稱為脈絡(context)。模型的脈絡視窗(context window)是它最多能讀取的詞元數量,這個數字固定,單位是其自身詞彙表中的詞片段。
為什麼不把整個目錄放進提示詞、省去檢索?首先,內容必須放得下;即使放得下,模型讀取長提示詞時也並不均勻。Liu 等人發現,當相關資訊位於輸入的開頭或結尾時,模型往往表現最好;當相關資訊位於長脈絡的中間時,表現會顯著下降(Liu et al., 2024)。後續的一項研究把這稱為中間遺失(lost-in-the-middle,LiM)效應。該研究發現,當輸入佔模型脈絡視窗最多達 50% 時,這種效應最強;接近上限時,則轉為偏向結尾的偏誤(Veseli et al., 2025)。段落越多也不一定越好。Jin 等人給長脈絡 LLM 提供更多檢索段落時,輸出品質先是提升,之後下降;他們將檢索到的困難負例(看似相關、實際上無關的段落)視為造成下降的關鍵因素(Jin et al., 2024)。某家廠商的工程文章對小型資料集提出相反的建議:當知識庫少於 200,000 個詞元(約 500 頁的資料)時,就把整個知識庫放進提示詞(Anthropic, 2024)。那是廠商的經驗法則,不是實測結果。
商店使用的 LLM 是開放模型 gemma4:12b-it-q4_K_M。它的錄製檔(recordings)recordings/rag_answers.py 與 recordings/agent_trace.py,如同第 4 章的錄製檔一樣,每次模型呼叫只錄製一次,溫度為 0、種子固定、思考功能關閉(第 9 章),並將回覆儲存下來,範例透過 lib.recorded 讀取這些回覆。每份錄製檔的資訊清單(manifest),也就是描述該錄製檔的檔案,將模型的授權標示為 Apache-2.0。
商店的錄製回答(為每個回答評分)在每個提示詞中放入五項產品,並置於下列指示之後:「You help shoppers at an online furniture and home-goods store. Using only the products listed below, answer the shopper. Recommend up to three products that fit, cite each one by its ID, and say plainly if none of them fit. Don't mention products that aren't listed.」目錄相當於多少個提示詞的份量?以 LLM 讀取的方式計算目錄的字數,就能得到答案。
1"""Counts the catalog's words as the LLM reads each product, then sets them against a median prompt of the2instructions and five products, and against the recording's context window. The point rests on the "catalog3against one prompt" line."""4from statistics import median56from lib.agreement import as_the_llm_saw7from lib.recorded import manifest, rag_answers8from lib.store import load_products910recording = manifest("rag-answers")11# num_ctx is the context window, in tokens, that the recording set12window = recording["options"]["num_ctx"]13print("model:", recording["model"])14print("licence in the manifest:", recording["model_licence"])15print("instructions:", recording["prompts"].split("\n")[0])16print("context window the recording set, in tokens:", window)1718products = load_products()19# words are whitespace-separated pieces, since this model's tokenizer isn't in the book's lib20words = {product_id: len(f"ID: {product_id}\n{as_the_llm_saw(product)}".split()) # each product as the LLM read it21 for product_id, product in products.iterrows()}22instructions = len(recording["prompts"].split()) # the prompt's own words, without the query23# one prompt per recorded answer: the instructions, then the five products' words24prompts = [instructions + sum(words[p] for p in ids) for ids in rag_answers()["context_ids"]]25catalog = sum(words.values())26print(f"words in the catalog as the LLM reads it: {catalog:,}")27print(f"words in one prompt with five products, median: {median(prompts):,.0f}")28print(f"the catalog against one prompt: {catalog / median(prompts):,.0f} times")29print(f"the catalog against the recording's window, at one token per word: {catalog / window:,.0f} times")30print(f"the catalog against 200,000 tokens, at one token per word: {catalog / 200_000:.1f} times")
python catalog-against-the-context.pymodel: gemma4:12b-it-q4_K_M
licence in the manifest: Apache-2.0
instructions: You help shoppers at an online furniture and home-goods store. Using only the products listed below, answer the shopper. Recommend up to three products that fit, cite each one by its ID, and say plainly if none of them fit. Don't mention products that aren't listed.
context window the recording set, in tokens: 8192
words in the catalog as the LLM reads it: 1,091,544
words in one prompt with five products, median: 536
the catalog against one prompt: 2,035 times
the catalog against the recording's window, at one token per word: 133 times
the catalog against 200,000 tokens, at one token per word: 5.5 times
即使本書的樣本已將描述截短至約 400 個字元(資料卡),目錄仍有 1,091,544 個字,而含五項產品的提示詞,字數的中位數為 536。目錄比單一提示詞大 2,035 times。
由於沒有此模型的分詞器,這些計數是以字為單位。錄製檔設定的脈絡視窗為 8192 個詞元。假設每個字恰好是一個詞元,目錄會填滿該視窗 133 times 次,而廠商所說的 200,000 個詞元則會被填滿 5.5 times 次。商店必須進行檢索。
實用原則: 提示詞中放入少數檢索到的項目,而不是整個資料集;要放幾項,應在訓練查詢上以數種數量衡量回答的表現後再決定,因為更多脈絡可能有幫助,也可能有害。商店的錄製檔只回答了測試查詢,每次五項產品,因此這裡的五是固定設定,並非實測所得。只有在整個資料集能輕鬆放進視窗內時,才考慮整份送入,並且要在相同查詢上與檢索方式比較測試。
此部分尚未翻譯,因此以英文呈現。
Score each answer by the products it cites
An LLM told to answer from a context should recommend only what the context holds, and the ID check tests that it does. If no exact product is in the context, no prompt makes a grounded answer recommend one. Retrieval sets a ceiling on the answer, and what else the context holds matters too.
Cuconasu et al. found that the retriever's highest-scoring documents that aren't directly relevant, ones that don't contain the answer, hurt answers. More surprisingly, adding random documents to the prompt improved LLM accuracy by up to 35% (Cuconasu et al., 2024). Products that look right but aren't are distractors, the store's version of those passages. In question answering, Joren et al. found a related pattern. When the context lacked what was needed, the larger models they tested, Gemini 1.5 Pro, GPT 4o and Claude 3.5, often gave incorrect answers instead of abstaining. Smaller ones, Mistral 3 and Gemma 2 among them, often hallucinated or abstained even when the context was sufficient (Joren et al., 2024).
A citation here is the ID an answer gives for a product it recommends. With judgments, you can score an answer without reading it: look up each cited product's grade. The exact share of an answer is the share of its cited products that people judged exact. An answer that cites nothing scores 0, because it gives the shopper nothing that fits, whether or not that was the right call. A separate count of answers that recommend nothing covers that case.
exact share = (cited products judged exact) / (products cited), 0 when the answer cites nothing
The store's recording answered each test query three times, with the query following "I'm looking for". Each answer came from a different context of five products: semantic retrieval's top 5, BM25's top 5, or five products drawn at random with a fixed seed. The random context is a floor, a context no retriever chose. The model, the prompt and the query stay the same, so the answers differ only because their contexts do. We read each context as the recording stored it, in the order the LLM read it. The BM25 contexts keep tied products in bm25s's own order, not by product_id as lib.lexical does, so 7 of the 30 test queries get a context that differs from what lib.lexical returns, 6 of them only in order. The recording asked for JSON with the answer and the cited IDs in separate fields. score_answers in lib.generation scores each answer on the human grades. The comparison of those scores is the first of the chapter's planned comparisons, established if its permutation p-value is below 0.025.
How many answers will recommend nothing, given five random products and a prompt that tells the LLM to say plainly if none of them fit?
1"""Scores the recorded answers on the human grades, context by context, and makes the first planned comparison. The2point rests on the mean exact share and recommend-nothing lines, and on the permutation p-value of the3comparison."""4from lib.generation import PLANNED_TEST_COMPARISONS, THRESHOLD, report_planned, score_answers5from lib.lexical import LexicalSearch6from lib.recorded import rag_answers7from lib.store import load_products, load_queries, qrels89answers = rag_answers()10stored = answers[answers["context"] == "bm25"].set_index("query_id")["context_ids"] # as the LLM read them11rerun = LexicalSearch(load_products()).run(load_queries("test"), k=5) # ties by product_id12differ = [q for q in stored.index if stored[q] != rerun[q]]13print("bm25 contexts that differ from lib.lexical:", len(differ))14# a match as sets means the same five products in another order: a tie-break, not a different context15print("bm25 contexts that differ from lib.lexical only in order:", sum(set(stored[q]) == set(rerun[q]) for q in differ))1617scores = score_answers(answers, qrels("test")) # one row per test query and context, on human grades18for context, rows in scores.groupby("context", sort=False):19 # held: the context holds an exact product; nothing: the answer cites no product20 held, nothing = rows["exact in context"] > 0, rows["cited"] == 021 print(f"{context}, unjudged products in the contexts: {rows['unjudged in context'].sum()}")22 print(f"{context}, contexts with an exact product: {held.sum()}")23 print(f"{context}, answers that cite an exact product: {(rows['exact cited'] > 0).sum()}")24 print(f"{context}, products cited: {rows['cited'].sum()}")25 print(f"{context}, cited products that are exact: {rows['exact cited'].sum()}")26 print(f"{context}, mean exact share: {rows['exact share'].mean():.3f}")27 print(f"{context}, answers that recommend nothing: {nothing.sum()}")28 print(f"{context}, recommend nothing, with an exact product in the context: {(nothing & held).sum()}")29 print(f"{context}, recommend something, with no exact product in the context: {(~nothing & ~held).sum()}")30 near = rows[~held] # answers whose context held no exact product31 print(f"{context}, products cited with no exact product in the context: {near['cited'].sum()}")32 print(f"{context}, of those, partial: {near['partial cited'].sum()}")33 print(f"{context}, unjudged products cited: {rows['unjudged cited'].sum()}")34print("answers that cite more exact products than their context holds:",35 (scores["exact cited"] > scores["exact in context"]).sum())3637print("planned test comparisons:", len(PLANNED_TEST_COMPARISONS))38print(f"p-value threshold: {THRESHOLD:.3f}")39share = scores.pivot(index="query_id", columns="context", values="exact share")40# the first planned comparison: semantic minus BM25 on the exact share, tested at THRESHOLD41report_planned("answers, semantic over bm25", share["semantic"], share["bm25"])
python rag-answers-by-context.pybm25 contexts that differ from lib.lexical: 7
bm25 contexts that differ from lib.lexical only in order: 6
semantic, unjudged products in the contexts: 0
semantic, contexts with an exact product: 25
semantic, answers that cite an exact product: 22
semantic, products cited: 80
semantic, cited products that are exact: 60
semantic, mean exact share: 0.658
semantic, answers that recommend nothing: 2
semantic, recommend nothing, with an exact product in the context: 1
semantic, recommend something, with no exact product in the context: 4
semantic, products cited with no exact product in the context: 11
semantic, of those, partial: 11
semantic, unjudged products cited: 0
bm25, unjudged products in the contexts: 10
bm25, contexts with an exact product: 24
bm25, answers that cite an exact product: 23
bm25, products cited: 73
bm25, cited products that are exact: 52
bm25, mean exact share: 0.639
bm25, answers that recommend nothing: 3
bm25, recommend nothing, with an exact product in the context: 0
bm25, recommend something, with no exact product in the context: 3
bm25, products cited with no exact product in the context: 7
bm25, of those, partial: 7
bm25, unjudged products cited: 1
random, unjudged products in the contexts: 147
random, contexts with an exact product: 1
random, answers that cite an exact product: 1
random, products cited: 1
random, cited products that are exact: 1
random, mean exact share: 0.033
random, answers that recommend nothing: 29
random, recommend nothing, with an exact product in the context: 0
random, recommend something, with no exact product in the context: 0
random, products cited with no exact product in the context: 0
random, of those, partial: 0
random, unjudged products cited: 0
answers that cite more exact products than their context holds: 0
planned test comparisons: 2
p-value threshold: 0.025
answers, semantic over bm25, mean difference: 0.019
answers, semantic over bm25, wins, losses, ties: 4, 5, 21
answers, semantic over bm25, permutation p: 0.765
answers, semantic over bm25, established at 0.025: no
answers, semantic over bm25, t-test p: 0.715
answers, semantic over bm25, t-test agrees: yes
answers, semantic over bm25, 97.5% interval: -0.078 to 0.161
answers, semantic over bm25, interval agrees: yes
| Measure | Semantic | BM25 | Random |
|---|---|---|---|
| Contexts with an exact product | 25 | 24 | 1 |
| Answers that cite an exact product | 22 | 23 | 1 |
| Products cited | 80 | 73 | 1 |
| Cited products that are exact | 60 | 52 | 1 |
| Mean exact share | 0.658 | 0.639 | 0.033 |
| Answers that recommend nothing | 2 | 3 | 29 |
With random products, the LLM recommends nothing in 29 of the 30 answers, so this model and prompt did not force a recommendation out of random products. The one product it cites is exact, and the random draw happened to include it. The random contexts also hold 147 products nobody judged, which count as not exact. We expect few of them to fit, but no judgment says so.
The ceiling holds: 0 answers cite more exact products than their context holds. Near misses are another matter. Where the context held no exact product, the answer still recommended something 4 times with semantic retrieval and 3 times with BM25, rather than say that no product fits exactly.
Every product those answers cite is partial: 11 of 11 with semantic retrieval and 7 of 7 with BM25, the right kind of product with a modifier wrong. Whether that counts as a failure depends on whether the shopper wants near matches.
The reverse failure is rarer: answers recommend nothing although their context held an exact product, 1 with semantic retrieval and 0 with BM25.
BM25's contexts hold 10 unjudged products, and semantic retrieval's hold 0. The score reads only cited products, and BM25's answers cite 1 unjudged product, which counts as not exact.
Semantic retrieval's lead on nDCG@10 is established (chapter 3). On the first planned comparison, though, its difference from BM25 in mean exact share, 0.019, isn't established: the permutation p-value is 0.765 (both comparisons).
Working rule: Score every answer by its cited products' human grades, and report beside it how many answers recommend nothing, split by whether the context held an exact product. Keep a random context as a floor whenever you change the model or the prompt: an answer that recommends something from random products has invented a fit. Test a retriever's gain on the answers too, because a gain in ranking needn't carry through.
逐一對照上下文檢查每個引用
接地(grounding)是指答案以所提供的上下文為依據,就像精確比對的份額所假設的那樣,而不是依據模型的記憶或憑空捏造的內容。引用讓接地可以被檢查,而歸因(attribution)要問的是,被引用的來源是否支持它所附著的那句陳述。稽核發現,引用往往並不支持。在 2023 年對生成式搜尋引擎的一項稽核中,只有 51.5% 的生成句子完全受到其引用的支持,而只有 74.5% 的引用支持它所附著的句子(Liu et al., 2023)。在 ELI5 上(這是 ALCE 基準中取自 Reddit 論壇 Explain Like I'm Five 的長篇問題),即使是 2023 年最好的模型,也有 50% 的時候缺乏完整的引用支持(Gao et al., 2023)。
引用要在兩個層次檢查,由成本最低的開始。第一層是精確的,而且不需要模型:每個被引用的 ID 都必須是上下文中有的 ID。來自上下文之外的 ID,指的是模型憑記憶或編造出來的商品,不論是哪一種,答案都沒有接地。第二層要問的是,被引用商品的文字是否支持答案對它的說法,這需要一個閱讀者(檢查被引用的商品支持什麼)。
錄製資料的結構描述要求把被引用的 ID 放在一個獨立欄位中,以整數清單表示(在資訊清單中為 schemas)。結構描述只描述欄位的形式,並未說明哪些 ID 可以放進去,也未說明可以放幾個。這項檢查會把每筆已記錄的答案對照它的上下文來讀,並將該欄位與答案文字中明確寫出的上下文 ID 相比較。文字無法揭露來自上下文之外的 ID,因為這種 ID 看起來就和答案中寫出的其他數字(例如尺寸)沒有差別。這就是 ID 應該放在欄位中的原因:
1"""Checks every recorded answer's cited IDs against its context, and its text against its cited field. The point2rests on the zero counts for IDs outside the context and for disagreeing text, and on the count of answers that3cite more than three products."""4from lib.generation import ids_in_text5from lib.recorded import rag_answers67answers = rag_answers()8outside = answers.apply(lambda a: [p for p in a["cited"] if p not in a["context_ids"]], axis=1)9# an ID cited twice makes the list longer than the set of its IDs10repeated = answers["cited"].map(lambda cited: len(cited) != len(set(cited)))11# the text is searched only for the context's IDs, so an ID from outside it can't match12written = answers.apply(lambda a: ids_in_text(a["answer"], a["context_ids"]), axis=1) # IDs the text writes out13print("answers:", len(answers))14print("answers that cite an ID outside their context:", (outside.map(len) > 0).sum())15print("answers that cite one ID twice:", repeated.sum())16print("answers that cite nothing:", (answers["cited"].map(len) == 0).sum())17# the prompt asks for up to three products18print("answers that cite more than three products:", (answers["cited"].map(len) > 3).sum())19print("answers whose text and cited IDs differ:", (written != answers["cited"].map(set)).sum())
python citations-inside-the-context.pyanswers: 90
answers that cite an ID outside their context: 0
answers that cite one ID twice: 0
answers that cite nothing: 34
answers that cite more than three products: 3
answers whose text and cited IDs differ: 0
在 90 筆已記錄的答案中,這項檢查發現有 0 筆答案引用了上下文之外的 ID,有 0 筆重複引用同一個 ID,有 0 筆的文字與欄位不一致。
有 34 筆答案完全沒有引用任何東西。答案確實違反的指示是推薦的數量:有 3 筆答案引用了超過三項商品,儘管提示寫的是「最多推薦三項」。
在這份錄製資料上,ID 檢查沒有發現任何未接地的引用。這是你想要的結果,但並不是放棄這項檢查的理由:它每筆答案只需要一次集合查詢,而不同的模型、提示或上下文長度都可能讓它失敗。通過檢查只能說明每項被推薦的商品都曾出現在模型面前,並不表示它適合購物者。
實用原則: 對每筆答案都執行 ID 檢查,在離線狀態下、並在購物者看到答案之前完成,並把上下文之外的 ID 視為失敗的答案:捨棄或重試,絕不顯示。把引用放在獨立欄位中,檢查就不必解析文字,同時也要檢查文字與欄位是否一致。統計答案違反指示的情形,例如推薦的數量,並在程式碼中強制執行其中重要的那些。
檢查被引用的商品支持什麼
第二層檢查要問的是:被引用的文字是否支持該項主張。忠實度(faithfulness)是指回答中的主張有多大比例受到脈絡支持。依 Ragas 文件的說法(其頁面未標示版本;所開啟的頁面日期為 2025-12-09),「若一則回應的所有主張都能由檢索到的脈絡支持,該回應即視為忠實」(Ragas documentation)。要檢查這一點,需要人來讀。ALCE 使用自然語言推論(natural language inference,NLI)模型,這項任務是判定一段文字是否蘊含另一段文字:它採用在一批 NLI 資料集上微調的 T5-11B 模型。只有當一則陳述的各項引用合起來蘊含該陳述時,其引用召回率才為 1(Gao et al., 2023, sections 3.2 and 3.3)。LLM 也能判斷支持程度,但有其限制。在 2024 年 TREC(Text REtrieval Conference,文字檢索會議的共同評測)的 RAG 賽道中,Thakur 等人發現,在三級量表上,人類與 GPT-4o 的預測完全一致的比例,從頭判斷時為 56%,而人從 GPT-4o 的支持標籤出發再加以編輯時為 72%。他們也發現,獨立的人類評審與 GPT-4o 的相關性,比與另一位人類的相關性更高,可見部分歧見存在於人與人之間(Thakur et al., 2025)。
支持不等於相關。引用可以忠實於商品的文字,卻仍然推薦了錯誤的商品:文字確實說了回答所主張的內容,但該商品並不是購物者要找的。只有人工判斷能發現這一點。
先以人工檢視商店的一則回答,也就是語意脈絡下測試查詢 medium size chandelier 的回答。接著統計已記錄的回答中,語意與 BM25 脈絡下每項被引用商品的人工評分。
1"""Reads one recorded answer by hand, checking its claim against the cited product's text and the judges' label,2then counts the grades of every product the two retrievers' answers cite. The point rests on the partial label,3the exact count in this answer's context, and the breakdown of cited products."""4from collections import Counter56from lib.agreement import LABELS7from lib.recorded import rag_answers8from lib.store import load_products, load_queries, qrels910products, queries, judgments = load_products(), load_queries("test"), qrels("test")11answers = rag_answers().set_index(["query_id", "context"])1213chandelier, claim = 58, "medium-sized" # one answer, read by hand14answer, judged = answers.loc[(chandelier, "semantic")], judgments[chandelier]15# the answer cites one product, so index 0 is that product16cited = answer["cited"][0]17print("query:", queries.at[chandelier, "query"])18print("answer:", answer["answer"])19print("cited product:", products.at[cited, "name"])20print(f"the cited product's text says {claim}:", "yes" if claim in products.at[cited, "description"] else "no")21print("the judges' label:", LABELS[judged[cited]])22# grade 2 is exact; a product nobody judged has no grade, so it counts as 023print("exact products in this answer's context:", sum(judged.get(p, 0) == 2 for p in answer["context_ids"]))2425retrieved = answers.drop("random", level="context") # every citation from the two retrievers' contexts26# LABELS names each grade; a product with no grade counts as unjudged27labels = Counter(LABELS.get(judgments[query_id].get(p), "unjudged")28 for (query_id, _), cited_ids in retrieved["cited"].items() for p in cited_ids)29print("cited products, semantic and bm25 contexts:", sum(labels.values()))30for label in ["exact", "partial", "irrelevant", "unjudged"]:31 print(f"cited products, {label}:", labels[label])
python faithful-but-partial.pyquery: medium size chandelier
answer: I recommend the sybil 12 - light unique / statement classic / traditional chandelier with antler accents (ID: 8171), as it is specifically described as a perfect medium-sized lighting fixture.
cited product: sybil 12 - light unique / statement classic / traditional chandelier with antler accents
the cited product's text says medium-sized: yes
the judges' label: partial
exact products in this answer's context: 2
cited products, semantic and bm25 contexts: 153
cited products, exact: 112
cited products, partial: 40
cited products, irrelevant: 0
cited products, unjudged: 1
這則回答推薦了一盞吊燈,並稱它「特別被描述為完美的中型照明燈具」。被引用商品的描述確實寫了 medium-sized(yes),因此支持檢查會通過這項主張。商店的評審將該商品標為 partial:依 WANDS 的定義,這是正確類型的商品,但缺少購物者要求的某個修飾條件。評審看到了商品圖片與商品頁面,而 LLM 沒有。其脈絡中有 2 件完全符合的商品,回答卻一件也沒有推薦。就這則回答而言,引用是忠實的,推薦仍然不合。
在兩種檢索器的回答中,共 153 件被引用商品,其分布為完全符合 112 件、部分符合 40 件、不相關 0 件,未評判 1 件。除了無人評判的 1 件之外,每件不屬完全符合的被引用商品都是部分符合:商品種類是購物者要找的,但依評審的判讀,某個修飾條件不對。
實用原則: 先以你的人工判斷為回答評分,因為它們能說明推薦是否合適。再針對回答對每項商品所做的主張,加上支持檢查,可用 NLI 模型或 LLM 評審。以人工樣本校準該評審,如第 4 章所做,也如 ARES 所做:它使用少量經人工標註的資料點進行預測驅動推論(prediction-powered inference,PPI)以減輕預測誤差,即以人工標籤修正評審的估計(Saad-Falcon et al., 2024)。對於沒有人工判斷的查詢,你只有支持檢查可用,因此在報告時要如實說明。
先衡量情境,再衡量答案
RAG 系統可能在檢索階段失敗,也可能在生成階段失敗,因此除了衡量答案之外,也要衡量情境。Ragas 是評估 RAG 的框架,它在不需要參考答案的情況下,對數項指標評分:檢索辨識出相關且聚焦的情境段落的程度、忠實度(faithfulness),以及答案的相關性。它的情境召回率(context recall)則需要參考答案(Es et al., 2024)。它的情境指標沿用了第 3 章的指標,並以 LLM 擔任評審。Ragas 的**情境精確率(context precision)是其情境指標中考量排名的一項,是在含有相關區塊的各個位置上,對 precision@k 取平均(Ragas 文件)。它的情境召回率(context recall)**是情境所支持的參考答案主張所占的比例;另有一種以 ID 為基礎的變體,會將檢索到的 ID 與參考 ID 比較(Ragas 文件)。在已有判定的情況下,參考就是精確商品的集合,而以 ID 為基礎的情境召回率就是 recall@k。
context recall@k = (exact products in the top k) / (exact products judged for the query)
context precision@k = mean of precision@i over the positions i ≤ k that hold an exact product
哪一項指標與答案的表現相符?Salemi 與 Zamani 發現,相關性標籤與 RAG 系統的下游表現只有微弱的相關,因此他們改以 LLM 根據每個檢索段落所給出的答案,來為該段落評分(Salemi and Zamani, 2024)。**斯皮爾曼等級相關(Spearman's rank correlation)**是兩項指標的等級之間的相關,而不是數值之間的相關;當兩者對各案例的排序相同時為 1,排序互不相關時接近 0。它能針對一次記錄回答這個問題。
我們針對已記錄的答案,衡量檢索器為每個測試查詢所建構的每一個情境,每個查詢各有一個語意檢索與一個 BM25 的情境。此範例以手動方式建構情境召回率與情境精確率,將結果與 lib.metrics.recall_at_k 以及 lib.generation 中的副本核對,並逐步處理其中一個情境:
1"""Builds context precision and context recall by hand and checks each against the library's version, then works2through one context. The point rests on the two "every time" lines and on the precision@5 and context precision3of the worked context."""4from lib import generation5from lib.generation import EXACT6from lib.metrics import precision_at_k, recall_at_k7from lib.recorded import rag_answers8from lib.store import load_queries, qrels91011def context_precision(context, judged, k=5):12 """Ragas's context precision, with the judges as the reference: mean precision@i over exact positions i."""13 # positions count from 1, as ranks do; context[:k] is the top k14 hits = [i for i, p in enumerate(context[:k], start=1) if judged.get(p, 0) == EXACT]15 return sum(precision_at_k(context, judged, i) for i in hits) / len(hits) if hits else 0.0161718def context_recall(context, judged, k=5):19 """Ragas's ID-based context recall: the share of the reference IDs that the context holds."""20 # the reference is every exact product judged for the query, not only those in the top k21 reference = {p for p, grade in judged.items() if grade == EXACT}22 return len(reference & set(context[:k])) / len(reference)232425judgments, queries = qrels("test"), load_queries("test")26answers = rag_answers().query("context != 'random'") # the two retrievers' contexts for each test query27contexts = [(answer.context_ids, judgments[answer.query_id]) for answer in answers.itertuples()]28print("contexts:", len(contexts))29same = {"recall_at_k": all(context_recall(c, j) == recall_at_k(c, j, 5) for c, j in contexts),30 "lib.generation": all(context_precision(c, j) == generation.context_precision(c, j) for c, j in contexts)}31print("context recall equals recall_at_k every time:", "yes" if same["recall_at_k"] else "no")32print("context precision equals lib.generation's every time:", "yes" if same["lib.generation"] else "no")3334# the first row: the semantic context for door jewelry organizer35first = answers.iloc[0] # one context, worked through36context, judged = first["context_ids"], judgments[first["query_id"]]37print(f"grades in order, {queries.at[first['query_id'], 'query']}, {first['context']}:", *[judged.get(p, 0) for p in context])38print(f"context precision@5: {context_precision(context, judged):.3f}")39print(f"precision@5: {precision_at_k(context, judged, 5):.3f}")40# exact products judged for each test query, sorted from the fewest to the most41exact = sorted(sum(grade == EXACT for grade in grades.values()) for grades in judgments.values())42print(f"exact products judged per test query: {exact[0]} to {exact[-1]}")43print(f"highest recall@5 possible for the query with the most: {5 / exact[-1]:.3f}")
python context-measures-by-hand.pycontexts: 60
context recall equals recall_at_k every time: yes
context precision equals lib.generation's every time: yes
grades in order, door jewelry organizer, semantic: 1 1 2 2 2
context precision@5: 0.478
precision@5: 0.600
exact products judged per test query: 4 to 170
highest recall@5 possible for the query with the most: 0.029
在 60 個情境中,情境召回率每次都等於 recall_at_k(yes),而情境精確率則每次都等於 lib.generation 的結果(yes)。
以 door jewelry organizer 的語意情境為例,依序的等級為 1 1 2 2 2,其中 2 表示精確,1 表示部分相符。這使 precision@5 為 0.600,情境精確率為 0.478,後者較低,是因為精確商品排在最後。
將每項指標與答案的精確占比做相關,即可回答該問題。此範例以等級的皮爾森相關計算斯皮爾曼相關,並與 SciPy 的 spearmanr 核對:
1"""Correlates three context measures with the exact share of each recorded answer, computing Spearman's correlation2by hand and checking it against SciPy. The point rests on the three Spearman lines and the weakest-measure line."""3import pandas as pd4from scipy import stats56from lib.generation import context_precision, context_recall, exact_share7from lib.metrics import precision_at_k8from lib.recorded import rag_answers9from lib.store import qrels1011judgments = qrels("test")12answers = rag_answers().query("context != 'random'") # the two retrievers' contexts for each test query13rows = pd.DataFrame([{"recall@5": context_recall(a.context_ids, judgments[a.query_id]),14 "precision@5": precision_at_k(a.context_ids, judgments[a.query_id], 5),15 "context precision@5": context_precision(a.context_ids, judgments[a.query_id]),16 "exact share": exact_share(a.cited, judgments[a.query_id])} for a in answers.itertuples()])17print("contexts:", len(rows))1819ranks, spearman = rows.rank(), {} # Spearman: Pearson's correlation of the ranks20for measure in ["recall@5", "precision@5", "context precision@5"]:21 spearman[measure] = ranks[measure].corr(ranks["exact share"])22 print(f"spearman with the exact share, {measure}: {spearman[measure]:.3f}")23# rounded to nine places, so floating-point differences in the last digits don't count24matches = all(round(value, 9) == round(stats.spearmanr(rows[measure], rows["exact share"]).statistic, 9)25 for measure, value in spearman.items())26print("matches scipy's spearmanr:", "yes" if matches else "no")27print("weakest with the exact share:", min(spearman, key=spearman.get))
python context-measures-against-answers.pycontexts: 60
spearman with the exact share, recall@5: 0.540
spearman with the exact share, precision@5: 0.797
spearman with the exact share, context precision@5: 0.808
matches scipy's spearmanr: yes
weakest with the exact share: recall@5
手動建構的斯皮爾曼相關與 SciPy 的結果相符(yes)。
與精確占比的相關最弱者是 recall@5,為 0.540;precision@5 為 0.797,情境精確率為 0.808。
Recall@5 的分母說明了原因。各測試查詢有 4 to 170 個經判定的精確商品,因此對於精確商品最多的查詢,五個商品的情境所能達到的最高 recall@5 為 0.029。無論判定的數量多少,答案都只需要三項推薦。Precision@5 計算五個商品中有多少符合需求,而情境精確率還會獎勵將它們排在前面。
請將這些相關性視為對一次記錄的描述,而不是檢定。兩邊使用相同的判定,而答案只能引用其情境所含的內容,因此本身就帶有一定的相關性。這些情境也是成對出現的,即來自相同查詢的語意與 BM25 情境。
實用原則: 對於大小固定的情境,應根據你的判定回報該情境的 precision@k 或情境精確率,並在旁邊列出 recall@k。當答案必須涵蓋每一個相關項目時,例如適用於同一台電器的每一個零件,就使用召回率。在沒有判定的情況下,Ragas 以 LLM 評判的版本可以估計類似的量;在信任它們之前,應先用人工樣本加以校準。選擇 k 時,應在訓練查詢上以數個 k 值衡量答案,這需要已為這些查詢記錄的答案。
此部分尚未翻譯,因此以英文呈現。
Choose the retrievable unit
A retrievable unit is the piece of text that retrieval scores and returns, and that a context holds: a whole document, a passage or a sentence. A long document gets cut into chunks, and chunking is the rule that cuts a document, such as a fixed number of tokens, sentences, paragraphs, or breaks chosen by meaning. Chen et al. call the retrieval unit an often-overlooked design choice, and in their study, indexing propositions, short self-contained statements, beat indexing passages (Chen et al., 2023). Beyond that, the studies disagree. One finds that the best size depends on the questions and the embedding model: 64 to 128 tokens for datasets with concise, fact-based answers, and 512 to 1,024 tokens for questions that need broader context (Bhat et al., 2025). Another finds that the computational costs of semantic chunking aren't justified by consistent performance gains (Qu et al., 2024). A comparison of 36 methods ranks chunking by groups of paragraphs first, and finds that simple fixed-size character chunks performed poorly as baselines (Shaukat et al., 2026).
A small unit can also lose what it is about. A vendor's engineering post reports that adding chunk-specific context to each chunk before embedding it cut the top-20-chunk retrieval failure rate by 35%. That is the vendor's claim about its own method (Anthropic, 2024).
The store's products are short, partly because the book's sample cuts every description to about 400 characters (data card). Of the descriptions, 5,933 end in that cut, and 1,568 products have none.
Non-empty descriptions run to a median of 68 words, and the whole text that search reads runs to at most 140 words per product. The store's full product pages would be longer.
Even so, each product is a short text about one thing, so cutting it into chunks isn't the store's question. Its question is which part of each product to index. A sentence from a description shows the risk: the last sentence of the chandelier's description, from the support check, reads the antler 's durability and beauty will last a lifetime ., which never says what the product is.
The recorded embeddings cover each product's whole text, so we try the units with BM25 only, on the 50 train queries, because choosing a unit is tuning. We index five units: the whole product (the text lib.lexical indexes), the name alone, the description alone, each of the 35,061 description sentences on its own, and each sentence with the product's name, class and category in front, a simple form of adding context to each chunk. The last sentence of a cut description is a fragment. The sentences with the other fields keep every field the whole product has, and a product with no description gets one unit of those fields, the same as its whole text, so the two units differ only in cutting the description. UnitSearch in lib.generation ranks each product by its best-scoring unit, so each run still lists products. We report the precision@5 and recall@5 a five-product context would have, with judged@5, the share of the top 5 with a judgment. UnitSearch orders equal scores by product_id, so the code also reverses that rule and counts the train queries whose top 5 changes:
1"""Compares five ways of cutting products into retrievable units under BM25 on the train queries, by precision@5,2recall@5 and judged@5, and counts how often ties change the top 5. The point rests on the precision@5 and3recall@5 lines for the whole product and for the sentences with the other fields."""4import numpy as np56from lib.generation import UnitSearch, sentences7from lib.metrics import judged_at_k, precision_at_k, recall_at_k8from lib.store import load_products, load_queries, product_text, qrels910products = load_products()11# words per description, counted by whitespace12lengths = products["description"].str.split().str.len()13print(f"products with no description: {(lengths == 0).sum():,}")14# the book's sample ends a description it cut with ' ...'15print(f"descriptions the sample cut short: {products['description'].str.endswith(' ...').sum():,}")16# empty descriptions are left out of the median17print(f"description words, median of the others: {lengths[lengths > 0].median():.0f}")18print(f"product text words per product, most: {product_text(products).str.split().str.len().max()}")1920header = products["name"] + ". " + products["product_class"] + ". " + products["category"] + ". " # the other fields21# a product with no description keeps one empty piece, so it still gets a unit22pieces = [(p, piece) for p, text in products["description"].items() for piece in sentences(text) or [""]]23# the empty placeholders for products with no description don't count24print(f"description sentences: {sum(piece != '' for _, piece in pieces):,}")25# 8171 is the chandelier from the support check26print("last sentence of product 8171:", sentences(products.at[8171, "description"])[-1])27units = { # unit name: (texts to index, the product each belongs to)28 "whole product": (product_text(products), products.index),29 "name only": (products["name"], products.index),30 "description only": (products["description"], products.index),31 "sentences": ([piece for _, piece in pieces], [p for p, _ in pieces]),32 "sentences with the other fields": ([header[p] + piece for p, piece in pieces], [p for p, _ in pieces]),33}34train, judgments = load_queries("train"), qrels("train") # choosing a unit is tuning: train queries only35scores = {}36for name, (texts, owners) in units.items():37 run = UnitSearch(list(texts), list(owners)).run(train, k=5) # equal scores by product_id38 for metric in [precision_at_k, recall_at_k, judged_at_k]:39 scores[name, metric] = np.mean([metric(run[q], judgments[q], 5) for q in train.index])40 print(f"{name}, train {metric.__name__.replace('_at_k', '')}@5: {scores[name, metric]:.3f}")41 flipped = UnitSearch(list(texts), [-p for p in owners]).run(train, k=5) # negated owners: ties go to the highest ID42 changed = sum(set(run[q]) != {-p for p in flipped[q]} for q in train.index)43 print(f"{name}, train queries whose top 5 changes with ties reversed: {changed}")44print("highest train precision@5:", max(units, key=lambda name: scores[name, precision_at_k]))45for metric in [precision_at_k, recall_at_k]:46 gap = scores["whole product", metric] - scores["sentences with the other fields", metric]47 print(f"whole product minus sentences with the other fields, train {metric.__name__.replace('_at_k', '')}@5: {gap:.3f}")
python retrievable-unit-on-train.pyproducts with no description: 1,568
descriptions the sample cut short: 5,933
description words, median of the others: 68
product text words per product, most: 140
description sentences: 35,061
last sentence of product 8171: the antler 's durability and beauty will last a lifetime .
whole product, train precision@5: 0.464
whole product, train recall@5: 0.124
whole product, train judged@5: 0.940
whole product, train queries whose top 5 changes with ties reversed: 4
name only, train precision@5: 0.368
name only, train recall@5: 0.091
name only, train judged@5: 0.960
name only, train queries whose top 5 changes with ties reversed: 28
description only, train precision@5: 0.304
description only, train recall@5: 0.049
description only, train judged@5: 0.828
description only, train queries whose top 5 changes with ties reversed: 4
sentences, train precision@5: 0.272
sentences, train recall@5: 0.034
sentences, train judged@5: 0.732
sentences, train queries whose top 5 changes with ties reversed: 18
sentences with the other fields, train precision@5: 0.428
sentences with the other fields, train recall@5: 0.106
sentences with the other fields, train judged@5: 0.944
sentences with the other fields, train queries whose top 5 changes with ties reversed: 8
highest train precision@5: whole product
whole product minus sentences with the other fields, train precision@5: 0.036
whole product minus sentences with the other fields, train recall@5: 0.018
| Unit | precision@5 | recall@5 | judged@5 | Top 5 changes with ties reversed |
|---|---|---|---|---|
| Whole product | 0.464 | 0.124 | 0.940 | 4 |
| Name only | 0.368 | 0.091 | 0.960 | 28 |
| Description only | 0.304 | 0.049 | 0.828 | 4 |
| Sentences | 0.272 | 0.034 | 0.732 | 18 |
| Sentences with the other fields | 0.428 | 0.106 | 0.944 | 8 |
The highest train precision@5 goes to the whole product. The fair comparison is with the sentences that carry the other fields: the same fields, the same products within reach, and judged@5 of 0.944 against 0.940 for the whole product.
Cutting the description into sentences costs 0.036 in train precision@5 and 0.018 in recall@5. Ties decide part of those gaps: where products tie across the fifth place, the tie rule picks which reach the context, and reversing it changes the top 5 for 8 train queries with the sentences with the other fields and 4 with the whole product. Read a gap this size as rough.
The other units also drop fields. The description alone can never return the products that have no description, and bare sentences surface many products nobody judged, so their precision is partly a guess. These are train numbers and untested: they inform a choice and prove nothing.
Working rule: When your items are short, index each whole item as one unit, as the store does. When they are long, chunk them, put the item's title in front of every chunk, and choose the chunk size on train queries by the context measures of the previous section. Pass the LLM the whole item, or the chunk with its title, so it knows what it is reading. Before you trust a small gap, count the queries whose top k changes when ties break the other way.
此部分尚未翻譯,因此以英文呈現。
Read an agent's searches from its trace
Unlike the recorded answers, an agent searches for itself: it is an LLM that runs in a loop and acts. A tool is a function the program offers the model, described by a name, a description and its parameters. A tool call is the model's request to run a tool with arguments. In the agent loop, the program runs each tool call, sends back the result, and the model decides what to do next. In Ollama, which ran the store's model, an agent loop lets the model decide when to invoke tools and fold their results into its replies (Ollama documentation). recordings/agent_trace.py is such a loop. ReAct agents interleave reasoning traces with task-specific actions. On question answering (HotpotQA) and fact verification (Fever), with a Wikipedia API as the tool, they overcome the hallucination and error propagation that are common in chain-of-thought reasoning (Yao et al., 2023). The trace is the record of every tool call and its result.
An agent's searches are queries, and they need not look like a shopper's. In a log of 14 million agent search requests, more than 90% of multi-turn sessions contain at most ten steps, and on average 54% of the query terms a search adds already appear in the accumulated evidence (Ning et al., 2026). The agent picks up words from what it has read.
The recording gave the agent each test query as the shopper's request, with one tool, search, described as "Search the store's catalog. Returns the top 5 products: ID, name, class and attributes." It may search up to 3 times, with any wording, then reply with three product IDs. The tool returns no description.
The recording ran twice, with the same model and prompt: once with BM25 behind the tool, and once with semantic retrieval, which embedded the agent's search text with the same embedding model as the products. The traces store each BM25 search's results in bm25s's own tie order, as the answers' contexts do, so 7 of the 32 BM25 searches get results that differ from what lib.lexical returns, 6 of them only in order. Here is what the agent actually searched for:
1"""Reads the recorded agent traces for both tools: each agent's first search, how many searches each tool took, and2whether BM25's stored results match lib.lexical. The point rests on the first-search lines and the one-search3trace counts."""4from lib.lexical import LexicalSearch5from lib.recorded import agent_traces, manifest, rag_answers6from lib.store import load_products, load_queries78recording, queries = manifest("agent-trace"), load_queries("test")9print("tool description:", recording["prompts"]["tool"]["function"]["description"])10print("searches allowed:", recording["max_searches"])11traces = {(t["retriever"], t["query_id"]): t for t in agent_traces()}12print("traces:", len(traces))13# index 0 is the agent's first search14first = {key: t["searches"][0]["query"] for key, t in traces.items()}15print("first searches the same under both tools:", sum(first[("bm25", q)] == first[("semantic", q)] for q in queries.index))16print("first searches that keep the query as typed:", sum(first[("bm25", q)] == queries.at[q, "query"] for q in queries.index))17for q in queries.index:18 if first[("bm25", q)] != queries.at[q, "query"]:19 print(f"first search for {queries.at[q, 'query']}: {first[('bm25', q)]}")20# the five products each recorded answer read, keyed by context (semantic, bm25 or random) and query21context = {(a.context, a.query_id): a.context_ids for a in rag_answers().itertuples()}22print("traces whose first search returns the answer recording's context:",23 # lists compared with ==: the same five products, in the same order24 sum(t["searches"][0]["results"] == context[key] for key, t in traces.items()))25for tool in ["bm25", "semantic"]:26 mine = [t for (retriever, _), t in traces.items() if retriever == tool]27 print(f"{tool}, searches: {sum(len(t['searches']) for t in mine)}")28 print(f"{tool}, traces with one search: {sum(len(t['searches']) == 1 for t in mine)}")2930# `shown` is how many products the search tool returns per search (its top 5)31lexical, shown = LexicalSearch(load_products()), recording["results_shown"]32stored = [s for (tool, _), t in traces.items() if tool == "bm25" for s in t["searches"]] # what the agent saw33rerun = [lexical.search(s["query"], shown).index.tolist() for s in stored] # ties by product_id34differ = [(s["results"], r) for s, r in zip(stored, rerun) if s["results"] != r]35print("bm25 searches whose results differ from lib.lexical:", len(differ))36# the same products in another order: a tie-break, not a different result37print("bm25 searches whose results differ from lib.lexical only in order:", sum(set(a) == set(b) for a, b in differ))
python agent-searches-in-traces.pytool description: Search the store's catalog. Returns the top 5 products: ID, name, class and attributes.
searches allowed: 3
traces: 60
first searches the same under both tools: 30
first searches that keep the query as typed: 26
first search for 48" sliding single track , barn door for laundry: 48" sliding single track barn door laundry
first search for counter top one cup hot water dispenser: counter top hot water dispenser one cup
first search for refrigerator with ice an water in door: refrigerator with ice and water in door
first search for tye dye duvet cover: tie dye duvet cover
traces whose first search returns the answer recording's context: 55
bm25, searches: 32
bm25, traces with one search: 29
semantic, searches: 35
semantic, traces with one search: 27
bm25 searches whose results differ from lib.lexical: 7
bm25 searches whose results differ from lib.lexical only in order: 6
The first search is the same under both tools for 30 queries, as expected: the agent writes it before the tool has returned anything, from the same prompt and tool description, at temperature 0. It keeps the query as typed for 26.
Of the rest, one fixes a misspelling: tye dye duvet cover becomes tie dye duvet cover. Another fixes a real-word error that chapter 8's corrector misses: an becomes and in refrigerator with ice and water in door. Both an and and are stop words for BM25 (chapter 5), so only semantic retrieval sees that change. The other two become 48" sliding single track barn door laundry and counter top hot water dispenser one cup.
In 55 of the 60 traces, the first search returns exactly the five products that the answer recording put in its context.
The agent rarely looks again. With BM25 it makes 32 searches in all, and 29 traces stop after one. With semantic retrieval it makes 35, and 27 stop after one.
Where do the words of a later search come from? To find out, set each later search against the words already searched, and against the names, classes and attributes the tool had shown:
1"""Counts the words each later search adds to the agent's query so far, and how many of those the tool had already2shown. The point rests on the "words the tool had shown" line for each tool."""3from lib.generation import shown, words4from lib.recorded import agent_traces5from lib.store import load_products, load_queries67products, queries = load_products(), load_queries("test")8for tool in ["bm25", "semantic"]:9 later = new = from_results = 010 for trace in (t for t in agent_traces() if t["retriever"] == tool):11 # words are lowercase runs of letters and digits, so case and punctuation don't count12 asked, seen = words(queries.at[trace["query_id"], "query"]), set()13 for number, search in enumerate(trace["searches"]):14 if number > 0: # a later search: the words it adds, and where from15 added = words(search["query"]) - asked16 later, new, from_results = later + 1, new + len(added), from_results + len(added & seen)17 # every word the agent has searched so far18 asked |= words(search["query"])19 # what the tool showed for that search: name, class and attributes, but no description20 seen |= words(shown(search["results"], products))21 print(f"{tool}, later searches: {later}")22 print(f"{tool}, words that later searches add: {new}")23 print(f"{tool}, of those, words the tool had shown: {from_results}")
python agent-later-searches.pybm25, later searches: 2
bm25, words that later searches add: 2
bm25, of those, words the tool had shown: 1
semantic, later searches: 5
semantic, words that later searches add: 4
semantic, of those, words the tool had shown: 4
Later searches add few words of their own. Under BM25, 2 later searches add 2 words, 1 of them shown by the tool before. Under semantic retrieval, 5 later searches add 4 words, 4 of them shown before. Counts this small describe these traces; they are not a rate to compare with Ning et al.'s.
Working rule: Read the traces before you score an agent. Count its searches per task, compare each search with the shopper's words, and find where added words come from. Add the agent's searches to the queries you test your retriever on, scored with the judgments for the shopper's request or judged afresh, because a retriever tuned on people's queries hasn't been tested on an agent's. Expect an agent that mostly passes the shopper's words through to be bounded by the same retrieval as a RAG answer.
比較代理人工具背後的檢索器
能夠重新搜尋的代理人,看起來似乎會讓檢索器變得不那麼重要。但有一項基準測試的結果並非如此。BrowseComp-Plus 固定了代理人所搜尋的文件集合,因此可以在同一個代理人內抽換檢索器。使用 BM25 時,GPT-5 達到 55.9%;使用 Qwen3-Embedding-8B 檢索器時則達到 70.1%,而且平均搜尋呼叫次數更少,分別為 21.74 次與 23.23 次(Chen et al., 2025, Table 1)。購物是代理人的經典任務。WebShop 收錄了 118 萬件真實世界的商品,其作者在 2022 年提出的最佳模型,任務成功率為 29%,而人類專家為 59%(Yao et al., 2022)。在由四間模擬商店組成的 WebMall 上,其作者回報最佳的代理人完成任務的比率低於 65%,任務內容包括搜尋最便宜的商品,以及搜尋描述模糊的商品(Peeters et al., 2025)。
若要求代理人從搜尋結果中挑選,它就應該只挑選搜尋結果所回傳的內容,而它的軌跡可以顯示它是否做到。因此,它的挑選結果受限於檢索,就像 RAG 的回答一樣。
商店的已記錄代理人以單一工具搜尋商品目錄,最多搜尋 3 次,然後挑選三項商品。依各個工具,以 precision@3 為這些挑選結果評分,是本章規劃的比較中的第二項;若其排列檢定的 p 值低於 0.025,則視為成立。Precision@3 是完全符合的挑選數除以 3,因此缺少的挑選會算作未命中。首先,針對每個工具,計算挑選在哪些地方出現缺漏:工具從未回傳的挑選、無人評判的挑選、挑選少於三項的軌跡、搜尋結果中沒有任何完全符合商品的軌跡,以及看過完全符合商品卻一項也沒挑選的軌跡。接著,將挑選結果與該工具在第一次搜尋時自己的前 3 名比較,這是一項探索性檢查:
1"""Scores each tool's agent picks with precision@3 and counts how the traces went, with an exploratory check2against the tool's first search. The point rests on the precision@3 lines and the picks-the-tool-never-returned3counts."""4import pandas as pd56from lib.generation import EXACT, report_planned7from lib.metrics import precision_at_k8from lib.recorded import agent_traces9from lib.significance import compare10from lib.store import qrels1112judgments, picks, top_three, below = qrels("test"), {}, {}, 013for tool in ["bm25", "semantic"]:14 traces = [t for t in agent_traces() if t["retriever"] == tool]15 # every product any of the agent's searches returned16 returned = [{p for search in t["searches"] for p in search["results"]} for t in traces]17 # exact products each trace's searches returned; a product nobody judged counts as not exact18 seen = [sum(judgments[t["query_id"]].get(p, 0) == EXACT for p in r) for t, r in zip(traces, returned)]19 picked = [sum(judgments[t["query_id"]].get(p, 0) == EXACT for p in t["picked"]) for t in traces]20 print(f"{tool}, picks the tool never returned: {sum(len(set(t['picked']) - r) for t, r in zip(traces, returned))}")21 print(f"{tool}, picks nobody judged: {sum(p not in judgments[t['query_id']] for t in traces for p in t['picked'])}")22 print(f"{tool}, traces with fewer than three picks: {sum(len(t['picked']) < 3 for t in traces)}")23 print(f"{tool}, traces whose searches returned no exact product: {sum(s == 0 for s in seen)}")24 print(f"{tool}, traces that saw an exact product and picked none: {sum(s > 0 and k == 0 for s, k in zip(seen, picked))}")25 # precision_at_k divides by 3, so a trace with fewer picks scores lower26 picks[tool] = pd.Series({t["query_id"]: precision_at_k(t["picked"], judgments[t["query_id"]], 3) for t in traces})27 # index 0 is the agent's first search28 top_three[tool] = pd.Series({t["query_id"]: precision_at_k(t["searches"][0]["results"], judgments[t["query_id"]], 3)29 for t in traces}) # the tool's own top 3 for the first search30 print(f"{tool}, precision@3 of the picks: {picks[tool].mean():.3f}")31 print(f"{tool}, precision@3 of the first search's top 3: {top_three[tool].mean():.3f}")32 p_value = compare(picks[tool], top_three[tool])["permutation p"]33 print(f"{tool}, picks against the top 3, exploratory permutation p: {p_value:.3f}")34 # uncorrected: this check is exploratory, not one of the planned comparisons35 below += p_value < 0.053637print(f"picks against the top 3, p-values below 0.05 uncorrected: {below} of 2")38report_planned("agent picks, semantic over bm25", picks["semantic"], picks["bm25"])
python agent-picks-by-tool.pybm25, picks the tool never returned: 0
bm25, picks nobody judged: 3
bm25, traces with fewer than three picks: 2
bm25, traces whose searches returned no exact product: 5
bm25, traces that saw an exact product and picked none: 6
bm25, precision@3 of the picks: 0.567
bm25, precision@3 of the first search's top 3: 0.500
bm25, picks against the top 3, exploratory permutation p: 0.152
semantic, picks the tool never returned: 0
semantic, picks nobody judged: 0
semantic, traces with fewer than three picks: 0
semantic, traces whose searches returned no exact product: 5
semantic, traces that saw an exact product and picked none: 2
semantic, precision@3 of the picks: 0.622
semantic, precision@3 of the first search's top 3: 0.633
semantic, picks against the top 3, exploratory permutation p: 1.000
picks against the top 3, p-values below 0.05 uncorrected: 0 of 2
agent picks, semantic over bm25, mean difference: 0.056
agent picks, semantic over bm25, wins, losses, ties: 5, 4, 21
agent picks, semantic over bm25, permutation p: 0.422
agent picks, semantic over bm25, established at 0.025: no
agent picks, semantic over bm25, t-test p: 0.305
agent picks, semantic over bm25, t-test agrees: yes
agent picks, semantic over bm25, 97.5% interval: -0.033 to 0.209
agent picks, semantic over bm25, interval agrees: yes
兩個工具的挑選結果沒有顯示出已成立的差異。語意檢索的 precision@3 為 0.622,BM25 為 0.567,兩者差異的排列檢定 p 值為 0.422(兩項比較)。就這個模型、提示詞與這些測試查詢而言,這並不能證明檢索器無關緊要,只能說明這些查詢沒有顯示出差異。
以相同方式為每個工具計算,未命中可分為以下幾類。工具從未回傳的挑選,在 BM25 下有 0 項,在語意檢索下有 0 項,因此上述的限制成立。檢索也限制了挑選結果:在 BM25 下有 5 條軌跡、在語意檢索下有 5 條軌跡,沒有任何一次搜尋回傳完全符合的商品,因此不可能挑選到完全符合的商品。
其他的未命中出在挑選本身。在 BM25 下,代理人看過完全符合的商品卻一項也沒挑選的軌跡有 6 條,語意檢索下則為 2 條;其挑選中未經評判的有 3 項,語意檢索下為 0 項;挑選少於三項的軌跡有 2 條,語意檢索下為 0 條。未經評判的挑選與缺少的挑選都算作未命中。
在兩個工具下,代理人的挑選結果呈現相同的樣貌。工具在第一次搜尋時自己的前 3 名,在 BM25 下得分為 0.500,在語意檢索下為 0.633。以該前 3 名為基準來衡量挑選結果,排列檢定的 p 值在 BM25 下為 0.152,在語意檢索下為 1.000,而 0 of 2 低於未校正的 0.05。這項檢查是在看過測試結果之後才選定的,因此絕不能視為已成立。
實用原則: 測試代理人工具背後的檢索器,方式與為人類測試時相同:針對代理人自己的搜尋與最終挑選,並以人工判斷評分。確認每項挑選都來自工具的回傳結果。計算工具從未回傳相關商品的任務數量,因為這些任務沒有任何代理人能夠補救,只有檢索能夠改善。
此部分尚未翻譯,因此以英文呈現。
What goes wrong in production
A live assistant answers every shopper with a fresh model call, on a context that retrieval builds for that request.
The model, the prompt and the context are one system. A different model version, a reworded prompt, another context size or a different order of products can change every answer. The store's manifests pin the model's digest, the prompts, the answer and tool schemas and the context window. Change any of them and score the answers again, against the same judgments and the random floor. Keep the raw replies and each context, as the recordings do.
The citation check belongs on the answer's path. The ID check is cheap, so run it on every answer before the shopper sees it, and log each failure with the answer and its context.
Abstentions move with retrieval. Track the share of answers that recommend nothing. On the store's random contexts, 29 of the 30 answers recommend nothing, so a rise in that share can mean that retrieval or the catalog changed. Read those answers before you assume the model did.
Product text reaches the prompt. Product descriptions may come from outside your team, and the LLM reads them as part of its input. Chapter 4 cites a study in which words and instructions planted in a text misled LLM judges. Test the assistant on products whose text contains instructions before you trust it with a catalog you don't write.
Longer contexts change the answer. Raising k lengthens the context and brings in more distractors, and both can hurt (Liu et al., 2024; Jin et al., 2024). Measure answers at each setting on train queries.
Agent traffic is a different workload. In the logs that Ning et al. studied, agents send many machine-written queries per task, while the store's agent mostly sent one. One study warns that with agent traffic, caching may lose effectiveness, and query pre-processing may add overhead without improving results (Pezzuti et al., 2026). Log agents' searches apart from people's, and test the retriever on both.
Working rule: Version the model, prompt and context settings with the answers, check citations on every answer, track abstentions, and keep agents' searches in a test set of their own.
本章做出的決定
只有成立的計畫比較,才可能改變商店的檢索器;表中「成立」一列說明各項比較是否成立。
本章的 2 項計畫比較,是在測試查詢上以語意檢索減去 BM25,其成立的門檻為 0.025:
- **脈絡:語意檢索取得的五件完整商品,**即待超越的排序器。五件是錄製時固定且未經檢驗的設定。回答顯示語意檢索與 BM25 之間沒有成立的差異,因此檢索器仍以第 3 章成立的 nDCG@10 領先為依據。
- **以人工評分為回答打分:**計算每則回答所引用內容的精確占比,並一併計入未推薦任何商品的回答,再以隨機脈絡作為下限。
- **每則回答都依 ID 檢查引用。**錄製的回答通過此檢查,此檢查予以保留。以 NLI 模型或 LLM 評審進行的支持度檢查,必須先以人工樣本校準才算數。
- **以 precision@5 和依判斷計算的脈絡精確率衡量脈絡,**並列出 recall@5,因為當一個查詢的精確商品多於五件時,五件商品脈絡的 recall@5 不可能達到 1。
- **可檢索單位:整件商品。**它的訓練 precision@5 最高;把描述切成句子,即使在前面附上其他欄位,也會損失 0.036。
- **代理人的工具:相同的檢索器,在代理人的搜尋與選品上檢驗。**選品顯示工具之間沒有成立的差異,而且每一個選品都來自工具所回傳的內容。
- **範圍:一個模型、兩個提示、測試查詢。**所有關於 LLM 的發現,都只限於使用所錄製提示的
gemma4:12b-it-q4_K_M。更換模型或提示,就是一項需要重新評分的變更,如同更換檢索器一樣。 - **本章所展示的:「**引用檢查只能顯示某件商品出現在模型面前,不能顯示它是否合適;而排序上的進步,必須在它所供應的回答上加以檢驗。
本章,於本版次中
版次 2026.11.10範例於 2026年11 月 19 通過測試本章 10 個範例 個範例中有 10 個通過
譯自英文書,版次 2026.11.10。