面向 LLM 应用的检索
阅读约需 88分钟 分钟,10节中含10个示例。
学完本章,你将能够衡量 LLM 作答时所依赖的检索:按答案引用的商品为答案打分,将每条引用与检索返回的结果核对,衡量 LLM 读到的商品,选择把什么作为一个可检索单元建立索引,并从工具调用记录中读出智能体的搜索。
商店里的每次搜索,最终都会得到一份按相关度排序的家具和家居用品列表,由购物者自己阅读。购物助手也可以改用文字作答。购物者写下自己想要什么,LLM推荐几件商品,并用商品 ID 逐一标明出处。提示词要求 LLM 只推荐搜索呈现给它的商品,因此助手的表现不可能超过其检索的水平。智能体也是如此:它是自行执行搜索并决定搜索次数的 LLM。
因此,本章对照商店的人工判断,为答案、引用、上下文和智能体的搜索打分。关于 LLM 的每一项发现,都只适用于这个模型、这些提示词和这 30 条测试查询,而不适用于一般的 LLM。在生成答案时,LLM 以文本形式读取每件商品的名称、类别、分类、描述和属性,而智能体的搜索工具向它展示的内容更少。人工评审者还看到了每件商品的图片和网页(见LLM 看到了什么)。语义检索依据 Snowflake/snowflake-arctic-embed-m-v1.5 的已记录嵌入向量排序,而 BM25 是本书未做词干提取的基线,即 lib.lexical。
在查看任何结果之前,我们就计划了 2 项比较,全部基于测试查询,每项比较中都是语义检索对阵 BM25:
这 2 项计划比较共同分摊本书的 0.05,因此每项都需要做配对置换检验,且 p 值须低于 0.025。本章还报告配对 t 检验和 97.5% 的自助法区间。测试查询上的其他结果只是描述,或标注为探索性结果。
把检索到的产品放进提示词
检索增强生成(retrieval-augmented generation,RAG)是购物助手背后的设计,它把大语言模型与检索器配对使用。对每个请求,检索器先从索引中找出几段文本,大语言模型再据此写出回答。Lewis 等人的 RAG 模型把非参数记忆保存为维基百科的稠密向量索引,由一个预训练的神经检索器来搜索。这些作者还指出,语言模型单独使用时有两个问题仍是开放的研究问题:为其决策提供出处,以及更新其世界知识(Lewis et al., 2020)。商店同时面对这两个问题:商品目录会变化,回答也应说明推荐了哪些产品。检索到的文本被放进提示词(prompt),即模型读取的输入;提示词中检索到的文本就是上下文(context)。模型的上下文窗口(context window)是它最多能读取的词元数,词元是其自身词表中的词片,数量固定。
为什么不把整个商品目录放进提示词,省去检索?首先,它必须放得下;其次,即使放得下,模型阅读长提示词时也并不均匀。Liu 等人发现,当相关信息位于输入的开头或结尾时,模型往往表现最好;当相关信息位于长上下文中间时,性能会显著下降(Liu et al., 2024)。后来的一项研究把这称为“中间迷失”(lost-in-the-middle,LiM)效应。该研究发现,当输入占据模型上下文窗口的比例最高达 50% 时,这种效应最强;接近上限时,它让位于偏向结尾的倾向(Veseli et al., 2025)。段落越多也并不总是越好。Jin 等人给长上下文大语言模型提供更多检索到的段落时,输出质量先是提高,随后下降,他们把检索到的困难负例(看起来相关但实际不相关的段落)视为造成下降的一个关键因素(Jin et al., 2024)。一家厂商的工程博客文章对小型集合给出了相反的建议:当知识库不足 200,000 个词元(约 500 页资料)时,建议把整个知识库放进提示词(Anthropic, 2024)。这是厂商的经验法则,不是测量结果。
商店使用的大语言模型是开源模型 gemma4:12b-it-q4_K_M。它的录制文件(recordings)recordings/rag_answers.py 和 recordings/agent_trace.py 与第 4 章的录制一样,对每次模型调用只做一次,使用温度 0、固定随机种子并关闭思考(第 9 章),并保存了回复,示例通过 lib.recorded 读取这些回复。每个录制文件的清单(manifest,即描述它的文件)把该模型的许可证写为 Apache-2.0。
商店的录制回答(为每个回答打分)在每个提示词中放入五个产品,并在其前面加上这些指令:“You help shoppers at an online furniture and home-goods store. Using only the products listed below, answer the shopper. Recommend up to three products that fit, cite each one by its ID, and say plainly if none of them fit. Don't mention products that aren't listed.”商品目录相当于多少个提示词?按大语言模型读取的方式数一数它的词数,就能回答这个问题。
1"""Counts the catalog's words as the LLM reads each product, then sets them against a median prompt of the2instructions and five products, and against the recording's context window. The point rests on the "catalog3against one prompt" line."""4from statistics import median56from lib.agreement import as_the_llm_saw7from lib.recorded import manifest, rag_answers8from lib.store import load_products910recording = manifest("rag-answers")11# num_ctx is the context window, in tokens, that the recording set12window = recording["options"]["num_ctx"]13print("model:", recording["model"])14print("licence in the manifest:", recording["model_licence"])15print("instructions:", recording["prompts"].split("\n")[0])16print("context window the recording set, in tokens:", window)1718products = load_products()19# words are whitespace-separated pieces, since this model's tokenizer isn't in the book's lib20words = {product_id: len(f"ID: {product_id}\n{as_the_llm_saw(product)}".split()) # each product as the LLM read it21 for product_id, product in products.iterrows()}22instructions = len(recording["prompts"].split()) # the prompt's own words, without the query23# one prompt per recorded answer: the instructions, then the five products' words24prompts = [instructions + sum(words[p] for p in ids) for ids in rag_answers()["context_ids"]]25catalog = sum(words.values())26print(f"words in the catalog as the LLM reads it: {catalog:,}")27print(f"words in one prompt with five products, median: {median(prompts):,.0f}")28print(f"the catalog against one prompt: {catalog / median(prompts):,.0f} times")29print(f"the catalog against the recording's window, at one token per word: {catalog / window:,.0f} times")30print(f"the catalog against 200,000 tokens, at one token per word: {catalog / 200_000:.1f} times")
python catalog-against-the-context.pymodel: gemma4:12b-it-q4_K_M
licence in the manifest: Apache-2.0
instructions: You help shoppers at an online furniture and home-goods store. Using only the products listed below, answer the shopper. Recommend up to three products that fit, cite each one by its ID, and say plainly if none of them fit. Don't mention products that aren't listed.
context window the recording set, in tokens: 8192
words in the catalog as the LLM reads it: 1,091,544
words in one prompt with five products, median: 536
the catalog against one prompt: 2,035 times
the catalog against the recording's window, at one token per word: 133 times
the catalog against 200,000 tokens, at one token per word: 5.5 times
即使本书的样本已把描述截短到约 400 个字符(数据卡),商品目录仍有 1,091,544 个词,而含五个产品的提示词词数的中位数为 536。商品目录比一个提示词大 2,035 times。
由于没有这个模型的分词器,这里的计数以词为单位。录制时设定的上下文窗口为 8192 个词元。假设每个词恰好是一个词元,商品目录会把该窗口填满 133 times 次,而对于厂商所说的 200,000 个词元则是 5.5 times 次。商店必须进行检索。
工作准则: 提示词中放入少量检索到的条目,而不是整个集合;放入多少条,要通过在训练查询上以多种规模测量回答来决定,因为更多的上下文可能有帮助,也可能有害。商店的录制只回答了测试查询,每个查询五个产品,所以这里的五是固定设置,而不是测量得出的。只有当整个集合能轻松放进窗口内时,才考虑把它整体发送,并且要在相同的查询上将其与检索进行对比测试。
这部分尚未翻译,因此以英文显示。
Score each answer by the products it cites
An LLM told to answer from a context should recommend only what the context holds, and the ID check tests that it does. If no exact product is in the context, no prompt makes a grounded answer recommend one. Retrieval sets a ceiling on the answer, and what else the context holds matters too.
Cuconasu et al. found that the retriever's highest-scoring documents that aren't directly relevant, ones that don't contain the answer, hurt answers. More surprisingly, adding random documents to the prompt improved LLM accuracy by up to 35% (Cuconasu et al., 2024). Products that look right but aren't are distractors, the store's version of those passages. In question answering, Joren et al. found a related pattern. When the context lacked what was needed, the larger models they tested, Gemini 1.5 Pro, GPT 4o and Claude 3.5, often gave incorrect answers instead of abstaining. Smaller ones, Mistral 3 and Gemma 2 among them, often hallucinated or abstained even when the context was sufficient (Joren et al., 2024).
A citation here is the ID an answer gives for a product it recommends. With judgments, you can score an answer without reading it: look up each cited product's grade. The exact share of an answer is the share of its cited products that people judged exact. An answer that cites nothing scores 0, because it gives the shopper nothing that fits, whether or not that was the right call. A separate count of answers that recommend nothing covers that case.
exact share = (cited products judged exact) / (products cited), 0 when the answer cites nothing
The store's recording answered each test query three times, with the query following "I'm looking for". Each answer came from a different context of five products: semantic retrieval's top 5, BM25's top 5, or five products drawn at random with a fixed seed. The random context is a floor, a context no retriever chose. The model, the prompt and the query stay the same, so the answers differ only because their contexts do. We read each context as the recording stored it, in the order the LLM read it. The BM25 contexts keep tied products in bm25s's own order, not by product_id as lib.lexical does, so 7 of the 30 test queries get a context that differs from what lib.lexical returns, 6 of them only in order. The recording asked for JSON with the answer and the cited IDs in separate fields. score_answers in lib.generation scores each answer on the human grades. The comparison of those scores is the first of the chapter's planned comparisons, established if its permutation p-value is below 0.025.
How many answers will recommend nothing, given five random products and a prompt that tells the LLM to say plainly if none of them fit?
1"""Scores the recorded answers on the human grades, context by context, and makes the first planned comparison. The2point rests on the mean exact share and recommend-nothing lines, and on the permutation p-value of the3comparison."""4from lib.generation import PLANNED_TEST_COMPARISONS, THRESHOLD, report_planned, score_answers5from lib.lexical import LexicalSearch6from lib.recorded import rag_answers7from lib.store import load_products, load_queries, qrels89answers = rag_answers()10stored = answers[answers["context"] == "bm25"].set_index("query_id")["context_ids"] # as the LLM read them11rerun = LexicalSearch(load_products()).run(load_queries("test"), k=5) # ties by product_id12differ = [q for q in stored.index if stored[q] != rerun[q]]13print("bm25 contexts that differ from lib.lexical:", len(differ))14# a match as sets means the same five products in another order: a tie-break, not a different context15print("bm25 contexts that differ from lib.lexical only in order:", sum(set(stored[q]) == set(rerun[q]) for q in differ))1617scores = score_answers(answers, qrels("test")) # one row per test query and context, on human grades18for context, rows in scores.groupby("context", sort=False):19 # held: the context holds an exact product; nothing: the answer cites no product20 held, nothing = rows["exact in context"] > 0, rows["cited"] == 021 print(f"{context}, unjudged products in the contexts: {rows['unjudged in context'].sum()}")22 print(f"{context}, contexts with an exact product: {held.sum()}")23 print(f"{context}, answers that cite an exact product: {(rows['exact cited'] > 0).sum()}")24 print(f"{context}, products cited: {rows['cited'].sum()}")25 print(f"{context}, cited products that are exact: {rows['exact cited'].sum()}")26 print(f"{context}, mean exact share: {rows['exact share'].mean():.3f}")27 print(f"{context}, answers that recommend nothing: {nothing.sum()}")28 print(f"{context}, recommend nothing, with an exact product in the context: {(nothing & held).sum()}")29 print(f"{context}, recommend something, with no exact product in the context: {(~nothing & ~held).sum()}")30 near = rows[~held] # answers whose context held no exact product31 print(f"{context}, products cited with no exact product in the context: {near['cited'].sum()}")32 print(f"{context}, of those, partial: {near['partial cited'].sum()}")33 print(f"{context}, unjudged products cited: {rows['unjudged cited'].sum()}")34print("answers that cite more exact products than their context holds:",35 (scores["exact cited"] > scores["exact in context"]).sum())3637print("planned test comparisons:", len(PLANNED_TEST_COMPARISONS))38print(f"p-value threshold: {THRESHOLD:.3f}")39share = scores.pivot(index="query_id", columns="context", values="exact share")40# the first planned comparison: semantic minus BM25 on the exact share, tested at THRESHOLD41report_planned("answers, semantic over bm25", share["semantic"], share["bm25"])
python rag-answers-by-context.pybm25 contexts that differ from lib.lexical: 7
bm25 contexts that differ from lib.lexical only in order: 6
semantic, unjudged products in the contexts: 0
semantic, contexts with an exact product: 25
semantic, answers that cite an exact product: 22
semantic, products cited: 80
semantic, cited products that are exact: 60
semantic, mean exact share: 0.658
semantic, answers that recommend nothing: 2
semantic, recommend nothing, with an exact product in the context: 1
semantic, recommend something, with no exact product in the context: 4
semantic, products cited with no exact product in the context: 11
semantic, of those, partial: 11
semantic, unjudged products cited: 0
bm25, unjudged products in the contexts: 10
bm25, contexts with an exact product: 24
bm25, answers that cite an exact product: 23
bm25, products cited: 73
bm25, cited products that are exact: 52
bm25, mean exact share: 0.639
bm25, answers that recommend nothing: 3
bm25, recommend nothing, with an exact product in the context: 0
bm25, recommend something, with no exact product in the context: 3
bm25, products cited with no exact product in the context: 7
bm25, of those, partial: 7
bm25, unjudged products cited: 1
random, unjudged products in the contexts: 147
random, contexts with an exact product: 1
random, answers that cite an exact product: 1
random, products cited: 1
random, cited products that are exact: 1
random, mean exact share: 0.033
random, answers that recommend nothing: 29
random, recommend nothing, with an exact product in the context: 0
random, recommend something, with no exact product in the context: 0
random, products cited with no exact product in the context: 0
random, of those, partial: 0
random, unjudged products cited: 0
answers that cite more exact products than their context holds: 0
planned test comparisons: 2
p-value threshold: 0.025
answers, semantic over bm25, mean difference: 0.019
answers, semantic over bm25, wins, losses, ties: 4, 5, 21
answers, semantic over bm25, permutation p: 0.765
answers, semantic over bm25, established at 0.025: no
answers, semantic over bm25, t-test p: 0.715
answers, semantic over bm25, t-test agrees: yes
answers, semantic over bm25, 97.5% interval: -0.078 to 0.161
answers, semantic over bm25, interval agrees: yes
| Measure | Semantic | BM25 | Random |
|---|---|---|---|
| Contexts with an exact product | 25 | 24 | 1 |
| Answers that cite an exact product | 22 | 23 | 1 |
| Products cited | 80 | 73 | 1 |
| Cited products that are exact | 60 | 52 | 1 |
| Mean exact share | 0.658 | 0.639 | 0.033 |
| Answers that recommend nothing | 2 | 3 | 29 |
With random products, the LLM recommends nothing in 29 of the 30 answers, so this model and prompt did not force a recommendation out of random products. The one product it cites is exact, and the random draw happened to include it. The random contexts also hold 147 products nobody judged, which count as not exact. We expect few of them to fit, but no judgment says so.
The ceiling holds: 0 answers cite more exact products than their context holds. Near misses are another matter. Where the context held no exact product, the answer still recommended something 4 times with semantic retrieval and 3 times with BM25, rather than say that no product fits exactly.
Every product those answers cite is partial: 11 of 11 with semantic retrieval and 7 of 7 with BM25, the right kind of product with a modifier wrong. Whether that counts as a failure depends on whether the shopper wants near matches.
The reverse failure is rarer: answers recommend nothing although their context held an exact product, 1 with semantic retrieval and 0 with BM25.
BM25's contexts hold 10 unjudged products, and semantic retrieval's hold 0. The score reads only cited products, and BM25's answers cite 1 unjudged product, which counts as not exact.
Semantic retrieval's lead on nDCG@10 is established (chapter 3). On the first planned comparison, though, its difference from BM25 in mean exact share, 0.019, isn't established: the permutation p-value is 0.765 (both comparisons).
Working rule: Score every answer by its cited products' human grades, and report beside it how many answers recommend nothing, split by whether the context held an exact product. Keep a random context as a floor whenever you change the model or the prompt: an answer that recommends something from random products has invented a fit. Test a retriever's gain on the answers too, because a gain in ranking needn't carry through.
对照上下文检查每条引用
依据性(grounding)是指,回答建立在所给的上下文之上,正如精确份额所假定的那样,而不是建立在模型记忆或编造的内容之上。引用使依据性可以检查,而归因(attribution)要问的是:被引用的来源是否支持它所附着的那条陈述。审计发现,引用往往并不支持。在 2023 年对生成式搜索引擎的一项审计中,只有 51.5% 的生成句子完全得到其引用的支持,只有 74.5% 的引用支持了它所附着的句子(Liu et al., 2023)。ELI5 是 ALCE 基准中的长文本问题,取自 Reddit 论坛 Explain Like I'm Five。在这个数据集上,即便是 2023 年最好的模型,也有 50% 的时候缺乏完整的引用支持(Gao et al., 2023)。
引用要分两个层次检查,先做成本最低的一层。第一层是精确的,不需要模型:每个被引用的 ID 都必须是上下文中包含的 ID。来自上下文之外的 ID,指向的是模型记忆中的或编造出来的商品,无论哪种情况,回答都没有依据。第二层要问被引用商品的文本是否支持回答对它的说法,这需要一个阅读者(检查被引用商品支持什么)。
录制时的模式(schema)要求把被引用的 ID 作为整数列表放在单独的字段中(清单中的 schemas)。模式描述的是字段的形式,并没有规定哪些 ID 可以放进去,也没有规定放多少个。这项检查把每条已录制的回答与其上下文对照,并将该字段与回答文本中写出的上下文 ID 作比较。文本无法暴露来自上下文之外的 ID,因为这样的 ID 看起来和回答中写出的任何其他数字(例如尺寸)没有区别。这正是 ID 应当放在字段中的原因:
1"""Checks every recorded answer's cited IDs against its context, and its text against its cited field. The point2rests on the zero counts for IDs outside the context and for disagreeing text, and on the count of answers that3cite more than three products."""4from lib.generation import ids_in_text5from lib.recorded import rag_answers67answers = rag_answers()8outside = answers.apply(lambda a: [p for p in a["cited"] if p not in a["context_ids"]], axis=1)9# an ID cited twice makes the list longer than the set of its IDs10repeated = answers["cited"].map(lambda cited: len(cited) != len(set(cited)))11# the text is searched only for the context's IDs, so an ID from outside it can't match12written = answers.apply(lambda a: ids_in_text(a["answer"], a["context_ids"]), axis=1) # IDs the text writes out13print("answers:", len(answers))14print("answers that cite an ID outside their context:", (outside.map(len) > 0).sum())15print("answers that cite one ID twice:", repeated.sum())16print("answers that cite nothing:", (answers["cited"].map(len) == 0).sum())17# the prompt asks for up to three products18print("answers that cite more than three products:", (answers["cited"].map(len) > 3).sum())19print("answers whose text and cited IDs differ:", (written != answers["cited"].map(set)).sum())
python citations-inside-the-context.pyanswers: 90
answers that cite an ID outside their context: 0
answers that cite one ID twice: 0
answers that cite nothing: 34
answers that cite more than three products: 3
answers whose text and cited IDs differ: 0
在 90 条已录制的回答中,检查发现有 0 条回答引用了上下文之外的 ID,0 条把同一个 ID 引用了两次,0 条的文本与字段不一致。
有 34 条回答完全没有引用。回答确实违反的指令是推荐的数量:有 3 条回答引用了三个以上的商品,而提示词写的是“最多推荐三个”。
在这份录制数据上,ID 检查没有发现无依据的引用。这是你想要的结果,但不是放弃这项检查的理由:每条回答只需一次集合查找,而换用不同的模型、提示词或上下文长度,都可能使它不通过。通过检查只说明每个被推荐的商品都出现在模型面前,并不说明它合适。
工作准则: 对每条回答运行 ID 检查,离线进行,并在购物者看到之前完成,把上下文之外的 ID 视为失败的回答:丢弃或重试,绝不展示。要求模型把引用放在单独的字段中,使检查不必解析正文,并检查文本与字段是否一致。统计回答违反指令的情况,例如推荐的数量,并用代码强制执行其中重要的那些。
检查被引用的商品支持什么
第二层检查要问的是:被引用的文本是否支持答案中的论断。忠实度(faithfulness)是答案的论断中得到上下文支持的比例。Ragas 文档(其页面未标明版本;所查阅的页面日期为 2025-12-09)的说法是:“如果一个回答的所有论断都能得到检索到的上下文支持,则该回答被视为忠实的”(Ragas 文档)。要检查这一点,需要有人来读。ALCE 使用自然语言推理(natural language inference,NLI)模型。NLI 的任务是判断一段文本是否蕴含另一段文本;ALCE 所用的是在多个 NLI 数据集的合集上微调过的 T5-11B 模型。只有当一条陈述的各条引用合起来蕴含该陈述时,它的引用召回率才为 1(Gao 等,2023,第 3.2 节和第 3.3 节)。LLM 也可以判断支持关系,但有其局限。在 2024 年 TREC(Text REtrieval Conference,文本检索会议的共同评测)的 RAG 赛道中,Thakur 等人发现,在三级量表上,人与 GPT-4o 的预测完全一致的比例,对从零开始做出的判断为 56%,对人从 GPT-4o 的支持标签出发再做修改的判断为 72%。他们还发现,一位独立的人类评判者与 GPT-4o 的相关性高于另一个人与其的相关性,因此分歧有一部分存在于人与人之间(Thakur 等,2025)。
支持不等于相关。一条引用可以忠实于商品的文本,却仍然推荐了错误的商品:文本确实说了答案所称的内容,但该商品并不是购物者要找的。只有评判标注才能发现这一点。
先手工检查商店的一条答案,即语义上下文中测试查询 medium size chandelier 的答案。然后统计已记录的答案中,语义上下文和 BM25 上下文里每个被引用商品的人工评级。
1"""Reads one recorded answer by hand, checking its claim against the cited product's text and the judges' label,2then counts the grades of every product the two retrievers' answers cite. The point rests on the partial label,3the exact count in this answer's context, and the breakdown of cited products."""4from collections import Counter56from lib.agreement import LABELS7from lib.recorded import rag_answers8from lib.store import load_products, load_queries, qrels910products, queries, judgments = load_products(), load_queries("test"), qrels("test")11answers = rag_answers().set_index(["query_id", "context"])1213chandelier, claim = 58, "medium-sized" # one answer, read by hand14answer, judged = answers.loc[(chandelier, "semantic")], judgments[chandelier]15# the answer cites one product, so index 0 is that product16cited = answer["cited"][0]17print("query:", queries.at[chandelier, "query"])18print("answer:", answer["answer"])19print("cited product:", products.at[cited, "name"])20print(f"the cited product's text says {claim}:", "yes" if claim in products.at[cited, "description"] else "no")21print("the judges' label:", LABELS[judged[cited]])22# grade 2 is exact; a product nobody judged has no grade, so it counts as 023print("exact products in this answer's context:", sum(judged.get(p, 0) == 2 for p in answer["context_ids"]))2425retrieved = answers.drop("random", level="context") # every citation from the two retrievers' contexts26# LABELS names each grade; a product with no grade counts as unjudged27labels = Counter(LABELS.get(judgments[query_id].get(p), "unjudged")28 for (query_id, _), cited_ids in retrieved["cited"].items() for p in cited_ids)29print("cited products, semantic and bm25 contexts:", sum(labels.values()))30for label in ["exact", "partial", "irrelevant", "unjudged"]:31 print(f"cited products, {label}:", labels[label])
python faithful-but-partial.pyquery: medium size chandelier
answer: I recommend the sybil 12 - light unique / statement classic / traditional chandelier with antler accents (ID: 8171), as it is specifically described as a perfect medium-sized lighting fixture.
cited product: sybil 12 - light unique / statement classic / traditional chandelier with antler accents
the cited product's text says medium-sized: yes
the judges' label: partial
exact products in this answer's context: 2
cited products, semantic and bm25 contexts: 153
cited products, exact: 112
cited products, partial: 40
cited products, irrelevant: 0
cited products, unjudged: 1
这条答案推荐了一盏吊灯,称它“被明确描述为一款完美的中等大小照明灯具”。被引用商品的描述确实写了 medium-sized(yes),因此支持检查会通过这一论断。商店的评判者把该商品标为 partial:按 WANDS 的定义,它是对的商品类型,但缺少购物者要求的某个修饰语。评判者看到了它的图片和商品页面,而 LLM 没有看到。其上下文中有 2 件完全匹配的商品,而答案一件也没有推荐。就这条答案而言,引用是忠实的,推荐却仍然没有命中。
在两个检索器的答案中,被引用的 153 件商品的分布为:完全匹配 112 件,部分匹配 40 件,不相关 0 件,未评判 1 件。除了无人评判的 1 件之外,每件不是完全匹配的被引用商品都是部分匹配:商品类型是购物者要找的,但按评判者的判断,某个修饰语不对。
工作准则: 先用你的评判标注给答案打分,因为它们说明一条推荐是否合适。再对答案中关于每件商品的论断加一道支持检查,可以用 NLI 模型或 LLM 评判者。用一份人工样本校准该评判者,就像第 4 章所做的,也像 ARES 所做的:它用一小组人工标注的数据点做预测驱动推断(prediction-powered inference,PPI)以减轻预测误差,即用人工标签修正评判者的估计(Saad-Falcon 等,2024)。对于没有评判标注的查询,你手里只有支持检查,所以在报告结果时要说明这一点。
先衡量上下文,再衡量答案
RAG 系统既可能在检索阶段失败,也可能在生成阶段失败,因此除了衡量答案,还要衡量上下文。Ragas 是一个用于评估 RAG 的框架,它在没有参考答案的情况下给出多项评分:检索识别出相关且集中的上下文段落的程度、忠实度,以及答案的相关性。它的上下文召回率则需要参考答案(Es et al., 2024)。它的上下文度量沿用了第 3 章的指标,由 LLM 担任评判者。Ragas 的上下文精确率(context precision)是其上下文度量中考虑排名的一项,即在包含相关片段的各个位置上,对 precision@k 取平均值(Ragas 文档)。它的上下文召回率(context recall)是上下文所支持的参考答案论断所占的比例;另有一种基于 ID 的变体,把检索到的 ID 与参考 ID 相比较(Ragas 文档)。有了判定结果之后,参考就是确切匹配商品的集合,基于 ID 的上下文召回率就是 recall@k。
context recall@k = (exact products in the top k) / (exact products judged for the query)
context precision@k = mean of precision@i over the positions i ≤ k that hold an exact product
哪一项度量能跟踪答案?Salemi 和 Zamani 发现,相关性标签与 RAG 系统的下游表现之间只有较弱的相关性,他们转而根据 LLM 基于每个检索段落给出的答案,为该段落评分(Salemi and Zamani, 2024)。斯皮尔曼等级相关(Spearman's rank correlation)是两项度量的名次之间的相关,而不是它们取值之间的相关:两者对各个案例的排序相同时为 1,排序互不相关时接近 0。它可以就一次记录回答这个问题。
我们对检索器为已记录的答案所构建的每个上下文进行度量,每个测试查询对应一个语义检索的上下文和一个 BM25 的上下文。本示例手工构建上下文召回率和上下文精确率,将它们与 lib.metrics.recall_at_k 以及 lib.generation 中的副本核对,并逐步演算其中一个上下文:
1"""Builds context precision and context recall by hand and checks each against the library's version, then works2through one context. The point rests on the two "every time" lines and on the precision@5 and context precision3of the worked context."""4from lib import generation5from lib.generation import EXACT6from lib.metrics import precision_at_k, recall_at_k7from lib.recorded import rag_answers8from lib.store import load_queries, qrels91011def context_precision(context, judged, k=5):12 """Ragas's context precision, with the judges as the reference: mean precision@i over exact positions i."""13 # positions count from 1, as ranks do; context[:k] is the top k14 hits = [i for i, p in enumerate(context[:k], start=1) if judged.get(p, 0) == EXACT]15 return sum(precision_at_k(context, judged, i) for i in hits) / len(hits) if hits else 0.0161718def context_recall(context, judged, k=5):19 """Ragas's ID-based context recall: the share of the reference IDs that the context holds."""20 # the reference is every exact product judged for the query, not only those in the top k21 reference = {p for p, grade in judged.items() if grade == EXACT}22 return len(reference & set(context[:k])) / len(reference)232425judgments, queries = qrels("test"), load_queries("test")26answers = rag_answers().query("context != 'random'") # the two retrievers' contexts for each test query27contexts = [(answer.context_ids, judgments[answer.query_id]) for answer in answers.itertuples()]28print("contexts:", len(contexts))29same = {"recall_at_k": all(context_recall(c, j) == recall_at_k(c, j, 5) for c, j in contexts),30 "lib.generation": all(context_precision(c, j) == generation.context_precision(c, j) for c, j in contexts)}31print("context recall equals recall_at_k every time:", "yes" if same["recall_at_k"] else "no")32print("context precision equals lib.generation's every time:", "yes" if same["lib.generation"] else "no")3334# the first row: the semantic context for door jewelry organizer35first = answers.iloc[0] # one context, worked through36context, judged = first["context_ids"], judgments[first["query_id"]]37print(f"grades in order, {queries.at[first['query_id'], 'query']}, {first['context']}:", *[judged.get(p, 0) for p in context])38print(f"context precision@5: {context_precision(context, judged):.3f}")39print(f"precision@5: {precision_at_k(context, judged, 5):.3f}")40# exact products judged for each test query, sorted from the fewest to the most41exact = sorted(sum(grade == EXACT for grade in grades.values()) for grades in judgments.values())42print(f"exact products judged per test query: {exact[0]} to {exact[-1]}")43print(f"highest recall@5 possible for the query with the most: {5 / exact[-1]:.3f}")
python context-measures-by-hand.pycontexts: 60
context recall equals recall_at_k every time: yes
context precision equals lib.generation's every time: yes
grades in order, door jewelry organizer, semantic: 1 1 2 2 2
context precision@5: 0.478
precision@5: 0.600
exact products judged per test query: 4 to 170
highest recall@5 possible for the query with the most: 0.029
在 60 个上下文中,上下文召回率每次都等于 recall_at_k(yes),上下文精确率每次都等于 lib.generation 中的结果(yes)。
对于 door jewelry organizer 的语义上下文,按顺序的等级为 1 1 2 2 2,其中 2 表示确切匹配,1 表示部分匹配。由此得到的 precision@5 为 0.600,上下文精确率为 0.478,后者更低,因为确切匹配的商品排在最后。
把每项度量与答案的确切匹配占比做相关分析,就能回答这个问题。本示例把斯皮尔曼相关计算为名次的皮尔逊相关,并与 SciPy 的 spearmanr 核对:
1"""Correlates three context measures with the exact share of each recorded answer, computing Spearman's correlation2by hand and checking it against SciPy. The point rests on the three Spearman lines and the weakest-measure line."""3import pandas as pd4from scipy import stats56from lib.generation import context_precision, context_recall, exact_share7from lib.metrics import precision_at_k8from lib.recorded import rag_answers9from lib.store import qrels1011judgments = qrels("test")12answers = rag_answers().query("context != 'random'") # the two retrievers' contexts for each test query13rows = pd.DataFrame([{"recall@5": context_recall(a.context_ids, judgments[a.query_id]),14 "precision@5": precision_at_k(a.context_ids, judgments[a.query_id], 5),15 "context precision@5": context_precision(a.context_ids, judgments[a.query_id]),16 "exact share": exact_share(a.cited, judgments[a.query_id])} for a in answers.itertuples()])17print("contexts:", len(rows))1819ranks, spearman = rows.rank(), {} # Spearman: Pearson's correlation of the ranks20for measure in ["recall@5", "precision@5", "context precision@5"]:21 spearman[measure] = ranks[measure].corr(ranks["exact share"])22 print(f"spearman with the exact share, {measure}: {spearman[measure]:.3f}")23# rounded to nine places, so floating-point differences in the last digits don't count24matches = all(round(value, 9) == round(stats.spearmanr(rows[measure], rows["exact share"]).statistic, 9)25 for measure, value in spearman.items())26print("matches scipy's spearmanr:", "yes" if matches else "no")27print("weakest with the exact share:", min(spearman, key=spearman.get))
python context-measures-against-answers.pycontexts: 60
spearman with the exact share, recall@5: 0.540
spearman with the exact share, precision@5: 0.797
spearman with the exact share, context precision@5: 0.808
matches scipy's spearmanr: yes
weakest with the exact share: recall@5
手工构建的斯皮尔曼相关与 SciPy 的结果一致(yes)。
与确切匹配占比的相关性最弱的是 recall@5,为 0.540,而 precision@5 为 0.797,上下文精确率为 0.808。
recall@5 的分母说明了原因。测试查询各有 4 to 170 个被判定的确切匹配商品,因此对于确切匹配商品最多的查询,含五个商品的上下文所能达到的最高 recall@5 为 0.029。无论判定的商品有多少,一个答案只需要三条推荐。Precision@5 统计五个商品中有多少个合适,而上下文精确率还会奖励把它们排在前面。
应把这些相关性看作对一次记录的描述,而不是检验。双方使用的是同一套判定,而答案只能引用其上下文中包含的内容,因此一定程度的相关性是先天存在的。此外,这些上下文来自相同的查询,成对出现,即一个语义的和一个 BM25 的。
工作准则: 对于固定大小的上下文,应根据你的判定报告该上下文的 precision@k 或上下文精确率,并在旁边列出 recall@k。当答案必须涵盖每一个相关项时,例如适合某一台电器的所有部件,就使用召回率。没有判定时,Ragas 由 LLM 评判的版本可以估计类似的量;在信任它们之前,应先用人工样本对其进行校准。选择 k 的办法是,在训练查询上以多个 k 值衡量答案,这需要为这些查询记录过答案。
选择可检索单元
可检索单元(retrievable unit)是检索打分并返回的那段文本,也是上下文所容纳的内容:可以是整篇文档、一个段落或一个句子。长文档会被切成块(chunk),而分块(chunking)就是切分文档的规则,例如按固定的词元数、句子数、段落数切分,或按语义选择断点。Chen 等人称检索单元是一项常被忽视的设计选择,在他们的研究中,以命题(简短而自成一体的陈述)建立索引,效果优于以段落建立索引(Chen et al., 2023)。除此之外,各项研究的结论并不一致。有一项研究发现,最佳大小取决于问题和嵌入模型:对于答案简洁、以事实为主的数据集,适宜 64 到 128 个词元;对于需要更广泛上下文的问题,适宜 512 到 1,024 个词元(Bhat et al., 2025)。另一项研究发现,语义分块的计算成本并不能被稳定的性能提升所抵消(Qu et al., 2024)。一项对 36 种方法的比较把按段落组分块排在第一位,并发现简单的固定大小字符块作为基线表现不佳(Shaukat et al., 2026)。
较小的单元还可能丢失自己所讲的对象。一家厂商的工程博文报告说,在嵌入每个块之前为其加上针对该块的上下文,可使前 20 个块的检索失败率降低 35%。这是厂商对其自身方法的说法(Anthropic, 2024)。
商店的商品文本较短,部分原因是本书的样本把每条描述都截断到约 400 个字符(数据卡)。在这些描述中,有 5,933 条以这种截断结尾,另有 1,568 个商品没有描述。
非空描述的词数中位数为 68,搜索所读取的全部文本每个商品最多为 140 个词。商店完整的商品页面会更长。
即便如此,每个商品都是一段关于单一对象的短文本,因此把它切成块并不是商店要解决的问题。它要解决的问题是:索引每个商品的哪一部分。描述中的一个句子就显示了这种风险:吊灯描述的最后一句,出自支持度检查,内容为 the antler 's durability and beauty will last a lifetime .,这句话从未说明该商品是什么。
已记录的嵌入覆盖的是每个商品的完整文本,所以我们只用 BM25 在 50 条训练查询上试验这些单元,因为选择单元就是调参。我们为五种单元建立索引:整个商品(即 lib.lexical 所索引的文本)、仅名称、仅描述、35,061 个描述句子中的每一句单独成一个单元,以及在每个句子前加上商品名称、类和类别的单元,这是为每个块添加上下文的一种简单形式。被截断的描述的最后一句是一个残句。带其他字段的句子保留了整个商品所具有的每个字段,没有描述的商品则得到一个由这些字段构成的单元,与其完整文本相同,因此这两种单元的差别只在于是否切分描述。UnitSearch(位于 lib.generation)按得分最高的单元为每个商品排名,所以每次运行得到的仍是商品列表。我们报告包含五个商品的上下文所具有的 precision@5 和 recall@5,以及 judged@5,即前 5 个结果中有判定的比例。UnitSearch 按 product_id 对得分相同的结果排序,所以代码还会反转该规则,并统计前 5 个结果发生变化的训练查询数:
1"""Compares five ways of cutting products into retrievable units under BM25 on the train queries, by precision@5,2recall@5 and judged@5, and counts how often ties change the top 5. The point rests on the precision@5 and3recall@5 lines for the whole product and for the sentences with the other fields."""4import numpy as np56from lib.generation import UnitSearch, sentences7from lib.metrics import judged_at_k, precision_at_k, recall_at_k8from lib.store import load_products, load_queries, product_text, qrels910products = load_products()11# words per description, counted by whitespace12lengths = products["description"].str.split().str.len()13print(f"products with no description: {(lengths == 0).sum():,}")14# the book's sample ends a description it cut with ' ...'15print(f"descriptions the sample cut short: {products['description'].str.endswith(' ...').sum():,}")16# empty descriptions are left out of the median17print(f"description words, median of the others: {lengths[lengths > 0].median():.0f}")18print(f"product text words per product, most: {product_text(products).str.split().str.len().max()}")1920header = products["name"] + ". " + products["product_class"] + ". " + products["category"] + ". " # the other fields21# a product with no description keeps one empty piece, so it still gets a unit22pieces = [(p, piece) for p, text in products["description"].items() for piece in sentences(text) or [""]]23# the empty placeholders for products with no description don't count24print(f"description sentences: {sum(piece != '' for _, piece in pieces):,}")25# 8171 is the chandelier from the support check26print("last sentence of product 8171:", sentences(products.at[8171, "description"])[-1])27units = { # unit name: (texts to index, the product each belongs to)28 "whole product": (product_text(products), products.index),29 "name only": (products["name"], products.index),30 "description only": (products["description"], products.index),31 "sentences": ([piece for _, piece in pieces], [p for p, _ in pieces]),32 "sentences with the other fields": ([header[p] + piece for p, piece in pieces], [p for p, _ in pieces]),33}34train, judgments = load_queries("train"), qrels("train") # choosing a unit is tuning: train queries only35scores = {}36for name, (texts, owners) in units.items():37 run = UnitSearch(list(texts), list(owners)).run(train, k=5) # equal scores by product_id38 for metric in [precision_at_k, recall_at_k, judged_at_k]:39 scores[name, metric] = np.mean([metric(run[q], judgments[q], 5) for q in train.index])40 print(f"{name}, train {metric.__name__.replace('_at_k', '')}@5: {scores[name, metric]:.3f}")41 flipped = UnitSearch(list(texts), [-p for p in owners]).run(train, k=5) # negated owners: ties go to the highest ID42 changed = sum(set(run[q]) != {-p for p in flipped[q]} for q in train.index)43 print(f"{name}, train queries whose top 5 changes with ties reversed: {changed}")44print("highest train precision@5:", max(units, key=lambda name: scores[name, precision_at_k]))45for metric in [precision_at_k, recall_at_k]:46 gap = scores["whole product", metric] - scores["sentences with the other fields", metric]47 print(f"whole product minus sentences with the other fields, train {metric.__name__.replace('_at_k', '')}@5: {gap:.3f}")
python retrievable-unit-on-train.pyproducts with no description: 1,568
descriptions the sample cut short: 5,933
description words, median of the others: 68
product text words per product, most: 140
description sentences: 35,061
last sentence of product 8171: the antler 's durability and beauty will last a lifetime .
whole product, train precision@5: 0.464
whole product, train recall@5: 0.124
whole product, train judged@5: 0.940
whole product, train queries whose top 5 changes with ties reversed: 4
name only, train precision@5: 0.368
name only, train recall@5: 0.091
name only, train judged@5: 0.960
name only, train queries whose top 5 changes with ties reversed: 28
description only, train precision@5: 0.304
description only, train recall@5: 0.049
description only, train judged@5: 0.828
description only, train queries whose top 5 changes with ties reversed: 4
sentences, train precision@5: 0.272
sentences, train recall@5: 0.034
sentences, train judged@5: 0.732
sentences, train queries whose top 5 changes with ties reversed: 18
sentences with the other fields, train precision@5: 0.428
sentences with the other fields, train recall@5: 0.106
sentences with the other fields, train judged@5: 0.944
sentences with the other fields, train queries whose top 5 changes with ties reversed: 8
highest train precision@5: whole product
whole product minus sentences with the other fields, train precision@5: 0.036
whole product minus sentences with the other fields, train recall@5: 0.018
| 单元 | precision@5 | recall@5 | judged@5 | 平局反转后前 5 个结果的变化数 |
|---|---|---|---|---|
| 整个商品 | 0.464 | 0.124 | 0.940 | 4 |
| 仅名称 | 0.368 | 0.091 | 0.960 | 28 |
| 仅描述 | 0.304 | 0.049 | 0.828 | 4 |
| 句子 | 0.272 | 0.034 | 0.732 | 18 |
| 带其他字段的句子 | 0.428 | 0.106 | 0.944 | 8 |
训练集 precision@5 最高的是whole product。公平的比较对象是带其他字段的句子:字段相同,可触及的商品相同,其 judged@5 为 0.944,而整个商品为 0.940。
把描述切成句子,使训练集 precision@5 下降 0.036,recall@5 下降 0.018。这些差距有一部分由平局决定:当商品在第五位前后得分相同时,由平局规则选出哪些商品进入上下文,反转该规则后,带其他字段的句子有 8 条训练查询的前 5 个结果发生变化,整个商品则有 4 条。对这样大小的差距,只能当作粗略的参考。
其他单元还丢掉了一些字段。仅描述的单元永远无法返回没有描述的商品,而单纯的句子会带出许多无人判定的商品,因此它们的 precision 有一部分是猜测。这些是训练集数字,未经测试:它们可为选择提供参考,但不能证明任何结论。
**工作准则:**当条目较短时,像商店那样,把每个完整条目作为一个单元建立索引。当条目较长时,对其分块,在每个块前加上条目的标题,并依据上一节的上下文度量,在训练查询上选择块的大小。把整个条目,或带标题的块交给 LLM,使它知道自己读的是什么。在相信一个小差距之前,先统计在平局以相反方式裁决时前 k 个结果发生变化的查询数。
这部分尚未翻译,因此以英文显示。
Read an agent's searches from its trace
Unlike the recorded answers, an agent searches for itself: it is an LLM that runs in a loop and acts. A tool is a function the program offers the model, described by a name, a description and its parameters. A tool call is the model's request to run a tool with arguments. In the agent loop, the program runs each tool call, sends back the result, and the model decides what to do next. In Ollama, which ran the store's model, an agent loop lets the model decide when to invoke tools and fold their results into its replies (Ollama documentation). recordings/agent_trace.py is such a loop. ReAct agents interleave reasoning traces with task-specific actions. On question answering (HotpotQA) and fact verification (Fever), with a Wikipedia API as the tool, they overcome the hallucination and error propagation that are common in chain-of-thought reasoning (Yao et al., 2023). The trace is the record of every tool call and its result.
An agent's searches are queries, and they need not look like a shopper's. In a log of 14 million agent search requests, more than 90% of multi-turn sessions contain at most ten steps, and on average 54% of the query terms a search adds already appear in the accumulated evidence (Ning et al., 2026). The agent picks up words from what it has read.
The recording gave the agent each test query as the shopper's request, with one tool, search, described as "Search the store's catalog. Returns the top 5 products: ID, name, class and attributes." It may search up to 3 times, with any wording, then reply with three product IDs. The tool returns no description.
The recording ran twice, with the same model and prompt: once with BM25 behind the tool, and once with semantic retrieval, which embedded the agent's search text with the same embedding model as the products. The traces store each BM25 search's results in bm25s's own tie order, as the answers' contexts do, so 7 of the 32 BM25 searches get results that differ from what lib.lexical returns, 6 of them only in order. Here is what the agent actually searched for:
1"""Reads the recorded agent traces for both tools: each agent's first search, how many searches each tool took, and2whether BM25's stored results match lib.lexical. The point rests on the first-search lines and the one-search3trace counts."""4from lib.lexical import LexicalSearch5from lib.recorded import agent_traces, manifest, rag_answers6from lib.store import load_products, load_queries78recording, queries = manifest("agent-trace"), load_queries("test")9print("tool description:", recording["prompts"]["tool"]["function"]["description"])10print("searches allowed:", recording["max_searches"])11traces = {(t["retriever"], t["query_id"]): t for t in agent_traces()}12print("traces:", len(traces))13# index 0 is the agent's first search14first = {key: t["searches"][0]["query"] for key, t in traces.items()}15print("first searches the same under both tools:", sum(first[("bm25", q)] == first[("semantic", q)] for q in queries.index))16print("first searches that keep the query as typed:", sum(first[("bm25", q)] == queries.at[q, "query"] for q in queries.index))17for q in queries.index:18 if first[("bm25", q)] != queries.at[q, "query"]:19 print(f"first search for {queries.at[q, 'query']}: {first[('bm25', q)]}")20# the five products each recorded answer read, keyed by context (semantic, bm25 or random) and query21context = {(a.context, a.query_id): a.context_ids for a in rag_answers().itertuples()}22print("traces whose first search returns the answer recording's context:",23 # lists compared with ==: the same five products, in the same order24 sum(t["searches"][0]["results"] == context[key] for key, t in traces.items()))25for tool in ["bm25", "semantic"]:26 mine = [t for (retriever, _), t in traces.items() if retriever == tool]27 print(f"{tool}, searches: {sum(len(t['searches']) for t in mine)}")28 print(f"{tool}, traces with one search: {sum(len(t['searches']) == 1 for t in mine)}")2930# `shown` is how many products the search tool returns per search (its top 5)31lexical, shown = LexicalSearch(load_products()), recording["results_shown"]32stored = [s for (tool, _), t in traces.items() if tool == "bm25" for s in t["searches"]] # what the agent saw33rerun = [lexical.search(s["query"], shown).index.tolist() for s in stored] # ties by product_id34differ = [(s["results"], r) for s, r in zip(stored, rerun) if s["results"] != r]35print("bm25 searches whose results differ from lib.lexical:", len(differ))36# the same products in another order: a tie-break, not a different result37print("bm25 searches whose results differ from lib.lexical only in order:", sum(set(a) == set(b) for a, b in differ))
python agent-searches-in-traces.pytool description: Search the store's catalog. Returns the top 5 products: ID, name, class and attributes.
searches allowed: 3
traces: 60
first searches the same under both tools: 30
first searches that keep the query as typed: 26
first search for 48" sliding single track , barn door for laundry: 48" sliding single track barn door laundry
first search for counter top one cup hot water dispenser: counter top hot water dispenser one cup
first search for refrigerator with ice an water in door: refrigerator with ice and water in door
first search for tye dye duvet cover: tie dye duvet cover
traces whose first search returns the answer recording's context: 55
bm25, searches: 32
bm25, traces with one search: 29
semantic, searches: 35
semantic, traces with one search: 27
bm25 searches whose results differ from lib.lexical: 7
bm25 searches whose results differ from lib.lexical only in order: 6
The first search is the same under both tools for 30 queries, as expected: the agent writes it before the tool has returned anything, from the same prompt and tool description, at temperature 0. It keeps the query as typed for 26.
Of the rest, one fixes a misspelling: tye dye duvet cover becomes tie dye duvet cover. Another fixes a real-word error that chapter 8's corrector misses: an becomes and in refrigerator with ice and water in door. Both an and and are stop words for BM25 (chapter 5), so only semantic retrieval sees that change. The other two become 48" sliding single track barn door laundry and counter top hot water dispenser one cup.
In 55 of the 60 traces, the first search returns exactly the five products that the answer recording put in its context.
The agent rarely looks again. With BM25 it makes 32 searches in all, and 29 traces stop after one. With semantic retrieval it makes 35, and 27 stop after one.
Where do the words of a later search come from? To find out, set each later search against the words already searched, and against the names, classes and attributes the tool had shown:
1"""Counts the words each later search adds to the agent's query so far, and how many of those the tool had already2shown. The point rests on the "words the tool had shown" line for each tool."""3from lib.generation import shown, words4from lib.recorded import agent_traces5from lib.store import load_products, load_queries67products, queries = load_products(), load_queries("test")8for tool in ["bm25", "semantic"]:9 later = new = from_results = 010 for trace in (t for t in agent_traces() if t["retriever"] == tool):11 # words are lowercase runs of letters and digits, so case and punctuation don't count12 asked, seen = words(queries.at[trace["query_id"], "query"]), set()13 for number, search in enumerate(trace["searches"]):14 if number > 0: # a later search: the words it adds, and where from15 added = words(search["query"]) - asked16 later, new, from_results = later + 1, new + len(added), from_results + len(added & seen)17 # every word the agent has searched so far18 asked |= words(search["query"])19 # what the tool showed for that search: name, class and attributes, but no description20 seen |= words(shown(search["results"], products))21 print(f"{tool}, later searches: {later}")22 print(f"{tool}, words that later searches add: {new}")23 print(f"{tool}, of those, words the tool had shown: {from_results}")
python agent-later-searches.pybm25, later searches: 2
bm25, words that later searches add: 2
bm25, of those, words the tool had shown: 1
semantic, later searches: 5
semantic, words that later searches add: 4
semantic, of those, words the tool had shown: 4
Later searches add few words of their own. Under BM25, 2 later searches add 2 words, 1 of them shown by the tool before. Under semantic retrieval, 5 later searches add 4 words, 4 of them shown before. Counts this small describe these traces; they are not a rate to compare with Ning et al.'s.
Working rule: Read the traces before you score an agent. Count its searches per task, compare each search with the shopper's words, and find where added words come from. Add the agent's searches to the queries you test your retriever on, scored with the judgments for the shopper's request or judged afresh, because a retriever tuned on people's queries hasn't been tested on an agent's. Expect an agent that mostly passes the shopper's words through to be bounded by the same retrieval as a RAG answer.
比较智能体所用工具背后的检索器
能够再次搜索的智能体,似乎会降低检索器的重要性。但有一项基准测试得出了相反的结果。BrowseComp-Plus 固定了智能体所搜索的文档集合,因此可以在同一个智能体内更换检索器。使用 BM25 时,GPT-5 的准确率为 55.9%,使用 Qwen3-Embedding-8B 检索器时则达到 70.1%,而且平均搜索调用次数更少,分别为 21.74 次和 23.23 次(Chen et al., 2025, Table 1)。购物是智能体的经典任务。WebShop 收录了 118 万件真实商品,其作者在 2022 年提出的最佳模型,任务成功率为 29%,而人类专家为 59%(Yao et al., 2022)。在由四家模拟商店组成的 WebMall 上,其作者报告称,最佳智能体完成任务的比率低于 65%,所涉任务包括搜索最便宜的商品和描述含糊的商品(Peeters et al., 2025)。
被要求从搜索结果中挑选的智能体,应当只挑选结果中返回的内容,而它的轨迹会显示它是否做到了这一点。因此,它的挑选受检索所限,正如 RAG 的回答一样。
商店的已记录智能体使用一个工具最多搜索目录 3 次,然后挑选三件商品。按各个工具对这些挑选评分,采用 precision@3,这是本章计划中的比较的第二项;若其置换检验 p 值低于 0.025,则该差异成立。Precision@3 等于准确挑选的数量除以 3,因此缺失的挑选按未命中计算。首先,针对每个工具,统计挑选在哪些地方缺失:工具从未返回的挑选、无人评判的挑选、挑选少于三件的轨迹、搜索未返回任何准确商品的轨迹,以及见到了准确商品却一件也未挑选的轨迹。然后,把这些挑选与该工具对首次搜索给出的前 3 项相比较,这是一项探索性检查:
1"""Scores each tool's agent picks with precision@3 and counts how the traces went, with an exploratory check2against the tool's first search. The point rests on the precision@3 lines and the picks-the-tool-never-returned3counts."""4import pandas as pd56from lib.generation import EXACT, report_planned7from lib.metrics import precision_at_k8from lib.recorded import agent_traces9from lib.significance import compare10from lib.store import qrels1112judgments, picks, top_three, below = qrels("test"), {}, {}, 013for tool in ["bm25", "semantic"]:14 traces = [t for t in agent_traces() if t["retriever"] == tool]15 # every product any of the agent's searches returned16 returned = [{p for search in t["searches"] for p in search["results"]} for t in traces]17 # exact products each trace's searches returned; a product nobody judged counts as not exact18 seen = [sum(judgments[t["query_id"]].get(p, 0) == EXACT for p in r) for t, r in zip(traces, returned)]19 picked = [sum(judgments[t["query_id"]].get(p, 0) == EXACT for p in t["picked"]) for t in traces]20 print(f"{tool}, picks the tool never returned: {sum(len(set(t['picked']) - r) for t, r in zip(traces, returned))}")21 print(f"{tool}, picks nobody judged: {sum(p not in judgments[t['query_id']] for t in traces for p in t['picked'])}")22 print(f"{tool}, traces with fewer than three picks: {sum(len(t['picked']) < 3 for t in traces)}")23 print(f"{tool}, traces whose searches returned no exact product: {sum(s == 0 for s in seen)}")24 print(f"{tool}, traces that saw an exact product and picked none: {sum(s > 0 and k == 0 for s, k in zip(seen, picked))}")25 # precision_at_k divides by 3, so a trace with fewer picks scores lower26 picks[tool] = pd.Series({t["query_id"]: precision_at_k(t["picked"], judgments[t["query_id"]], 3) for t in traces})27 # index 0 is the agent's first search28 top_three[tool] = pd.Series({t["query_id"]: precision_at_k(t["searches"][0]["results"], judgments[t["query_id"]], 3)29 for t in traces}) # the tool's own top 3 for the first search30 print(f"{tool}, precision@3 of the picks: {picks[tool].mean():.3f}")31 print(f"{tool}, precision@3 of the first search's top 3: {top_three[tool].mean():.3f}")32 p_value = compare(picks[tool], top_three[tool])["permutation p"]33 print(f"{tool}, picks against the top 3, exploratory permutation p: {p_value:.3f}")34 # uncorrected: this check is exploratory, not one of the planned comparisons35 below += p_value < 0.053637print(f"picks against the top 3, p-values below 0.05 uncorrected: {below} of 2")38report_planned("agent picks, semantic over bm25", picks["semantic"], picks["bm25"])
python agent-picks-by-tool.pybm25, picks the tool never returned: 0
bm25, picks nobody judged: 3
bm25, traces with fewer than three picks: 2
bm25, traces whose searches returned no exact product: 5
bm25, traces that saw an exact product and picked none: 6
bm25, precision@3 of the picks: 0.567
bm25, precision@3 of the first search's top 3: 0.500
bm25, picks against the top 3, exploratory permutation p: 0.152
semantic, picks the tool never returned: 0
semantic, picks nobody judged: 0
semantic, traces with fewer than three picks: 0
semantic, traces whose searches returned no exact product: 5
semantic, traces that saw an exact product and picked none: 2
semantic, precision@3 of the picks: 0.622
semantic, precision@3 of the first search's top 3: 0.633
semantic, picks against the top 3, exploratory permutation p: 1.000
picks against the top 3, p-values below 0.05 uncorrected: 0 of 2
agent picks, semantic over bm25, mean difference: 0.056
agent picks, semantic over bm25, wins, losses, ties: 5, 4, 21
agent picks, semantic over bm25, permutation p: 0.422
agent picks, semantic over bm25, established at 0.025: no
agent picks, semantic over bm25, t-test p: 0.305
agent picks, semantic over bm25, t-test agrees: yes
agent picks, semantic over bm25, 97.5% interval: -0.033 to 0.209
agent picks, semantic over bm25, interval agrees: yes
两个工具的挑选没有显示出成立的差异。语义检索的 precision@3 为 0.622,BM25 为 0.567,二者之差的置换检验 p 值为 0.422(两项比较)。就此模型、提示词和这些测试查询而言,这并不能证明检索器无关紧要,只能说明这些查询没有显示出差异。
对每个工具按同样方式统计,未命中分为以下几类。工具从未返回的挑选,在 BM25 下为 0 件,在语义检索下为 0 件,因此上述界限成立。检索也限制了挑选:在 BM25 下有 5 条轨迹、在语义检索下有 5 条轨迹,没有任何一次搜索返回准确商品,因此任何挑选都不可能准确。
其余的未命中出在挑选本身。在 BM25 下,智能体见到准确商品却一件也未挑选的轨迹有 6 条,语义检索下为 2 条;其挑选中无人评判的有 3 件,语义检索下为 0 件;挑选少于三件的轨迹有 2 条,语义检索下为 0 条。无人评判的挑选和缺失的挑选都按未命中计算。
在各个工具下,智能体的挑选呈现相同的态势。工具对首次搜索给出的前 3 项,在 BM25 下得分为 0.500,在语义检索下为 0.633。以该前 3 项为参照来衡量挑选,置换检验 p 值在 BM25 下为 0.152,在语义检索下为 1.000,未经校正时低于 0.05 的有 0 of 2。这项检查是在看到测试结果之后才选定的,因此它绝不能算作成立。
工作准则: 检验智能体工具背后的检索器,要像为人检验检索器那样,针对智能体自己的搜索和最终挑选,并用人工判断来评分。检查每一项挑选是否都来自工具结果。统计工具从未返回相关商品的任务数,因为这类任务任何智能体都无法补救,只有检索才能解决。
这部分尚未翻译,因此以英文显示。
What goes wrong in production
A live assistant answers every shopper with a fresh model call, on a context that retrieval builds for that request.
The model, the prompt and the context are one system. A different model version, a reworded prompt, another context size or a different order of products can change every answer. The store's manifests pin the model's digest, the prompts, the answer and tool schemas and the context window. Change any of them and score the answers again, against the same judgments and the random floor. Keep the raw replies and each context, as the recordings do.
The citation check belongs on the answer's path. The ID check is cheap, so run it on every answer before the shopper sees it, and log each failure with the answer and its context.
Abstentions move with retrieval. Track the share of answers that recommend nothing. On the store's random contexts, 29 of the 30 answers recommend nothing, so a rise in that share can mean that retrieval or the catalog changed. Read those answers before you assume the model did.
Product text reaches the prompt. Product descriptions may come from outside your team, and the LLM reads them as part of its input. Chapter 4 cites a study in which words and instructions planted in a text misled LLM judges. Test the assistant on products whose text contains instructions before you trust it with a catalog you don't write.
Longer contexts change the answer. Raising k lengthens the context and brings in more distractors, and both can hurt (Liu et al., 2024; Jin et al., 2024). Measure answers at each setting on train queries.
Agent traffic is a different workload. In the logs that Ning et al. studied, agents send many machine-written queries per task, while the store's agent mostly sent one. One study warns that with agent traffic, caching may lose effectiveness, and query pre-processing may add overhead without improving results (Pezzuti et al., 2026). Log agents' searches apart from people's, and test the retriever on both.
Working rule: Version the model, prompt and context settings with the answers, check citations on every answer, track abstentions, and keep agents' searches in a test set of their own.
本章做出的决定
只有成立的计划内比较才可能改变商店的检索器,而表中“成立”一行说明了每项比较是否成立。
本章的 2 项计划内比较,即在测试查询上用语义检索减去 BM25,在 0.025 的水平下得到如下结果:
- **上下文:来自语义检索的五件完整商品,**即待超越的排序器。五件是记录中固定且未经检验的设置。回答显示,语义检索与 BM25 之间没有成立的差异,因此检索器仍以第 3 章中在 nDCG@10 上成立的领先为依据。
- **依据人工评分对回答打分:**统计每个回答所引用内容中的精确占比,同时统计什么都不推荐的回答,并以随机上下文作为下限。
- **对每个回答按 ID 检查引用。**记录中的回答通过了该检查,这项检查予以保留。用 NLI 模型或 LLM 评审做的支持度检查,只有在经人工样本校准后才算数。
- **用判定得出的 precision@5 和上下文精确率来度量上下文,**并列出 recall@5。原因是当一个查询的精确商品多于五件时,五件商品上下文的 recall@5 不可能达到 1。
- **可检索单元:完整商品。**它的训练集 precision@5 最高,而把描述切成句子,即便在前面加上其他字段,也会损失 0.036。
- **智能体的工具:同样的检索器,在智能体的搜索与选择上检验。**这些选择显示,两种工具之间没有成立的差异,而且每个选择都来自工具返回的结果。
- **范围:一个模型、两个提示词、测试查询。**关于 LLM 的每一项发现,都针对采用所记录提示词的
gemma4:12b-it-q4_K_M。更换模型或提示词属于需要重新评分的改动,正如更换检索器一样。 - **本章所示:**引用检查只能表明某件商品出现在模型面前,不能表明它合适;排序上的提升必须在它所供给的回答上加以检验。
本章,在本版次中
版次 2026.11.10示例于 2026年11月19日 通过本章 10个示例 个示例中有 10 个通过
译自英文版图书,版次 2026.11.10。