在无法回答时明确说明
阅读约需 85分钟 分钟,7节中含5个示例。
读完本章,你可以把拒答变成一种正式的回答:区分仓库无法回答的问题和能够回答的问题;以已知类别的理由拒答;检查所问时间段的数据是否已经到达;展示每个回答背后的数据来源;并用精确率和召回率衡量拒答,使智能体既不乱猜,也不过于频繁地拒绝。
商店的数据截止于 2026 年 6 月的最后一天,它的智能体需要知道自己何时无法回答。第 6 章构建了这样一个智能体:由模型选择指标、维度和过滤条件,再由语义层编写 SQL。在黄金问题的开发集上,该智能体答对了大部分可回答的问题,而它的两个失败正是本章的主题。对于“哪个地区表现最好?”,它给出了每个地区的历史总收入,并按从小到大排列,也就是为一个没有唯一答案的问题给出了一个数字。对于下一季度的净收入,它反问对方指的是日历季度还是财年季度,仿佛这两种理解都能从这些数据中得到答案。
这两个失败看起来都像是回答,而仓库给出的错误数字比没有数字更糟,因为它会被粘贴进演示文稿。关于文本到 SQL 的研究也指出了同一点:不可回答的问题可能生成能够运行、却给出误导性结果的 SQL(Ren 等人)。本章为智能体提供说“不”的方法,以及区分恰当拒答与拒绝工作的度量。本章建立在第 6 章以指标优先的智能体之上,而不是建立在第 7 章带防护的 SQL 路径之上,因此没有任何指标覆盖的问题在这里仍会被拒答,度量也会把每一次这样的拒答计为一项代价。
这部分尚未翻译,因此以英文显示。
Kinds of questions the warehouse can't answer
A question that shouldn't get a single number is one of two kinds, and they need different replies. An ambiguous question has several valid answers: the words fit more than one metric, period or filter, and the right reply is a question back to the person or a stated assumption. An unanswerable question has no valid answer in the warehouse: the data doesn't hold what it needs, or it asks for something SQL can't do, such as a forecast.
The research gives both kinds names you can use. PRACTIQ, a conversational text-to-SQL dataset, sorts them into four ambiguous categories, such as an ambiguous SELECT column, where several valid SQLs differ in the columns they select, and four unanswerable ones, such as a nonexistent SELECT column, where "The column that contains the results asked in the question do not exist in the database" (Dong et al.). TrustSQL adds questions that fail because the schema is incompatible or because they need something SQL can't do (Lee et al.). Knowing the schema doesn't remove the ambiguity: AMBROSIA's authors found that the ambiguity persists even when the database context is provided (Saparina and Lapata).
We can audit each golden decline against the warehouse before trusting its label. The shop's golden questions, from chapter 3, include eight whose right answer is a decline, four in each split, and each carries a decline_type: two ambiguous and two unanswerable per split. The audit prints each one with its category in PRACTIQ's terms, or TrustSQL's for the forecast, and searches both schemas for a column whose name matches the words the question needs. Then it answers two ambiguous questions in each of their readings, from the semantic layer. One of the ambiguous questions has no reading to answer: "Who are our best customers?", gq-019, is ambiguous in PRACTIQ's terms, but no metric or dimension ranks customers, so its right reply is a decline, not a question back.
1"""The golden questions whose right answer is a decline: which are ambiguous, which are unanswerable, and why.2The point that unanswerable questions have nothing to compute from rests on the "columns matching" lines, and the point3that ambiguous ones have too many answers rests on the "best region" and "net revenue" lines."""4from lib.golden import load_golden5from lib.semantic import MetricRequest, load6from lib.warehouse import connect78con, layer = connect(), load()9# Each decline question's category: PRACTIQ's names, and TrustSQL's for the forecast. Words to look for in the schema.10CATEGORIES = {"gq-005": ("beyond SQL: a forecast", "forecast|predict"), "gq-019": ("ambiguous SELECT column", None),11 "gq-020": ("ambiguous SELECT column", None), "gq-021": ("nonexistent SELECT column", "ship"),12 "gq-042": ("ambiguous filter criteria", None), "gq-043": ("ambiguous values within a column", None),13 "gq-044": ("unsupported join", "view|page"), "gq-045": ("nonexistent WHERE column", "deliver|promise")}14for q in load_golden():15 if q.kind == "decline":16 category, words = CATEGORIES[q.question_id]17 print(f"{q.question_id} ({q.split}, {q.decline_type}): {category}")18 if words:19 # raw and analytics are the only schemas with data; the layer reads from both20 found = con.execute("""SELECT string_agg(table_name || '.' || column_name, ', ') FROM information_schema.columns21 WHERE table_schema IN ('raw', 'analytics') AND regexp_matches(column_name, ?)""",22 [words]).fetchall()[0][0]23 print(f" columns matching {words!r}: {found or 'none'}")2425# the match is on the name: a column named for cost is the product's unit cost, which is not the shipping the question asks for26cost = con.execute("""SELECT string_agg(table_name || '.' || column_name, ', ') FROM information_schema.columns27 WHERE table_schema IN ('raw', 'analytics') AND regexp_matches(column_name, 'cost')""").fetchall()28print(f"columns named for cost: {cost[0][0] or 'none'}")29reasons = [r for (r,) in con.execute("SELECT DISTINCT reason FROM analytics.fct_refunds ORDER BY 1").fetchall()]30print(f"refund reasons: {', '.join(reasons)}")31print(f"last order day: {con.execute('SELECT max(order_date_local) FROM analytics.fct_orders').fetchall()[0][0]}")3233# "Which region is doing best?" in fiscal 2026, by three metrics the layer defines.34fiscal_2026 = {"metric_time__fiscal_year": 2026}35for metric, order in (("net_revenue", "-net_revenue"), ("orders", "-orders"), ("refund_rate", "refund_rate")):36 # a leading minus sorts descending, so net revenue and orders put the largest first; refund_rate sorts ascending,37 # so its first row is the lowest rate; the limit of 1 keeps that one row38 answer = layer.query(con, MetricRequest((metric,), ("order__region",), fiscal_2026, (order,), 1))39 print(f"best region by {order.lstrip('-')}{' (lowest)' if order == metric else ''}: {answer.rows[0][0]}")4041# Q2 read as a calendar quarter and as a fiscal quarter: calendar Q2 starts on April 1, fiscal Q2 2026 runs May to July.42for name, value in (("metric_time__quarter", "2025-04-01"), ("metric_time__fiscal_quarter", "2026-Q2")):43 # one row with one column, so the unpacking gives the revenue total for the period44 (revenue,), = layer.query(con, MetricRequest(("net_revenue",), (), {name: value})).rows45 print(f"net revenue, {name.removeprefix('metric_time__')} {value}: {revenue:,.2f}")
python unanswerable-cases.pygq-005 (dev, unanswerable): beyond SQL: a forecast
columns matching 'forecast|predict': none
gq-019 (dev, ambiguous): ambiguous SELECT column
gq-020 (dev, ambiguous): ambiguous SELECT column
gq-021 (dev, unanswerable): nonexistent SELECT column
columns matching 'ship': none
gq-042 (test, ambiguous): ambiguous filter criteria
gq-043 (test, ambiguous): ambiguous values within a column
gq-044 (test, unanswerable): unsupported join
columns matching 'view|page': none
gq-045 (test, unanswerable): nonexistent WHERE column
columns matching 'deliver|promise': none
columns named for cost: dim_products.unit_cost, products.unit_cost
refund reasons: changed_mind, damaged, late_delivery, other, wrong_size
last order day: 2026-06-30
best region by net_revenue: EMEA
best region by orders: EMEA
best region by refund_rate (lowest): APAC
net revenue, quarter 2025-04-01: 415,495.84
net revenue, fiscal_quarter 2026-Q2: 467,081.90
No column in raw or analytics is named for shipping, product views, delivery promises or forecasts, and the last order day is 2026-06-30. So the four unanswerable questions have nothing to compute from, but each has something nearby that a model could reach for. The products table has unit_cost (dim_products.unit_cost, products.unit_cost), which is the product's cost, not shipping. The refunds table has a reason called late_delivery, which counts refunds, not late deliveries. A model that trusts the question's premise will write runnable SQL against those columns: PremiseSQL's authors found that models "are strongly premise-trusting by default and rarely critique flawed premises" (Lin et al.).
The ambiguous questions have answers, too many of them. For "Which region is doing best?" in fiscal 2026, the layer names EMEA by net revenue and by orders, and APAC by the lowest refund rate. For revenue in Q2 of 2025, it gives 415,495.84 euros of net revenue for the calendar quarter, and 467,081.90 for the fiscal one, which runs from May to July. Each of those numbers is right for one reading and wrong for the others, and nothing in the result says which reading produced it.
A third kind looks unanswerable and isn't: a nonexistent filter value, in PRACTIQ's terms, is a filter value no row holds, such as a region called Europe. In the shop, chapter 6's validation refuses it and names the regions that exist, so the model can correct its request instead of declining.
Working rule: Sort every question your golden set marks as a decline into ambiguous or unanswerable, and name its category, so you know which reply it needs. Ask back on an ambiguous question only when every reading is answerable, and decline an unanswerable one with what's missing. For each unanswerable question, find the column or value nearby that a model could mistake for the answer, because that's the wrong number you're guarding against.
这部分尚未翻译,因此以英文显示。
Decline with a reason
The two kinds of question need a reply that isn't a number, and a reply the code can read. A decline is an answer: it ends the run, it says why, and the person can act on it. The reason has two parts. One is a sentence for the person, naming what's missing. A decline's kind, the reason's other part, comes from a fixed list, for the code, so declines can be counted, routed and scored by kind instead of read one at a time. In chapter 6, the forecast's question back about the calendar or the fiscal quarter scored as correct because it returned no number, and only its trace showed that the reason was wrong.
This chapter's agent, which we call the declining agent, lives in lib/declining.py, a module this chapter adds. Its DecliningToolbox keeps chapter 6's five metric tools and changes what two of them do. cannot_answer takes a kind as well as a reason, and refuses a kind outside the list. query_metric runs two checks in code before it answers: chapter 6's ranking check, which refuses a request that ranks its result over no period, and the freshness check that the section on freshness builds. The prompt, DECLINING in lib/prompts.py, differs from chapter 6's in three places. It lists the kinds the model may give; it allows ask_back only when every option is a request the metrics can answer, chapter 6's working rule; and it states what a wrong number costs, in the form Kalai and colleagues propose: "Answer only if you are >t confident, since mistakes are penalized t/(1-t) points" (Kalai et al.). With t at 0.75, the formula makes a wrong number cost 0.75/0.25 = 3 points and a decline 0.
The recording decline-runs holds the pinned model's replies to that prompt for the dev split's answer and decline questions, 21 of them. It was made by recordings/record_decline_runs.py with the same model file, sampling and seed as every recording in this book, and at most four model calls a run, as in chapter 6. The three forbidden questions on dev aren't recorded, because their decline belongs below the model, in chapter 9. We chose the prompt's changes by reading chapter 6's dev traces, so the dev split informed them. We recorded the dev split once, after writing the prompt, and changed nothing afterwards. The recording's manifest says so in its tuning field, and every replay checks it in part: a reply is found by a hash of the messages that produced it, prompt included, so a changed prompt finds no reply.
A kind helps only when the agent picks it, so we print the kinds and who decides each, send cannot_answer a call without one, and replay the four dev decline questions and one answerable question the agent declined.
1"""Decline with a kind from a fixed list and a reason, so declines can be checked and counted like answers.2The prose's point about "Which region is doing best?" rests on the gq-020 request and the ranking check's reply to it."""3from lib.agent import run_agent4from lib.declining import DECLINE_KINDS, MODEL_KINDS, DecliningToolbox, decline_kind5from lib.golden import load_golden6from lib.metric_tools import catalog7from lib.model import RecordedModel8from lib.prompts import declining_prompt9from lib.warehouse import connect1011con = connect()12toolbox = DecliningToolbox(con)13for kind, meaning in DECLINE_KINDS.items():14 print(f"{kind}{' (model)' if kind in MODEL_KINDS else ' (code)'}: {meaning}")15# without a kind the call is refused, and the error lists the kinds the model may give16print(toolbox.call("cannot_answer", {"reason": "no shipping costs"}).content)17print(toolbox.call("cannot_answer", {"kind": "not_in_data", "reason": "no shipping costs"}).answer.show())1819system, model = declining_prompt(catalog(toolbox.layer)), RecordedModel("decline-runs")20for q in load_golden():21 # gq-016 is answerable; it is replayed too, to show the agent declining it22 if q.split == "dev" and q.kind == "decline" or q.question_id == "gq-016":23 trace = run_agent(q.question, system, model, DecliningToolbox(con, toolbox.layer, role=q.role), max_steps=4)24 calls = [step for step in trace.steps if step.kind == "model"]25 print(f"{q.question_id} ({q.decline_type or q.kind}): {q.question}")26 print(f"{q.question_id} ended with: {calls[-1].tool}, kind {decline_kind(trace.answer)}")27 print(f"{q.question_id} model calls: {len(calls)}")28 label = "asks" if trace.answer.provenance.source == "ask_back" else "reason"29 print(f" {label}: {trace.answer.reason.splitlines()[0]}")30 if q.question_id == "gq-021":31 # the first model call is the reply that lacked the tool's name, which the loop answered with an error32 print(f" first reply: {calls[0].content}")33 if q.question_id == "gq-020":34 asked = next(i for i, step in enumerate(trace.steps) if step.tool == "query_metric")35 request = trace.steps[asked].arguments36 print(f"gq-020 request: {', '.join(request['metrics'])} by {', '.join(request['group_by'])}, ordered by "37 f"{', '.join(request['order_by'])}, period: {request.get('filters') or 'none'}")38 # the step right after the request is the ranking check's refusal of it, so the offset of one is the reply39 print(f"gq-020 check: {trace.steps[asked + 1].content}")40 # the options sit in the arguments of the ask_back call, the step before the run's last step41 print(f"gq-020 options: {', '.join(trace.steps[-2].arguments['options'])}")
python cannot-answer-tool.pynot_in_data (model): no metric or dimension holds what the question needs
forecast (model): the question asks about a period that hasn't happened, or for a prediction
no_definition (model): the question uses a term the shop hasn't defined, and no metric settles it
stale (code): the data for the period hasn't all arrived: not loaded yet, or not happened yet
forbidden (code): the role asking may not see the answer (chapter 9)
error: kind must be one of not_in_data, forecast, no_definition; got ''
cannot answer: not_in_data: no shipping costs
gq-005 (unanswerable): What will our net revenue be next quarter?
gq-005 ended with: cannot_answer, kind forecast
gq-005 model calls: 1
reason: forecast: The question asks about a future period that hasn't happened yet, and no data is available for forecasting.
gq-016 (answer): How many customers placed at least one order in 2025?
gq-016 ended with: cannot_answer, kind not_in_data
gq-016 model calls: 2
reason: not_in_data: The metric 'new_customers' cannot use 'order__order_date' as a filter, and no alternative metric or dimension is available to answer the question about customers placing at least one order in 2025.
gq-019 (ambiguous): Who are our best customers?
gq-019 ended with: cannot_answer, kind no_definition
gq-019 model calls: 1
reason: no_definition: The term 'best' is not defined in the available metrics or dimensions, and no metric or dimension can determine customer performance based on this vague criterion.
gq-020 (ambiguous): Which region is doing best?
gq-020 ended with: ask_back, kind None
gq-020 model calls: 3
asks: Which time period are you interested in? For example, the most recent month, the last quarter, or a specific year?
gq-020 request: net_revenue by customer__region, ordered by net_revenue, period: none
gq-020 check: error: request rejected: the request ranks its result but names no period; filter or group by a metric_time grain; or ask the person which period they mean
gq-020 options: last month, last quarter, 2026
gq-021 (unanswerable): How much did shipping cost us in fiscal year 2026?
gq-021 ended with: cannot_answer, kind not_in_data
gq-021 model calls: 2
reason: not_in_data: The shop's data does not include a metric for shipping costs.
first reply: {"kind": "not_in_data", "reason": "The shop's data does not include a metric for shipping costs."}
The model may give three kinds: not_in_data, forecast and no_definition. Code gives the other two: stale, when the data for the period hasn't arrived, and forbidden, which chapter 9's access rules will give. A call without a kind comes back as an error that lists the allowed kinds, so the model can correct it in its next call.
In the declining agent's run, the forecast, gq-005, ends in a decline of kind forecast, in 1 model call. "Who are our best customers?", gq-019, is declined with kind no_definition, which is the honest reply chapter 6 said was due: the shop's metrics don't rank customers, so no option could have been answered. The shipping question, gq-021, took 2 calls. The model's first reply was the decline's arguments without the tool's name, the loop sent back an error, and the second reply declined with kind not_in_data.
"Which region is doing best?", gq-020, is the one to read closely. The model asked for net revenue by region, ordered, with no period (net_revenue by customer__region, ordered by net_revenue, period: none), the request chapter 6's agent ran. The ranking check refused it because it ranks its result but names no period, and the model then asked back which period the person meant, offering last month, last quarter, 2026. That counts as a correct reply, and it settles half the question. A period doesn't say what "best" means, and the readings give different regions.
The last question, gq-016, asks how many customers placed at least one order in 2025. The warehouse can answer it, but no metric in the layer counts customers who ordered, and chapter 6's agent answered it with the count of new customers, a wrong number. The declining agent declined it as not_in_data, which is honest about the layer, and gave a reason about a filter the metric can't use, which is beside the point.
Working rule: Make declining a tool that ends the run with a kind from a fixed list and a sentence for the person. Let the model choose among the kinds it can judge, such as missing data, a forecast or an undefined term, and let code decide the kinds code can check, such as stale data or a forbidden role. Count declines by kind, and read the reasons in traces, because a decline of the right kind can still give the wrong reason.
这部分尚未翻译,因此以英文显示。
Check freshness before answering
A decline the model chooses covers questions the data can't answer at all. A subtler case is data that can answer the question but hasn't all arrived. A partial total looks complete: the SQL runs, the number has the right shape, and it's too small. Freshness is how far a source's events have arrived, read from when its rows were loaded. dbt's source freshness check works that way: it reads the newest load time from a loaded_at_field, or from a loaded_at_query, which is "A SQL expression that returns the most recent loaded timestamp", and compares it with thresholds such as warn_after, the age the newest data can reach before a warning (dbt). dbt can also read the warehouse's own metadata when you give it neither; we don't, for the reason below.
Freshness has to come from the data, not from the warehouse's table metadata. Snowflake's LAST_ALTERED records the "Date and time the object was last altered by a DML, DDL, or background metadata operation" (Snowflake), so maintenance moves it too. DuckDB's table catalog, duckdb_tables(), has no time column at all (duckdb_tables() time columns: none).
The shop's generator gives every raw table a _loaded_at column, the time the nightly load brought the row in, at 02:00 UTC the day after the event, and the marts carry it through. Its refunds feed stopped after June 26, 2026, while orders run to June 30. Some metrics keep changing after their period ends. The refund rate, refund_rate, divides refunded euros by the gross revenue of the same orders, and counts each refund on its order's day. A refund comes weeks after its order, so a month's rate keeps rising until every refund on its orders has come. A metric's settling window, as we call it, is the time until every event it counts for a period has come. The freshness check in lib/declining.py compares each feed's newest load with the end of the period a request filters on, plus the settling window of the metrics that read the feed. FEEDS names, for each table a metric reads, the feed whose loads decide its freshness, and SETTLING_DAYS gives the window of each metric counted on an earlier date than its events; every other metric's is zero. Loads are in UTC and the shop's days are local, so the period ends when its last local day ends in the region furthest behind UTC. We measure the window from the data, run the check with it, and estimate why May and June fall short.
1"""Check freshness before answering: refunds stopped loading after June 26, 2026, and arrive weeks after the order.2The prose's point rests on the unchecked June rate, the shortfall lines that split June's gap, and the 'stale' decline."""3from datetime import date45from lib.declining import (FEEDS, SETTLING_DAYS, DecliningToolbox, check_freshness, refund_delays, refund_shortfall,6 settled_refund_rate)7from lib.semantic import MetricRequest, load8from lib.warehouse import connect, data_as_of910con, layer = connect(), load()11# the catalog's column types, filtered for any type containing TIME: DuckDB's catalog has none, so the list is empty12catalog = [c for (c, t, *_) in con.execute("DESCRIBE SELECT * FROM duckdb_tables()").fetchall() if "TIME" in t]13print(f"duckdb_tables() time columns: {', '.join(catalog) or 'none'}")14for table in ("analytics.fct_orders", "analytics.fct_refunds", "analytics.dim_customers"):15 print(f"data as of, {table}: {data_as_of(con, table)}")16delays = refund_delays(con) # the longest delay on settled orders is what SETTLING_DAYS uses17print(f"refunds per day: {delays['per day']:.1f}\nmedian days from order to refund: {delays['median']}\n"18 f"refunds within 30 days of the order: {delays['within 30']:.0%}\n"19 f"refunds within 45 days of the order: {delays['within 45']:.0%}\n"20 f"longest days from order to refund: {delays['longest']}\n"21 f"settling window, order_refunds: {SETTLING_DAYS['order_refunds']} days")22for month in ("2026-04", "2026-05", "2026-06"): # without a check, the layer answers for any month23 (rate,), = layer.query(con, MetricRequest(("refund_rate",), (), {"metric_time__month": month})).rows24 print(f"refund rate, {month}, unchecked: {rate:.2%}")25# Why May and June are short, estimated from the twelve months whose refunds have all come.26print(f"refund rate, settled months: {settled_refund_rate(con):.2%}")27for month, end in (("2026-05", date(2026, 6, 1)), ("2026-06", date(2026, 7, 1))):28 # refund_shortfall takes the first day after the month, so the month's events are all before it29 short = refund_shortfall(con, end)30 print(f"{month}, settled months' rate with refunds as loaded: {short['as loaded']:.2%}\n"31 f"{month}, points short, refunds not loaded: {short['not loaded'] * 100:.2f}\n"32 f"{month}, points short, refunds not happened yet: {short['not happened'] * 100:.2f}\n"33 f"{month}, share of the shortfall not loaded: {short['share not loaded']:.0%}")3435print(f"new customers, feed for dim_customers: {FEEDS['analytics.dim_customers']}") # customers load at sign-up36for name, metric in (("refund rate", "refund_rate"), ("new customers", "new_customers")):37 for check in check_freshness(con, layer, MetricRequest((metric,), (), {"metric_time__month": "2026-06"})):38 print(f"{name}, 2026-06: {check.describe()}")39for month in ("2026-06", "2026-05", "2026-04"):40 query = {"metrics": ["refund_rate"], "filters": {"metric_time__month": month}}41 # r keeps the tool's result, so a declined call can print its reason in place of a number42 a = (r := DecliningToolbox(con, layer).call("query_metric", query)).answer43 print(f"refund rate, {month}, checked: "44 + (f"{a.rows[0][0]:.2%}, data as of {a.provenance.data_as_of}" if a.answered else r.content))
python stale-refunds.pyduckdb_tables() time columns: none
data as of, analytics.fct_orders: 2026-07-02 02:00:00
data as of, analytics.fct_refunds: 2026-06-27 02:00:00
data as of, analytics.dim_customers: 2026-06-15 02:00:00
refunds per day: 1.2
median days from order to refund: 27.5
refunds within 30 days of the order: 56%
refunds within 45 days of the order: 93%
longest days from order to refund: 49
settling window, order_refunds: 49 days
refund rate, 2026-04, unchecked: 6.74%
refund rate, 2026-05, unchecked: 6.85%
refund rate, 2026-06, unchecked: 0.54%
refund rate, settled months: 7.21%
2026-05, settled months' rate with refunds as loaded: 5.78%
2026-05, points short, refunds not loaded: 0.48
2026-05, points short, refunds not happened yet: 0.95
2026-05, share of the shortfall not loaded: 33%
2026-06, settled months' rate with refunds as loaded: 1.17%
2026-06, points short, refunds not loaded: 0.46
2026-06, points short, refunds not happened yet: 5.57
2026-06, share of the shortfall not loaded: 8%
new customers, feed for dim_customers: analytics.fct_orders
refund rate, 2026-06: analytics.fct_orders last loaded 2026-07-02 02:00, period ends 2026-06-30: complete
refund rate, 2026-06: analytics.fct_refunds last loaded 2026-06-27 02:00, period ends 2026-06-30, its rows can arrive through 2026-08-18 (49 days later): STALE (not loaded, not happened yet)
new customers, 2026-06: analytics.fct_orders last loaded 2026-07-02 02:00, period ends 2026-06-30: complete
refund rate, 2026-06, checked: declined: stale: analytics.fct_refunds was last loaded at 2026-06-27 02:00, and rows for the period ending 2026-06-30 can arrive through 2026-08-18, 49 days later (not loaded, not happened yet), so the answer would be missing rows
refund rate, 2026-05, checked: declined: stale: analytics.fct_refunds was last loaded at 2026-06-27 02:00, and rows for the period ending 2026-05-31 can arrive through 2026-07-19, 49 days later (not loaded, not happened yet), so the answer would be missing rows
refund rate, 2026-04, checked: 6.74%, data as of 2026-06-27 02:00:00
On orders placed before March 2026, old enough for all their refunds to have come, 56% of refunds came within 30 days of the order and 93% within 45, and the longest took 49 days. So the refund rate's window is 49 days, the longest delay the shop has seen.
Without a check, the layer reports June's refund rate as 0.54%, against 6.85% for May, which looks normal. June's fall has two causes: some of June's refunds were made but never loaded, and most haven't been made yet. The twelve settled months, May 2025 to April 2026, estimate each. Their refund rate is 7.21%, and with each month's refunds counted only to the point the refunds feed's last load reaches for June, they give 1.17%.
Most of June's shortfall is refunds not yet made: the stale feed explains only 8% of it. May is short too, for the same two reasons, and the stale feed explains 33% of its shortfall. These are estimates from other months, not measurements.
The check finds June's orders complete and its refunds short for both causes, and the tool declines May and June with kind stale and a reason that names the feed, its last load and the last day the period's refunds can come. The model never sees a number to report. April's window ended before the refunds' last load, so its rate is answered, with the refunds' load time in its provenance.
The naive version of the check misfires. The customers table was last loaded at 2026-06-15 02:00:00, because a customer's row loads when they sign up, and nobody signed up after mid-June. Checked against its own loads, June's new customers would be declined. But a customer's first order date comes from the orders, so FEEDS maps the customers table to the orders feed, and June's new customers pass. Which feed decides a table's freshness is a fact about your pipeline, and it belongs in a mapping a person writes, as dbt sources declare their own loaded_at_field.
We decline a period that hasn't settled rather than answer with a warning, as a rule of thumb. A warning travels with the answer only until someone copies the number. A decline that names the last load and the last day the period's events can come lets the person ask for a period that has settled.
Working rule: Check freshness in code before every answer that names a period: compare the newest load of each feed the answer reads with the end of the period plus the metric's settling window, and decline as stale when the feed stops short, whether its rows haven't loaded or haven't happened. Measure each window from your own data, as the delay that covers the events you've seen. Read load times from your own load columns or a load log, never from table metadata, and map each table to the feed that decides its freshness. Answer a question with no period, and show its data as of in the provenance.
每个答案都要展示出处
拒答向提问者说明为什么没有数字。答案也欠提问者同样的交代:这个数字从何而来。答案的出处(provenance)包括:其背后的指标或 SQL、筛选条件、计算时所用的角色,以及数据的新鲜程度(数据有多新)。自第 1 章起,店铺的 Answer 一直带有 answer.provenance。本节讲的是如何把它展示出来。我们没有找到研究来衡量展示出处是否能让人发现更多错误的数字,因此这是我们的设计选择,而不是经过测量的效果。理由是:核对一个指标的定义只需几秒,核对一条查询则需要几分钟。
provenance_card()(位于 lib/declining.py)会打印带有出处的答案:结果、每个指标的名称及其负责人在语义层中写下的定义、筛选条件、角色、数据截止时间、每个数据源的新鲜度检查,以及可选的 SQL。拒答的卡片则包含其类型、原因和角色。下面回放拒答型智能体的三次运行及其卡片,看看卡片能发现什么:2026 年 3 月按地区划分的净收入,它答对了;2025 年按销售区域划分的总收入,它答错了;以及运费成本,它拒答了。
1"""Show the provenance with every answer: the metric's definition, filters, role, freshness and the SQL that ran.2The prose's point rests on the first card's SQL and the second card's definition line, which names the tax."""3from lib.agent import run_agent4from lib.declining import DecliningToolbox, provenance_card5from lib.golden import golden_result, grade, load_golden6from lib.metric_tools import catalog7from lib.model import RecordedModel8from lib.prompts import declining_prompt9from lib.semantic import load10from lib.warehouse import connect1112con, layer = connect(), load()13system, model = declining_prompt(catalog(layer)), RecordedModel("decline-runs")14golden = {q.question_id: q for q in load_golden()}15# one right answer, one near-miss metric, one decline16for question_id in ("gq-002", "gq-012", "gq-021"):17 q = golden[question_id]18 toolbox = DecliningToolbox(con, layer, role=q.role)19 trace = run_agent(q.question, system, model, toolbox, max_steps=4)20 print(f"{question_id}: {q.question}")21 # only the first card shows its SQL, which the prose walks through22 print(provenance_card(trace.answer, layer, toolbox.freshness, sql=question_id == "gq-002"))23 print(f"{question_id} outcome: {grade(q, trace.answer, golden_result(con, q))}")24 print()
python answer-with-provenance.pygq-002: What was net revenue in each region in March 2026?
order__region net_revenue
AMER 51,762.14
APAC 35,296.93
EMEA 63,023.76
metric: net_revenue: Revenue in EUR after discounts and before tax, from real orders that weren't cancelled, counted on the order's local business day. Returned orders count; refunds are not subtracted.
filters: {'metric_time__month': '2026-03'}
role: analyst
data as of: 2026-07-02 02:00:00
freshness: analytics.fct_orders last loaded 2026-07-02 02:00, period ends 2026-03-31: complete
sql:
SELECT
orders.region AS order__region,
SUM(orders.net_revenue_eur) AS net_revenue
FROM analytics.fct_orders AS orders
WHERE
orders.status <> 'cancelled'
AND CAST(DATE_TRUNC('MONTH', CAST(orders.order_date_local AS DATE)) AS DATE) = CAST('2026-03-01' AS DATE)
GROUP BY
orders.region
ORDER BY
order__region ASC
gq-002 outcome: correct
gq-012: What was total revenue for each sales region in 2025?
order__region gross_revenue
AMER 538,429.66
APAC 444,472.81
EMEA 897,698.19
metric: gross_revenue: Revenue in EUR including tax, before refunds, from real orders that weren't cancelled.
filters: {'metric_time__year': 2025}
role: analyst
data as of: 2026-07-02 02:00:00
freshness: analytics.fct_orders last loaded 2026-07-02 02:00, period ends 2025-12-31: complete
gq-012 outcome: wrong
gq-021: How much did shipping cost us in fiscal year 2026?
declined: not_in_data: The shop's data does not include a metric for shipping costs.
role: finance
gq-021 outcome: correct
第一张卡片 gq-002 是一个答案连同其背后的全部内容。净收入的定义说明了这个数字包含什么,筛选条件写明 2026 年 3 月,新鲜度一行表明订单数据源在该期间是完整的,SQL 则显示已取消的订单被排除在外。它的结果是 correct。
第二张卡片 gq-012 正是这种卡片存在的原因。问题问的是总收入,而店铺的财务团队用这个词指的是净收入。智能体选择了 gross_revenue,卡片的定义一行写着“Revenue in EUR including tax”。只看数字的人会看到三个看似合理的地区总计;读了数字下面第一行的人会发现含税,并重新提问。结果是 wrong,卡片无需借助 SQL 就让这个错误显现出来。
第三张卡片是一次拒答,列出了它的类型和原因 not_in_data,以及计算时所用的角色 finance。拒答不需要 SQL,但需要角色:当第 9 章加入访问规则后,同一个问题可能对一个角色给出答案,而对另一个角色拒答。
工作准则: 每个答案都要连同其出处一起展示,并把指标的名称和定义放在最前面,因为差一点就对的指标正是在那里暴露的。然后展示筛选条件、期间、角色和数据截止时间,并把 SQL 放在多点一步即可看到的位置,供需要核对的人查看。拒答也以同样的方式展示,包括其类型、原因和角色。
这部分尚未翻译,因此以英文显示。
Measure declines, not only accuracy
Declines, the freshness check and provenance change how the agent fails, and a score has to show which way. Accuracy doesn't. It counts a decline on an answerable question as a miss, the same as a wrong number, so an agent that guesses scores no worse than one that declines. Kalai and colleagues argue that grading this way rewards guessing (Kalai et al.). Two pairs of measures separate the cases, and one score weighs them.
Decline precision and recall treat a question whose right answer is a decline as a positive, and a run that ended without a number, by a decline or a question back, as a predicted positive. Decline precision asks how many of the agent's declines were due; decline recall asks how many of the due declines the agent made. Coverage and selective risk come from selective prediction. Coverage is the share of questions the system doesn't reject, and selective risk is the error rate on the questions the system answers (Geifman and El-Yaniv). Lowering the bar for answering raises coverage and usually risk. TrustSQL's penalty score weighs them. In the penalty score, each question scores one point for a right answer or a right decline, nothing for declining an answerable question, and minus c for a wrong number, including an answer to a question it should have declined; the score is the mean over the questions (Lee et al.). TrustSQL's questions to decline are unanswerable ones only, and it leaves resolving ambiguity out of its scope. We score the ambiguous questions too, as questions to decline, and count a question back on one as a right decline, so our score is TrustSQL's adapted to the shop's golden set. With c at 0 it is the share of correct replies, as accuracy is; with c large, one wrong number cancels many right ones. As formulas, where a wrong number includes any number given for a question that should be declined:
decline precision = correct declines / all declines
decline recall = correct declines / questions whose right answer is a decline
coverage = questions answered with a number / questions asked
selective risk = wrong numbers / questions answered with a number
penalty score = (right answers + correct declines − c × wrong numbers) / questions asked
A run that hit the step limit is neither a decline nor an answer: it adds nothing to the score and counts as unanswered for coverage. Tally in lib/declining.py computes all five. We run chapter 6's metrics-first agent and the declining agent on the same 21 dev questions, 17 answerable and 4 to decline, so the measures compare like with like, and print each one with c at 3, the cost the prompt states, and at 10, a value TrustSQL uses. Choosing c is a judgement about what a wrong number costs, not a measurement.
1"""Measure declines, not only accuracy: precision, recall, coverage, selective risk and a penalty score, on dev.2The prose's point rests on each agent's decline recall, decline precision and penalty score lines."""3from lib.agent import run_agent4from lib.declining import DecliningToolbox, Tally5from lib.golden import golden_result, grade, load_golden6from lib.metric_tools import MetricToolbox, catalog7from lib.model import RecordedModel8from lib.prompts import declining_prompt, metrics_first_prompt9from lib.semantic import load10from lib.warehouse import connect1112con, layer = connect(), load()13agents = {"metrics-first": (metrics_first_prompt(catalog(layer)), RecordedModel("metric-agent-runs"), MetricToolbox),14 "declining": (declining_prompt(catalog(layer)), RecordedModel("decline-runs"), DecliningToolbox)}15# The dev split's answer and decline questions; forbidden questions are decided below the model, in chapter 9.16questions = [q for q in load_golden() if q.split == "dev" and q.kind in ("answer", "decline")]17for name, (system, model, Toolbox) in agents.items():18 tally, by_type = Tally(), {"ambiguous": Tally(), "unanswerable": Tally()}19 for q in questions:20 answer = run_agent(q.question, system, model, Toolbox(con, layer, role=q.role), max_steps=4).answer21 outcome = grade(q, answer, golden_result(con, q))22 tally.add(q.question_id, q.kind, answer, outcome)23 if q.kind == "decline":24 by_type[q.decline_type].add(q.question_id, q.kind, answer, outcome)25 for ending in ("right", "wrong", "abstained", "step limit"):26 print(f"{name} {ending}: {tally.count(ending=ending)} of {len(questions)}")27 # None for the kind means every question counts, so the wrong numbers include a number given for any question28 for label, kind, ending in (("wrong numbers", None, "wrong"), ("declined answerable", "answer", "abstained")):29 ids = [i for i, k, e in tally.rows if kind in (None, k) and e == ending]30 print(f"{name} {label}: {', '.join(ids) or 'none'}")31 print(f"{name} decline precision: {tally.precision:.2f}")32 print(f"{name} decline recall: {tally.recall:.2f}")33 for kind, part in by_type.items():34 # an abstention is a decline or a question back; this counts those on the questions whose right answer is a decline35 print(f"{name} recall, {kind}: {part.count('decline', 'abstained')} of {part.count('decline')}")36 print(f"{name} coverage: {tally.coverage:.2f}")37 print(f"{name} selective risk: {tally.selective_risk:.2f}")38 # c is the penalty for a wrong number, in points: 3 is the cost the prompt states, 10 is one of the penalties TrustSQL reports39 for c in (3, 10):40 print(f"{name} penalty score, c={c}: {tally.penalty_score(c):.2f}")
python decline-precision-recall.pymetrics-first right: 12 of 21
metrics-first wrong: 5 of 21
metrics-first abstained: 3 of 21
metrics-first step limit: 1 of 21
metrics-first wrong numbers: gq-012, gq-016, gq-017, gq-018, gq-020
metrics-first declined answerable: none
metrics-first decline precision: 1.00
metrics-first decline recall: 0.75
metrics-first recall, ambiguous: 1 of 2
metrics-first recall, unanswerable: 2 of 2
metrics-first coverage: 0.81
metrics-first selective risk: 0.29
metrics-first penalty score, c=3: 0.00
metrics-first penalty score, c=10: -1.67
declining right: 11 of 21
declining wrong: 4 of 21
declining abstained: 5 of 21
declining step limit: 1 of 21
declining wrong numbers: gq-008, gq-012, gq-017, gq-018
declining declined answerable: gq-016
declining decline precision: 0.80
declining decline recall: 1.00
declining recall, ambiguous: 2 of 2
declining recall, unanswerable: 2 of 2
declining coverage: 0.71
declining selective risk: 0.27
declining penalty score, c=3: 0.14
declining penalty score, c=10: -1.19
The declining agent declines more and answers less, and the decline measures show the trade. Decline recall rose from 0.75 to 1.00: the declining agent declined or asked back on all four questions, where chapter 6's agent answered "Which region is doing best?". Decline precision fell from 1.00 to 0.80, because it declined one answerable question, gq-016, the count of customers who ordered.
Coverage fell from 0.81 to 0.71, and selective risk moved from 0.29 to 0.27.
The penalty score at c = 3 went from 0.00 to 0.14, and at c = 10 from -1.67 to -1.19.
The wrong numbers tell the rest. Chapter 6's agent gave gq-012, gq-016, gq-017, gq-018, gq-020; the declining agent gave gq-008, gq-012, gq-017, gq-018. Two left the list by the routes this chapter built: gq-020 through the ranking check and a question back, and gq-016 through a decline. One joined it: gq-008, total revenue in 2025, which chapter 6's agent answered with net revenue and this one with gross revenue, the near-miss pair chapter 6 described. The prompt's sentence that a metric with a similar name isn't the same metric didn't prevent it, though with one sample per call, a single flip such as gq-008 may be chance. The rest are the near-miss metrics of gq-012 and gq-018, and gq-017, which needs the asker's region, chapter 9's work. Both agents score below zero at c = 10: a handful of wrong numbers outweighs every right one, which matches TrustSQL's own finding, that "achieving high scores under severe penalties requires significant effort".
Read these as small numbers. The dev split has four questions to decline, so recall moves in steps of a quarter, and one question of 21 changes coverage by about five points. The dev numbers are optimistic for the reason given in Decline with a reason, and this chapter records nothing on the test split. The direction agrees with larger studies: prompting helps abstention but doesn't fix it (Kirichenko et al.), and prompts that push models to critique questions also raise "false invalid judgments" on valid ones (Lin et al.).
Working rule: Score an agent on declines as well as answers: report decline precision and recall, coverage and selective risk, and a penalty score with a c your stakeholders agree to, stated as a judgement. Compare agents on the same questions, and list the wrong numbers each gave, because a change that removes one wrong number can add another. When the decline questions number in single digits, read the runs, not only the rates.
生产环境中会出什么问题
这家商店的拒答智能体会拒答应当拒答的开发类问题,代价是漏答了一个本可回答的问题。对真实数据仓库做拒答的智能体,会以下列方式出错,每一种都有相应的发现办法。
**安静的数据源看起来像是过期了。**新鲜度检查读取数据中最新的 _loaded_at,而某天没有事件的数据源,当天不会加载任何行。商店的退款大约每天到达 1.2 笔,因此一个安静的周末就会让一个完整的周期看起来已经过期。记录每次运行及其行数的加载日志可以解决这个问题;dbt 的 loaded_at_query 就是为这类数据源而设。你可以按数据源统计过期拒答的次数来发现这种情况:一个在大多数日子里都被拒答的数据源,是安静,而不是延迟。
**沉降窗口会过时。**商店有一半的退款发生在下单之后 27.5 天以上,而窗口取的是迄今所见的最长延迟。更长的退货期,或更慢的支付服务商,都会使窗口过短,检查就会放行仍在上升的月份。应按计划重新测量每个窗口,所用订单要足够久远,其全部事件都已到齐,并关注在窗口之后到达的事件所占的比例。
**序列的最后一个周期会逃过检查。**检查读取的是请求所筛选的周期。按月分组而不带周期筛选的请求会返回每个月,最后一个月并不完整,却能通过检查。应将任何时间序列的最后一组与数据源核对,或要求每个按时间分组的请求都带有周期。
**拒答会在指标层止步的地方增多。**拒答智能体拒答了 gq-016,因为没有任何指标统计下过单的客户。每一次对数据仓库本可回答的问题给出的 not_in_data 拒答,都是该层的一个缺口:把它们当作待办清单,用来决定要编写的指标,或交给第 7 章的受保护 SQL处理。
类别可能正确而理由错误。gq-016 的理由谈的是一个筛选条件,而第 6 章的预测回复问错了事情。类别用于计数;那句话是写给人看的,由模型来写。每周阅读一批理由样本,并让类别的数量少到人能够区分它们。
**提示中的置信度目标不是校准。**提示要求模型只在置信度高于 75% 时作答,而模型没有可靠的办法知道自己的置信度:大语言模型口头表达的置信度“往往过于自信”(Xiong et al.),而一项未经评审的研究指出,“查询可执行性”在把正确 SQL 排在错误 SQL 之前的各项信号中,AUROC 仅约为 0.61 到 0.68(Richardson)。应优先采用代码中的检查,例如未知指标、没有周期的排名或过期的数据源,并把提示中的目标仅当作一种提示。
反问可能只解决了问题的一半。gq-020 询问了周期,却没有定义“最好”,而评分把它算作正确。应在追踪记录中阅读反问的内容,并让反问工具给出点明各指标的选项,由用户来选择一种定义。
**工作准则:**在更多人使用该智能体之前,为新鲜度检查配备加载日志,按计划重新测量每个指标的沉降窗口,检查每个时间序列的最后一个周期,按类别和数据源统计拒答,并阅读一批理由和反问的样本。先从对可回答问题的 not_in_data 拒答入手,因为每一次都指明了一个要编写的指标。
本章做出的决定
- **拒答是一个会结束本次运行的工具,**它带有取自固定清单的类别,以及一句写给提问者的话。模型在
not_in_data、forecast和no_definition之间选择;stale由代码给出,从第 9 章起forbidden也由代码给出。 - **有歧义的问题与无法回答的问题要区分开,**各自按 PRACTIQ 或 TrustSQL 的术语命名。只有在每一种理解都可以回答时,才向提问者反问。
- **代码中的检查先于模型的判断:**排名检查会拒绝没有限定时间段的排名;新鲜度检查会拒答这样的时间段:其数据源的数据截止时间早于该时间段的结束时间加上指标的稳定窗口。
- **新鲜度来自数据中的加载时间,**由每张表映射到决定它的数据源,绝不取自表的元数据。
- **尚未稳定的时间段应当拒答,而不是附带一条警告作答,**这是一条经验法则,拒答理由要指明该数据源最近一次加载的时间,以及其事件最晚可能出现的日期。退款率的窗口取该店铺退款所呈现的最长延迟。
- **每个回答都展示其出处,**先是指标和定义,然后是筛选条件、角色、数据截止时间和新鲜度,SQL 则只需再点一步即可查看。
- **拒答用精确率和召回率、覆盖率和选择性风险,以及惩罚分数来衡量,**惩罚分数取 c = 3 和 c = 10,即 TrustSQL 的均值,其中有歧义的问题按拒答计分;评测基于开发集中的 21 个应答与拒答问题,并与第 6 章的智能体并列比较。越权问题留待第 9 章处理。
- **提示词依据第 6 章的开发集轨迹编写,并只记录一次,**本章没有记录测试集。
本章,在本版次中
版次 2026.11.15示例于 2026年11月19日 通过本章 5个示例 个示例中有 5 个通过
译自英文版图书,版次 2026.11.15。