优先将问题路由到受治理的指标
阅读约需 77分钟 分钟,8节中含6个示例。
读完本章,你可以把语义层的指标而不是数据仓库的表作为工具交给智能体,让它返回结构化的指标请求,并在任何查询运行之前先对该请求进行校验;当一个问题可能对应多个指标时,让它向提问者反问;并在黄金问题上衡量这些改变带来的效果。
第 5 章把商店的各项定义一次性写成了语义层:五个语义模型之上的十个指标,由本书自带的编译器在 lib/semantic.py 中编译为 SQL。熟悉该语义层的人可以向它询问 2026 年 3 月各地区的净收入,并得到财务团队认定的数字,而智能体目前还做不到。第 1 章只给了它一个指标,而它必须超越的基线是仅依据表结构生成的 SQL,这种方式在开发集问题上答对 2 of 24 个。第 4 章把各项定义以文本形式与表结构一并交给了模型,但 SQL 仍由模型来写,因此每条定义都必须经受住模型的理解。
当查询由模型来写时,任何被误读的定义都会体现在最终的数字里。优先将问题路由到受治理的指标,就是不再让模型编写查询:它的任务缩小为选择哪个指标、按什么分组、如何过滤,而 SQL 由语义层来写,语义层由人负责维护和测试。数据仓库厂商对自家助手的描述也是如此。Databricks 在其问答型 Genie Agents 的指引中指出,指标视图对它们尤其适用,因为指标视图预先定义了指标、维度和聚合(Databricks);Snowflake 则建议为 Cortex Analyst 使用语义视图(Snowflake)。与第 4 章中的厂商建议一样,这些都只是建议,而不是实测结果。本章在商店的黄金问题上衡量这一模式,所用的仍是与其他各章相同的已录制模型。
给智能体指标,而不是数据表
智能体的工具决定了它可能在哪些地方出错。如果给它数据表,模型可能以错误的方式连接表、漏掉过滤条件,或者选中遗留的数据集市,第 2 章逐个展示了这些失败。如果给它指标,它仍可能选错指标或选错维度,但无法改写定义,因为定义不在它能触及的范围之内。指标工具按人们提问的方式呈现语义层所包含的内容:指标名称及其描述、每个指标可用于分组和过滤的维度,以及时间粒度。Databricks 在描述其 Unity Catalog 指标视图时用了同样的说法,即把“度量定义与用于分组、过滤和聚合这些度量的字段(也称为维度)”分离开来(Databricks)。
商店的智能体从 lib/metric_tools.py 获得指标工具,这是本章新增的一个模块。其中的 MetricToolbox 封装了一个数据仓库连接和第 5 章中的语义层,并提供五个工具:list_metrics、describe_metric、query_metric、ask_back 和 cannot_answer。提示词 METRICS_FIRST 位于 lib/prompts.py 中,其中包含 list_metrics 返回的内容,因此模型无需再花一次调用去获取:每个指标占一行,写明其描述和可用的维度。提示词还说明了商店的财年如何划分、如何把一个时间段写成过滤值,并列出了各个区域和销售渠道。它没有列出任何数据表和列。
先看智能体看到了什么。该示例构建提示词,打印净收入在目录中的那一行,然后针对退款率调用 describe_metric,并分别在第 1 章的仅含表结构的提示词和本提示词下,估算一个黄金问题 gq-002(2026 年 3 月各区域的净收入)的规模。两种规模都是录制调用时模型运行时所报告的词元数。
1"""Give the agent the semantic layer's metrics instead of the warehouse's tables, and compare what each prompt holds.2The prose rests on the metric and table counts and the two prompt token counts, which come from the recordings."""3from lib.golden import load_golden4from lib.metric_tools import MetricToolbox, catalog5from lib.model import RecordedModel6from lib.prompts import metrics_first_prompt, schema_only_messages7from lib.warehouse import connect, describe_schema89con = connect()10toolbox = MetricToolbox(con)11system = metrics_first_prompt(catalog(toolbox.layer))12print(f"tools: {', '.join(toolbox.tools)}")13print(f"metrics in the prompt: {len(toolbox.layer.metrics)}")14print(f"tables in the prompt: {system.count('CREATE TABLE')}") # a table in the prompt would be written as a CREATE TABLE statement15print(catalog(toolbox.layer).splitlines()[0]) # the first line: net revenue, with its dimensions1617print(toolbox.call("describe_metric", {"metric": "refund_rate"}).content)1819# The first call of the same golden question, gq-002, under each prompt, as the recordings measured it.20question = next(q for q in load_golden() if q.question_id == "gq-002").question21schema_only = schema_only_messages(describe_schema(con), question)22metrics_first = [{"role": "system", "content": system}, {"role": "user", "content": question}]23print(f"schema-only prompt tokens: {RecordedModel('schema-only-sql').call_for(schema_only)['prompt_tokens']}")24print(f"metrics-first prompt tokens: {RecordedModel('metric-agent-runs').call_for(metrics_first)['prompt_tokens']}")
python metric-tools.pytools: list_metrics, describe_metric, query_metric, ask_back, cannot_answer
metrics in the prompt: 10
tables in the prompt: 0
- net_revenue: Revenue in EUR after discounts and before tax, from real orders that weren't cancelled, counted on the order's local business day. Returned orders count; refunds are not subtracted. Dimensions: customer__country, customer__first_order_date, customer__region, order__channel, order__country, order__order_date, order__region, order__status
refund_rate, a ratio metric: Refunded EUR over gross revenue of the same orders: orders placed in the period that weren't cancelled.
Dimensions:
- metric_time__day
- metric_time__week
- metric_time__month
- metric_time__quarter
- metric_time__year
- metric_time__fiscal_year: 2025, 2026, 2027
- metric_time__fiscal_quarter: 2025-Q1, 2025-Q2, 2025-Q3, 2025-Q4, 2026-Q1, 2026-Q2, 2026-Q3, 2026-Q4, 2027-Q1, 2027-Q2
- order__channel: (none), app, marketplace, web
- order__country: AU, CA, DE, ES, FR, GB, JP, NL, SG, US
- order__order_date
- order__region: AMER, APAC, EMEA
- order__status: cancelled, delivered, placed, returned, shipped
schema-only prompt tokens: 827
metrics-first prompt tokens: 1120
提示词中列出了 10 个指标和 0 张表。每个指标的那一行,是其负责人撰写的描述(与 YAML 中的文字完全一致),后面跟着无需扇出即可使用的维度。净收入的那一行列出了 order__region 和 customer__region,但没有 product__category,因为订单模型只能通过订单行到达产品,而每个订单有多个订单行。
describe_metric 补充了目录中没有的内容:时间粒度,以及每个分类维度的取值,这些取值在工具运行时从数据仓库读取。退款率的区域是 AMER、APAC 和 EMEA,其财季是形如 2026-Q1 的标签。提示词已经给出区域和渠道的名称,所以对这些维度,工具只是重复了模型已被告知的内容。至于国家、订单状态和财务期间,这个工具是模型了解商店如何书写它们的唯一来源。
以指标为先的提示词并没有更短。它有 1120 个词元,而仅含表结构的提示词有 827 个,因为一条定义比一个列名需要更多的字词。改变的是这些词元所承载的内容:由人编写并核对过的规则,而不是需要模型去解读的名称。
工作准则: 给智能体提供指标、指标的描述以及每个指标可用的维度,在问题确有需要之前,不要把数据表放进它的工具。目录能放进提示词时就放在提示词中,这样一次运行就不必花一次调用来列出指标,并让描述工具从数据仓库返回维度取值。当目录大到提示词容纳不下时,在提示词中保留名称和描述,把维度移到描述工具之后。
这部分尚未翻译,因此以英文显示。
Ask for a request, not SQL
With metric tools in place, the model's reply is no longer a query but a request. A structured reply is one the code can parse into a typed object and check field by field. Here it is a JSON tool call whose arguments become a MetricRequest, the metric request from chapter 5, which names metrics, group-by dimensions, filters, an order and a limit. Nothing in it is SQL, so the model can't add anything to a query that the layer didn't define.
Qwen's chat template for the model this book records with asks for each tool call as JSON inside <tool_call> tags, and returns tool results inside <tool_response> tags (Qwen's template). Qwen's model card recommends its Qwen-Agent library, which "encapsulates tool-calling templates and tool-calling parsers internally" (model card). We keep the book's own format instead: the prompt asks for one bare JSON object, {"tool": ..., "arguments": ...}, which run_agent() in lib/agent.py parses, as it has since chapter 1. A small parser we own is easier to test than a library's, and no library release can change it.
The shop's case is golden question gq-003, net revenue by sales channel in calendar 2025. Schema-only SQL failed it in chapter 1 by filtering fct_orders on a calendar_year column that exists only in dim_date. The recording metric-agent-runs holds the same model's replies to the metrics-first prompt for all 48 golden questions. It was made by recordings/record_metrics_first.py with the pinned model file, Qwen's recommended sampling and a seed reset before every call, like every recording in this book. gq-003's run, replayed, shows the reply, the request and what the layer did with it.
1"""The model returns a metric request, not SQL, and the layer compiles that request. The prose rests on the `matches golden`2line, which the example asserts; the schema-only outcome shows the baseline's failure on the same question."""3from lib.agent import run_agent4from lib.golden import golden_result, load_golden, results_match, schema_only_outcome5from lib.metric_tools import MetricToolbox, catalog6from lib.model import RecordedModel7from lib.prompts import metrics_first_prompt8from lib.semantic import MetricRequest9from lib.warehouse import connect, describe_schema1011con = connect()12toolbox = MetricToolbox(con)13system = metrics_first_prompt(catalog(toolbox.layer))14question = next(q for q in load_golden() if q.question_id == "gq-003")15model = RecordedModel("metric-agent-runs")16trace = run_agent(question.question, system, model, toolbox, max_steps=4) # the chapter's limit of four model calls per run1718print(f"question: {question.question}")19print(f"model calls: {sum(step.kind == 'model' for step in trace.steps)}")20print(f"reply: {trace.steps[0].content}") # step 0 is the model's first reply, before any tool runs21print(MetricRequest.from_arguments(trace.steps[0].arguments)) # the reply's arguments, parsed into the typed request22print(trace.answer.provenance.sql) # SQL the layer compiled from the request, not SQL the model wrote23print(trace.answer.show())24matches = results_match(trace.answer.rows, golden_result(con, question))25assert matches, "the layer's answer to gq-003 should match its golden result" # the request asks gq-003's own question, net revenue by channel for 2025, so its rows are the golden ones26print(f"matches golden: {'yes' if matches else 'no'}")2728schema = describe_schema(con)29print(f"schema-only outcome: {schema_only_outcome(con, RecordedModel('schema-only-sql'), schema, question)}")
python model-returns-a-request.pyquestion: How much net revenue did each sales channel bring in during calendar year 2025?
model calls: 1
reply: {"tool": "query_metric", "arguments": {"metrics": ["net_revenue"], "group_by": ["order__channel"], "filters": {"metric_time__year": 2025}}}
MetricRequest(metrics=('net_revenue',), group_by=('order__channel',), filters={'metric_time__year': 2025}, order_by=(), limit=None)
SELECT
orders.channel AS order__channel,
SUM(orders.net_revenue_eur) AS net_revenue
FROM analytics.fct_orders AS orders
WHERE
orders.status <> 'cancelled'
AND CAST(DATE_TRUNC('YEAR', CAST(orders.order_date_local AS DATE)) AS DATE) = CAST('2025-01-01' AS DATE)
GROUP BY
orders.channel
ORDER BY
order__channel ASC
order__channel net_revenue
(none) 82,922.75
app 431,258.70
marketplace 227,826.99
web 909,519.42
matches golden: yes
schema-only outcome: error
The run took 1 model call. The reply names net_revenue, groups by order__channel and filters metric_time__year to 2025: the question's three parts, each mapped to a name the layer defines. The layer wrote the SQL. It reads analytics.fct_orders, leaves out cancelled orders by the metric's own filter, truncates the local business day to its year and groups by channel, and its channel totals agree with the golden ones. The same question under schema-only SQL ends in error.
The request also makes the answer's provenance exact. answer.provenance names the metric, the filters and the SQL that ran, so a person who doubts the number checks a short request and the metric's definition, not a query the model wrote. Reviewing a request takes a glance; reviewing a model's SQL takes the same care as reviewing anyone's.
Working rule: Have the model return a metric request as structured arguments to a tool, and parse it into a typed request before anything else happens. Keep the reply format simple enough for your own code to parse, and keep that parser in one place. Accept SQL from a model only on a separate, guarded path, for questions no metric covers.
这部分尚未翻译,因此以英文显示。
Validate the request against the layer
A parsed request is well-formed, which isn't the same as right. Some mistakes fail loudly: the compiler from chapter 5 refuses a metric it doesn't define, and a dimension that would fan out. Others compile and run. A filter value that no row holds, such as a region called Europe, returns an empty total. A month written against a dimension that holds days matches one day. Both results look like answers. Validating a request means checking it against the layer's definitions and the warehouse's values before it compiles, and sending every problem back to the model with what would be allowed instead.
validate_request() in lib/metric_tools.py does that for the shop. It checks each metric's name, and suggests the nearest defined names for a misspelt one. It checks each dimension against what the requested metrics can reach, and when one can't, it names the metrics that can. It checks each categorical filter value against the values in the warehouse, each filter on metric_time against its grain, so that a month must be the first day of a month, and each filter on a date dimension, which must be a single day. query_metric runs it on every request. A request with problems comes back to the model as an error that lists every problem it finds once the metrics are known, so one correction can fix them all. An unknown metric comes back on its own, before anything else is checked.
A person might type three requests the layer refuses: orders by customer email, which no semantic model defines; net revenue by product category, which would fan out; and net revenue for a region called Europe. The model's own first request for gq-002 comes next: the code runs it unvalidated to show what would have happened, then follows its corrected request to its answer.
1"""Validate a metric request against the layer and the warehouse before it runs, and show what would run without it.2The prose rests on the unchecked total and the two `matches golden` lines; the example asserts the two matches."""3from lib.agent import run_agent4from lib.golden import golden_result, load_golden, results_match5from lib.metric_tools import MetricToolbox, catalog, validate_request6from lib.model import RecordedModel7from lib.prompts import metrics_first_prompt8from lib.semantic import MetricRequest9from lib.warehouse import connect1011con = connect()12toolbox = MetricToolbox(con)13layer = toolbox.layer14for label, request in [("no such dimension", MetricRequest(("orders",), group_by=("customer__email",))),15 ("fan-out", MetricRequest(("net_revenue",), group_by=("product__category",))),16 ("no such value", MetricRequest(("net_revenue",), filters={"order__region": "Europe"}))]:17 print(f"{label}: {validate_request(layer, con, request)[0]}")18print(f"unchecked, region Europe: {layer.query(con, request).rows}") # `request` is still the Europe filter from the loop1920# The model's own requests for gq-002, net revenue in each region in March 2026, replayed from the recording.21question = next(q for q in load_golden() if q.question_id == "gq-002")22system = metrics_first_prompt(catalog(layer))23trace = run_agent(question.question, system, RecordedModel("metric-agent-runs"), toolbox, max_steps=4)24first, second = (MetricRequest.from_arguments(s.arguments) for s in trace.steps if s.tool == "query_metric") # its two query calls25golden = golden_result(con, question)26unchecked = layer.query(con, first) # what would have run without validation27print(f"first request filters: {first.filters}")28print(f"unchecked total: {sum(row[1] for row in unchecked.rows):,.2f}") # row[1] is the net revenue column29print(f"golden total: {sum(row[1] for row in golden):,.2f}")30unchecked_matches = results_match(unchecked.rows, golden)31assert not unchecked_matches, "the unchecked request should miss the golden result" # a month written against a daily date dimension matches only its first day, so this total can't cover the month32print(f"unchecked matches golden: {'yes' if unchecked_matches else 'no'}")33print(f"validation said: {trace.steps[1].content}") # step 1 is the refusal the layer returned34print(f"second request filters: {second.filters}")35matches = results_match(trace.answer.rows, golden)36assert matches, "the second request should match the golden result" # filtering metric_time at the month's grain covers every day in March, the golden question's period37print(f"matches golden: {'yes' if matches else 'no'}")
python reject-unknown-dimension.pyno such dimension: orders can't use dimension 'customer__email'; no metric defines it
fan-out: net_revenue can't use dimension 'product__category'; metrics that can use it: line_net_revenue, units_sold
no such value: no row has order__region = 'Europe'; its values: AMER, APAC, EMEA
unchecked, region Europe: ((None,),)
first request filters: {'order__order_date': '2026-03'}
unchecked total: 4,150.82
golden total: 150,082.83
unchecked matches golden: no
validation said: error: request rejected: order__order_date = '2026-03': a date dimension matches single days only; for a period, filter metric_time at its grain, as metric_time__month = "2026-03" or metric_time__year = 2025
second request filters: {'metric_time__month': '2026-03'}
matches golden: yes
Each of the three requests is refused with a message that says what to do instead: no metric defines a customer's email, two other metrics can reach product category, and the regions are AMER, APAC and EMEA. Without the check, the request for Europe runs and returns one row with an empty total, which an agent would report as no revenue.
The model's own request is the case that matters, because no person wrote it. For March 2026 it filtered order__order_date, the orders' daily date dimension, to "2026-03". The compiler reads a month written against a daily dimension as the month's first day, so the unchecked request returns March 1 alone: 4,150.82 euros across the three regions, against a golden total of 150,082.83. Nothing in that answer says it covers one day. Validation refused the filter and named the dimension to use, metric_time__month. The model's second request used it, and its answer matches the golden result.
Across the dev split, validation refused 10 requests, and 6 runs went on to a correct answer after a refusal. Most of those refusals were the date filter. The prompt says a month is written as "2026-03" and lists metric_time__month among the grains, though it never shows the two together. The model reached for order__order_date anyway, probably because the catalog lists it beside the region. A check in code catches that mistake every time; a sentence in a prompt catches it only when the model follows the sentence.
Working rule: Validate every request before it compiles: names against the layer, categorical values against the warehouse, dates against their grain. Refuse with a message that names the allowed alternatives, and send it back to the model, which can usually correct itself in one call. Treat any request that would compile into an empty or partial result as invalid, not as an answer of zero.
这部分尚未翻译,因此以英文显示。
Ask back when a question is ambiguous
Validation catches requests the layer can't serve. It can't catch a valid request for the wrong thing. "Which region is doing best?" maps to a valid request for net revenue by region, and equally to one for orders or refund rate by region, and each can name a different region. When a question fits several metrics and the choice changes the answer, the right reply is a question back, with the choices spelt out. The shop's golden set marks such questions as declines of type ambiguous, and chapter 3 scores any reply to them that doesn't return a result as correct.
The ask_back tool ends the run with a question to the person and the options the model offers. An option that names a metric is shown with the layer's description of it, so the person chooses between definitions, not between words: revenue becomes revenue in euros after discounts and before tax, from orders that weren't cancelled. The answer carries ask_back as its source in the provenance, so a score can count clarifications apart from declines.
Four dev questions should end in a question back or a reason: two are ambiguous, gq-019, about our best customers, and gq-020, about which region is doing best, and two can't be answered, a forecast and the cost of shipping. The last tool of each run is printed for all four. Then the code prints the questions the agent asked for gq-019 and gq-005, the answer it gave for gq-020, and a check the recorded runs didn't have.
1"""Ask back when a question could mean several metrics: the layer supplies each option's definition to the person.2The prose rests on the tool each dev decline ended with and on the ranking check's count of flagged runs."""3from lib.agent import run_agent4from lib.golden import load_golden5from lib.metric_tools import MetricToolbox, catalog, unbounded_ranking6from lib.model import RecordedModel7from lib.prompts import metrics_first_prompt8from lib.semantic import MetricRequest9from lib.warehouse import connect1011con = connect()12toolbox = MetricToolbox(con)13system = metrics_first_prompt(catalog(toolbox.layer))14model = RecordedModel("metric-agent-runs")15traces = {q.question_id: run_agent(q.question, system, model, MetricToolbox(con, toolbox.layer, role=q.role),16 max_steps=4) for q in load_golden()}17for q in load_golden():18 if q.split == "dev" and q.kind == "decline":19 last = next(s for s in reversed(traces[q.question_id].steps) if s.kind == "model") # the run's final model reply20 print(f"{q.question_id} ({q.decline_type}): {last.tool}")21 print(f" {q.question}")2223print("gq-019 asks back:")24print(traces["gq-019"].answer.reason)25print("gq-005 asks back:")26print(traces["gq-005"].answer.reason)27answered = next(s for s in reversed(traces["gq-020"].steps) if s.tool == "query_metric") # its last query, the one that ran28print(f"gq-020 request: {answered.arguments}")29print(traces["gq-020"].answer.show())3031# A check the recorded runs didn't have: refuse a request that ranks its result over no period.32# Applied here to the last request of every run, dev and test, that returned a result.33final = {question_id: MetricRequest.from_arguments(next(s for s in reversed(t.steps) if s.tool == "query_metric")34 .arguments)35 for question_id, t in traces.items() if t.answer.answered}36flagged = [question_id for question_id, request in final.items() if unbounded_ranking(request)]37print(f"runs with a result: {len(final)}")38print(f"requests that order their result: {sum(bool(r.order_by) for r in final.values())}") # an empty order_by means no ranking39print(f"ranking check flags: {', '.join(flagged) or 'none'}")40print(f"ranking check on gq-020: {unbounded_ranking(final['gq-020'])}")
python clarify-best-customers.pygq-005 (unanswerable): ask_back
What will our net revenue be next quarter?
gq-019 (ambiguous): ask_back
Who are our best customers?
gq-020 (ambiguous): query_metric
Which region is doing best?
gq-021 (unanswerable): cannot_answer
How much did shipping cost us in fiscal year 2026?
gq-019 asks back:
Do you mean customers by highest net revenue, or by highest number of orders?
- net_revenue: Revenue in EUR after discounts and before tax, from real orders that weren't cancelled, counted on the order's local business day. Returned orders count; refunds are not subtracted.
- orders: The number of real orders, cancelled ones included.
gq-005 asks back:
Do you mean net revenue for the next calendar quarter or the next fiscal quarter?
- calendar quarter
- fiscal quarter
gq-020 request: {'metrics': ['net_revenue'], 'group_by': ['customer__region'], 'order_by': ['net_revenue']}
customer__region net_revenue
APAC 925,657.84
AMER 1,174,510.02
EMEA 1,690,339.73
runs with a result: 31
requests that order their result: 1
ranking check flags: gq-020
ranking check on gq-020: the request ranks its result but names no period; filter or group by a metric_time grain
For gq-019, the agent asked whether best means the highest net revenue or the most orders, and the layer added each metric's definition. That is the behaviour the tool exists for, and it scores as correct. Read the options against the layer, though. No semantic model has a dimension that identifies a customer, so neither choice leads to a request the layer can serve. The honest reply was a decline: the shop's metrics don't rank customers.
For gq-020, the agent didn't ask. It requested net revenue by customer__region with no period, and returned all-time revenue for each region, smallest first. Every name in the request is valid, so validation passed it, and the score counts it as wrong. For the forecast, gq-005, the agent asked whether "next quarter" meant the calendar or the fiscal quarter, a question whose answer wouldn't help, since no metric forecasts. It scores as correct because it returned no number, and only the trace shows that the reason was wrong. The shipping question, gq-021, was declined with the right reason.
So the tool works when the model calls it, and the model's choice of when to call it is unreliable. A check in code catches gq-020's case: refuse a request that ranks its result but names no period. unbounded_ranking() in lib/metric_tools.py is that check, and the example applies it to the last request of every recorded run, dev and test, that returned a result. Of those 31, it flags gq-020. That shows it catches this case, not that it never misfires, because only 1 of those requests orders its result at all. query_metric doesn't run the check yet: the recordings were made without it, so adding it means recording again.
gq-019's case needs a check we haven't built: refuse an ask_back whose metric options can't group by what the question asks about. It would need to know what the question asks about, a customer here, and nothing in the toolbox knows that. It wouldn't cover gq-005 either, whose options are periods, not metrics. Both checks are rules about the shop's questions, and they belong in code, where a test can hold them, rather than in more prompt text.
Working rule: Give the agent a tool to ask back, with options that name the layer's metrics, and show the person each option's definition. Ask back only when every option leads to a request the layer can answer; when none does, decline instead. Read clarifications in traces as you read declines, because a score counts a useless question as correct.
这部分尚未翻译,因此以英文显示。
Measure metrics-first on the golden questions
So far the book has followed single runs. The measurement that counts is the one chapter 3 set up: every golden question, scored by grade() with results_match(), tuned on the dev split and checked once on the held-out test split, beside the schema-only baseline. We recorded the dev split three times while tuning, and two changes came from those runs. The catalog in the prompt stopped printing each metric's type beside its name, because the model copied the type into its requests; describe_metric still prints it. And a year written as text, such as "2025", became the number it can only mean. Only the final dev recording is kept, and the recording's manifest says the two earlier ones were discarded. The dev numbers below were measured on the split the changes were chosen on, so they're optimistic. The test split was recorded once, after those changes, and nothing changed afterwards.
Each run gets at most four model calls. A run that uses all four model calls without an answer is a failure, and grade() scores it as step limit, apart from the declines the model chose. Every run on both splits, for each approach, is replayed and counted by tool: model calls, refused requests, runs that reached a correct answer after a refusal, and clarifications by kind of question.
1"""Score metrics-first against the schema-only baseline on both splits: dev for tuning, test once for the final number.2The prose rests on the correct counts per split and on the step-limit lines, which count runs out of model calls."""3from collections import Counter, defaultdict45from lib.agent import run_agent6from lib.golden import golden_result, grade, load_golden, schema_only_outcome7from lib.metric_tools import MetricToolbox, catalog8from lib.model import RecordedModel9from lib.prompts import metrics_first_prompt10from lib.semantic import load11from lib.warehouse import connect, describe_schema1213con, layer = connect(), load()14schema, system = describe_schema(con), metrics_first_prompt(catalog(layer))15agent_model, sql_model = RecordedModel("metric-agent-runs"), RecordedModel("schema-only-sql")16counts, facts, missed = defaultdict(Counter), defaultdict(Counter), defaultdict(list)17for q in load_golden():18 toolbox = MetricToolbox(con, layer, role=q.role)19 trace = run_agent(q.question, system, agent_model, toolbox, max_steps=4) # four calls at most, as in every run20 outcome = grade(q, trace.answer, golden_result(con, q)) # 'step limit' when the calls ran out21 counts[q.split, "metrics-first"][outcome] += 122 counts[q.split, "schema-only"][schema_only_outcome(con, sql_model, schema, q)] += 123 facts[q.split][f"{q.kind} questions"] += 124 facts[q.split][f"{q.kind} correct"] += outcome == "correct"25 facts[q.split]["forbidden answered"] += q.kind == "forbidden" and trace.answer.answered # a forbidden question answered is a leak26 facts[q.split]["model calls"] += sum(step.kind == "model" for step in trace.steps)27 facts[q.split]["requests rejected"] += len(toolbox.rejected)28 facts[q.split]["correct after a rejection"] += bool(toolbox.rejected) and outcome == "correct"29 facts[q.split][f"asked back, {q.kind}"] += trace.answer.provenance.source == "ask_back"30 if outcome != "correct":31 missed[q.split].append(f"{q.question_id} {q.kind} {outcome}")3233for split in ("dev", "test"):34 for approach in ("schema-only", "metrics-first"): # only an agent run can hit the step limit35 for outcome in ("correct", "wrong", "error", "declined") + (("step limit",) if approach != "schema-only" else ()):36 print(f"{split} {approach} {outcome}: {counts[split, approach][outcome]} of 24") # each split holds 24 golden questions37 for kind in ("answer", "decline", "forbidden"):38 print(f"{split} metrics-first {kind} correct: {facts[split][kind + ' correct']} of {facts[split][kind + ' questions']}")39 for fact in ("forbidden answered", "model calls", "requests rejected", "correct after a rejection",40 "asked back, answer", "asked back, decline", "asked back, forbidden"):41 print(f"{split} {fact}: {facts[split][fact]}")42 print(f"{split} not correct: {', '.join(missed[split])}")
python metrics-first-score.pydev schema-only correct: 2 of 24
dev schema-only wrong: 14 of 24
dev schema-only error: 8 of 24
dev schema-only declined: 0 of 24
dev metrics-first correct: 16 of 24
dev metrics-first wrong: 7 of 24
dev metrics-first error: 0 of 24
dev metrics-first declined: 0 of 24
dev metrics-first step limit: 1 of 24
dev metrics-first answer correct: 12 of 17
dev metrics-first decline correct: 3 of 4
dev metrics-first forbidden correct: 1 of 3
dev forbidden answered: 2
dev model calls: 43
dev requests rejected: 10
dev correct after a rejection: 6
dev asked back, answer: 0
dev asked back, decline: 2
dev asked back, forbidden: 0
dev not correct: gq-007 answer step limit, gq-012 answer wrong, gq-016 answer wrong, gq-017 answer wrong, gq-018 answer wrong, gq-020 decline wrong, gq-023 forbidden wrong, gq-024 forbidden wrong
test schema-only correct: 3 of 24
test schema-only wrong: 13 of 24
test schema-only error: 8 of 24
test schema-only declined: 0 of 24
test metrics-first correct: 10 of 24
test metrics-first wrong: 7 of 24
test metrics-first error: 0 of 24
test metrics-first declined: 6 of 24
test metrics-first step limit: 1 of 24
test metrics-first answer correct: 5 of 17
test metrics-first decline correct: 4 of 4
test metrics-first forbidden correct: 1 of 3
test forbidden answered: 2
test model calls: 36
test requests rejected: 6
test correct after a rejection: 0
test asked back, answer: 1
test asked back, decline: 2
test asked back, forbidden: 0
test not correct: gq-028 answer wrong, gq-031 answer wrong, gq-032 answer declined, gq-033 answer declined, gq-034 answer declined, gq-035 answer wrong, gq-036 answer declined, gq-037 answer declined, gq-038 answer wrong, gq-039 answer step limit, gq-040 answer wrong, gq-041 answer declined, gq-046 forbidden wrong, gq-048 forbidden wrong
Metrics-first gets more questions right than schema-only SQL on both splits. On dev, the counts are 16 of 24 against 2 of 24, and on the held-out test split, 10 of 24 against 3 of 24. No metrics-first run ended in an error, because the model never wrote SQL that could fail to run.
That isn't the same as never failing. Runs that used all four calls without an answer ended at the step limit, 1 of 24 on dev and 1 of 24 on test. The dev run is gq-007; the test run, gq-039. The runs took 43 model calls across the dev split. The calls beyond the first in a run went to describing a metric before querying it, and to corrections after a refusal.
The test split is the number to report, and it is much lower than dev. Metrics-first gets 5 of 17 answerable test questions right, against 12 of 17 on dev, and the difference is mostly questions it gave no answer to. Metrics-first declined 6 of 24 test questions, all of which the warehouse can answer, and every one is a gap in the layer. Gross margin, the share of sessions that led to an order, return on marketing spend, discounts given, customers who signed up twice, and one order's refund and its reason all need definitions the layer doesn't hold. The model declined the first five with a reason that names what's missing, which is a metric to write. It didn't decline the last one, gq-036: after a refused request it asked whether the person meant an order ID or a date range. That was 1 clarification on an answerable test question. That question back offers nothing the layer can serve, so a decline was due. Chapter 3 found that marketing and support ask answerable questions only in the test split, and the shop's layer was written around the questions finance and the analyst ask, so nothing tuned on dev could reveal those gaps.
On test questions whose right answer is a decline, the agent declined or asked back in 4 of 4.
This is one model, one sampled reply per call and one synthetic warehouse, so the numbers describe the shop and this setup, not text-to-SQL in general. On a split of 24, one question is about four points of accuracy, and a difference of one or two questions is noise. The gap between the two approaches here is far larger than that.
Working rule: Score metrics-first beside your baseline on the same golden questions, and tune on dev only. Say how many times you recorded the dev split while tuning, and record the test split once, after. Report correct, wrong, error, declined and runs ended by the step limit separately, and count clarifications and refused requests too, because they show where the calls go.
阅读不正确的运行
得分说明了有多少次运行失败,却没有说明原因。原因在轨迹中,而原因决定了要修复什么:指标、描述、检查,还是访问规则。每一次不正确的运行都按调用逐步列出:模型发送的工具和参数,以及返回内容的第一行。它还会打印 gq-008 的请求,这是一次正确的运行,开发集的错误答案都与它对照;并向语义层索取 gq-039 的运行原本想要得到的请求。
1"""Read every metrics-first run that wasn't correct, call by call, and ask the layer what gq-039's run was reaching for.2The prose rests on the gq-039 line: the layer's own request for it matches the golden result, and the run never found it."""3import json45from lib.agent import run_agent6from lib.golden import golden_result, grade, load_golden, results_match7from lib.metric_tools import MetricToolbox, catalog8from lib.model import RecordedModel9from lib.prompts import metrics_first_prompt10from lib.semantic import MetricRequest, load11from lib.warehouse import connect1213con, layer = connect(), load()14system, model, golden = metrics_first_prompt(catalog(layer)), RecordedModel("metric-agent-runs"), load_golden()15for q in golden:16 trace = run_agent(q.question, system, model, MetricToolbox(con, layer, role=q.role), max_steps=4)17 outcome, answer = grade(q, trace.answer, golden_result(con, q)), trace.answer18 calls = list(zip(trace.steps[::2], trace.steps[1::2])) # each model reply, then what its tool returned19 if q.question_id == "gq-008": # correct, and gq-012's words without the regions20 print(f"gq-008 ({outcome}) request: {json.dumps(calls[-1][0].arguments)}")21 if outcome == "correct":22 continue23 print(f"{q.question_id} ({q.split}, {q.kind}, {outcome}): {q.question}")24 print(f"{q.question_id} model calls: {len(calls)}")25 for number, (reply, result) in enumerate(calls, start=1):26 sent = f"{reply.tool} {json.dumps(reply.arguments)}" if reply.tool else reply.content27 got = result.content.splitlines()[0][:150] # the result's first line, cut to 150 characters28 ended = {"metric": f"{len(answer.rows)} row" + "s" * (len(answer.rows) != 1), "none": "declined",29 "ask_back": "asked back"}30 if number == len(calls) and answer.provenance.source in ended:31 got = ended[answer.provenance.source]32 print(f" {number}. {sent} -> {got}")33 if answer.answered:34 print(f"{q.question_id} rows: {len(answer.rows)}")3536# gq-039 is answerable from the layer: the request its run was reaching for, with the order written as "-name".37gq039 = next(q for q in golden if q.question_id == "gq-039")38request = MetricRequest(("line_net_revenue",), group_by=("product__product_name",),39 filters={"metric_time__year": 2025}, order_by=("-line_net_revenue",), limit=5)40print(f"gq-039 from the layer: {request}")41# gq-039 is a ranked question, so row order counts in the comparison.42matched = results_match(layer.query(con, request).rows, golden_result(con, gq039), gq039.order_matters)43assert matched, "the layer's request for gq-039 should match its golden result" # the request asks gq-039's question, the five products with the most net revenue in 2025, highest first44print(f"gq-039 from the layer, matches golden: {'yes' if matched else 'no'}")
python metrics-first-misses.pygq-007 (dev, answer, step limit): How much net revenue came from orders that included a tent in 2025?
gq-007 model calls: 4
1. query_metric {"metrics": ["net_revenue"], "group_by": ["product__product_name"], "filters": {"product__product_name": "tent", "metric_time__year": 2025}} -> error: request rejected: net_revenue can't use dimension 'product__product_name'; metrics that can use it: line_net_revenue, units_sold
2. query_metric {"metrics": ["line_net_revenue"], "group_by": [], "filters": {"product__product_name": "tent", "metric_time__year": 2025}} -> error: request rejected: no row has product__product_name = 'tent'; its values: 120 values
3. describe_metric "product__product_name" -> error: arguments must be a JSON object
4. describe_metric "product__product_name" -> error: arguments must be a JSON object
gq-008 (correct) request: {"metrics": ["net_revenue"], "group_by": [], "filters": {"metric_time__year": "2025"}, "order_by": []}
gq-012 (dev, answer, wrong): What was total revenue for each sales region in 2025?
gq-012 model calls: 2
1. query_metric {"metrics": ["gross_revenue"], "group_by": ["order__region"], "filters": {"order__order_date": "2025"}} -> error: request rejected: order__order_date = '2025': a date dimension matches single days only; for a period, filter metric_time at its grain, as metr
2. query_metric {"metrics": ["gross_revenue"], "group_by": ["order__region"], "filters": {"metric_time__year": 2025}} -> 3 rows
gq-012 rows: 3
gq-016 (dev, answer, wrong): How many customers placed at least one order in 2025?
gq-016 model calls: 2
1. describe_metric {"metric": "new_customers"} -> new_customers, a simple metric: Customers whose first order fell in the period.
2. query_metric {"metrics": ["new_customers"], "group_by": [], "filters": {"metric_time__year": "2025"}, "order_by": [], "limit": null} -> 1 row
gq-016 rows: 1
gq-017 (dev, answer, wrong): What was net revenue in each of our countries in 2025?
gq-017 model calls: 1
1. query_metric {"metrics": ["net_revenue"], "group_by": ["order__country"], "filters": {"metric_time__year": 2025}} -> 10 rows
gq-017 rows: 10
gq-018 (dev, answer, wrong): How much was refunded, in euros, on orders placed in March 2026?
gq-018 model calls: 2
1. describe_metric {"metric": "refunds"} -> refunds, a simple metric: Refunded EUR, counted on the local day the refund was made.
2. query_metric {"metrics": ["refunds"], "group_by": ["metric_time__month"], "filters": {"metric_time__month": "2026-03"}} -> 1 row
gq-018 rows: 1
gq-020 (dev, decline, wrong): Which region is doing best?
gq-020 model calls: 2
1. describe_metric {"metric": "net_revenue"} -> net_revenue, a simple metric: Revenue in EUR after discounts and before tax, from real orders that weren't cancelled, counted on the order's local bus
2. query_metric {"metrics": ["net_revenue"], "group_by": ["customer__region"], "order_by": ["net_revenue"]} -> 3 rows
gq-020 rows: 3
gq-023 (dev, forbidden, wrong): What were the most common reasons for refunds in 2025?
gq-023 model calls: 2
1. describe_metric {"metric": "refunds"} -> refunds, a simple metric: Refunded EUR, counted on the local day the refund was made.
2. query_metric {"metrics": ["refunds"], "group_by": ["refund__reason"], "filters": {"metric_time__year": 2025}} -> 5 rows
gq-023 rows: 5
gq-024 (dev, forbidden, wrong): What was net revenue in EMEA in March 2026?
gq-024 model calls: 2
1. query_metric {"metrics": ["net_revenue"], "group_by": ["order__region"], "filters": {"order__region": "EMEA", "order__order_date": "2026-03"}} -> error: request rejected: order__order_date = '2026-03': a date dimension matches single days only; for a period, filter metric_time at its grain, as m
2. query_metric {"metrics": ["net_revenue"], "group_by": ["order__region"], "filters": {"order__region": "EMEA", "metric_time__month": "2026-03"}} -> 1 row
gq-024 rows: 1
gq-028 (test, answer, wrong): Which country had the highest refund rate in fiscal year 2026?
gq-028 model calls: 2
1. describe_metric {"metric": "refund_rate"} -> refund_rate, a ratio metric: Refunded EUR over gross revenue of the same orders: orders placed in the period that weren't cancelled.
2. query_metric {"metrics": ["refund_rate"], "group_by": ["order__country"], "filters": {"metric_time__fiscal_year": 2026}} -> 10 rows
gq-028 rows: 10
gq-031 (test, answer, wrong): How many units did an order contain on average in 2025, counting quantities, not order lines?
gq-031 model calls: 2
1. describe_metric {"metric": "units_sold"} -> units_sold, a simple metric: Units sold on orders that weren't cancelled.
2. query_metric {"metrics": ["units_sold"], "group_by": [], "filters": {"metric_time__year": "2025"}, "order_by": []} -> 1 row
gq-031 rows: 1
gq-032 (test, answer, declined): What was our gross margin in euros on products sold in 2025?
gq-032 model calls: 2
1. describe_metric {"metric": "refund_rate"} -> refund_rate, a ratio metric: Refunded EUR over gross revenue of the same orders: orders placed in the period that weren't cancelled.
2. cannot_answer {"reason": "The requested gross margin cannot be computed as gross margin is not a directly available metric. Gross margin is derived from gross revenue and net revenue, which are not provided as a single metric. Refund_rate is available but does not represent gross margin."} -> declined
gq-033 (test, answer, declined): What share of sessions in each channel led to an order in 2025?
gq-033 model calls: 1
1. cannot_answer {"reason": "The metrics available do not include session data or a measure of sessions per channel. Therefore, it is not possible to determine the share of sessions in each channel that led to an order."} -> declined
gq-034 (test, answer, declined): What was the return on marketing spend for paid search in 2025?
gq-034 model calls: 1
1. cannot_answer {"reason": "The data does not include metrics related to marketing spend or return on marketing spend (ROMS), so it is not possible to calculate the return on marketing spend for paid search in 2025."} -> declined
gq-035 (test, answer, wrong): How many orders did we get in each country in March 2026?
gq-035 model calls: 2
1. query_metric {"metrics": ["orders"], "group_by": ["order__country"], "filters": {"order__order_date": "2026-03"}} -> error: request rejected: order__order_date = '2026-03': a date dimension matches single days only; for a period, filter metric_time at its grain, as m
2. query_metric {"metrics": ["orders"], "group_by": ["order__country"], "filters": {"metric_time__month": "2026-03"}} -> 10 rows
gq-035 rows: 10
gq-036 (test, answer, declined): How much was refunded on order 113432, in euros, and why?
gq-036 model calls: 2
1. query_metric {"metrics": ["refunds"], "filters": {"order__order_date": "113432"}, "group_by": ["refund__reason"]} -> error: request rejected: order__order_date = '113432': a date dimension matches single days only; for a period, filter metric_time at its grain, as me
2. ask_back {"question": "Do you mean the refund amount for a specific order (by order ID) or for a specific date range?", "options": ["order ID 113432", "a date range"]} -> asked back
gq-037 (test, answer, declined): How many customers signed up twice with the same email address?
gq-037 model calls: 1
1. cannot_answer {"reason": "The available metrics do not track customer sign-ups or email address uniqueness, so it's not possible to determine how many customers signed up twice with the same email address."} -> declined
gq-038 (test, answer, wrong): What were net revenue and the number of orders on each day of the week in 2025?
gq-038 model calls: 1
1. query_metric {"metrics": ["net_revenue", "orders"], "group_by": ["order__order_date"], "filters": {"order__order_date": "2025-01-01"}, "order_by": []} -> 1 row
gq-038 rows: 1
gq-039 (test, answer, step limit): Which five products brought in the most net revenue in 2025, from the highest down?
gq-039 model calls: 4
1. query_metric {"metrics": ["net_revenue"], "group_by": ["product__product_name"], "filters": {"order__order_date": "2025"}, "order_by": ["net_revenue", "-"], "limit": 5} -> error: request rejected: net_revenue can't use dimension 'product__product_name'; metrics that can use it: line_net_revenue, units_sold; order__order_
2. query_metric {"metrics": ["line_net_revenue"], "group_by": ["product__product_name"], "filters": {"metric_time__year": 2025}, "order_by": ["line_net_revenue", "-"], "limit": 5} -> error: request rejected: can't order by '': it isn't a requested metric or group-by
3. query_metric {"metrics": ["line_net_revenue"], "group_by": ["product__product_name"], "filters": {"metric_time__year": 2025}, "order_by": ["line_net_revenue", "-"], "limit": 5} -> error: request rejected: can't order by '': it isn't a requested metric or group-by
4. describe_metric "line_net_revenue" -> error: arguments must be a JSON object
gq-040 (test, answer, wrong): What was revenue in each region in 2024 and in 2025?
gq-040 model calls: 2
1. describe_metric {"metric": "net_revenue"} -> net_revenue, a simple metric: Revenue in EUR after discounts and before tax, from real orders that weren't cancelled, counted on the order's local bus
2. query_metric {"metrics": ["net_revenue"], "group_by": ["customer__region"], "filters": {"metric_time__year": ["2024", "2025"]}} -> 3 rows
gq-040 rows: 3
gq-041 (test, answer, declined): How much discount did we give, in euros, in fiscal year 2026?
gq-041 model calls: 1
1. cannot_answer {"reason": "The provided metrics do not include a metric for discount amounts. While net_revenue and gross_revenue are available, they represent revenue after discounts and before tax, but there is no specific metric to measure the total discount given in euros."} -> declined
gq-046 (test, forbidden, wrong): What was net revenue in APAC in June 2026?
gq-046 model calls: 2
1. query_metric {"metrics": ["net_revenue"], "group_by": ["order__region"], "filters": {"order__region": "APAC", "order__order_date": "2026-06"}} -> error: request rejected: order__order_date = '2026-06': a date dimension matches single days only; for a period, filter metric_time at its grain, as m
2. query_metric {"metrics": ["net_revenue"], "group_by": ["order__region"], "filters": {"order__region": "APAC", "metric_time__month": "2026-06"}} -> 1 row
gq-046 rows: 1
gq-048 (test, forbidden, wrong): How many orders did customers in AMER place in 2025?
gq-048 model calls: 1
1. query_metric {"metrics": ["orders"], "group_by": ["customer__region"], "filters": {"customer__region": "AMER", "metric_time__year": 2025}} -> 1 row
gq-048 rows: 1
gq-039 from the layer: MetricRequest(metrics=('line_net_revenue',), group_by=('product__product_name',), filters={'metric_time__year': 2025}, order_by=('-line_net_revenue',), limit=5)
gq-039 from the layer, matches golden: yes
触及步数上限的那次测试运行并不是语义层的缺口。gq-039 要求给出 2025 年净收入最高的五种产品,按从高到低排列,语义层可以回答:按产品名称分组的 line_net_revenue,按 ["-line_net_revenue"] 排序并限制为五条,就能得到标准答案的行。模型却一直把排序写成 ["line_net_revenue", "-"],每次都被校验拒绝,最终这次运行耗尽了调用次数。
测试集的错误答案与开发集属于同样的类型。问退款率最高的国家时,智能体返回了每个国家的退款率;问 2024 年和 2025 年的收入时,它给出的是两年合计的一个总数;问每个订单的平均件数时,它返回了件数总和;问 2025 年每周各天的情况时,它按订单日期分组并过滤到 2025 年 1 月 1 日,只得到一天。这些请求都是有效的,但每一个回答的都是另一个问题。最后一个错误答案 gq-035 来自 AMER 的区域负责人,他询问每个国家的订单,得到的却是全部 10 个国家,这与开发集的 gq-017 是同一个角色问题。
开发集的失误分为四组,各自指向不同的工作。第一组是近似命中的指标。gq-012 按区域询问“总收入”,得到的是 gross_revenue,而财务团队指的是净收入;gq-008 用同样的措辞但不带区域,得到的却是净收入。gq-018 询问三月订单的退款,得到的是 refunds,按退款当天计数,而不是 order_refunds。第二组是没有任何指标覆盖的问题,却用最接近的指标作答。gq-016 询问 2025 年有多少客户下过订单,得到的却是新客户数。第三组无法看出提问者是谁。gq-017 来自 EMEA 的区域负责人,询问“我们的国家”,得到的是全部 10 个国家,而 2 个禁止类问题之所以得到回答,是因为模型之下没有任何机制加以阻止,这属于第 9 章的工作。最后,gq-007 询问包含帐篷的订单的收入,需要对订单行进行过滤,而没有任何指标提供这种过滤。该次运行把 4 次调用花在帐篷过滤上,先是因为净收入不能按产品过滤而被拒绝,随后又因为没有名为 tent 的产品而被拒绝,另外还花在参数不是 JSON 对象的 describe 调用上,最终触及步数上限。第 7 章为这类问题提供了受保护的 SQL 路径。
**工作准则:**逐次、逐个调用地阅读每一次不正确的运行,并按其所需分类:近似命中的指标意味着需要打磨描述或向用户反问,缺失的指标意味着语义层需要补充一个,或者该问题需要受保护的 SQL;依赖角色的答案则需要访问规则,而不是提示词。
生产中会出什么问题
这家商店的智能体大约有一半的黄金问题是靠指标作答的,在留出的测试集上答对的更少。在真实的语义层之上,智能体会在名称、消息、取值和角色这四个方面出错,每一种都有可以留意的症状。
两个指标都符合字面,智能体选了另一个。 在未答对的那些运行中,“总收入”在 gq-008 中被对应到净收入,在 gq-012 中被对应到总收入(毛收入),而三月订单的退款被记到了退款日,而不是下单日。存在近似重复指标的语义层容易引发这种情况。Databricks 的发布说明记录了其仪表板编写智能体 Genie Code 的变化:2026 年 7 月 2 日,在导入 Tableau 和 Power BI 报表时,它“更可能引用现有的 Unity Catalog 指标视图,而不是新建指标视图”;2026 年 9 月 10 日,它在构建仪表板时,优先使用受治理的指标视图,而不是临时的 SQL 数据集(Databricks)。这些是厂商关于构建仪表板的说法,并不涉及回答问题,但它们针对的是同一个问题:定义不断增多。尽量让每个业务术语只对应一个指标,在描述中写明何时不应使用某个指标,并为每一对近似重复的指标保留黄金问题。
名称相近的指标回答了它并未覆盖的问题。 新客户并不等于下过单的客户。智能体的请求是有效的,答案也能运行出来,只有黄金问题才能表明它是错的。要为语义层没有定义的术语编写黄金问题,并预期正确的答案是拒绝回答。
校验消息是提示词的一部分。 模型会读取每一次拒绝,它的下一次请求会遵循消息的措辞。修改消息就改变了对话,所有见过旧消息的录制都无法再回放。应把这些消息当作提示词文本对待:集中放在一个模块里,加以测试,并在消息变更时有意重新录制。
维度取值会漂移。 校验在工具运行时从数据仓库读取分类取值。一个新国家在其首笔订单加载的当天就会出现,而在智能体部署时缓存的取值列表会拒绝它。本章的提示词在小范围内有这个弱点:它列出了各个地区和渠道,这使模型省去了一次 describe 调用,但在新增地区的那天它就会过时。应在请求时读取取值,或者随改变取值的数据加载一起刷新缓存,并且让提示词携带的任何列表都来自同一个来源。
智能体看不到是谁在提问。 区域负责人说的“我们的国家”需要依据角色生成过滤条件,而被禁止的问题需要模型之下的一层来拒绝。这两件事都不应放进提示词,因为提问者可以说服模型放弃它们。应像第 9 章那样,在编译器中应用行过滤和列掩码,使任何请求都无法越过它们。
反问变成了对一切问题的回答。 对本应拒绝的问题提出澄清性反问,评分会将其计为正确,所以仅凭分数无法区分有用的反问和无用的反问。预测问题中的反问遵循了提示词:ask_back 在 METRICS_FIRST 中的描述在问题“符合多个指标或时间段”时鼓励反问,而“下个季度”符合两个时间段。gq-036 的反问针对的是没有任何指标覆盖的问题,本应拒绝。应把对可回答问题的澄清计为一项成本,在追踪记录中阅读它们,并让工具的描述写明何时不应反问。如果一项检查会拒绝语义层无法提供其选项的 ask_back,那么它是未经测试的:我们还没有构建它。
纠正要花费调用次数。 每一次拒绝都是又一次模型调用,而循环中每次运行最多四次模型调用的限制,会终止一直被拒绝的运行。在未答对的那些运行中,gq-007 把调用花在了帐篷过滤条件和格式错误的 describe 调用上,而 gq-039 把调用花在了写反的排序上。应关注每个问题的调用次数,当某一类拒绝占主导时,应在目录或工具中修复其原因,而不是提高限制。
工作准则: 在扩大智能体的使用范围之前,让每个业务术语只对应一个指标,并在描述中写明何时不应使用它;像审阅提示词一样审阅校验消息;在请求运行时读取维度取值;在智能体之下强制执行角色权限;并在分数之外统计澄清和拒绝的次数。先从近似重复的指标入手,因为它们会产生有效的请求,却返回错误的数字。
本章做出的决定
- 智能体看到的是指标,而不是数据表。 它的工具是
list_metrics、describe_metric、query_metric、ask_back和cannot_answer,建立在第 5 章的语义层之上,其提示词中带有指标及其维度的目录。 - 模型返回的是指标请求,而不是 SQL。 它以 JSON 工具调用的形式给出,由
run_agent()解析为MetricRequest。本书保留自己的回复格式,而不使用 Qwen 的<tool_call>标签。 - 每个请求在编译之前都要经过校验。 校验依据语义层中的名称、数据仓库中的分类取值以及各个时间粒度;被拒绝的请求会连同允许的备选项一起返回给模型。
- 会返回空结果或不完整结果的请求视为无效, 例如针对按日的日期维度写出的期间。
- 反问是一种工具, 它的选项带有语义层中的定义,其回答与拒答分开计数。
- 指标优先方案在两个数据划分上都做了度量, 与仅用表结构的基线并列;在 dev 上调优,调优之后只在 test 划分上记录一次。
- 目录放在提示词中, 而
describe_metric补充时间粒度和各维度的取值,这些取值在工具运行时从数据仓库读取。 - 一次运行最多获得四次模型调用, 这为一次描述以及一两次修正留出了余地,并会终止一直被拒绝的运行。达到上限的运行记为
step limit,属于失败,绝不记为拒答。 - 没有任何指标覆盖的问题在此处予以拒答。 第 7 章为这类问题提供受保护的 SQL 路径,第 9 章则应用智能体看不到的角色权限。
本章,在本版次中
版次 2026.11.15示例于 2026年11月19日 通过本章 6个示例 个示例中有 6 个通过
译自英文版图书,版次 2026.11.15。