優先將問題導向受治理的指標

閱讀約需 85 分鐘 分鐘,8 節 中共有 6 個範例。

版次 2026.11.15 範例於 2026年11 月 19 通過測試 78 個範例中有 78 個通過 變更內容

由 AI 翻譯。經人工以英文驗證。 我們如何製作書籍

譯自英文書,版次 2026.11.15。 閱讀原文

讀完本章後,你將能夠:把語意層的指標而非資料倉儲的資料表,作為工具提供給代理;讓代理回傳結構化的指標請求,並在執行任何查詢之前先加以驗證;當一個問題可能對應多個指標時,讓代理向使用者反問;並在黃金問題上衡量這些改變帶來的差異。

第 5 章把商店的各項定義一次寫定,成為一個語意層:在五個語意模型之上定義十個指標,並由本書自己的編譯器在 lib/semantic.py 中編譯成 SQL。熟悉這個語意層的人,可以向它查詢 2026 年 3 月各區域的淨營收,並得到財務團隊認可的數字,但代理目前還做不到。第 1 章只給了它一個指標,而它必須超越的基準是僅依據結構描述(schema)產生的 SQL,這個做法在開發集問題上答對 2 of 24。第 4 章把各項定義以文字形式與結構描述一併提供給模型,但 SQL 仍由模型撰寫,因此每一項定義都必須經得起模型的解讀。

由模型撰寫查詢時,任何被誤讀的定義最終都會反映在數字上。優先將問題導向受治理的指標,就是不再讓模型撰寫查詢:它的工作縮減為選擇哪個指標、依什麼分組、如何篩選,而 SQL 則由人們負責維護並測試的語意層來產生。資料倉儲廠商也以相同方式描述自家的助理。Databricks 針對其問答式 Genie Agents 的指引指出,指標檢視(metric view)特別適合它們,因為這類檢視預先定義了指標、維度與聚合方式(Databricks);Snowflake 則建議將語意檢視(semantic view)用於 Cortex Analyst(Snowflake)。如同第 4 章中的廠商建議,這些都只是建議,而非實測結果。本章在商店的黃金問題上衡量這個模式,使用的是與其他各章相同的已錄製模型。

此部分尚未翻譯,因此以英文呈現。

Give the agent metrics, not tables

最近一次變動於第 2026.11.12 版

An agent's tools decide what it can get wrong. Given tables, a model can join them the wrong way, miss a filter or pick the legacy mart, as chapter 2 showed one failure at a time. Given metrics, it can still pick the wrong metric or the wrong dimension, but it can't rewrite a definition, because definitions aren't among the things it can touch. A metric tool exposes what a semantic layer holds in the shape a person asks for it: metric names with their descriptions, the dimensions each can be grouped and filtered by, and the time grains. Databricks describes its Unity Catalog metric views in the same terms, as separating "measure definitions from the fields (also called dimensions) used to group, filter, and aggregate them" (Databricks).

The shop's agent gets its metric tools from lib/metric_tools.py, a module this chapter adds. Its MetricToolbox wraps one warehouse connection and the semantic layer from chapter 5, and offers five tools: list_metrics, describe_metric, query_metric, ask_back and cannot_answer. The prompt, METRICS_FIRST in lib/prompts.py, carries what list_metrics returns, so the model doesn't spend a call asking for it: one line per metric, with its description and the dimensions it can use. The prompt also says how the shop's fiscal year runs and how to write a period as a filter value, and it names the regions and sales channels. It lists no tables and no columns.

Look first at what the agent sees. The example builds the prompt and prints net revenue's line of the catalog, then calls describe_metric for the refund rate, and sizes one golden question, gq-002, net revenue in each region in March 2026, under the schema-only prompt from chapter 1 and under this one. Both sizes are the token counts the model's runtime reported when the calls were recorded.

metric-tools.py Python 3.12.15 · duckdb 2.0.0 · pyyaml 6.0.3 · sqlglot 30.31.0 通過
"""Give the agent the semantic layer's metrics instead of the warehouse's tables, and compare what each prompt holds.The prose rests on the metric and table counts and the two prompt token counts, which come from the recordings."""from lib.golden import load_goldenfrom lib.metric_tools import MetricToolbox, catalogfrom lib.model import RecordedModelfrom lib.prompts import metrics_first_prompt, schema_only_messagesfrom lib.warehouse import connect, describe_schemacon = connect()toolbox = MetricToolbox(con)system = metrics_first_prompt(catalog(toolbox.layer))print(f"tools: {', '.join(toolbox.tools)}")print(f"metrics in the prompt: {len(toolbox.layer.metrics)}")print(f"tables in the prompt: {system.count('CREATE TABLE')}")       # a table in the prompt would be written as a CREATE TABLE statementprint(catalog(toolbox.layer).splitlines()[0])                          # the first line: net revenue, with its dimensionsprint(toolbox.call("describe_metric", {"metric": "refund_rate"}).content)# The first call of the same golden question, gq-002, under each prompt, as the recordings measured it.question = next(q for q in load_golden() if q.question_id == "gq-002").questionschema_only = schema_only_messages(describe_schema(con), question)metrics_first = [{"role": "system", "content": system}, {"role": "user", "content": question}]print(f"schema-only prompt tokens: {RecordedModel('schema-only-sql').call_for(schema_only)['prompt_tokens']}")print(f"metrics-first prompt tokens: {RecordedModel('metric-agent-runs').call_for(metrics_first)['prompt_tokens']}")
輸出python metric-tools.py
tools: list_metrics, describe_metric, query_metric, ask_back, cannot_answer
metrics in the prompt: 10
tables in the prompt: 0
- net_revenue: Revenue in EUR after discounts and before tax, from real orders that weren't cancelled, counted on the order's local business day. Returned orders count; refunds are not subtracted. Dimensions: customer__country, customer__first_order_date, customer__region, order__channel, order__country, order__order_date, order__region, order__status
refund_rate, a ratio metric: Refunded EUR over gross revenue of the same orders: orders placed in the period that weren't cancelled.
Dimensions:
- metric_time__day
- metric_time__week
- metric_time__month
- metric_time__quarter
- metric_time__year
- metric_time__fiscal_year: 2025, 2026, 2027
- metric_time__fiscal_quarter: 2025-Q1, 2025-Q2, 2025-Q3, 2025-Q4, 2026-Q1, 2026-Q2, 2026-Q3, 2026-Q4, 2027-Q1, 2027-Q2
- order__channel: (none), app, marketplace, web
- order__country: AU, CA, DE, ES, FR, GB, JP, NL, SG, US
- order__order_date
- order__region: AMER, APAC, EMEA
- order__status: cancelled, delivered, placed, returned, shipped
schema-only prompt tokens: 827
metrics-first prompt tokens: 1120

The prompt lists 10 metrics and 0 tables. Each metric's line is its owner's description, word for word from the YAML, followed by the dimensions it can reach without fan-out. Net revenue's line names order__region and customer__region, but not product__category, because the orders model reaches products only through the order lines, several per order.

describe_metric adds what the catalog leaves out: the time grains, and the values of each categorical dimension, read from the warehouse when the tool runs. The refund rate's regions are AMER, APAC and EMEA, and its fiscal quarters are labels such as 2026-Q1. The prompt already names the regions and channels, so for those the tool repeats what the model was told. For the countries, the order statuses and the fiscal periods, it's the only place the model sees how the shop writes them.

The metrics-first prompt isn't smaller. It is 1120 tokens against the schema-only prompt's 827, because a definition takes more words than a column name. What changed is what the tokens hold: rules a person wrote and checked, instead of names the model has to interpret.

Working rule: Give the agent the metrics, their descriptions and the dimensions each can use, and keep tables out of its tools until a question needs them. Put the catalog in the prompt while it fits, so a run doesn't spend a call listing metrics, and let a describe tool return dimension values from the warehouse. When the catalog outgrows the prompt, keep names and descriptions there and move the dimensions behind the describe tool.

此部分尚未翻譯,因此以英文呈現。

Ask for a request, not SQL

最近一次變動於第 2026.11.12 版

With metric tools in place, the model's reply is no longer a query but a request. A structured reply is one the code can parse into a typed object and check field by field. Here it is a JSON tool call whose arguments become a MetricRequest, the metric request from chapter 5, which names metrics, group-by dimensions, filters, an order and a limit. Nothing in it is SQL, so the model can't add anything to a query that the layer didn't define.

Qwen's chat template for the model this book records with asks for each tool call as JSON inside <tool_call> tags, and returns tool results inside <tool_response> tags (Qwen's template). Qwen's model card recommends its Qwen-Agent library, which "encapsulates tool-calling templates and tool-calling parsers internally" (model card). We keep the book's own format instead: the prompt asks for one bare JSON object, {"tool": ..., "arguments": ...}, which run_agent() in lib/agent.py parses, as it has since chapter 1. A small parser we own is easier to test than a library's, and no library release can change it.

The shop's case is golden question gq-003, net revenue by sales channel in calendar 2025. Schema-only SQL failed it in chapter 1 by filtering fct_orders on a calendar_year column that exists only in dim_date. The recording metric-agent-runs holds the same model's replies to the metrics-first prompt for all 48 golden questions. It was made by recordings/record_metrics_first.py with the pinned model file, Qwen's recommended sampling and a seed reset before every call, like every recording in this book. gq-003's run, replayed, shows the reply, the request and what the layer did with it.

model-returns-a-request.py Python 3.12.15 · duckdb 2.0.0 · pyyaml 6.0.3 · sqlglot 30.31.0 通過
"""The model returns a metric request, not SQL, and the layer compiles that request. The prose rests on the `matches golden`line, which the example asserts; the schema-only outcome shows the baseline's failure on the same question."""from lib.agent import run_agentfrom lib.golden import golden_result, load_golden, results_match, schema_only_outcomefrom lib.metric_tools import MetricToolbox, catalogfrom lib.model import RecordedModelfrom lib.prompts import metrics_first_promptfrom lib.semantic import MetricRequestfrom lib.warehouse import connect, describe_schemacon = connect()toolbox = MetricToolbox(con)system = metrics_first_prompt(catalog(toolbox.layer))question = next(q for q in load_golden() if q.question_id == "gq-003")model = RecordedModel("metric-agent-runs")trace = run_agent(question.question, system, model, toolbox, max_steps=4)     # the chapter's limit of four model calls per runprint(f"question: {question.question}")print(f"model calls: {sum(step.kind == 'model' for step in trace.steps)}")print(f"reply: {trace.steps[0].content}")                                   # step 0 is the model's first reply, before any tool runsprint(MetricRequest.from_arguments(trace.steps[0].arguments))               # the reply's arguments, parsed into the typed requestprint(trace.answer.provenance.sql)                                          # SQL the layer compiled from the request, not SQL the model wroteprint(trace.answer.show())matches = results_match(trace.answer.rows, golden_result(con, question))assert matches, "the layer's answer to gq-003 should match its golden result"   # the request asks gq-003's own question, net revenue by channel for 2025, so its rows are the golden onesprint(f"matches golden: {'yes' if matches else 'no'}")schema = describe_schema(con)print(f"schema-only outcome: {schema_only_outcome(con, RecordedModel('schema-only-sql'), schema, question)}")
輸出python model-returns-a-request.py
question: How much net revenue did each sales channel bring in during calendar year 2025?
model calls: 1
reply: {"tool": "query_metric", "arguments": {"metrics": ["net_revenue"], "group_by": ["order__channel"], "filters": {"metric_time__year": 2025}}}
MetricRequest(metrics=('net_revenue',), group_by=('order__channel',), filters={'metric_time__year': 2025}, order_by=(), limit=None)
SELECT
  orders.channel AS order__channel,
  SUM(orders.net_revenue_eur) AS net_revenue
FROM analytics.fct_orders AS orders
WHERE
  orders.status <> 'cancelled'
  AND CAST(DATE_TRUNC('YEAR', CAST(orders.order_date_local AS DATE)) AS DATE) = CAST('2025-01-01' AS DATE)
GROUP BY
  orders.channel
ORDER BY
  order__channel ASC
order__channel  net_revenue
(none)          82,922.75
app             431,258.70
marketplace     227,826.99
web             909,519.42
matches golden: yes
schema-only outcome: error

The run took 1 model call. The reply names net_revenue, groups by order__channel and filters metric_time__year to 2025: the question's three parts, each mapped to a name the layer defines. The layer wrote the SQL. It reads analytics.fct_orders, leaves out cancelled orders by the metric's own filter, truncates the local business day to its year and groups by channel, and its channel totals agree with the golden ones. The same question under schema-only SQL ends in error.

The request also makes the answer's provenance exact. answer.provenance names the metric, the filters and the SQL that ran, so a person who doubts the number checks a short request and the metric's definition, not a query the model wrote. Reviewing a request takes a glance; reviewing a model's SQL takes the same care as reviewing anyone's.

Working rule: Have the model return a metric request as structured arguments to a tool, and parse it into a typed request before anything else happens. Keep the reply format simple enough for your own code to parse, and keep that parser in one place. Accept SQL from a model only on a separate, guarded path, for questions no metric covers.

此部分尚未翻譯,因此以英文呈現。

Validate the request against the layer

最近一次變動於第 2026.11.12 版

A parsed request is well-formed, which isn't the same as right. Some mistakes fail loudly: the compiler from chapter 5 refuses a metric it doesn't define, and a dimension that would fan out. Others compile and run. A filter value that no row holds, such as a region called Europe, returns an empty total. A month written against a dimension that holds days matches one day. Both results look like answers. Validating a request means checking it against the layer's definitions and the warehouse's values before it compiles, and sending every problem back to the model with what would be allowed instead.

validate_request() in lib/metric_tools.py does that for the shop. It checks each metric's name, and suggests the nearest defined names for a misspelt one. It checks each dimension against what the requested metrics can reach, and when one can't, it names the metrics that can. It checks each categorical filter value against the values in the warehouse, each filter on metric_time against its grain, so that a month must be the first day of a month, and each filter on a date dimension, which must be a single day. query_metric runs it on every request. A request with problems comes back to the model as an error that lists every problem it finds once the metrics are known, so one correction can fix them all. An unknown metric comes back on its own, before anything else is checked.

A person might type three requests the layer refuses: orders by customer email, which no semantic model defines; net revenue by product category, which would fan out; and net revenue for a region called Europe. The model's own first request for gq-002 comes next: the code runs it unvalidated to show what would have happened, then follows its corrected request to its answer.

reject-unknown-dimension.py Python 3.12.15 · duckdb 2.0.0 · pyyaml 6.0.3 · sqlglot 30.31.0 通過
"""Validate a metric request against the layer and the warehouse before it runs, and show what would run without it.The prose rests on the unchecked total and the two `matches golden` lines; the example asserts the two matches."""from lib.agent import run_agentfrom lib.golden import golden_result, load_golden, results_matchfrom lib.metric_tools import MetricToolbox, catalog, validate_requestfrom lib.model import RecordedModelfrom lib.prompts import metrics_first_promptfrom lib.semantic import MetricRequestfrom lib.warehouse import connectcon = connect()toolbox = MetricToolbox(con)layer = toolbox.layerfor label, request in [("no such dimension", MetricRequest(("orders",), group_by=("customer__email",))),                       ("fan-out", MetricRequest(("net_revenue",), group_by=("product__category",))),                       ("no such value", MetricRequest(("net_revenue",), filters={"order__region": "Europe"}))]:    print(f"{label}: {validate_request(layer, con, request)[0]}")print(f"unchecked, region Europe: {layer.query(con, request).rows}")           # `request` is still the Europe filter from the loop# The model's own requests for gq-002, net revenue in each region in March 2026, replayed from the recording.question = next(q for q in load_golden() if q.question_id == "gq-002")system = metrics_first_prompt(catalog(layer))trace = run_agent(question.question, system, RecordedModel("metric-agent-runs"), toolbox, max_steps=4)first, second = (MetricRequest.from_arguments(s.arguments) for s in trace.steps if s.tool == "query_metric")   # its two query callsgolden = golden_result(con, question)unchecked = layer.query(con, first)                # what would have run without validationprint(f"first request filters: {first.filters}")print(f"unchecked total: {sum(row[1] for row in unchecked.rows):,.2f}")           # row[1] is the net revenue columnprint(f"golden total: {sum(row[1] for row in golden):,.2f}")unchecked_matches = results_match(unchecked.rows, golden)assert not unchecked_matches, "the unchecked request should miss the golden result"   # a month written against a daily date dimension matches only its first day, so this total can't cover the monthprint(f"unchecked matches golden: {'yes' if unchecked_matches else 'no'}")print(f"validation said: {trace.steps[1].content}")                               # step 1 is the refusal the layer returnedprint(f"second request filters: {second.filters}")matches = results_match(trace.answer.rows, golden)assert matches, "the second request should match the golden result"              # filtering metric_time at the month's grain covers every day in March, the golden question's periodprint(f"matches golden: {'yes' if matches else 'no'}")
輸出python reject-unknown-dimension.py
no such dimension: orders can't use dimension 'customer__email'; no metric defines it
fan-out: net_revenue can't use dimension 'product__category'; metrics that can use it: line_net_revenue, units_sold
no such value: no row has order__region = 'Europe'; its values: AMER, APAC, EMEA
unchecked, region Europe: ((None,),)
first request filters: {'order__order_date': '2026-03'}
unchecked total: 4,150.82
golden total: 150,082.83
unchecked matches golden: no
validation said: error: request rejected: order__order_date = '2026-03': a date dimension matches single days only; for a period, filter metric_time at its grain, as metric_time__month = "2026-03" or metric_time__year = 2025
second request filters: {'metric_time__month': '2026-03'}
matches golden: yes

Each of the three requests is refused with a message that says what to do instead: no metric defines a customer's email, two other metrics can reach product category, and the regions are AMER, APAC and EMEA. Without the check, the request for Europe runs and returns one row with an empty total, which an agent would report as no revenue.

The model's own request is the case that matters, because no person wrote it. For March 2026 it filtered order__order_date, the orders' daily date dimension, to "2026-03". The compiler reads a month written against a daily dimension as the month's first day, so the unchecked request returns March 1 alone: 4,150.82 euros across the three regions, against a golden total of 150,082.83. Nothing in that answer says it covers one day. Validation refused the filter and named the dimension to use, metric_time__month. The model's second request used it, and its answer matches the golden result.

Across the dev split, validation refused 10 requests, and 6 runs went on to a correct answer after a refusal. Most of those refusals were the date filter. The prompt says a month is written as "2026-03" and lists metric_time__month among the grains, though it never shows the two together. The model reached for order__order_date anyway, probably because the catalog lists it beside the region. A check in code catches that mistake every time; a sentence in a prompt catches it only when the model follows the sentence.

Working rule: Validate every request before it compiles: names against the layer, categorical values against the warehouse, dates against their grain. Refuse with a message that names the allowed alternatives, and send it back to the model, which can usually correct itself in one call. Treat any request that would compile into an empty or partial result as invalid, not as an answer of zero.

問題含糊時要反問

最近一次變動於第 2026.11.12 版

驗證能攔下語意層無法處理的請求(requests),卻攔不下一個有效但指向錯誤對象的請求。「哪個地區表現最好?」可以對應到依地區統計淨營收的有效請求,同樣也可以對應到依地區統計訂單數或退款率的請求,而且每一種都可能指出不同的地區。當一個問題符合多個指標,而選擇會改變答案時,正確的回應是反問,並把各個選項說清楚。商店的黃金集(golden set)把這類問題標記為 ambiguous 類型的拒答,而第 3 章會把對這類問題未回傳結果的任何回覆都計為正確。

ask_back 工具會以一個向使用者提出的問題,以及模型提供的選項來結束這次執行。若選項指名某個指標,就會連同語意層對該指標的說明一併呈現,讓使用者在各種定義之間做選擇,而不是在字面之間做選擇:營收會變成「扣除折扣後、未計稅、以歐元計價、且不含已取消訂單的營收」。這個回答在出處紀錄(provenance)中以 ask_back 作為來源,因此評分時可以把澄清與拒答分開計算。

開發集中有四個問題,應該以反問或說明理由作結:其中兩個含糊,一是 gq-019,詢問最佳客戶;二是 gq-020,詢問哪個地區表現最好;另外兩個無法回答,一是預測,一是運費成本。四次執行的最後一個工具都會印出來。接著程式碼會印出代理人針對 gq-019 與 gq-005 所提出的問題、它對 gq-020 給出的答案,以及一項已記錄的執行所沒有的檢查。

clarify-best-customers.py Python 3.12.15 · duckdb 2.0.0 · pyyaml 6.0.3 · sqlglot 30.31.0 通過
"""Ask back when a question could mean several metrics: the layer supplies each option's definition to the person.The prose rests on the tool each dev decline ended with and on the ranking check's count of flagged runs."""from lib.agent import run_agentfrom lib.golden import load_goldenfrom lib.metric_tools import MetricToolbox, catalog, unbounded_rankingfrom lib.model import RecordedModelfrom lib.prompts import metrics_first_promptfrom lib.semantic import MetricRequestfrom lib.warehouse import connectcon = connect()toolbox = MetricToolbox(con)system = metrics_first_prompt(catalog(toolbox.layer))model = RecordedModel("metric-agent-runs")traces = {q.question_id: run_agent(q.question, system, model, MetricToolbox(con, toolbox.layer, role=q.role),                                   max_steps=4) for q in load_golden()}for q in load_golden():    if q.split == "dev" and q.kind == "decline":        last = next(s for s in reversed(traces[q.question_id].steps) if s.kind == "model")     # the run's final model reply        print(f"{q.question_id} ({q.decline_type}): {last.tool}")        print(f"  {q.question}")print("gq-019 asks back:")print(traces["gq-019"].answer.reason)print("gq-005 asks back:")print(traces["gq-005"].answer.reason)answered = next(s for s in reversed(traces["gq-020"].steps) if s.tool == "query_metric")   # its last query, the one that ranprint(f"gq-020 request: {answered.arguments}")print(traces["gq-020"].answer.show())# A check the recorded runs didn't have: refuse a request that ranks its result over no period.# Applied here to the last request of every run, dev and test, that returned a result.final = {question_id: MetricRequest.from_arguments(next(s for s in reversed(t.steps) if s.tool == "query_metric")                                                   .arguments)         for question_id, t in traces.items() if t.answer.answered}flagged = [question_id for question_id, request in final.items() if unbounded_ranking(request)]print(f"runs with a result: {len(final)}")print(f"requests that order their result: {sum(bool(r.order_by) for r in final.values())}")   # an empty order_by means no rankingprint(f"ranking check flags: {', '.join(flagged) or 'none'}")print(f"ranking check on gq-020: {unbounded_ranking(final['gq-020'])}")
輸出python clarify-best-customers.py
gq-005 (unanswerable): ask_back
  What will our net revenue be next quarter?
gq-019 (ambiguous): ask_back
  Who are our best customers?
gq-020 (ambiguous): query_metric
  Which region is doing best?
gq-021 (unanswerable): cannot_answer
  How much did shipping cost us in fiscal year 2026?
gq-019 asks back:
Do you mean customers by highest net revenue, or by highest number of orders?
- net_revenue: Revenue in EUR after discounts and before tax, from real orders that weren't cancelled, counted on the order's local business day. Returned orders count; refunds are not subtracted.
- orders: The number of real orders, cancelled ones included.
gq-005 asks back:
Do you mean net revenue for the next calendar quarter or the next fiscal quarter?
- calendar quarter
- fiscal quarter
gq-020 request: {'metrics': ['net_revenue'], 'group_by': ['customer__region'], 'order_by': ['net_revenue']}
customer__region  net_revenue
APAC              925,657.84
AMER              1,174,510.02
EMEA              1,690,339.73
runs with a result: 31
requests that order their result: 1
ranking check flags: gq-020
ranking check on gq-020: the request ranks its result but names no period; filter or group by a metric_time grain

在 gq-019 中,代理人詢問「最佳」是指淨營收最高還是訂單數最多,而語意層為每個指標補上了定義。這正是此工具存在的目的,也計為正確。不過,請對照語意層來讀這些選項。語意模型中沒有任何維度可以識別客戶,所以兩種選擇都無法導向語意層能處理的請求。誠實的回應是拒答:商店的指標不對客戶排名。

在 gq-020 中,代理人沒有反問。它請求依 customer__region 統計淨營收,且未指定期間,結果回傳每個地區的歷來營收,並由小到大排列。請求中的每個名稱都有效,所以驗證放行了它,而評分把它算作錯誤。至於預測問題 gq-005,代理人詢問「下一季」是指曆年季度還是會計季度,但這個問題的答案毫無幫助,因為沒有任何指標能做預測。它被計為正確,是因為沒有回傳任何數字,只有追蹤紀錄(trace)才顯示理由是錯的。運費問題 gq-021 則以正確的理由被拒答。

因此,只要模型呼叫這個工具,它就能發揮作用,但模型何時呼叫它並不可靠。程式碼中的檢查可以攔下 gq-020 的情況:拒絕對結果排序卻未指定期間的請求。unbounded_ranking() 位於 lib/metric_tools.py,就是這項檢查,範例把它套用到每一次已記錄執行(含開發集與測試集)中最後一個有回傳結果的請求。在這 31 個請求中,它標記了 gq-020 個。這只能說明它攔得到這個情況,不能說明它不會誤判,因為這些請求中只有 1 個會對結果排序。query_metric 目前還不執行這項檢查:這些錄製是在沒有它的情況下完成的,所以要加入它就得重新錄製。

gq-019 的情況需要一項我們尚未建立的檢查:拒絕指標選項無法依問題所問對象分組的 ask_back。這項檢查得知道問題問的是什麼,此處是客戶,而工具箱中沒有任何東西知道這一點。它也無法涵蓋 gq-005,因為該題的選項是期間,而不是指標。這兩項檢查都是針對商店問題的規則,應該放在程式碼中,讓測試可以把關,而不是加進更多提示文字。

實用原則: 給代理人一個反問的工具,選項要指名語意層的指標,並向使用者顯示每個選項的定義。只有在每個選項都能導向語意層可回答的請求時才反問;若沒有任何選項做得到,就改為拒答。閱讀追蹤紀錄中的澄清時,要像閱讀拒答一樣仔細,因為評分會把無用的提問算作正確。

此部分尚未翻譯,因此以英文呈現。

Measure metrics-first on the golden questions

最近一次變動於第 2026.11.15 版

So far the book has followed single runs. The measurement that counts is the one chapter 3 set up: every golden question, scored by grade() with results_match(), tuned on the dev split and checked once on the held-out test split, beside the schema-only baseline. We recorded the dev split three times while tuning, and two changes came from those runs. The catalog in the prompt stopped printing each metric's type beside its name, because the model copied the type into its requests; describe_metric still prints it. And a year written as text, such as "2025", became the number it can only mean. Only the final dev recording is kept, and the recording's manifest says the two earlier ones were discarded. The dev numbers below were measured on the split the changes were chosen on, so they're optimistic. The test split was recorded once, after those changes, and nothing changed afterwards.

Each run gets at most four model calls. A run that uses all four model calls without an answer is a failure, and grade() scores it as step limit, apart from the declines the model chose. Every run on both splits, for each approach, is replayed and counted by tool: model calls, refused requests, runs that reached a correct answer after a refusal, and clarifications by kind of question.

metrics-first-score.py Python 3.12.15 · duckdb 2.0.0 · pyyaml 6.0.3 · sqlglot 30.31.0 通過
"""Score metrics-first against the schema-only baseline on both splits: dev for tuning, test once for the final number.The prose rests on the correct counts per split and on the step-limit lines, which count runs out of model calls."""from collections import Counter, defaultdictfrom lib.agent import run_agentfrom lib.golden import golden_result, grade, load_golden, schema_only_outcomefrom lib.metric_tools import MetricToolbox, catalogfrom lib.model import RecordedModelfrom lib.prompts import metrics_first_promptfrom lib.semantic import loadfrom lib.warehouse import connect, describe_schemacon, layer = connect(), load()schema, system = describe_schema(con), metrics_first_prompt(catalog(layer))agent_model, sql_model = RecordedModel("metric-agent-runs"), RecordedModel("schema-only-sql")counts, facts, missed = defaultdict(Counter), defaultdict(Counter), defaultdict(list)for q in load_golden():    toolbox = MetricToolbox(con, layer, role=q.role)    trace = run_agent(q.question, system, agent_model, toolbox, max_steps=4)      # four calls at most, as in every run    outcome = grade(q, trace.answer, golden_result(con, q))                       # 'step limit' when the calls ran out    counts[q.split, "metrics-first"][outcome] += 1    counts[q.split, "schema-only"][schema_only_outcome(con, sql_model, schema, q)] += 1    facts[q.split][f"{q.kind} questions"] += 1    facts[q.split][f"{q.kind} correct"] += outcome == "correct"    facts[q.split]["forbidden answered"] += q.kind == "forbidden" and trace.answer.answered   # a forbidden question answered is a leak    facts[q.split]["model calls"] += sum(step.kind == "model" for step in trace.steps)    facts[q.split]["requests rejected"] += len(toolbox.rejected)    facts[q.split]["correct after a rejection"] += bool(toolbox.rejected) and outcome == "correct"    facts[q.split][f"asked back, {q.kind}"] += trace.answer.provenance.source == "ask_back"    if outcome != "correct":        missed[q.split].append(f"{q.question_id} {q.kind} {outcome}")for split in ("dev", "test"):    for approach in ("schema-only", "metrics-first"):     # only an agent run can hit the step limit        for outcome in ("correct", "wrong", "error", "declined") + (("step limit",) if approach != "schema-only" else ()):            print(f"{split} {approach} {outcome}: {counts[split, approach][outcome]} of 24")    # each split holds 24 golden questions    for kind in ("answer", "decline", "forbidden"):        print(f"{split} metrics-first {kind} correct: {facts[split][kind + ' correct']} of {facts[split][kind + ' questions']}")    for fact in ("forbidden answered", "model calls", "requests rejected", "correct after a rejection",                 "asked back, answer", "asked back, decline", "asked back, forbidden"):        print(f"{split} {fact}: {facts[split][fact]}")    print(f"{split} not correct: {', '.join(missed[split])}")
輸出python metrics-first-score.py
dev schema-only correct: 2 of 24
dev schema-only wrong: 14 of 24
dev schema-only error: 8 of 24
dev schema-only declined: 0 of 24
dev metrics-first correct: 16 of 24
dev metrics-first wrong: 7 of 24
dev metrics-first error: 0 of 24
dev metrics-first declined: 0 of 24
dev metrics-first step limit: 1 of 24
dev metrics-first answer correct: 12 of 17
dev metrics-first decline correct: 3 of 4
dev metrics-first forbidden correct: 1 of 3
dev forbidden answered: 2
dev model calls: 43
dev requests rejected: 10
dev correct after a rejection: 6
dev asked back, answer: 0
dev asked back, decline: 2
dev asked back, forbidden: 0
dev not correct: gq-007 answer step limit, gq-012 answer wrong, gq-016 answer wrong, gq-017 answer wrong, gq-018 answer wrong, gq-020 decline wrong, gq-023 forbidden wrong, gq-024 forbidden wrong
test schema-only correct: 3 of 24
test schema-only wrong: 13 of 24
test schema-only error: 8 of 24
test schema-only declined: 0 of 24
test metrics-first correct: 10 of 24
test metrics-first wrong: 7 of 24
test metrics-first error: 0 of 24
test metrics-first declined: 6 of 24
test metrics-first step limit: 1 of 24
test metrics-first answer correct: 5 of 17
test metrics-first decline correct: 4 of 4
test metrics-first forbidden correct: 1 of 3
test forbidden answered: 2
test model calls: 36
test requests rejected: 6
test correct after a rejection: 0
test asked back, answer: 1
test asked back, decline: 2
test asked back, forbidden: 0
test not correct: gq-028 answer wrong, gq-031 answer wrong, gq-032 answer declined, gq-033 answer declined, gq-034 answer declined, gq-035 answer wrong, gq-036 answer declined, gq-037 answer declined, gq-038 answer wrong, gq-039 answer step limit, gq-040 answer wrong, gq-041 answer declined, gq-046 forbidden wrong, gq-048 forbidden wrong

Metrics-first gets more questions right than schema-only SQL on both splits. On dev, the counts are 16 of 24 against 2 of 24, and on the held-out test split, 10 of 24 against 3 of 24. No metrics-first run ended in an error, because the model never wrote SQL that could fail to run.

That isn't the same as never failing. Runs that used all four calls without an answer ended at the step limit, 1 of 24 on dev and 1 of 24 on test. The dev run is gq-007; the test run, gq-039. The runs took 43 model calls across the dev split. The calls beyond the first in a run went to describing a metric before querying it, and to corrections after a refusal.

The test split is the number to report, and it is much lower than dev. Metrics-first gets 5 of 17 answerable test questions right, against 12 of 17 on dev, and the difference is mostly questions it gave no answer to. Metrics-first declined 6 of 24 test questions, all of which the warehouse can answer, and every one is a gap in the layer. Gross margin, the share of sessions that led to an order, return on marketing spend, discounts given, customers who signed up twice, and one order's refund and its reason all need definitions the layer doesn't hold. The model declined the first five with a reason that names what's missing, which is a metric to write. It didn't decline the last one, gq-036: after a refused request it asked whether the person meant an order ID or a date range. That was 1 clarification on an answerable test question. That question back offers nothing the layer can serve, so a decline was due. Chapter 3 found that marketing and support ask answerable questions only in the test split, and the shop's layer was written around the questions finance and the analyst ask, so nothing tuned on dev could reveal those gaps.

On test questions whose right answer is a decline, the agent declined or asked back in 4 of 4.

This is one model, one sampled reply per call and one synthetic warehouse, so the numbers describe the shop and this setup, not text-to-SQL in general. On a split of 24, one question is about four points of accuracy, and a difference of one or two questions is noise. The gap between the two approaches here is far larger than that.

Working rule: Score metrics-first beside your baseline on the same golden questions, and tune on dev only. Say how many times you recorded the dev split while tuning, and record the test split once, after. Report correct, wrong, error, declined and runs ended by the step limit separately, and count clarifications and refused requests too, because they show where the calls go.

閱讀未正確的執行結果

最近一次變動於第 2026.11.12 版

分數說明了有多少次執行失敗,卻沒有說明原因。原因在追蹤紀錄裡,而原因也決定了該修正什麼:指標、描述、檢查,或是存取規則。每一次未正確的執行都會逐次呼叫攤開來看:模型送出的工具與引數,以及回傳內容的第一行。它也會印出gq-008的請求,這是一次正確的執行,開發集的錯誤答案會與它並列比較;同時也會向該層索取gq-039的那次執行原本想取得的請求。

metrics-first-misses.py Python 3.12.15 · duckdb 2.0.0 · pyyaml 6.0.3 · sqlglot 30.31.0 通過
"""Read every metrics-first run that wasn't correct, call by call, and ask the layer what gq-039's run was reaching for.The prose rests on the gq-039 line: the layer's own request for it matches the golden result, and the run never found it."""import jsonfrom lib.agent import run_agentfrom lib.golden import golden_result, grade, load_golden, results_matchfrom lib.metric_tools import MetricToolbox, catalogfrom lib.model import RecordedModelfrom lib.prompts import metrics_first_promptfrom lib.semantic import MetricRequest, loadfrom lib.warehouse import connectcon, layer = connect(), load()system, model, golden = metrics_first_prompt(catalog(layer)), RecordedModel("metric-agent-runs"), load_golden()for q in golden:    trace = run_agent(q.question, system, model, MetricToolbox(con, layer, role=q.role), max_steps=4)    outcome, answer = grade(q, trace.answer, golden_result(con, q)), trace.answer    calls = list(zip(trace.steps[::2], trace.steps[1::2]))      # each model reply, then what its tool returned    if q.question_id == "gq-008":                               # correct, and gq-012's words without the regions        print(f"gq-008 ({outcome}) request: {json.dumps(calls[-1][0].arguments)}")    if outcome == "correct":        continue    print(f"{q.question_id} ({q.split}, {q.kind}, {outcome}): {q.question}")    print(f"{q.question_id} model calls: {len(calls)}")    for number, (reply, result) in enumerate(calls, start=1):        sent = f"{reply.tool} {json.dumps(reply.arguments)}" if reply.tool else reply.content        got = result.content.splitlines()[0][:150]                              # the result's first line, cut to 150 characters        ended = {"metric": f"{len(answer.rows)} row" + "s" * (len(answer.rows) != 1), "none": "declined",                 "ask_back": "asked back"}        if number == len(calls) and answer.provenance.source in ended:            got = ended[answer.provenance.source]        print(f"  {number}. {sent} -> {got}")    if answer.answered:        print(f"{q.question_id} rows: {len(answer.rows)}")# gq-039 is answerable from the layer: the request its run was reaching for, with the order written as "-name".gq039 = next(q for q in golden if q.question_id == "gq-039")request = MetricRequest(("line_net_revenue",), group_by=("product__product_name",),                        filters={"metric_time__year": 2025}, order_by=("-line_net_revenue",), limit=5)print(f"gq-039 from the layer: {request}")# gq-039 is a ranked question, so row order counts in the comparison.matched = results_match(layer.query(con, request).rows, golden_result(con, gq039), gq039.order_matters)assert matched, "the layer's request for gq-039 should match its golden result"   # the request asks gq-039's question, the five products with the most net revenue in 2025, highest firstprint(f"gq-039 from the layer, matches golden: {'yes' if matched else 'no'}")
輸出python metrics-first-misses.py
gq-007 (dev, answer, step limit): How much net revenue came from orders that included a tent in 2025?
gq-007 model calls: 4
  1. query_metric {"metrics": ["net_revenue"], "group_by": ["product__product_name"], "filters": {"product__product_name": "tent", "metric_time__year": 2025}} -> error: request rejected: net_revenue can't use dimension 'product__product_name'; metrics that can use it: line_net_revenue, units_sold
  2. query_metric {"metrics": ["line_net_revenue"], "group_by": [], "filters": {"product__product_name": "tent", "metric_time__year": 2025}} -> error: request rejected: no row has product__product_name = 'tent'; its values: 120 values
  3. describe_metric "product__product_name" -> error: arguments must be a JSON object
  4. describe_metric "product__product_name" -> error: arguments must be a JSON object
gq-008 (correct) request: {"metrics": ["net_revenue"], "group_by": [], "filters": {"metric_time__year": "2025"}, "order_by": []}
gq-012 (dev, answer, wrong): What was total revenue for each sales region in 2025?
gq-012 model calls: 2
  1. query_metric {"metrics": ["gross_revenue"], "group_by": ["order__region"], "filters": {"order__order_date": "2025"}} -> error: request rejected: order__order_date = '2025': a date dimension matches single days only; for a period, filter metric_time at its grain, as metr
  2. query_metric {"metrics": ["gross_revenue"], "group_by": ["order__region"], "filters": {"metric_time__year": 2025}} -> 3 rows
gq-012 rows: 3
gq-016 (dev, answer, wrong): How many customers placed at least one order in 2025?
gq-016 model calls: 2
  1. describe_metric {"metric": "new_customers"} -> new_customers, a simple metric: Customers whose first order fell in the period.
  2. query_metric {"metrics": ["new_customers"], "group_by": [], "filters": {"metric_time__year": "2025"}, "order_by": [], "limit": null} -> 1 row
gq-016 rows: 1
gq-017 (dev, answer, wrong): What was net revenue in each of our countries in 2025?
gq-017 model calls: 1
  1. query_metric {"metrics": ["net_revenue"], "group_by": ["order__country"], "filters": {"metric_time__year": 2025}} -> 10 rows
gq-017 rows: 10
gq-018 (dev, answer, wrong): How much was refunded, in euros, on orders placed in March 2026?
gq-018 model calls: 2
  1. describe_metric {"metric": "refunds"} -> refunds, a simple metric: Refunded EUR, counted on the local day the refund was made.
  2. query_metric {"metrics": ["refunds"], "group_by": ["metric_time__month"], "filters": {"metric_time__month": "2026-03"}} -> 1 row
gq-018 rows: 1
gq-020 (dev, decline, wrong): Which region is doing best?
gq-020 model calls: 2
  1. describe_metric {"metric": "net_revenue"} -> net_revenue, a simple metric: Revenue in EUR after discounts and before tax, from real orders that weren't cancelled, counted on the order's local bus
  2. query_metric {"metrics": ["net_revenue"], "group_by": ["customer__region"], "order_by": ["net_revenue"]} -> 3 rows
gq-020 rows: 3
gq-023 (dev, forbidden, wrong): What were the most common reasons for refunds in 2025?
gq-023 model calls: 2
  1. describe_metric {"metric": "refunds"} -> refunds, a simple metric: Refunded EUR, counted on the local day the refund was made.
  2. query_metric {"metrics": ["refunds"], "group_by": ["refund__reason"], "filters": {"metric_time__year": 2025}} -> 5 rows
gq-023 rows: 5
gq-024 (dev, forbidden, wrong): What was net revenue in EMEA in March 2026?
gq-024 model calls: 2
  1. query_metric {"metrics": ["net_revenue"], "group_by": ["order__region"], "filters": {"order__region": "EMEA", "order__order_date": "2026-03"}} -> error: request rejected: order__order_date = '2026-03': a date dimension matches single days only; for a period, filter metric_time at its grain, as m
  2. query_metric {"metrics": ["net_revenue"], "group_by": ["order__region"], "filters": {"order__region": "EMEA", "metric_time__month": "2026-03"}} -> 1 row
gq-024 rows: 1
gq-028 (test, answer, wrong): Which country had the highest refund rate in fiscal year 2026?
gq-028 model calls: 2
  1. describe_metric {"metric": "refund_rate"} -> refund_rate, a ratio metric: Refunded EUR over gross revenue of the same orders: orders placed in the period that weren't cancelled.
  2. query_metric {"metrics": ["refund_rate"], "group_by": ["order__country"], "filters": {"metric_time__fiscal_year": 2026}} -> 10 rows
gq-028 rows: 10
gq-031 (test, answer, wrong): How many units did an order contain on average in 2025, counting quantities, not order lines?
gq-031 model calls: 2
  1. describe_metric {"metric": "units_sold"} -> units_sold, a simple metric: Units sold on orders that weren't cancelled.
  2. query_metric {"metrics": ["units_sold"], "group_by": [], "filters": {"metric_time__year": "2025"}, "order_by": []} -> 1 row
gq-031 rows: 1
gq-032 (test, answer, declined): What was our gross margin in euros on products sold in 2025?
gq-032 model calls: 2
  1. describe_metric {"metric": "refund_rate"} -> refund_rate, a ratio metric: Refunded EUR over gross revenue of the same orders: orders placed in the period that weren't cancelled.
  2. cannot_answer {"reason": "The requested gross margin cannot be computed as gross margin is not a directly available metric. Gross margin is derived from gross revenue and net revenue, which are not provided as a single metric. Refund_rate is available but does not represent gross margin."} -> declined
gq-033 (test, answer, declined): What share of sessions in each channel led to an order in 2025?
gq-033 model calls: 1
  1. cannot_answer {"reason": "The metrics available do not include session data or a measure of sessions per channel. Therefore, it is not possible to determine the share of sessions in each channel that led to an order."} -> declined
gq-034 (test, answer, declined): What was the return on marketing spend for paid search in 2025?
gq-034 model calls: 1
  1. cannot_answer {"reason": "The data does not include metrics related to marketing spend or return on marketing spend (ROMS), so it is not possible to calculate the return on marketing spend for paid search in 2025."} -> declined
gq-035 (test, answer, wrong): How many orders did we get in each country in March 2026?
gq-035 model calls: 2
  1. query_metric {"metrics": ["orders"], "group_by": ["order__country"], "filters": {"order__order_date": "2026-03"}} -> error: request rejected: order__order_date = '2026-03': a date dimension matches single days only; for a period, filter metric_time at its grain, as m
  2. query_metric {"metrics": ["orders"], "group_by": ["order__country"], "filters": {"metric_time__month": "2026-03"}} -> 10 rows
gq-035 rows: 10
gq-036 (test, answer, declined): How much was refunded on order 113432, in euros, and why?
gq-036 model calls: 2
  1. query_metric {"metrics": ["refunds"], "filters": {"order__order_date": "113432"}, "group_by": ["refund__reason"]} -> error: request rejected: order__order_date = '113432': a date dimension matches single days only; for a period, filter metric_time at its grain, as me
  2. ask_back {"question": "Do you mean the refund amount for a specific order (by order ID) or for a specific date range?", "options": ["order ID 113432", "a date range"]} -> asked back
gq-037 (test, answer, declined): How many customers signed up twice with the same email address?
gq-037 model calls: 1
  1. cannot_answer {"reason": "The available metrics do not track customer sign-ups or email address uniqueness, so it's not possible to determine how many customers signed up twice with the same email address."} -> declined
gq-038 (test, answer, wrong): What were net revenue and the number of orders on each day of the week in 2025?
gq-038 model calls: 1
  1. query_metric {"metrics": ["net_revenue", "orders"], "group_by": ["order__order_date"], "filters": {"order__order_date": "2025-01-01"}, "order_by": []} -> 1 row
gq-038 rows: 1
gq-039 (test, answer, step limit): Which five products brought in the most net revenue in 2025, from the highest down?
gq-039 model calls: 4
  1. query_metric {"metrics": ["net_revenue"], "group_by": ["product__product_name"], "filters": {"order__order_date": "2025"}, "order_by": ["net_revenue", "-"], "limit": 5} -> error: request rejected: net_revenue can't use dimension 'product__product_name'; metrics that can use it: line_net_revenue, units_sold; order__order_
  2. query_metric {"metrics": ["line_net_revenue"], "group_by": ["product__product_name"], "filters": {"metric_time__year": 2025}, "order_by": ["line_net_revenue", "-"], "limit": 5} -> error: request rejected: can't order by '': it isn't a requested metric or group-by
  3. query_metric {"metrics": ["line_net_revenue"], "group_by": ["product__product_name"], "filters": {"metric_time__year": 2025}, "order_by": ["line_net_revenue", "-"], "limit": 5} -> error: request rejected: can't order by '': it isn't a requested metric or group-by
  4. describe_metric "line_net_revenue" -> error: arguments must be a JSON object
gq-040 (test, answer, wrong): What was revenue in each region in 2024 and in 2025?
gq-040 model calls: 2
  1. describe_metric {"metric": "net_revenue"} -> net_revenue, a simple metric: Revenue in EUR after discounts and before tax, from real orders that weren't cancelled, counted on the order's local bus
  2. query_metric {"metrics": ["net_revenue"], "group_by": ["customer__region"], "filters": {"metric_time__year": ["2024", "2025"]}} -> 3 rows
gq-040 rows: 3
gq-041 (test, answer, declined): How much discount did we give, in euros, in fiscal year 2026?
gq-041 model calls: 1
  1. cannot_answer {"reason": "The provided metrics do not include a metric for discount amounts. While net_revenue and gross_revenue are available, they represent revenue after discounts and before tax, but there is no specific metric to measure the total discount given in euros."} -> declined
gq-046 (test, forbidden, wrong): What was net revenue in APAC in June 2026?
gq-046 model calls: 2
  1. query_metric {"metrics": ["net_revenue"], "group_by": ["order__region"], "filters": {"order__region": "APAC", "order__order_date": "2026-06"}} -> error: request rejected: order__order_date = '2026-06': a date dimension matches single days only; for a period, filter metric_time at its grain, as m
  2. query_metric {"metrics": ["net_revenue"], "group_by": ["order__region"], "filters": {"order__region": "APAC", "metric_time__month": "2026-06"}} -> 1 row
gq-046 rows: 1
gq-048 (test, forbidden, wrong): How many orders did customers in AMER place in 2025?
gq-048 model calls: 1
  1. query_metric {"metrics": ["orders"], "group_by": ["customer__region"], "filters": {"customer__region": "AMER", "metric_time__year": 2025}} -> 1 row
gq-048 rows: 1
gq-039 from the layer: MetricRequest(metrics=('line_net_revenue',), group_by=('product__product_name',), filters={'metric_time__year': 2025}, order_by=('-line_net_revenue',), limit=5)
gq-039 from the layer, matches golden: yes

測試集中達到步數上限的那次執行,並不是該層的缺口。gq-039要求的是 2025 年淨營收最高的五項產品,由高到低排列,而該層可以回答:依產品名稱分組的line_net_revenue,以["-line_net_revenue"]排序並限制為五筆,就會得到標準答案的資料列。模型卻一直把排序寫成["line_net_revenue", "-"],驗證每次都拒絕,最後這次執行用完了呼叫次數。

測試集的錯誤答案與開發集屬於相同的類型。問退款率最高的國家時,代理人回傳了每個國家的退款率;問 2024 年與 2025 年的營收時,只回傳兩年合計的單一總數;問每筆訂單的平均件數時,回傳的是總件數;問 2025 年每個星期幾的情況時,它依訂單日期分組,並篩選到 2025 年 1 月 1 日,也就是只有一天。這些請求都是有效的,但各自回答了不同的問題。最後一個錯誤答案gq-035來自 AMER 的區域主管,他詢問各國的訂單,卻得到全部10個國家,這與開發集的gq-017一樣,是角色造成的問題。

開發集的失誤分成四組,各自指向不同的工作。第一組是接近但不正確的指標。gq-012依地區詢問「總營收」,得到的是gross_revenue,而財務團隊指的是淨額;gq-008用了相同的字詞但沒有地區,得到的卻是淨營收。gq-018詢問三月訂單的退款,得到的是refunds,以退款當天計算,而不是order_refunds。第二組是沒有任何指標涵蓋的問題,卻以最接近的指標作答。gq-016詢問 2025 年有多少客戶下過訂單,得到的卻是新客戶數。第三組是看不出提問者是誰。gq-017來自 EMEA 的區域主管,詢問「我們的國家」,卻得到全部10個國家;而2個禁止的問題之所以被回答,是因為模型之下沒有任何機制加以阻擋,這是第 9 章要處理的工作。最後,gq-007是包含帳篷的訂單營收,需要對訂單明細設定篩選條件,而沒有任何指標提供這種篩選。那次執行把它的4次呼叫花在帳篷篩選上,先是因為淨營收無法依產品篩選而被拒絕,接著又因為沒有任何產品名為tent而被拒絕,另外還花在引數不是 JSON 物件的 describe 呼叫上,最後達到步數上限。第 7 章會為這類問題提供受防護的 SQL 路徑。

實用原則: 逐次呼叫閱讀每一次未正確的執行,並依其需求分類:接近但不正確的指標,表示需要讓描述更精確,或需要反問提問者;缺少指標,表示該層需要新增一個,或該問題需要受防護的 SQL;而取決於角色的答案,需要的是存取規則,而不是提示詞。

此部分尚未翻譯,因此以英文呈現。

What goes wrong in production

最近一次變動於第 2026.11.12 版

The shop's agent answers about half of its golden questions from metrics, and fewer on the held-out test split. Over a real semantic layer, the agent goes wrong through names, messages, values and roles, and each has a symptom you can watch for.

Two metrics fit the words, and the agent picks the other one. In the runs that weren't correct, "total revenue" went to net revenue for gq-008 and to gross revenue for gq-012, and refunds on March's orders went to the refund day instead of the order day. A layer with near-duplicates invites this. Databricks' release notes record its dashboard-authoring agent, Genie Code, becoming "more likely to reference existing Unity Catalog metric views instead of creating new ones" when importing Tableau and Power BI reports, on Jul 2, 2026, and, on Sep 10, 2026, preferring governed metric views over ad-hoc SQL datasets when it builds a dashboard (Databricks). Those are a vendor's statements about building dashboards, not about answering questions, but they address the same problem: definitions that multiply. Keep one metric per business term where you can, write descriptions that say when not to use a metric, and keep golden questions for each pair of near-duplicates.

A metric with a nearby name answers a question it doesn't cover. New customers aren't customers who ordered. The agent's request is valid, the answer runs, and only a golden question shows it's wrong. Write golden questions for terms the layer doesn't define, and expect the right answer to be a decline.

Validation messages are part of the prompt. The model reads every refusal, and its next request follows the wording. Changing a message changes the conversation, and every recording that saw the old message stops replaying. Treat the messages as prompt text: keep them in one module, test them, and record again on purpose when they change.

Dimension values drift. Validation reads categorical values from the warehouse when the tool runs. A new country appears the day its first order loads, and a value list cached when the agent was deployed would refuse it. This chapter's prompt has that weakness on a small scale: it names the regions and channels, which spares the model a describe call, and it would go stale the day a region is added. Read values at request time, or refresh a cache with the load that changes them, and build any list the prompt carries from the same source.

The agent can't see who is asking. A regional lead's "our countries" needs a filter from the role, and a forbidden question needs a refusal from below the model. Neither belongs in the prompt, where a question can talk the model out of it. Apply row filters and column masks in the compiler, as chapter 9 does, so no request can reach past them.

Asking back becomes the answer to everything. A score counts a clarifying question on a question that should be declined as correct, so the score alone can't tell a useful question back from a useless one. The forecast's question back followed the prompt: ask_back's description in METRICS_FIRST invites one when a question "fits several metrics or periods", and "next quarter" fits two periods. gq-036's question back, on a question no metric covers, should have been a decline. Count clarifications on answerable questions as a cost, read them in traces, and write the tool's description to say when not to ask. A check that refuses an ask_back whose options the layer can't serve is untested: we haven't built it.

Corrections cost calls. Every refusal is another model call, and the loop's limit of four model calls per run ends a run that keeps being refused. In the runs that weren't correct, gq-007 spent its calls on a tent filter and on malformed describe calls, and gq-039 spent its calls on an order written the wrong way. Watch the calls per question, and when one kind of refusal dominates, fix its cause in the catalog or the tools rather than raising the limit.

Working rule: Before you widen the agent's audience, keep one metric per business term with descriptions that say when not to use it, review validation messages as you review prompts, read dimension values when requests run, enforce roles below the agent, and count clarifications and refusals beside the score. Start with the near-duplicate metrics, because they produce valid requests that return wrong numbers.

本章的決定

  • 代理只看得到指標,看不到資料表。 它的工具是 list_metrics、describe_metric、query_metric、ask_back 與 cannot_answer,建立在第 5 章的語意層之上,而它的提示詞帶有指標及其維度的目錄。
  • 模型回傳的是指標請求,而不是 SQL, 形式為 JSON 工具呼叫,由 run_agent() 解析成 MetricRequest。本書沿用自己的回覆格式,而不採用 Qwen 的 <tool_call> 標籤。
  • 每個請求在編譯之前都先經過驗證, 驗證依據語意層的名稱、倉儲中的類別值,以及各個時間粒度;被拒絕的請求會連同允許的替代選項一併退回給模型。
  • 會回傳空結果或不完整結果的請求視為無效, 例如針對每日日期維度所寫的期間。
  • 反問是一項工具, 其選項帶有語意層的定義,而它的回答與拒答分開計算。
  • 以指標優先的做法在兩個資料切分上都有量測, 並與僅提供綱要的基準線並列,在開發集上調校,調校後只記錄一次測試集的結果。
  • 目錄放在提示詞中, 而 describe_metric 會補上時間粒度與各維度的值,這些值於工具執行時從倉儲讀取。
  • 每次執行最多有四次模型呼叫, 這足以容納一次描述與一兩次修正,也會讓一再被拒絕的執行終止。觸及上限的執行會計為 step limit,屬於失敗,絕不計為拒答。
  • 沒有任何指標涵蓋的問題在此一律拒答。 第 7 章會為這類問題提供受防護的 SQL 路徑,第 9 章則套用代理看不到的角色。

本章,於本版次中

版次 2026.11.15範例於 2026年11 月 19 通過測試本章 6 個範例 個範例中有 6 個通過

譯自英文書,版次 2026.11.15。

引用本節

格式

各種格式皆引用英文原書,本譯本即依循該原書。

每種格式都會標示本版。其連結會開啟目前版次,節標題會說明該節內文最後一次變動的時間。存取日期與擷取日期即為你開啟此頁的當日。

本頁內容

第 6 章,共 11 章: 優先將問題導向受治理的指標

  1. Give the agent metrics, not tables
  2. Ask for a request, not SQL
  3. Validate the request against the layer
  4. 問題含糊時要反問
  5. Measure metrics-first on the golden questions
  6. 閱讀未正確的執行結果
  7. What goes wrong in production
  8. 本章的決定

返回頂端所有章節