測試伺服器並觀察其運作
閱讀約需 103 分鐘 分鐘,8 節 中共有 6 個範例。
讀完本章後,你可以在記憶體內與透過 HTTP 測試 Model Context Protocol 伺服器的工具、在發布前找出會破壞用戶端的 schema 變更、將錄製的主機端執行重播為迴歸測試、以 OpenTelemetry 追蹤工具呼叫、衡量模型選對工具的頻率,以及在不破壞既有用戶端的情況下淘汰一項工具。本章關於協定的敘述適用於規格的 2026-07-28 版,關於 SDK 的敘述則適用於 Python SDK,即 mcp 2.3.0。
建置處理工單的主機結尾時,商店的客服團隊所使用的主機 lib.ticket_host 已能針對兩部伺服器(商店自己的伺服器與合作夥伴的包裹追蹤伺服器)處理商店的全部 48 張工單,而模型的回覆取自錄製檔 ticket-runs。錄製檔是存放在 examples/data/recorded/ 之下的已儲存模型輸出,因此這次執行在每台機器上都會以相同方式重播,也不需要呼叫模型。
那次執行留下了兩個問題。主機為 36 of 48 張工單選對了第一個工具,卻沒有任何資訊指出它在哪些工單上選錯,以及原因為何。在退款工單上,模型對客戶的措辭彷彿退款已經完成,但要求退款的工單中,實際呼叫過 issue_refund 的有 0 of 7 張。商店的伺服器與主機持續在變動:描述被改寫、欄位被改名、工具被替換。這些變動中的任何一項,都可能在伺服器端沒有任何呼叫失敗的情況下,讓用戶端出問題。
要建立檢查機制,團隊可從 Model Context Protocol(MCP)取得幾項依據:每個工具的具型別 schema、每個請求的 _meta 中的追蹤脈絡,以及工具清單的變更通知。本章就以這些為基礎,建立商店的檢查機制。三個新模組負責這些設定。lib.testing 會執行一個 pytest 模組,並為每個測試印出一行穩定的輸出。lib.contract 會比較某個工具 schema 的兩個發行版本。lib.queue_replay 會重播第 10 章的佇列執行,並將每則回覆所做的承諾與主機實際發出的呼叫逐一核對。本章有兩個範例會重播錄製檔 ticket-runs,其餘範例不使用模型,而本章本身不會呼叫模型。
此部分尚未翻譯,因此以英文呈現。
Test tools in memory
A test of a server's tools should run with no port and no subprocess, so that it can run on every change. The Python SDK's Client gives you that: passed a server object instead of a URL or launch parameters, it connects to the server in the same process. The SDK's testing guide calls this the way to test a server, with the line "No subprocess. No port. Nothing on a wire" (testing). In the default mode="auto", the connection is a pair of DirectDispatchers, which hand each request to the server's handler as a Python object, with no streams, no JSON-RPC framing and no initialize handshake (mcp/client/client.py, _connect_inproc; mcp/shared/direct_dispatcher.py).
Two details matter in a test. raise_exceptions=True makes a failure outside a tool's body, such as a bug in a handler, come back with its real message instead of a sanitised "Internal server error". It doesn't change what a tool's own exception does: that is still a result with is_error=True, as it is for any client (testing). The SDK also stamps the server's identity into each result's _meta, so a test that compares whole results has to clear result.meta first (same page). Asserting on the fields a test is about avoids both the stamp and the SDK guide's snapshot library, inline-snapshot, which the book doesn't pin. Async tests need no extra plugin: anyio, which the SDK depends on, ships a pytest plugin that runs any test marked pytest.mark.anyio (AnyIO testing).
What the in-memory path skips matters as much as what it covers. With no HTTP layer, an in-memory test never meets the checks a Streamable HTTP server makes on headers, Origin and Host, never sees a server-sent event, and never carries trace context, as Trace calls with OpenTelemetry shows. The SDK's documentation adds that the in-process path doesn't deliver standalone notifications, such as a change to the tool list (client caching). A suite that is all in memory can pass while every HTTP client fails.
The shop's tests run against chapter 3's server, lib.shop_tools, from Design tools a model can use, with the shop's API on loopback for each test, served by the fixture shop_api_env from lib.testing. Three tests run in memory: an order's status comes back as structured fields, an unknown order comes back as a tool execution error that names find_orders, and an order ID in the wrong form is refused before the tool runs. The fourth serves the same tools over Streamable HTTP on loopback and sends a tools/call whose Mcp-Name header names a different tool from its body: a request the SDK's client never sends, and one an in-memory client can't express. Run as a script, the file runs its own tests with pytest 9.2.0.
1"""Four pytest tests of the shop's tools: three through the SDK's in-process client, and one over Streamable HTTP on2loopback, which reaches a check an in-memory test never makes. The prose's point rests on the unknown-order test's3line and on the last line, which counts the passes."""4import pytest5from mcp import Client67from lib import http_mcp, loopback, shop_tools8from lib.testing import anyio_backend, run, shop_api_env910pytestmark = pytest.mark.anyio # runs the async tests on anyio's pytest plugin111213async def test_order_status_fields(shop_api_env): # shop_api_env serves the shop's API on loopback for this test14 # raise_exceptions=True: a bug outside a tool's body raises here, with its own message15 async with Client(shop_tools.create_server(), raise_exceptions=True) as client:16 result = await client.call_tool("get_order_status", {"order_id": "O-1873"})17 assert not result.is_error18 assert result.structured_content["order_id"] == "O-1873"19 assert result.structured_content["status"] == "delivered"202122async def test_unknown_order_is_a_tool_error(shop_api_env):23 async with Client(shop_tools.create_server(), raise_exceptions=True) as client:24 result = await client.call_tool("quote_refund", {"order_id": "O-7144"})25 assert result.is_error # raise_exceptions doesn't make it raise26 assert "find_orders" in result.content[0].text # the model is told what to do instead272829async def test_malformed_order_id_is_refused(shop_api_env):30 async with Client(shop_tools.create_server(), raise_exceptions=True) as client:31 result = await client.call_tool("get_order_status", {"order_id": "1873"})32 assert result.is_error33 text = result.content[0].text # only argument validation writes this error:34 assert "order_id" in text and "string_pattern_mismatch" in text # the field and pydantic's pattern check353637def test_http_refuses_a_mismatched_header(shop_api_env):38 with loopback.serve(shop_tools.create_server().streamable_http_app()) as base_url:39 # the header names check_stock while the body calls get_order_status: the mismatch the server must refuse40 reply = http_mcp.post(base_url + "/mcp", "tools/call", {"name": "get_order_status",41 "arguments": {"order_id": "O-1873"}}, headers={"Mcp-Name": "check_stock"})42 assert reply.status_code == 40043 assert reply.json()["error"]["code"] == -32020 # the JSON-RPC code the server returns for the mismatch444546if __name__ == "__main__":47 run(__file__)
python test-tools-in-memory.pytest_order_status_fields: passed
test_unknown_order_is_a_tool_error: passed
test_malformed_order_id_is_refused: passed
test_http_refuses_a_mismatched_header: passed
tests passed: 4 of 4
A failing tool is a result a model can act on. The run reports 4 of 4, and the unknown order's test passed with raise_exceptions=True on: the tool raised ToolError, and the client got an ordinary result with the error flag set and the server's advice in its text, which is what a model needs to correct itself (Report failures as tool results).
The malformed ID never reached the tool. The test asserts that the error names order_id and pydantic's string_pattern_mismatch, which only argument validation writes. The SDK validated the arguments against the tool's parameter model, which its input schema is generated from, before calling the tool (mcp/server/mcpserver/server.py, _handle_call_tool). The HTTP test covers the layer the others skip: the server refused the mismatched header with status 400 and the JSON-RPC error -32020, as Read one HTTP exchange describes, before any tool code ran.
Working rule: Test each tool in memory with Client(server, raise_exceptions=True), and assert on structured_content fields and is_error, not on whole results. Keep at least one test per transport you ship that goes over it on loopback, for the header, origin and streaming checks an in-memory test can't reach, and put the tests of trace propagation and list changes there too.
此部分尚未翻譯,因此以英文呈現。
Check schemas as a contract
The in-memory tests from the previous section check that a tool does what its own release says. They don't check that the release still fits the clients using the last one. A client depends on a tool through its schemas: it sends arguments that its copy of the inputSchema accepts, and it reads the fields that its copy of the outputSchema promises. When a tool declares an output schema, the server MUST return structured content that conforms to it, and clients SHOULD validate it (tools, "Structured Content").
The Python SDK's client does validate, and it validates against its own copy. It keeps one compiled validator per tool, built from the output schema in the last tools/list it absorbed, and replaces it only when a later listing brings a different schema (mcp/client/session.py, _output_schema_validator and _absorb_tool_listing). A non-error result that doesn't match raises RuntimeError in the caller (mcp/client/session.py, validate_tool_result). So a server that renames an output field breaks every connected client that listed its tools before the release, at the moment each one next calls the tool, and the server sees nothing wrong.
In the shop's case, release 2 of the server renames the per-warehouse on_hand field of check_stock to count. To see the break, a client connects to release 1 in memory and lists the tools. It keeps its connection while the server swaps check_stock for release 2's version, as a rolling deploy would. Then the client calls the tool, lists again and calls once more. Last, a contract test runs the two checks it makes before a release ships: lib.contract.breaking_changes, a diff of the two releases' schemas, and a validation of release 2's real result against release 1's published output schema with jsonschema, the library the client uses.
1"""What a client connected before a release sees when the release renames check_stock's on_hand to count, and what2the contract checks report about the same change. The prose's point rests on the client's error line, which names3the missing field, and on the contract check's line."""4import asyncio56from mcp import Client7from pydantic import BaseModel89from lib import confirmations, contract, shop_tools10from lib.testing import shop_api_here111213class Warehouse(BaseModel):14 warehouse: str15 count: int # release 1 called it on_hand16 last_counted: str171819class StockReport(BaseModel):20 sku: str21 available: int22 warehouses: list[Warehouse]232425def check_stock(sku: shop_tools.Sku) -> StockReport:26 report = confirmations.chapter3_tools()["check_stock"].fn(sku) # release 1's tool does the reading27 return StockReport(sku=sku, available=report.available, warehouses=[Warehouse(28 warehouse=w.warehouse, count=w.on_hand, last_counted=str(w.last_counted)) for w in report.warehouses])293031async def main() -> None:32 server = shop_tools.create_server()33 with shop_api_here(): # the server's tools run in this process34 async with Client(server) as client:35 listing = await client.list_tools()36 print("release 1's tools/list cache hint:", f"ttlMs {listing.ttl_ms}")37 server.remove_tool("check_stock") # the release, swapped in while connected38 server.add_tool(check_stock, description=shop_tools.DESCRIPTIONS["check_stock"])39 try:40 await client.call_tool("check_stock", {"sku": "KG-1031"})41 except RuntimeError as error:42 print("client that listed before the release:", error.args[0].splitlines()[0]) # the message's first line43 new = next(tool for tool in (await client.list_tools()).tools if tool.name == "check_stock") # the release 2 schema44 result = await client.call_tool("check_stock", {"sku": "KG-1031"})45 print("client after listing again:", "result" if not result.is_error else "tool error")46 old = next(tool for tool in listing.tools if tool.name == "check_stock")47 print("contract check:", "; ".join(contract.breaking_changes(old, new)) or "no breaking change")48 validator = contract.validator(old.output_schema)(old.output_schema) # by the dialect the schema declares49 print("validator for release 1's schema:", type(validator).__name__ + ", as it declares",50 old.output_schema.get("$schema", "no $schema"))51 # the first error the schema finds in release 2's real result52 print("release 2's result against release 1's schema:", next(validator.iter_errors(result.structured_content)).message)535455asyncio.run(main())
python schema-change-breaks-a-client.pyrelease 1's tools/list cache hint: ttlMs 0
client that listed before the release: Invalid structured content returned by tool check_stock: 'on_hand' is a required property
client after listing again: result
contract check: output warehouses[].on_hand: removed
validator for release 1's schema: Draft202012Validator, as it declares no $schema
release 2's result against release 1's schema: 'on_hand' is a required property
Release 1's tool list carried the SDK's default cache hint, ttlMs 0, so the list was stale the moment it arrived, and the client still validated against it. The call after the release failed inside the client: "Invalid structured content returned by tool check_stock: 'on_hand' is a required property". The server had returned a correct release 2 result. After the client listed the tools again, the same call came back as a result. A zero TTL doesn't protect a connected client, because the validator follows the last listing, not the cache: a client that lists once at startup holds that schema for as long as it runs.
Both contract checks caught the change without a client. The schema diff reported output warehouses[].on_hand: removed, and validating release 2's result against release 1's schema failed with the client's own message: "'on_hand' is a required property". The diff is a reviewer's aid: it compares field paths, types and required inputs, and its docstring lists what it leaves out, such as patterns and enums. The validation is the test to keep, because it checks the server's real output against the exact schema that clients hold.
Breaking changes aren't symmetric. On the output side, removing, renaming or retyping a field breaks clients, and adding an optional field doesn't. On the input side, adding a required argument breaks old callers, and removing one breaks them quietly, because the SDK's server drops arguments it doesn't know without an error, as Treat ticket text as untrusted data shows. The pinned revision allows any JSON Schema 2020-12 keyword in both schemas, and reads a schema with no $schema as 2020-12 (tools). A schema may name another dialect in $schema, and clients and servers MUST validate by the declared dialect, or by 2020-12 when none is declared (base protocol, "JSON Schema Usage"). So choose the validator from the schema, as the SDK's client does with jsonschema's validator_for (mcp/client/session.py, _output_schema_validator). lib.contract.validator does the same for the shop's checks, with 2020-12 as the default. pydantic writes no $schema, so release 1's schema got Draft202012Validator, as it declares no $schema.
Working rule: Keep the last release's tools/list in the repository, and in CI validate every structured result your tests produce against it as well as against the new schema, each by the dialect it declares and by 2020-12 when it declares none. Treat a removed, renamed or retyped output field and a new required input as a breaking change: add the new field beside the old one, or add a new tool, and retire the old shape through a window (Change tools without breaking clients). Don't rely on a short TTL to protect clients that are already connected.
此部分尚未翻譯,因此以英文呈現。
Replay recorded host runs
Tool tests and contract tests cover the server. The support host from Build a host that works through tickets has logic of its own between the model and the servers: it routes names, asks for consent, sends results back and caps turns, and a change there can make every ticket go wrong without any tool failing. A replay covers that layer. It runs the real host against the real servers, with the model's replies taken from a recording, so a test can assert on what the host did for each ticket.
A replay tests the host and the servers against fixed model outputs. It doesn't test the model: a different model, or the same one with other sampling, needs a new recording. And it holds only while the messages stay as recorded. ReplayModel from lib.host looks each reply up by the SHA-256 of the exact messages, which include every tool definition and every tool result so far, and raises RecordingMismatch when they differ (Run the host loop). So a replay doubles as a tripwire: any change to what the model would see stops it, and a team then decides whether to record again.
The replay also gives the queue's results a second reading. Work through the ticket queue found that the model told customers their refunds were on the way for orders it never refunded. A reply is the model's claim, and the host's call log is the record of what happened, so the host can compare them before a reply reaches a customer. lib.queue_replay holds a small version of that check, built on the phrase check chapter 10 counted with, ticket_host.promises_refund, which looks for phrases such as "will be refunded" or "we'll refund". Reusing it keeps this chapter's counts and chapter 10's the same measurement. Its unbacked_refund is true when a run's last reply makes that promise and no issue_refund call in the run returned a result. The phrase list is a rule of thumb written against this recording's replies, not a classifier, and the module's docstring says what it misses.
queue_replay.replay_queue replays the whole queue, with the support host set up exactly as chapter 10's work-the-ticket-queue sets it up. The run prints one ticket's tool sequence, counts the refunds and the replies that promise one, and replays the first ticket after one sentence has been added to quote_refund's description.
1"""Replay chapter 10's queue run, the recording ticket-runs, through the support host, and check each reply's promise2against the host's call log. The prose's point rests on the refund lines, which compare refunds issued with replies3that promise one, and on the last line, which shows what a one-sentence rewording does to the replay."""4import asyncio56from lib import queue_replay, shop_tools, ticket_host7from lib.host import RecordingMismatch8from lib.shop_data import load_shop91011def reword_quote_refund(host) -> None: # the server's next release adds a sentence to one description12 changed = {**shop_tools.DESCRIPTIONS}13 changed["quote_refund"] += " Quotes are in the order's currency."14 host.listed["shop"] = shop_tools.with_descriptions(host.listed["shop"], changed)151617async def main() -> None:18 tickets = {ticket.ticket_id: ticket for ticket in load_shop().tickets}19 runs = await queue_replay.replay_queue()20 print("tickets replayed:", len(runs))21 print("T-0009 tools:", " then ".join(turn.tool for turn in runs["T-0009"].calls)) # its calls, in order22 print("refunds issued:", sum(queue_replay.refunded(run) for run in runs.values()))23 # the phrase check from chapter 10, a rule of thumb: see the module's docstring for what it misses24 promised = [ticket_id for ticket_id, run in runs.items() if ticket_host.promises_refund(run.answer)]25 # a promise with no refund behind it: no issue_refund call in the run returned a result26 unbacked = [ticket_id for ticket_id in promised if queue_replay.unbacked_refund(runs[ticket_id])]27 print("replies that promise a refund:", len(promised))28 print("of which no call made:", len(unbacked))29 for tag in ("issue_refund", "quote_refund"): # the tickets tagged with each refund tool30 group = [t for t in tickets.values() if t.expected_tool == tag]31 print(f"{tag} tickets flagged:", f"{sum(t.ticket_id in unbacked for t in group)} of {len(group)}")32 print("flagged:", ", ".join(unbacked))33 try:34 await queue_replay.replay_queue(change=reword_quote_refund, tickets=[tickets["T-0001"]])35 except RecordingMismatch:36 print("after rewording quote_refund, T-0001 replays:", "no, RecordingMismatch")373839asyncio.run(main())
python replay-ticket-runs.pytickets replayed: 48
T-0009 tools: shop.quote_refund then shop.find_orders
refunds issued: 0
replies that promise a refund: 11
of which no call made: 11
issue_refund tickets flagged: 6 of 7
quote_refund tickets flagged: 5 of 9
flagged: T-0007, T-0020, T-0029, T-0034, T-0036, T-0037, T-0040, T-0042, T-0045, T-0046, T-0048
after rewording quote_refund, T-0001 replays: no, RecordingMismatch
All 48 tickets replayed, and ticket T-0009 took the path it took in chapter 10: shop.quote_refund then shop.find_orders. An assertion on that sequence is a regression test for the host. If a change to routing or error handling sent the first tool execution error back differently, the model's second turn would no longer match the recording, and the replay would stop.
lib.queue_replay flags the promises that no call backs, and a host should send those replies to a person instead of to the customer. The host issued 0 refunds, yet 11 replies promise one, and 11 of them have no refund behind them.
Of the tickets asking for a refund, 6 of 7 are flagged, and of the tickets asking only what a refund would be, 5 of 9 are, each reply telling the customer the money was on its way. The one refund request not flagged, T-0024, ended with the model asking which product the customer meant, which promises nothing. The flagged tickets are T-0007, T-0020, T-0029, T-0034, T-0036, T-0037, T-0040, T-0042, T-0045, T-0046, T-0048.
The rewording at the end shows the tripwire. After one sentence was added to one description, the first ticket's replay gave no, RecordingMismatch. Every ticket's prompt holds every tool's description, so changing any description changes every prompt. That is the cost of replays, and also their use: a reworded description is the change that can alter what the model chooses, and the replay won't let it through unmeasured.
Working rule: Replay a recorded run of the whole queue in CI, assert on each ticket's tool sequence, and run a check that compares each reply's promises with the calls that returned a result. Keep generated IDs and clocks fixed in the servers a replay runs against, or it stops for reasons that aren't changes. Then treat RecordingMismatch as a signal, not as a flaky test: when you change a description, a prompt or a tool's output, record again with the recording script and compare the new run's scores with the old run's before you ship.
此部分尚未翻譯,因此以英文呈現。
Trace calls with OpenTelemetry
Tests run before a release; traces show what a server does after it. OpenTelemetry is the common way to get them: an application creates spans, each a named, timed operation with attributes and a parent, and an exporter sends them to a tracing backend. MCP adds one thing of its own. At the pinned revision, the _meta keys traceparent, tracestate and baggage are reserved for trace context, and their values MUST follow the W3C Trace Context and Baggage formats (base protocol). That is how a client's trace and a server's trace join into one, whatever the transport.
The Python SDK traces with no setup in your server's code. Every server installs OpenTelemetryMiddleware, which wraps each inbound message in a SERVER span and does nothing until an OpenTelemetry SDK and exporter are installed (mcp/server/lowlevel/server.py; OpenTelemetry). The source marks the middleware provisional, so its details may change in a 2.x minor release. On the client side, the JSON-RPC paths, which are URLs, stdio and the legacy in-memory mode, open a CLIENT span named MCP send {method} {target} and inject traceparent into the request's _meta, and the server extracts it as the parent of its span (mcp/shared/jsonrpc_dispatcher.py, mcp/shared/_otel.py). The in-process path from Test tools in memory sends no JSON-RPC, so it creates no client span and injects nothing.
The shop wants a trace for each tool call, findable by tool name and showing whether it failed. The script installs OpenTelemetry's SDK, opentelemetry-sdk 1.45.2, with an InMemorySpanExporter, which keeps finished spans in a list that a test can read (opentelemetry/sdk/trace/export/in_memory_span_exporter.py). The script serves chapter 3's tools from Design tools a model can use over Streamable HTTP on loopback in the same process, so the client's and the server's spans land in the same list. It calls find_orders and then quote_refund for an order that doesn't exist. Then it makes the same find_orders call in memory twice, the second time inside a span of the test's own.
1"""The OpenTelemetry spans the SDK emits for the shop's tool calls, over Streamable HTTP on loopback and in memory.2The prose's point rests on two printed lines: the HTTP server span, which shows the trace context crossing the wire,3and the in-memory server span parents, which show what happens without it."""4import asyncio56from mcp import Client7from opentelemetry import trace8from opentelemetry.sdk.trace import TracerProvider9from opentelemetry.sdk.trace.export import SimpleSpanProcessor10from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter11from opentelemetry.semconv._incubating.attributes.mcp_attributes import McpMethodNameValues1213from lib import loopback, shop_tools14from lib.testing import shop_api_here1516spans = InMemorySpanExporter()17trace.set_tracer_provider(provider := TracerProvider())18provider.add_span_processor(SimpleSpanProcessor(spans)) # each span, as it ends, into memory192021def finished(name: str) -> list: # every finished span with this name, in the order they ended22 return [span for span in spans.get_finished_spans() if span.name == name]232425def parent(span) -> str:26 return next((s.name for s in spans.get_finished_spans() if span.parent and s.context.span_id ==27 span.parent.span_id), "none") # "none" for a span with no parent, or whose parent never finished282930async def main() -> None:31 server = shop_tools.create_server()32 with shop_api_here(), loopback.serve(server.streamable_http_app()) as base_url:33 async with Client(base_url + "/mcp") as client:34 await client.call_tool("find_orders", {"customer_id": "C-0127"})35 await client.call_tool("quote_refund", {"order_id": "O-7144"}) # a tool error36 print("HTTP spans:", ", ".join(sorted(span.name for span in spans.get_finished_spans())))37 # the test makes one call of each, so the first finished span of each name is the call's38 call, error = finished("tools/call find_orders")[0], finished("tools/call quote_refund")[0]39 print("HTTP server span:", call.name, f"({call.kind.name}), parent: {parent(call)}")40 for key in sorted(set(call.attributes) - {"jsonrpc.request.id"}): # the request ID changes on every run41 print(f"attribute {key}:", call.attributes[key])42 print("tool error span status:", error.status.status_code.name)43 print("tool error span error.type:", error.attributes["error.type"])44 spans.clear()45 async with Client(server) as client:46 await client.call_tool("find_orders", {"customer_id": "C-0127"})47 # a span of the test's own, which the in-memory server falls back to as the parent48 with trace.get_tracer("shop-tests").start_as_current_span("test find_orders"):49 await client.call_tool("find_orders", {"customer_id": "C-0127"})50 print("in-memory spans:", ", ".join(sorted(span.name for span in spans.get_finished_spans())))51 print("in-memory server span parents:", ", ".join(parent(s) for s in finished("tools/call find_orders")))52 # the pinned conventions predate the pinned revision, and this list has no server/discover53 assert "server/discover" not in {m.value for m in McpMethodNameValues}545556asyncio.run(main())
python trace-a-tool-call.pyHTTP spans: MCP send server/discover, MCP send tools/call find_orders, MCP send tools/call quote_refund, MCP send tools/list, server/discover, tools/call find_orders, tools/call quote_refund, tools/list
HTTP server span: tools/call find_orders (SERVER), parent: MCP send tools/call find_orders
attribute gen_ai.operation.name: execute_tool
attribute gen_ai.tool.name: find_orders
attribute mcp.method.name: tools/call
attribute mcp.protocol.version: 2026-07-28
tool error span status: ERROR
tool error span error.type: tool_error
in-memory spans: server/discover, test find_orders, tools/call find_orders, tools/call find_orders, tools/list
in-memory server span parents: none, test find_orders
Over HTTP the spans came in pairs, each server span paired with a client span of the same method: MCP send server/discover, MCP send tools/call find_orders, MCP send tools/call quote_refund, MCP send tools/list, server/discover, tools/call find_orders, tools/call quote_refund, tools/list. The client listed the tools before its first call, because it validates the result against the output schema. The trace shows the server's span for the call, tools/call find_orders (SERVER), parent: MCP send tools/call find_orders, so the trace context crossed in _meta.
Its attributes name the method, tools/call, the operation, execute_tool, the tool, find_orders, and the revision (mcp/server/_otel.py).
The failed quote's span has the status ERROR and the error.type tool_error, although the call returned a result and not a JSON-RPC error, so a tracing backend can count tool execution errors without reading result bodies.
A test of trace propagation needs HTTP or stdio, because in memory the server has nothing to extract. In memory there were no client spans, only the server's spans and the test's own test find_orders: server/discover, test find_orders, tools/call find_orders, tools/call find_orders, tools/list. The two server spans' parents were none, test find_orders, in that order. With nothing to extract, the server falls back to whatever span is current, and only the test's own span gave it one (mcp/shared/_otel.py, extract_trace_context).
The names a server emits are the part to treat with care. OpenTelemetry's semantic conventions for MCP have the status Development; they recommend span names of the form {mcp.method.name} {target} and define metrics such as mcp.server.operation.duration (MCP semantic conventions, read 2026-10-31). They have also moved: the page the MCP specification links to shows a notice that the GenAI conventions moved to a repository of their own, which has no tagged release (moved notice). The SDK's server spans follow the recommended naming. Its client span name, MCP send …, doesn't, and the SDK emits no metrics: the source of mcp 2.3.0 never creates a meter. The pinned conventions package, opentelemetry-semantic-conventions 0.66b2, predates the pinned revision. Its list of method names has initialize and logging/setLevel, it still defines mcp.session.id, and it has no server/discover (opentelemetry/semconv/_incubating/attributes/mcp_attributes.py).
For logs, the pinned revision deprecates MCP's own Logging messages, notifications/message, and points servers to standard error on stdio and to OpenTelemetry otherwise (changelog). Replace sampling, roots and logging covers the move.
Working rule: Install an OpenTelemetry SDK and exporter in every server and host you run, and let the SDK's middleware make the server spans. Build dashboards on the attributes the SDK emits, mcp.method.name, gen_ai.tool.name and error.type, and label them as following a Development-status convention at a named version, because the names may change. Don't validate method names against the pinned conventions package. Test propagation over a real transport, and wrap in-memory calls in a span of your own when a test needs a parent.
衡量選對工具的頻率
重播已記錄的主機執行中的重播,問的是主機是否仍照原樣運作。第二個問題是:在商店提供的工具與描述之下,模型選得好不好。最常見的第一項衡量是工具選擇準確率:模型的第一次呼叫所用的工具,與正確的代理人會用的工具一致的案例所占的比例,常稱為 top-1 準確率。基準測試也會對引數評分。以 Berkeley Function Calling Leaderboard 為範例,它透過比對抽象語法樹,將每次呼叫的名稱與引數一併與預期的內容核對(BFCL)。
來自固定記錄的數字,衡量的範圍很窄:商店的工具名稱與描述,搭配一個模型 Qwen3-4B-Instruct-2507、以 Q4_K_M 量化、在溫度 0.7 下取樣(處理工單佇列),使用本書的一個提示詞與一個種子,涵蓋本書撰寫的 48 張工單。
這只是該設定下的一次抽樣,而且各組都很小:最小的一組只有 8 張工單,所以一張工單就占其分數的很大比例。它無法說明一般的模型;任何描述一經更動,就需要重新記錄,這個數字才有意義,重播的 RecordingMismatch 會強制執行這一點。每張工單都附有正確的代理人會最先呼叫的工具。如第 10 章所做,指令碼以 ticket_host.expected_first_tool 評分:退款請求若以 quote_refund 開頭就算正確,因為 issue_refund 需要先有報價。
為了替重播評分,指令碼取出每張工單的第一次工具呼叫,並印出整體準確率、各預期工具的準確率,以及每一種混淆。對於指明訂單的正確呼叫,它還以兩種方式對 order_id 引數評分:對照工單所寫的 ID,以及對照客戶真正指的訂單。這兩者在 5 張客戶輸入錯誤編號的工單上會不同。
1"""Score the first tool call of every ticket in chapter 10's queue run against the tool each ticket is tagged with,2overall, per tag and by confusion, and check the order IDs of the right calls. The prose's point rests on the3accuracy lines, the per-tag line for refunds, and the two order-ID lines, which differ."""4import asyncio5from collections import Counter67from lib import queue_replay, ticket_host8from lib.shop_data import load_shop91011async def main() -> None:12 tickets = load_shop().tickets13 runs = await queue_replay.replay_queue()14 scored = [] # (ticket, expected tool, first tool called)15 for ticket in tickets:16 calls = runs[ticket.ticket_id].calls17 # the first call, or None when the host made none; a refund ticket expects quote_refund first18 scored.append((ticket, ticket_host.expected_first_tool(ticket), calls[0] if calls else None))19 right = [(t, call) for t, expected, call in scored if call and call.tool == expected]20 print("first tool right:", f"{len(right)} of {len(scored)}")21 print("top-1 accuracy:", f"{len(right) / len(scored):.2f}")22 for expected, count in sorted(Counter(expected for _, expected, _ in scored).items()):23 hits = sum(1 for t, e, call in scored if e == expected and call and call.tool == expected)24 print(f"expected {expected}:", f"{hits} of {count}")25 confusions = Counter((expected, call.tool if call else "no tool") for _, expected, call in scored26 if not call or call.tool != expected)27 for (expected, chosen), count in confusions.most_common():28 print(f"expected {expected}, called {chosen}:", count)29 with_order = [(t, call) for t, call in right if t.order_id and "order_id" in call.arguments] # right calls naming an order30 as_written = sum(call.arguments["order_id"] == t.cited_order_id for t, call in with_order) # the ID the ticket gave31 meant = sum(call.arguments["order_id"] == t.order_id for t, call in with_order) # the customer's order32 print("right tool, order ID as the ticket wrote it:", f"{as_written} of {len(with_order)}")33 print("right tool, order the customer meant:", f"{meant} of {len(with_order)}")34 # tickets whose cited ID isn't the order the customer meant35 typos = [(t, e, call) for t, e, call in scored if t.cited_order_id and t.cited_order_id != t.order_id]36 print("tickets with a mistyped order ID:", len(typos))37 print("of which the right tool was called:", sum(1 for _, e, call in typos if call and call.tool == e))38 for t, _, call in [(t, e, call) for t, e, call in typos if not call or call.tool != e]:39 print(f"mistyped {t.ticket_id}, first tool:", call.tool if call else "no tool")40 print("smallest expected-tool group, tickets:", min(Counter(e for _, e, _ in scored).values()))41 manifest = queue_replay.manifest()42 print("recorded model:", manifest["model"].split("/")[-1]) # the model's name, without its publisher's path43 print("recorded quantization:", manifest["quantization"])444546asyncio.run(main())
python tool-choice-accuracy.pyfirst tool right: 36 of 48
top-1 accuracy: 0.75
expected shop.check_stock: 10 of 10
expected shop.find_orders: 8 of 8
expected shop.get_order_status: 12 of 14
expected shop.quote_refund: 6 of 16
expected shop.quote_refund, called shop.get_order_status: 8
expected shop.get_order_status, called partner.track_parcel: 2
expected shop.quote_refund, called shop.find_orders: 2
right tool, order ID as the ticket wrote it: 18 of 18
right tool, order the customer meant: 14 of 18
tickets with a mistyped order ID: 5
of which the right tool was called: 4
mistyped T-0042, first tool: shop.get_order_status
smallest expected-tool group, tickets: 8
recorded model: Qwen3-4B-Instruct-2507
recorded quantization: Q4_K_M
第一個工具正確的工單有 36 of 48 張,top-1 準確率為 0.75,與第 10 章計算的 36 of 48 相同。
整體數字掩蓋了錯誤出在哪裡。庫存問題得分 10 of 10,訂單搜尋得分 8 of 8,但退款工單只有 6 of 16,而且大多數失誤,即 8,都轉向了 get_order_status,儘管它的描述說退款要用 quote_refund,而 quote_refund 的描述說應先呼叫它。這一對描述是接下來要處理的,而佇列就是描述更動後要重新記錄的那組工單。轉向 partner.track_parcel 的兩個狀態問題,是第 10 章的追蹤編號案例(處理工單佇列)。
引數說明了為何參照基準很重要。在工具正確的情況下,模型在 18 of 18 次呼叫中使用工單所寫的訂單 ID,在 14 of 18 次呼叫中使用客戶真正指的訂單。
失誤來自客戶的輸入錯誤。在輸入錯誤的工單中,有 4 張到達了正確的工具而在此計分,另一張 T-0042 則先轉向 shop.get_order_status,因此改算在錯誤工具之中。沒有任何模型能在不查詢客戶訂單的情況下看穿輸入錯誤。以客戶真正的意思評分,模型顯得粗心;以它所得到的內容評分,它則是忠實的。這個落差屬於主機的流程,主機應在採取行動之前,先對照工單的客戶來核對訂單。
已發表的基準測試也指向同一方向。一篇預印本 RAG-MCP 報告指出,在模型選擇之前先檢索出一份工具短名單,使其基準任務上的選擇準確率從 13.62% 提高到 43.13%,由充當評審的模型評分(RAG-MCP)。MCP-Bench 涵蓋 28 個線上伺服器與 250 個工具,報告指出在指示未點名工具的任務上,挑戰依然持續存在(MCP-Bench)。兩者各是針對自身任務的單篇預印本,都無法說明商店的數字。
實用原則: 對每次已記錄的執行,依標籤對第一個工具評分,而不只看整體,並另外為引數評分,分別對照工單給模型的內容與真實情況。先處理最大混淆背後的描述,再重新記錄,並比較兩次記錄之間各標籤的分數。在每個分數旁註明模型、提示詞與工單集,且絕不跨不同設定比較分數。
此部分尚未翻譯,因此以英文呈現。
Change tools without breaking clients
The contract test in Check schemas as a contract says when a change breaks clients; it doesn't say how to make the change. The specification doesn't either. A tool definition has no deprecation field, and nothing in the pinned revision marks a tool as on its way out (tools). What follows is practice: the book's recommendation, built from parts the specification does define, not a protocol feature.
There are three such parts. A client asks for tool list changes with subscriptions/listen and hears about each one on that stream (subscriptions), as Tell clients when a resource changes shows for resources. A tools/list result carries a cache hint, ttlMs, and a change notification invalidates a cached list that is still fresh; clients MAY re-fetch early when a call fails as if the tool changed, for example with an unknown method or invalid parameters (caching). And a tool execution error is what clients SHOULD give the model so it can correct itself (tools, "Error Handling"). From those parts, the book recommends this procedure: add the new tool beside the old one; mark the old one "Deprecated: use …" in its description; announce the change; shorten the TTL while the change is under way; keep the old tool working through a window; and after the window, answer it with an execution error that names the replacement.
The shop is replacing find_orders with find_orders_v2, which can also filter by SKU. To walk through the three releases, the script serves chapter 3's tools from Design tools a model can use over Streamable HTTP on loopback, with a TTL of one minute on the tool list, and connects two clients: one that listens for list changes, and one that only caches. Release 2 adds find_orders_v2 and rewords find_orders, and the server publishes the change. Release 3 replaces find_orders with a function that raises ToolError. An in-memory client wouldn't do here, because the in-process path doesn't deliver list changes (Test tools in memory).
1"""Retire find_orders in favour of find_orders_v2 over three releases, watched by a listening client and a caching2client. The prose's point rests on the caching client's list and on the last line, the error the old name returns3after the window."""4import asyncio56from mcp import Client7from mcp.server.caching import CacheHint8from mcp.server.mcpserver import MCPServer9from mcp.server.mcpserver.exceptions import ToolError10from mcp.server.subscriptions import InMemorySubscriptionBus, ToolsListChanged1112from lib import confirmations, loopback, shop_tools13from lib.testing import shop_api_here1415bus, tools = InMemorySubscriptionBus(), confirmations.chapter3_tools()16server = MCPServer("shop-support", version="0.11.0", tools=list(tools.values()), subscriptions=bus,17 cache_hints={"tools/list": CacheHint(ttl_ms=60_000, scope="public")}) # a minute: short while a change is under way18old = tools["find_orders"]192021def find_orders_v2(customer_id: shop_tools.CustomerId, sku: shop_tools.Sku | None = None) -> shop_tools.OrderList:22 """Search one customer's orders, newest first, optionally only those with one product's SKU."""23 found = old.fn(customer_id, limit=50)24 return found.model_copy(update={"orders": [o for o in found.orders if sku is None or sku in o.skus]})252627def retired(customer_id: shop_tools.CustomerId) -> shop_tools.OrderList:28 raise ToolError("find_orders was retired. Call find_orders_v2 with the same customer_id.")293031async def replace_find_orders(fn, description: str) -> None:32 server.remove_tool("find_orders")33 server.add_tool(fn, name="find_orders", title=old.title, description=description, annotations=old.annotations)34 await bus.publish(ToolsListChanged()) # to every client listening for list changes353637async def main() -> None:38 with shop_api_here(), loopback.serve(server.streamable_http_app()) as base_url:39 async with Client(base_url + "/mcp") as listening, Client(base_url + "/mcp") as cached:40 listing, _ = await cached.list_tools(), await listening.list_tools() # both cache release 1's list41 print("release 1:", f"{len(listing.tools)} tools, ttlMs {listing.ttl_ms}")42 async with listening.listen(tools_list_changed=True) as changes: # a stream that carries list-change events43 server.add_tool(find_orders_v2) # release 2: the replacement first44 await replace_find_orders(old.fn, "Deprecated: use find_orders_v2. " + old.description)45 print("event:", type(await anext(changes)).__name__)46 print("listening client sees:", ", ".join(t.name for t in (await listening.list_tools()).tools))47 print("cached client sees:", ", ".join(t.name for t in (await cached.list_tools()).tools))48 during = await cached.call_tool("find_orders", {"customer_id": "C-0127"})49 print("find_orders during the window:", "tool error" if during.is_error else "result")50 await replace_find_orders(retired, "Retired: use find_orders_v2.") # release 3: the old name now errors51 after = await cached.call_tool("find_orders", {"customer_id": "C-0127"})52 print("find_orders after the window:", after.content[0].text)535455asyncio.run(main())
python deprecate-a-tool.pyrelease 1: 5 tools, ttlMs 60000
event: ToolsListChanged
listening client sees: get_order_status, quote_refund, issue_refund, check_stock, find_orders_v2, find_orders
cached client sees: find_orders, get_order_status, quote_refund, issue_refund, check_stock
find_orders during the window: result
find_orders after the window: Error executing tool find_orders: find_orders was retired. Call find_orders_v2 with the same customer_id.
Release 1 listed 5 tools, ttlMs 60000. When release 2 shipped, the listening client received ToolsListChanged. The SDK drops a cached list before it hands the client such an event (mcp/client/client.py), so the listening client's next listing showed both tools: get_order_status, quote_refund, issue_refund, check_stock, find_orders_v2, find_orders.
The caching client heard nothing and served release 1's list from its cache: find_orders, get_order_status, quote_refund, issue_refund, check_stock. Its call to find_orders during the window still returned a result. A model behind a caching client doesn't learn about find_orders_v2 until the TTL runs out or the client re-fetches, which is why the TTL should be short while a change is under way, and why the old tool has to keep working.
After release 3, the caching client's call to find_orders came back as an execution error the model can act on: "Error executing tool find_orders: find_orders was retired. Call find_orders_v2 with the same customer_id.". A removed tool would have produced the SDK's unknown-tool error instead, which tells the model the name was wrong but not what to call (Report failures as tool results). The SDK returns that error as a tool execution error, although the specification makes an unknown tool a protocol error (mcp/server/mcpserver/tools/tool_manager.py; tools, "Error Handling").
The listening client's list also shows a side effect: find_orders moved to the end. MCPServer lists tools in the order they were added, so the release's live swap, which removes the tool and adds it back, moves it (Cache list results). A changed order changes the start of every prompt that renders the list, which is what a model provider's prompt cache matches on. A server that restarts with the new code lists tools in the order its code registers them, so add new tools at the end and leave the existing ones where they are.
Working rule: Retire a tool in three releases: add the replacement and mark the old tool deprecated in its description, keep both working through a window you announce, then answer the old name with an execution error that names the replacement. Shorten ttlMs on tools/list while the change is under way, publish a list change at each release, and open a subscriptions/listen stream in every host you control. Say in your changelog that the window is your policy, since the protocol defines none.
正式環境中會出什麼問題
這裡的檢查只針對單一機器上的單一程序。在正式環境中,同樣的訊號來自許多複本與用戶端,而且每一種訊號都可能以各自的方式誤導你。
span 的數量暴增,超出後端能建立索引的範圍。 SDK 以方法與工具來命名伺服器端 span,使不同名稱的數量與工具清單一樣少。若在 span 名稱中放入訂單 ID、資源 URI 或客戶 ID,每個值就會產生一個名稱;MCP 慣例正是因此將資源 URI 放在屬性中(MCP semantic conventions)。徵兆是追蹤費用隨流量成長,而不是隨功能成長。
追蹤資料帶有商店的資料。 SDK 不會在其 span 中記錄工具引數或結果(mcp/server/_otel.py),而慣例將 gen_ai.tool.call.arguments 標示為選擇性啟用(同一來源)。商店的引數包含客戶與訂單 ID,而引數只說明呼叫者送出了什麼。若要記錄是誰執行了操作,請取用已驗證權杖的主體,在支援主機上即為工作人員,並搭配伺服器已檢查過的工單記錄中的客戶(避免伺服器淪為混淆代理人)。僅在設定開啟時才將引數加入 span,並讓它們遠離商店資料處理政策未涵蓋的任何後端。
慣例在你的儀表板底下變動。 MCP 慣例處於 Development 狀態,且已更換過一次儲存庫(搬遷通知),而 SDK 的中介軟體被標示為暫行(mcp/server/lowlevel/server.py)。SDK 的某個版本若重新命名了屬性,儀表板就會在沒有任何錯誤的情況下變空。請將屬性名稱集中放在設定中的同一處,並在每次 SDK 升級時檢查,如同處理任何更新觸發條件一樣。
部署期間各複本不一致。 在滾動部署中,一個複本提供第 1 版的結構描述,下一個複本提供第 2 版的,而從其中一個列出工具、卻對另一個發出呼叫的用戶端,會得到將結構描述當作合約來檢查中的 RuntimeError。徵兆是用戶端驗證錯誤在部署期間上升,並在部署結束時下降。請先發布僅新增的變更,等到每個複本都提供新的形式且過渡期已過,才移除舊的形式。
錄製內容會老化。 當正式環境主機背後的模型改變時,重播仍會持續通過,因此錄製內容的分數會逐漸偏離客戶實際看到的結果。請在每次更換模型時重新錄製,如處理工單佇列所建議,並對正式環境執行結果的抽樣進行相同的未兌現承諾檢查,因為重播無法找出錄製時的模型從未犯過的失敗。
實用原則: 在依賴某個追蹤或分數之前,先檢查它涵蓋了什麼:哪種傳輸方式、哪個模型,以及每個結構描述的哪個版本。讓 span 名稱保持有界,並預設不將引數放入 span,將儀表板使用的屬性名稱固定在具名的慣例版本上,並在每個版本抵達任何複本之前,對它執行合約測試與重播。
本章所做的決定
- 工具在記憶體內測試,使用
Client(server, raise_exceptions=True),針對欄位進行斷言,並為每種傳輸方式各寫一個迴路(loopback)測試,涵蓋行程內路徑所略過的部分。 - pytest 使用 anyio 自身的外掛執行,不使用
inline-snapshot,因為本書不固定其版本。lib.testing每個測試輸出一行,使輸出保持穩定。 - 上一版發行的 schema 即是契約。 CI 以各 schema 所宣告方言對應的驗證器來驗證新的結果,預設為 2020-12,並由
lib.contract比對 schema 的差異以供審查。 - 第 10 章錄製的
ticket-runs採重播,不再重新錄製。 描述變更之後若出現RecordingMismatch,就表示需要重新錄製。 - 主機端依據其呼叫記錄檢查回覆。 若回覆承諾退款,但沒有任何呼叫執行過退款,該回覆會交由人工處理;
lib.queue_replay中的關鍵字檢查只是經驗法則。 - 追蹤來自 SDK 的中介軟體,其所發出的屬性標示為開發中(Development)狀態的慣例,並以 HTTP 測試傳播機制。
- 工具選擇依標籤分別計分,引數另外計分,使用
expected_first_tool,做法與第 10 章的計分方式相同。訂單編號同時對照工單文字,以及顧客實際指的訂單來計分。 - 工具分三個版本逐步淘汰,每個版本都設定較短的 TTL 並更動一次清單,期限過後則回傳指出替代工具的執行錯誤。此程序是本書的做法,並非規格的規定。
本章,於本版次中
版次 2026.11.14範例於 2026年11 月 19 通過測試本章 6 個範例 個範例中有 6 個通過
譯自英文書,版次 2026.11.14。