追踪每次运行并统计其开销
阅读约需 30分钟 分钟,8节中含4个示例。
读完本章,你应当能够让每次运行为每一步写下一条记录,并保存到后续读者可以解析的文件中;为每条记录赋予 ID,把它与所属的运行和具体的调用关联起来;并能以词元数说明一次运行的开销。这里的开销指的是计数,从不指金额。
维护智能体在临时副本中的 unitcraft 上执行运行。一次运行可能顺利结束、糟糕地结束,或者始终没有结束,而停止原因只说明它是如何结束的。以第 1 章中任务集里的 T-01 这次运行为例。它读取了正确的文件并运行了测试,随后连续发出一串格式错误的调用,直到达到步数上限。停止原因显示为 step_cap,而停止原因所说明的也仅此而已。它没有说明这些格式错误的调用有什么共同点,测试框架对每个调用做了什么处理,也没有说明每一步用了多少词元。这些答案都在各个步骤之中,所以测试框架必须保存这些步骤。
每一步的记录就是追踪记录。第 10 章的评分已经在读取存储的计数,而想知道一次运行为何停止的读者,需要看到停止之前的那些步骤。本章把追踪记录写成数据,为其 ID 规定固定的形式,并统计其中的内容。这里的每个计数都读取自某份录制记录,或由某个脚本打印得出,没有一个来自在线模型,因此这些计数是存储的计数;而那项用模型重新推导这些计数的检查尚未运行。
每个事件一行
追踪记录是一个文件,每行是一个 JSON 对象。这种格式有个名字,叫 JSON Lines,定义很简短:每行包含一个 JSON 值,文件采用 UTF-8 编码,每行以换行符结尾。该页面建议在最后一行之后也加一个换行符,但并不强制要求,同时也接受回车加换行。JSON Lines 格式规定不得包含字节顺序标记。测试框架写出每条记录时,键按顺序排列,不含空格,只使用 ASCII 字符,因此同一条记录总是生成相同的字节。这一点的意义与打印输出的意义相同:读者可以逐行比较两份追踪记录。
重放 第 1 章中在 T-01 上的运行,会以这种形式写出它的各个步骤。重放时读取的是每一步已记录的回复,所以每次运行得到的追踪记录都相同。
1"""The chapter 1 recorded run of T-01 as JSON lines: one record per step, with run-scoped IDs and the recorded token2counts. The lines are written to a file and read back, and the reader counts the empty pieces it skips."""3import json4import tempfile5from pathlib import Path67from lib.case import load_tasks8from lib.loop import run_task9from lib.models import ReplayModel10from lib.tools import CH111from lib.trace import IdCounter, StepRecorder, dumps, read_lines, step_records1213task = load_tasks(only=["T-01"])[0]14model = StepRecorder(ReplayModel("chapter-1-t01", "T-01"))15run = run_task(model, task, CH1, max_steps=CH1.step_cap, ids=IdCounter())16records = step_records(run, model)17for record in records:18 print(dumps(record))19print("records:", len(records))20print("first call ID:", records[0]["calls"][0])21print("results:", ", ".join(f"step {r['step']} {r['result']}" for r in records))22print("prompt tokens, summed over the records:", sum(r["prompt_tokens"] for r in records))23print("completion tokens, summed over the records:", sum(r["completion_tokens"] for r in records))24try:25 json.loads("") # a blank line is not a JSON value26except json.JSONDecodeError as error:27 print("blank line:", error)28with tempfile.TemporaryDirectory() as folder:29 path = Path(folder) / "trace.jsonl"30 path.write_text("".join(dumps(record) + "\n" for record in records), encoding="utf-8")31 read_back, skipped = read_lines(path.read_text(encoding="utf-8"))32print("lines read back:", len(read_back))33print("empty pieces skipped:", skipped)34assert read_back == records and skipped == 1 # the one piece after the last line feed35assert run.ended == "step_cap" and len(records) == CH1.step_cap and records[-1]["stop"] == "step_cap"36assert sum(r["prompt_tokens"] for r in records) == run.prompt_tokens37run.close()
python trace-one-run.py{"arguments":{"path":"unitcraft/format.py"},"brief":"23 lines","calls":["run-0001-c1"],"completion_tokens":22,"gate":[],"malformed":[],"offered":["finish","read_file","run_tests"],"parent":null,"prompt_tokens":501,"result":"ok","run":"run-0001","step":1,"stop":"","tool":"read_file"}
{"arguments":{},"brief":"8 failed, 22 passed","calls":["run-0001-c2"],"completion_tokens":70,"gate":[],"malformed":[],"offered":["finish","read_file","run_tests"],"parent":null,"prompt_tokens":713,"result":"ok","run":"run-0001","step":2,"stop":"","tool":"run_tests"}
{"arguments":null,"brief":"error: malformed call","calls":["run-0001-c3"],"completion_tokens":560,"gate":[],"malformed":["not JSON (Expecting ',' delimiter)"],"offered":["finish","read_file","run_tests"],"parent":null,"prompt_tokens":941,"result":"refused","run":"run-0001","step":3,"stop":"","tool":"malformed"}
{"arguments":null,"brief":"error: malformed call","calls":["run-0001-c4"],"completion_tokens":292,"gate":[],"malformed":["not JSON (Expecting ',' delimiter)"],"offered":["finish","read_file","run_tests"],"parent":null,"prompt_tokens":1528,"result":"refused","run":"run-0001","step":4,"stop":"","tool":"malformed"}
{"arguments":null,"brief":"error: malformed call","calls":["run-0001-c5"],"completion_tokens":296,"gate":[],"malformed":["not JSON (Expecting ',' delimiter)"],"offered":["finish","read_file","run_tests"],"parent":null,"prompt_tokens":1847,"result":"refused","run":"run-0001","step":5,"stop":"","tool":"malformed"}
{"arguments":null,"brief":"error: malformed call","calls":["run-0001-c6"],"completion_tokens":296,"gate":[],"malformed":["not JSON (Expecting ',' delimiter)"],"offered":["finish","read_file","run_tests"],"parent":null,"prompt_tokens":2170,"result":"refused","run":"run-0001","step":6,"stop":"","tool":"malformed"}
{"arguments":null,"brief":"error: malformed call","calls":["run-0001-c7"],"completion_tokens":296,"gate":[],"malformed":["not JSON (Expecting ',' delimiter)"],"offered":["finish","read_file","run_tests"],"parent":null,"prompt_tokens":2493,"result":"refused","run":"run-0001","step":7,"stop":"","tool":"malformed"}
{"arguments":null,"brief":"error: malformed call","calls":["run-0001-c8"],"completion_tokens":296,"gate":[],"malformed":["not JSON (Expecting ',' delimiter)"],"offered":["finish","read_file","run_tests"],"parent":null,"prompt_tokens":2816,"result":"refused","run":"run-0001","step":8,"stop":"step_cap","tool":"malformed"}
records: 8
first call ID: run-0001-c1
results: step 1 ok, step 2 ok, step 3 refused, step 4 refused, step 5 refused, step 6 refused, step 7 refused, step 8 refused
prompt tokens, summed over the records: 13009
completion tokens, summed over the records: 2128
blank line: Expecting value: line 1 column 1 (char 0)
lines read back: 8
empty pieces skipped: 1
这次运行有 8 条记录,每个步骤一条,最后一条记录带有停止原因 step_cap。每条记录包含步骤编号、工具、参数、对返回内容的一行简述、结果类别、调用 ID、提供的工具、闸门在该步骤的输出行,以及该步骤的词元计数。第一次调用的 ID 是 run-0001-c1。
后面的步骤是格式错误的调用。模型的回复不是有效的 JSON,所以测试框架没有执行该调用。它把该步骤记录为 refused,并把解析错误写入 malformed 字段。refused 涵盖测试框架没有执行的所有调用,以及工具在运行前就拒绝的调用:无法解析或被截断的回复、测试框架未提供的工具名、与工具不匹配的参数、被闸门拦住的变更,以及工具自身检查拒绝的路径。第三类 error 则不同,它表示工具已运行并报告了失败。把 refused 当作工具错误,会让读者去一个从未被调用的工具里查找缺陷。
对所有记录求和,重放的这次运行使用了 13009 个提示词元和 2128 个补全词元。这些是录制时为每个步骤存下的计数,而不是重放期间取得的计数。
这个文件有一个陷阱,示例在结尾处展示了它。JSON Lines 不允许空行,因为空行不是一个值。解析空行会失败,Python 将其报告为 Expecting value: line 1 column 1 (char 0)。这些记录被写入文件,再通过读取器读回。按换行符拆分文本时,最后一个换行符之后会留下一个空片段,所以读取器有意跳过该片段并对其计数。这里读回了 8 条记录,跳过了 1 个片段。不跳过它的读取器会在每个文件的末尾停止。
工作准则: 追踪记录的读取器只在有意为之时才跳过空行,并对跳过的行数计数。否则它会在第一个空行处停止。写入方绝不在记录之间添加空行。
没有时钟的 ID
追踪中的一个步骤,必须能够按所属的运行、所应答的调用以及在顺序中的位置重新找到。追踪的 ID 只来自调用方传入的计数器,不来自其他任何来源。一次运行的 ID 是 run-0001,从 1 开始编号。一次工具调用的 ID 是其所属运行的 ID 加上 c 和一个数字,因此该运行的第一次调用是 run-0001-c1,与重放运行的第一条记录所带的 ID 相同。步骤从 1 开始,按发生的先后顺序编号。
没有任何记录带有时间。时间戳在每次运行时都会不同,任何显示时间戳的示例,其打印输出也会不同。每个示例都要与其录制时打印的输出相比较,因此时间戳会使这一比较在每次运行时都失败。追踪中保存的是计数、简短的结果和 ID,不含任何来自时钟的内容。
ID 还有第二个重要原因。运行 ID 和调用 ID 会写入测试框架发送给模型的请求,所以只有当重放对运行的编号方式与录制时一致,重放发出的请求才会与录制的请求相同。某个任务的重放代码会从该任务在录制中所处的位置接续其计数器,这就是 子运行示例 按录制的顺序而不是按自身的起点为其运行编号的原因。
工作准则: 运行和调用的编号应来自传入的计数器,绝不来自时钟或随机数生成器。在不同运行之间发生变化的 ID 会改变请求,而该运行的重放会在第一个无法与录制匹配的请求处停止。
作为计数的词元
轨迹中的每条记录都带有两个词元计数,按第 1 章所定义的、固定模型的词元来计。提示计数是模型读入的词元数,补全计数是模型写出的词元数。录制在运行时取得这两个计数。它把每一步的计数与该步的回复一起存入 replies.jsonl,并把每个任务的总数存入其清单。回放读取已存的计数,不再做词元化,因此回放不可能产生录制未曾写下的计数。
按任务以及对整个任务集汇总已存的各步计数,是下一个示例的工作。它把每个任务的汇总与清单为该任务存储的总数相核对,并把任务集的提示汇总与这些任务总数之和相核对。
1"""The token counts of the task-set recording, summed per task and for the set, read from the stored counts. Replay2never tokenizes, so these are the counts the recording wrote; the live check against a tokenizer is not run."""3from lib.models import load_recording4from lib.trace import OTEL_TOKEN_FIELDS56manifest, rows = load_recording("task-set") # rows: one per step, from replies.jsonl; the manifest has task totals7sums: dict[str, list[int]] = {}8for row in rows:9 total = sums.setdefault(row["run"], [0, 0, 0])10 total[0] += row["prompt_tokens"]11 total[1] += row["completion_tokens"]12 total[2] += 1 # each row is one step, so the third sum counts steps13for task in sorted(manifest["tasks"], key=lambda t: t["task"]):14 prompt, completion, steps = sums[task["task"]]15 print(f"{task['task']} prompt tokens: {prompt} completion tokens: {completion} steps: {steps}")16 assert (prompt, completion, steps) == (task["prompt_tokens"], task["completion_tokens"], task["steps"])17set_prompt = sum(total[0] for total in sums.values())18set_completion = sum(total[1] for total in sums.values())19print("set tasks:", len(sums))20print("set prompt tokens:", set_prompt)21print("set completion tokens:", set_completion)22print("live count check: not run")23for field, name in sorted(OTEL_TOKEN_FIELDS.items()): # the mapping table; the book's own names stay in the trace24 print(f"field {field} would map to {name} (data only; OpenTelemetry is not used)")25assert set_prompt == sum(t["prompt_tokens"] for t in manifest["tasks"])
python token-counts-from-recording.pyT-01 prompt tokens: 44836 completion tokens: 2025 steps: 16
T-02 prompt tokens: 43904 completion tokens: 2059 steps: 16
T-03 prompt tokens: 27562 completion tokens: 1463 steps: 12
T-04 prompt tokens: 8809 completion tokens: 1054 steps: 5
T-05 prompt tokens: 8494 completion tokens: 906 steps: 5
T-06 prompt tokens: 55109 completion tokens: 4670 steps: 16
T-07 prompt tokens: 24989 completion tokens: 3468 steps: 9
T-08 prompt tokens: 13713 completion tokens: 1154 steps: 7
T-09 prompt tokens: 10324 completion tokens: 701 steps: 6
T-10 prompt tokens: 51056 completion tokens: 3549 steps: 16
T-11 prompt tokens: 33951 completion tokens: 817 steps: 16
T-12 prompt tokens: 30787 completion tokens: 1617 steps: 12
T-13 prompt tokens: 49249 completion tokens: 942 steps: 16
T-14 prompt tokens: 12651 completion tokens: 972 steps: 7
T-15 prompt tokens: 20409 completion tokens: 1254 steps: 10
T-16 prompt tokens: 57696 completion tokens: 4621 steps: 16
T-17 prompt tokens: 17601 completion tokens: 1511 steps: 9
T-18 prompt tokens: 5122 completion tokens: 304 steps: 4
T-19 prompt tokens: 22713 completion tokens: 2154 steps: 9
T-20 prompt tokens: 33630 completion tokens: 2065 steps: 13
set tasks: 20
set prompt tokens: 572605
set completion tokens: 37306
live count check: not run
field completion_tokens would map to gen_ai.usage.output_tokens (data only; OpenTelemetry is not used)
field prompt_tokens would map to gen_ai.usage.input_tokens (data only; OpenTelemetry is not used)
每个任务的汇总都与其清单一致,所以下面任务集的总数就是录制自身的计数。该任务集有 20 个任务,合计使用了 572605 个提示词元和 37306 个补全词元。输出中的实时计数核对会用固定模型对已录制的提示做词元化,其报告为 not run。
该库的源码展示了这类计数是如何产生的。llama-cpp-python 的源码是按录制所固定的版本阅读的,并未在此运行。它的非流式补全会报告提示的词元列表长度和补全的词元列表长度,并以两者之和作为总数(llama.py at v0.3.36)。它的 tokenize 方法接收 UTF-8 字节,返回由整数词元 ID 组成的列表。除非被告知不要添加,否则它会加上一个序列开始词元,因此提示的计数既取决于模型的词表,也取决于这项设置。
来自不同 tokenizers 的计数不可比较。这是经验法则,而非测量结果:词元是某一词表的单位,同一段文本在另一词表下可能被切分成不同数量的单位。读者若要比较两个计数,必须先核对模型文件和设置是否一致。
录制的清单存储了模型文件及其 SHA-256,但没有存储影响提示计数的设置:库所应用的聊天模板,以及序列开始词元的设置。录制对这两项都使用了库的默认值,而本书尚未将它们写下来。
**工作准则:**存储每个计数时,要同时存储它所出自的模型文件,以及任何会改变它的设置,例如序列开始词元。不要比较来自不同分词器的计数。没有经过任何核对重新推导的计数,应报告为已存计数,绝不能报告为实时计数。
子运行的记录位于其父调用之下
第 8 章把一个子任务交给子循环,并取回一份摘要和一个停止原因。它还把子循环自己的记录留在子循环处,这样父运行的摘要就不是所发生之事的唯一痕迹。本节在追踪记录中兑现这一承诺。子运行的各个步骤作为它们自己的运行来写入,带有自己的运行 ID 和自己的调用 ID。它的每条记录都带有parent,即启动它的那次调用的 ID。子运行的记录紧接在启动它们的父调用之后打印。
在这里,父运行针对 T-04 启动一个子运行,该任务要求提供一个英寸单位。子运行所记录的运行以一次会修改文件的 finish 结束。一个脚本化的人对这次修改说了不,于是门控拦住了这次 finish,子运行带着一个问题交还控制。
1"""A parent starts a child on T-04, whose recorded run ends with a finish that changes files. The gate holds that finish2because the person says no, so the child hands back with ask_user. The child's records sit under the parent's start call;3the parent is scripted, so its records carry zero tokens."""4from lib.case import load_tasks5from lib.chapter1 import call6from lib.models import ReplayModel, load_recording7from lib.permissions import Gate, library_copy, replay_ids, scripted8from lib.subagents import START_CHILD9from lib.tasks import snapshot10from lib.tools import V111from lib.trace import StepRecorder, child_under_parent, dumps1213task = load_tasks(only=["T-04"])[0]14manifest, _ = load_recording("task-set")15stored = next(t for t in manifest["tasks"] if t["task"] == "T-04")16with library_copy() as library:17 before = snapshot(library, "unitcraft")18 gate = Gate(library, scripted((False, "not this change"))) # the scripted person says no, with this reason19 child_model = StepRecorder(ReplayModel("task-set", "T-04"), gate)20 replies = [call(START_CHILD, task_id="T-04"), call("ask_user", question="The child's change was held. Pass it on?")]21 parent, child, records = child_under_parent(task, replies, child_model, gate, library, replay_ids("T-04"))22 for record in records:23 print(dumps(record))24 print("child budget:", V1.step_cap)25 print("child ended:", child.handoff.ended)26 print("gate refusals:", gate.counts["refused"])27 print("parent ended:", parent.ended)28 print("copy unchanged by the held finish:", before == snapshot(library, "unitcraft"))29 child_records = [r for r in records if r["run"] == child.run.id]30 parent_records = [r for r in records if r["run"] != child.run.id]31 print("child run:", child.run.id)32 print("parent call of the child:", child_records[0]["parent"])33 print("held step:", next(r["step"] for r in child_records if r["result"] == "refused"))34 print("parent tokens, summed:", sum(r["prompt_tokens"] + r["completion_tokens"] for r in parent_records))35 child_prompt = sum(r["prompt_tokens"] for r in child_records)36 child_completion = sum(r["completion_tokens"] for r in child_records)37 print("child prompt tokens, summed:", child_prompt)38 print("child completion tokens, summed:", child_completion)39 assert child.handoff.ended == "ask_user" and sum(r["result"] == "refused" for r in records) == 140 assert parent.ended == "ask_user" and before == snapshot(library, "unitcraft")41 assert (child_prompt, child_completion) == (stored["prompt_tokens"], stored["completion_tokens"])42 parent.close()
python child-trace-under-parent.py{"arguments":{"task_id":"T-04"},"brief":"child ask_user at step 5","calls":["run-0001-c1"],"completion_tokens":0,"gate":[],"malformed":[],"offered":["ask_user","start_child"],"parent":null,"prompt_tokens":0,"result":"ok","run":"run-0001","step":1,"stop":"","tool":"start_child"}
{"arguments":{},"brief":"25 files","calls":["run-0004-c1"],"completion_tokens":51,"gate":[],"malformed":[],"offered":["ask_user","edit_file","finish","list_files","read_file","run_tests","write_file"],"parent":"run-0001-c1","prompt_tokens":1049,"result":"ok","run":"run-0004","step":1,"stop":"","tool":"list_files"}
{"arguments":{"path":"unitcraft/units.py"},"brief":"36 lines","calls":["run-0004-c2"],"completion_tokens":108,"gate":[],"malformed":[],"offered":["ask_user","edit_file","finish","list_files","read_file","run_tests","write_file"],"parent":"run-0001-c1","prompt_tokens":1253,"result":"ok","run":"run-0004","step":2,"stop":"","tool":"read_file"}
{"arguments":{"new":"Unit(\"m\", \"length\", Decimal(1)),\n Unit(\"km\", \"length\", Decimal(1000)),\n Unit(\"cm\", \"length\", Decimal(\"0.01\")),\n Unit(\"mm\", \"length\", Decimal(\"0.001\")),\n Unit(\"ft\", \"length\", Decimal(\"0.3048\")),\n Unit(\"in\", \"length\", Decimal(\"0.0254\")),\n Unit(\"kg\", \"mass\", Decimal(1)),\n Unit(\"g\", \"mass\", Decimal(\"0.001\")),\n Unit(\"lb\", \"mass\", Decimal(\"0.45359237\")),\n Unit(\"s\", \"time\", Decimal(1)),\n Unit(\"min\", \"time\", Decimal(60)),\n Unit(\"h\", \"time\", Decimal(3600)),\n Unit(\"C\", \"temperature\", None),\n Unit(\"F\", \"temperature\", None),\n Unit(\"K\", \"temperature\", None),","old":"Unit(\"m\", \"length\", Decimal(1)),\n Unit(\"km\", \"length\", Decimal(1000)),\n Unit(\"cm\", \"length\", Decimal(\"0.01\")),\n Unit(\"mm\", \"length\", Decimal(\"0.001\")),\n Unit(\"ft\", \"length\", Decimal(\"0.3048\")),\n Unit(\"kg\", \"mass\", Decimal(1)),\n Unit(\"g\", \"mass\", Decimal(\"0.001\")),\n Unit(\"lb\", \"mass\", Decimal(\"0.45359237\")),\n Unit(\"s\", \"time\", Decimal(1)),\n Unit(\"min\", \"time\", Decimal(60)),\n Unit(\"h\", \"time\", Decimal(3600)),\n Unit(\"C\", \"temperature\", None),\n Unit(\"F\", \"temperature\", None),\n Unit(\"K\", \"temperature\", None),","path":"unitcraft/units.py"},"brief":"edited unitcraft/units.py","calls":["run-0004-c3"],"completion_tokens":570,"gate":[],"malformed":[],"offered":["ask_user","edit_file","finish","list_files","read_file","run_tests","write_file"],"parent":"run-0001-c1","prompt_tokens":1686,"result":"ok","run":"run-0004","step":3,"stop":"","tool":"edit_file"}
{"arguments":{},"brief":"8 failed, 22 passed","calls":["run-0004-c4"],"completion_tokens":111,"gate":[],"malformed":[],"offered":["ask_user","edit_file","finish","list_files","read_file","run_tests","write_file"],"parent":"run-0001-c1","prompt_tokens":2276,"result":"ok","run":"run-0004","step":4,"stop":"","tool":"run_tests"}
{"arguments":{"summary":"Added the inch unit ('in') with a factor of 0.0254 to the units dictionary in unitcraft/units.py. This allows parse_quantity('2 in') to work correctly and enables the convert function to handle inch-to-meter conversions. The test suite failures indicate other issues in the codebase that are outside the scope of this specific task."},"brief":"p-1 was not applied: the person said no: not this change","calls":["run-0004-c5"],"completion_tokens":214,"gate":["p-1 1 files, diff 06ee73fd984f: asked","p-1 answer no, bound to p-1 and 06ee73fd984f","p-1 not applied: the person said no: not this change"],"malformed":[],"offered":["ask_user","edit_file","finish","list_files","read_file","run_tests","write_file"],"parent":"run-0001-c1","prompt_tokens":2545,"result":"refused","run":"run-0004","step":5,"stop":"ask_user","tool":"finish"}
{"arguments":{"question":"The child's change was held. Pass it on?"},"brief":"ask_user","calls":["run-0001-c2"],"completion_tokens":0,"gate":[],"malformed":[],"offered":["ask_user","start_child"],"parent":null,"prompt_tokens":0,"result":"ok","run":"run-0001","step":2,"stop":"ask_user","tool":"ask_user"}
child budget: 16
child ended: ask_user
gate refusals: 1
parent ended: ask_user
copy unchanged by the held finish: True
child run: run-0004
parent call of the child: run-0001-c1
held step: 5
parent tokens, summed: 0
child prompt tokens, summed: 8809
child completion tokens, summed: 1054
子运行采用录制数据给 T-04 的编号,即其中的第四次运行,因此它的请求与录制的请求相匹配。该运行为run-0004,它的第一条记录在parent中指明了父运行的调用:run-0001-c1。
子运行的记录带有录制数据为 T-04 存储的计数,即8809个提示词 token 和1054个补全 token,脚本会对照清单对它们进行断言。父运行是脚本化的,所以它的记录不带 token:0。
那个人的“不”正是子运行结束之处。门控在步骤5拦住了这次 finish,这是子运行的最后一条记录,子运行转而交还一个问题。
子运行以ask_user停止,门控拒绝了1次调用。这一停止来自门控,而不是来自16步的预算。同样是脚本化的父运行向那个人提出了它自己的问题,并以ask_user停止。运行结束后,库副本保持不变,这正是门控的全部意义所在:被拦住的修改从未到达库中。
这个示例没有展示耗尽预算的子运行。第 8 章讨论了那种情况,它的停止原因来自预算,而不是来自被拦住的修改。
工作准则: 给每个子运行各自的运行 ID 和各自的记录,每条记录的parent都设为启动它的那次调用。报告一次父运行的成本时,要加上其子运行的计数,并说明你这样做了。这份追踪记录把它们分开打印,没有把其中任何一项计入父运行。
成本以计数衡量,而非金额
有了按任务汇总的已存计数,一次运行的成本就是三个数:步数、工具调用次数和词元数。本书只报告这三项,不报告其他内容。价格表会过时,以美元计的数字在第一次价格变动后就不再正确,因此本书不会把计数乘以价格。需要金额的读者,可以把计数乘以其服务商当前收取的价格,而本书的数字不会因价格变动而失效。
按各次运行的结束方式对已存计数分组,可以把达到步数上限的运行与其他运行并列,这样每种终止原因的成本都显示在同一份输出中。
1"""The cost of the task-set runs as counts only: steps, tool calls and tokens, summed per way a run ended. The cost is2tokens, never money, and the tokens are the recording's stored counts; the groups are sorted, so the output is stable."""3from lib.models import load_recording45manifest, rows = load_recording("task-set")6tasks = sorted(manifest["tasks"], key=lambda t: (t["task"], t["kind"]))7for ending in sorted({t["ended"] for t in tasks}):8 group = [t for t in tasks if t["ended"] == ending]9 print(f"ended {ending}: runs {len(group)}, done {sum(t['done'] for t in group)}, "10 f"steps {sum(t['steps'] for t in group)}, tool calls {sum(t['tool_calls'] for t in group)}, "11 f"prompt tokens {sum(t['prompt_tokens'] for t in group)}, "12 f"completion tokens {sum(t['completion_tokens'] for t in group)}")13 print(" tasks:", ", ".join(t["task"] for t in group))14 print(" done:", ", ".join(t["task"] for t in group if t["done"]) or "none")15print(f"all: runs {len(tasks)}, done {sum(t['done'] for t in tasks)}, steps {sum(t['steps'] for t in tasks)}, "16 f"tool calls {sum(t['tool_calls'] for t in tasks)}, prompt tokens {sum(t['prompt_tokens'] for t in tasks)}, "17 f"completion tokens {sum(t['completion_tokens'] for t in tasks)}")18capped = [t for t in tasks if t["ended"] == "step_cap"]19print(f"step-capped runs done: {sum(t['done'] for t in capped)} of {len(capped)}")20print(f"step-capped runs, steps: {sum(t['steps'] for t in capped)} of {sum(t['steps'] for t in tasks)}")21print(f"step-capped runs, prompt tokens: {sum(t['prompt_tokens'] for t in capped)} of "22 f"{sum(t['prompt_tokens'] for t in tasks)}")23prompts: dict[str, list[int]] = {}24for row in sorted(rows, key=lambda r: (r["run"], r["step"])): # the stored step rows, in step order25 prompts.setdefault(row["run"], []).append(row["prompt_tokens"])26rising = [t["task"] for t in capped if all(b > a for a, b in zip(prompts[t["task"]], prompts[t["task"]][1:]))]27print(f"step-capped runs whose prompt count rises at every step: {len(rising)} of {len(capped)}")28assert all(t["tool_calls"] == t["steps"] for t in tasks), "one tool call per step in this recording"
python cost-as-counts.pyended ask_user: runs 7, done 1, steps 69, tool calls 69, prompt tokens 157824, completion tokens 10368
tasks: T-03, T-12, T-15, T-17, T-18, T-19, T-20
done: T-18
ended finish: runs 6, done 4, steps 39, tool calls 39, prompt tokens 78980, completion tokens 8255
tasks: T-04, T-05, T-07, T-08, T-09, T-14
done: T-04, T-05, T-09, T-14
ended step_cap: runs 7, done 0, steps 112, tool calls 112, prompt tokens 335801, completion tokens 18683
tasks: T-01, T-02, T-06, T-10, T-11, T-13, T-16
done: none
all: runs 20, done 5, steps 220, tool calls 220, prompt tokens 572605, completion tokens 37306
step-capped runs done: 0 of 7
step-capped runs, steps: 112 of 220
step-capped runs, prompt tokens: 335801 of 572605
step-capped runs whose prompt count rises at every step: 7 of 7
达到步数上限的运行共用了 112 of 220 步和 335801 of 572605 个提示词元。达到步数上限的运行所在的一行为 runs 7, done 0, steps 112, tool calls 112, prompt tokens 335801, completion tokens 18683。所有运行所在的一行为 runs 20, done 5, steps 220, tool calls 220, prompt tokens 572605, completion tokens 37306,它涵盖了上面的每一个分组。另有一条断言检查记录是否正确:每次运行每一步都只进行一次工具调用,因此工具调用次数与步数是同一个数。
达到步数上限的运行没有一次是完成的:计数为 0 of 7。达到步数上限的运行既没有以 finish 结束,也没有移交,因此无论其文件中有什么内容,零这个结果都由完成规则得出。ask_user 组中唯一的一次完成是 T-18,它的计数来自路径匹配(第 10 章)。零这个数并不说明任何一次达到步数上限的运行离完成有多近,本章也不把它当作关于这些任务的结论。
判断,而非测量:达到步数上限的那一行,是那些耗尽词元却没有完成的运行所付出的成本。针对运行总词元数设置词元预算(而不是第 5 章中针对每次请求的提示词预算),本可以让这些运行更早停止。其代价是可能会停掉一次本来会在后面某一步完成的运行,而这份记录无法显示这种可能性,因为每次达到步数上限的运行都停在上限处,没有写入更后面的步骤。使成本增长的是提示词计数:循环在每一步都会重新发送整份记录。统计提示词计数在每一步都上升的、达到步数上限的运行,结果为 7 of 7。步数预算只计算步数,因此对于较长的运行,它对成本的说明不如词元充分。若设置了词元预算,测试框架应当以其自身的终止原因和自身的计数来结束运行,以便区分由预算造成的截断和由步数上限造成的截断。
第 12 章将在这些相同的单位下比较智能体与固定流水线,而只有每次运行都以相同方式报告这些单位,它们才有用处。
**工作准则:**以每次运行的步数、工具调用次数和词元数报告成本,按结束方式分组,并把各组并列打印。不引用金额。把达到步数上限的组理解为未完成而耗费的词元,在第二种度量给出不同结论之前,不要把它当作关于任务的证据。
保留自己的字段名,在边界处做映射
本书记录中的词元字段是 prompt_tokens 和 completion_tokens,它们是本书自己的命名。OpenTelemetry 项目发布了面向生成式 AI 系统遥测数据的约定。其 GenAI span 页面在语义约定的 v1.37.0 修订版中,将这两项计数分别命名为 gen_ai.usage.input_tokens 和 gen_ai.usage.output_tokens(语义约定,GenAI spans)。在该修订版中,此页面将自身状态标记为“Development”(开发中),因此这些名称可能会变化。本书并不使用 OpenTelemetry。词元计数示例把这两组名称与本书的字段并排打印出来,仅作为数据呈现,本章不对 OpenTelemetry 工具如何读取追踪记录作任何断言。
本书保留自己的名称,出于实际的考虑。若读者把追踪记录导出到其他工具,只需要一张映射表,而依赖该标准的也只有这张表。标准变化时,本书的字段名不会随之改变。映射不会给计数增添任何内容。提示词计数与补全计数在两套名称下是相同的数值,追踪记录中的各条记录也保持写入时的原样。
工作准则: 在自己的项目中,按字段所统计的内容为追踪记录的字段命名。在边界处用一张表把它们映射到标准的名称,并在表旁写明所依据的标准修订版。标准的修订版变化时,重新核对这张表。
哪里会出错,以及如何发现
第一个错误出在 result 字段。refused 的调用从未运行过,如果读者把 refused 当成工具的错误,就会找错地方。你可以在记录中发现这一点:result 为 refused,malformed 给出解析错误,工具字段显示的是 malformed 而不是某个工具的名称,闸门的各行为空。这类步骤的简述以 error: 开头,读起来像是工具的失败,所以应当去读 result 和 malformed,而不要读简述。
第二个错误是为调试而添加的时间戳。它使每次运行的打印输出都不同,因此在改动之后第一次运行时,示例与其记录的输出的比对就会失败。解决办法是去掉时间戳,改用步骤编号。步骤编号在每次运行中都相同,足以让各行排出先后顺序。
第三个错误是第一节中的空行。读取器若把空行交给 JSON 解析器,就会在这里失败,而文件本身完好,这种失败看起来却像文件已损坏。你可以在解析因空字符串而失败时发现它。解决办法是制定一条规则,有意跳过空的片段并对跳过的次数计数,就像该示例所做的那样。
第四个错误是把追踪当作控制手段。追踪记录的是闸门和沙箱所作的决定。它本身不决定任何事,也无法撤销已经运行的调用。子示例中被扣住的结束操作,是由闸门扣住的,闸门在第 6 章中说明。对命令的限制来自第 7 章中的沙箱。在一次糟糕的运行之后阅读追踪,只能告诉你运行框架允许了什么,仅此而已。
第五点是本章的一个局限。本章只追踪内置工具。这里没有追踪任何 MCP 服务器,所以本章不对 MCP 调用在追踪中如何呈现作任何断言。第 9 章讲解工具本身。
工作准则: 在依据追踪作出决定之前,先读它的 result 和 stop 字段,再把它打印出的各项总和与存储的总数核对。与总数不符的追踪,其写入器存在缺陷,它所记录的那次运行也就还不能算作关于智能体的结论。
本章的决定
- 每次运行按步骤顺序为每一步写一行 JSON,键已排序,不含空格,仅使用 ASCII 字符。不写空行。
- ID 来自调用方传入的计数器:运行记为
run-0001,调用记为run-0001-c1。追踪记录中不引入时钟值或随机值。 - 词元计数采用录制时存储的数值。不运行实时计数检查,本章在相关之处会说明这一点。
- 子运行的记录带有自己的运行 ID 和一个
parent字段。只有报告明确说明时,父运行的成本才包含其子运行的计数。 - 成本由步骤数、工具调用数和词元数构成,并按各次运行的结束方式分组。本书不给出金额。
- 本书的字段名是自定的。OpenTelemetry 的名称仅作为数据打印出来,并不使用。
一条追踪记录每一步占一行,它换来的是对一个问题的回答,而停止原因回答不了这个问题:哪一步出了错,测试框架拒绝了什么,每一步用了多少词元。停止原因、得分和成本,其可核查程度只取决于统计它们所依据的那些行。请保留这些行,保留其中的 ID,并保留录制时写下的计数。
工作准则: 对你所报告的每一次运行,每一步保留一行,附带其 ID 和存储的计数。凡无法追溯到这些行的数字,一律不报告。
本章,在本版次中
版次 2026.11.2示例于 2026年11月20日 通过本章 4个示例 个示例中有 4 个通过
译自英文版图书,版次 2026.11.2。