Cover of Evaluating LLM Applications, edition 2026.11.3, code 4V0Y-VKTV Evaluating LLM Applications 4V0Y-VKTV 3 editions since Nov 2026 Rolling Edition 2026.11.3

Evaluating LLM Applications

Edition 2026.11.3 Examples passed Nov 21, 2026 54 of 54 examples pass What changed

Written with AI. Verified by people. How we make books

This book is open to contributions, so anyone can check it or correct it. Its repository · Report an error · How to fix one

Know whether an LLM application works, and whether a change made it better: test sets, checks, judges you can trust and statistics that hold up, on a documentation assistant for a small command-line tool.

Who it's for For engineers who have shipped, or are about to ship, a feature built on a model: an assistant, a question-answering service or an extraction step. They write Python and know prompts, embeddings and tests, but not statistics beyond a mean.

Start reading

Labs

Work through this book in the Lab

Run every example exactly as the book did, change the code to see what happens, and try your own ideas on the book's code and data. Nothing to install, and your work is saved. Opens in about 30 seconds.

Run every example exactly as the book did, change the code to see what happens, and try your own ideas on the book's code and data. Nothing to install, and your work is saved. Labs runs it for you: 40 hours a month.

Checking for your Lab…

Edition history3 editions since November 2026Every edition and what it changed
2026.11.12026.11.3
2026.11.3 · Nov 21, 2026Prose fix, New content27 sections changed

Contents

11 chapters, about 3 hours 19 minutes to read, with 54 examples.

  1. 1Measure an assistant before you change it

    By the end of this chapter you can measure a documentation assistant with a fixed set of questions, read the result with its interval, and say when twenty questions are too few to tell two prompts apart.

  2. 2Read the replies to find what to measure

    By the end of this chapter you can read a set of replies from an assistant, sort what is wrong with each one into a failure mode, and say which of those failures a check can see.

  3. 3Build a test set you can defend, and read public benchmarks

    By the end of this chapter you can build a test set whose score tells you something, check its schema and coverage with code, and say what a public benchmark score does and does not tell you about your own assistant.

  4. 4Write checks in code for the failures you found

    By the end of this chapter you can write the format and citation checks that run on every reply at no cost, say which of the failures from chapter 2 each one can and can't see, and show where two standard overlap scores rank a wrong answer above a right one.

  5. 5Write a rubric a judge can apply

    By the end of this chapter you can write a rubric that a judge model applies the same way to each reply, read its verdict with code that never counts anything it cannot read as a verdict, and say when a pairwise comparison or a scale is worth its cost.

  6. 6Measure a judge's errors against human ratings, then correct a pass rate

    By the end of this chapter you can measure how a judge's verdicts differ from human ratings, split those differences into misses and false passes, and correct an observed pass rate for them, with the assumption the correction rests on written beside it.

  7. 7Find where a judge's verdict stops being reliable

    By the end of this chapter you can test a judge for two biases that this book's data can show, position and length.

  8. 8Decide whether a change is better, and how many questions you need

    By the end of this chapter you can say whether a difference between two versions of the assistant is more than noise, using an interval on a paired comparison.

  9. 9Check that answers rest on the cited docs, in the asked version

    By the end of this chapter you can count whether an answer's citations support what it says, whether every claim in it rests on the sections the assistant was given, whether it is right for the version the reader asked about, and whether retrieval reached the right section on the way.

  10. 10Decline when the docs don't say, and measure both kinds of error

    By the end of this chapter you can count both ways an assistant fails to answer from its docs, read the two rates over their own denominators, and say when a rule that declines more often is worth what it costs.

  11. 11Put the evaluation in CI, and compare variants on traffic

    By the end of this chapter you can put the evaluation into a gate that blocks a change only when the change is measurably worse.

Cite this section

Format

Every format names this edition. Its link opens the current edition, where a section’s heading says when its text last changed. Accessed and retrieved dates are the day you open this.