Evaluating LLM Applications
Written with AI. Verified by people. How we make books
This book is open to contributions, so anyone can check it or correct it. Its repository · Report an error · How to fix one
Know whether an LLM application works, and whether a change made it better: test sets, checks, judges you can trust and statistics that hold up, on a documentation assistant for a small command-line tool.
Who it's for For engineers who have shipped, or are about to ship, a feature built on a model: an assistant, a question-answering service or an extraction step. They write Python and know prompts, embeddings and tests, but not statistics beyond a mean.
Read on Kindle or any e-reader
The EPUB also works in Apple Books, Kobo and most e-readers.
Send to Kindle
Sends this edition's EPUB in English.
-
Approve our address
Amazon delivers to a Kindle only from addresses you've approved, once for each service that emails it. Add ours to your Approved Personal Document E-mail List.
books@rollingedition.comOpen Amazon's Personal Document Settings. On Amazon, it's under Manage Your Content and Devices, Preferences, Personal Document Settings.
Sent. It'll appear in your Kindle library in a few minutes.
If it doesn't arrive, it's almost always the approval in step 1: check that books@rollingedition.com is on your Approved Personal Document E-mail List.
Each new edition of your saved books will go to your Kindle too.
That isn't a Kindle address. It ends in @kindle.com: find it in Amazon's Personal Document Settings.
You've sent {count} books today, the most we send for one account in a day. Try again tomorrow.
We've sent as many books as we can today. Try again tomorrow.
The book didn't send. Try again in a few minutes.
This edition is too large to email.
Send to Kindle comes with Labs.
That isn't a Kindle address. It ends in @kindle.com: find it in Amazon's Personal Document Settings.
The book didn't send. Try again in a few minutes.
Or download the EPUB and drop it on Amazon's Send to Kindle page.
Kindle is a trademark of Amazon.com, Inc.
Labs
Work through this book in the Lab
Run every example exactly as the book did, change the code to see what happens, and try your own ideas on the book's code and data. Nothing to install, and your work is saved. Opens in about 30 seconds.
Run every example exactly as the book did, change the code to see what happens, and try your own ideas on the book's code and data. Nothing to install, and your work is saved. Labs runs it for you: 40 hours a month.
Checking your hours…
Checking for your Lab…
Contents
11 chapters, about 3 hours 19 minutes to read, with 54 examples.
-
1Measure an assistant before you change it
By the end of this chapter you can measure a documentation assistant with a fixed set of questions, read the result with its interval, and say when twenty questions are too few to tell two prompts apart.
-
2Read the replies to find what to measure
By the end of this chapter you can read a set of replies from an assistant, sort what is wrong with each one into a failure mode, and say which of those failures a check can see.
-
3Build a test set you can defend, and read public benchmarks
By the end of this chapter you can build a test set whose score tells you something, check its schema and coverage with code, and say what a public benchmark score does and does not tell you about your own assistant.
-
4Write checks in code for the failures you found
By the end of this chapter you can write the format and citation checks that run on every reply at no cost, say which of the failures from chapter 2 each one can and can't see, and show where two standard overlap scores rank a wrong answer above a right one.
-
5Write a rubric a judge can apply
By the end of this chapter you can write a rubric that a judge model applies the same way to each reply, read its verdict with code that never counts anything it cannot read as a verdict, and say when a pairwise comparison or a scale is worth its cost.
-
6Measure a judge's errors against human ratings, then correct a pass rate
By the end of this chapter you can measure how a judge's verdicts differ from human ratings, split those differences into misses and false passes, and correct an observed pass rate for them, with the assumption the correction rests on written beside it.
-
7Find where a judge's verdict stops being reliable
By the end of this chapter you can test a judge for two biases that this book's data can show, position and length.
-
8Decide whether a change is better, and how many questions you need
By the end of this chapter you can say whether a difference between two versions of the assistant is more than noise, using an interval on a paired comparison.
-
9Check that answers rest on the cited docs, in the asked version
By the end of this chapter you can count whether an answer's citations support what it says, whether every claim in it rests on the sections the assistant was given, whether it is right for the version the reader asked about, and whether retrieval reached the right section on the way.
-
10Decline when the docs don't say, and measure both kinds of error
By the end of this chapter you can count both ways an assistant fails to answer from its docs, read the two rates over their own denominators, and say when a rule that declines more often is worth what it costs.
-
11Put the evaluation in CI, and compare variants on traffic
By the end of this chapter you can put the evaluation into a gate that blocks a change only when the change is measurably worse.