Small Open Models
Written with AI. Verified by people. How we make books
This book is open to contributions, so anyone can check it or correct it. Its repository · Report an error · How to fix one
How to replace a hosted model with one you run for a single job, and when it is the wrong choice: measuring the task, choosing a base model, prompting, LoRA fine-tuning, distillation, quantization, CPU serving, versions and hand-offs to people, on a bank's support messages from BANKING77.
Who it's for For engineers and data scientists who already call a hosted model's API for one narrow, repeated job, such as routing support messages, and want to know whether a model they run themselves can do it. You need Python and some use of an LLM API, not a transformer's internals.
Read on Kindle or any e-reader
The EPUB also works in Apple Books, Kobo and most e-readers.
Send to Kindle
Sends this edition's EPUB in English.
-
Approve our address
Amazon delivers to a Kindle only from addresses you've approved, once for each service that emails it. Add ours to your Approved Personal Document E-mail List.
books@rollingedition.comOpen Amazon's Personal Document Settings. On Amazon, it's under Manage Your Content and Devices, Preferences, Personal Document Settings.
Sent. It'll appear in your Kindle library in a few minutes.
If it doesn't arrive, it's almost always the approval in step 1: check that books@rollingedition.com is on your Approved Personal Document E-mail List.
Each new edition of your saved books will go to your Kindle too.
That isn't a Kindle address. It ends in @kindle.com: find it in Amazon's Personal Document Settings.
You've sent {count} books today, the most we send for one account in a day. Try again tomorrow.
We've sent as many books as we can today. Try again tomorrow.
The book didn't send. Try again in a few minutes.
This edition is too large to email.
Send to Kindle comes with Labs.
That isn't a Kindle address. It ends in @kindle.com: find it in Amazon's Personal Document Settings.
The book didn't send. Try again in a few minutes.
Or download the EPUB and drop it on Amazon's Send to Kindle page.
Kindle is a trademark of Amazon.com, Inc.
Labs
Work through this book in the Lab
Run every example exactly as the book did, change the code to see what happens, and try your own ideas on the book's code and data. Nothing to install, and your work is saved. Opens in about 30 seconds.
Run every example exactly as the book did, change the code to see what happens, and try your own ideas on the book's code and data. Nothing to install, and your work is saved. Labs runs it for you: 40 hours a month.
Checking your hours…
Checking for your Lab…
Contents
10 chapters, about 2 hours 50 minutes to read, with 36 examples.
-
1Build a router for the bank's inbox that hands off when unsure
By the end of this chapter, you can build a router that returns a team, a confidence and a hand-off flag for one message, score it on a fixed sample by three measures, and see why an untrained small model, prompted as it stands, can't be the router's first version.
-
2Measure the routing task before you choose a model
By the end of this chapter, you can say what each of the book's samples is for, compute the three measures the router is scored on with their denominators, and read a baseline that was recorded once and is reused for every comparison.
-
3Choose a base model by counting what one message costs
By the end of this chapter, you can name the two base models this book keeps, load each one at its pinned revision, and count what one routing message costs in tokens and parameters before you commit to a model.
-
4Prompt a small model and constrain its answers to the teams
By the end of this chapter, you can write a router prompt for a small open model, constrain its answer so that it can only be a team name, and measure how many of those answers are right.
-
5Fine-tune a classification head with LoRA
By the end of this chapter, you can train a classification head on the labelled messages with LoRA, count the parameters that training changes, choose a hand-off threshold on the dev split by a stated rule, and record that threshold beside the results it produced.
-
6Distill a larger model's answers into a small one
By the end of this chapter, you can train a small student on the labels a larger teacher gives your training rows, measure how often those labels agree with the gold labels, and decide from the same messages whether the student is worth using beside a student trained on gold labels.
-
7Quantize the router and measure what changes
By the end of this chapter, you can store the fine-tuned router's backbone weights as 8-bit integers in one call, count the predictions that change on a fixed sample, and say what that count does and does not show.
-
8Serve the router on a CPU, with the hand-off inside the route
By the end of this chapter, you can serve the router as one
POST /routeendpoint that returns a team, a confidence and a hand-off flag for each message. -
9Version the router, watch its hand-offs and fall back to people
By the end of this chapter, you can name a router by the parts that make its answers, check those parts before the route loads them, count the hand-off rate for each window of messages against a stated baseline, and say where each handed-off message goes.
-
10When a small model is the wrong choice
By the end of this chapter, you can say where a small router still loses to the stand-in and where a hosted model is expected to do better, count each option's monthly cost from inputs you state with their dates, find the volume at which a machine of your own becomes the cheaper route, and name the requirements that decide the question before any cost does.