Cover of Small Open Models, edition 2026.11.9, code ND0S-78AT Small Open Models ND0S-78AT 9 editions since Nov 2026 Rolling Edition 2026.11.9

Small Open Models

Edition 2026.11.9 Examples passed Nov 20, 2026 36 of 36 examples pass What changed

Written with AI. Verified by people. How we make books

This book is open to contributions, so anyone can check it or correct it. Its repository · Report an error · How to fix one

How to replace a hosted model with one you run for a single job, and when it is the wrong choice: measuring the task, choosing a base model, prompting, LoRA fine-tuning, distillation, quantization, CPU serving, versions and hand-offs to people, on a bank's support messages from BANKING77.

Who it's for For engineers and data scientists who already call a hosted model's API for one narrow, repeated job, such as routing support messages, and want to know whether a model they run themselves can do it. You need Python and some use of an LLM API, not a transformer's internals.

Start reading

Labs

Work through this book in the Lab

Run every example exactly as the book did, change the code to see what happens, and try your own ideas on the book's code and data. Nothing to install, and your work is saved. Opens in about 30 seconds.

Run every example exactly as the book did, change the code to see what happens, and try your own ideas on the book's code and data. Nothing to install, and your work is saved. Labs runs it for you: 40 hours a month.

Checking for your Lab…

Edition history9 editions since November 2026Every edition and what it changed
2026.11.12026.11.9
2026.11.9 · Nov 21, 2026Prose fix, New content19 sections changed

Contents

10 chapters, about 2 hours 50 minutes to read, with 36 examples.

  1. 1Build a router for the bank's inbox that hands off when unsure

    By the end of this chapter, you can build a router that returns a team, a confidence and a hand-off flag for one message, score it on a fixed sample by three measures, and see why an untrained small model, prompted as it stands, can't be the router's first version.

  2. 2Measure the routing task before you choose a model

    By the end of this chapter, you can say what each of the book's samples is for, compute the three measures the router is scored on with their denominators, and read a baseline that was recorded once and is reused for every comparison.

  3. 3Choose a base model by counting what one message costs

    By the end of this chapter, you can name the two base models this book keeps, load each one at its pinned revision, and count what one routing message costs in tokens and parameters before you commit to a model.

  4. 4Prompt a small model and constrain its answers to the teams

    By the end of this chapter, you can write a router prompt for a small open model, constrain its answer so that it can only be a team name, and measure how many of those answers are right.

  5. 5Fine-tune a classification head with LoRA

    By the end of this chapter, you can train a classification head on the labelled messages with LoRA, count the parameters that training changes, choose a hand-off threshold on the dev split by a stated rule, and record that threshold beside the results it produced.

  6. 6Distill a larger model's answers into a small one

    By the end of this chapter, you can train a small student on the labels a larger teacher gives your training rows, measure how often those labels agree with the gold labels, and decide from the same messages whether the student is worth using beside a student trained on gold labels.

  7. 7Quantize the router and measure what changes

    By the end of this chapter, you can store the fine-tuned router's backbone weights as 8-bit integers in one call, count the predictions that change on a fixed sample, and say what that count does and does not show.

  8. 8Serve the router on a CPU, with the hand-off inside the route

    By the end of this chapter, you can serve the router as one POST /route endpoint that returns a team, a confidence and a hand-off flag for each message.

  9. 9Version the router, watch its hand-offs and fall back to people

    By the end of this chapter, you can name a router by the parts that make its answers, check those parts before the route loads them, count the hand-off rate for each window of messages against a stated baseline, and say where each handed-off message goes.

  10. 10When a small model is the wrong choice

    By the end of this chapter, you can say where a small router still loses to the stand-in and where a hosted model is expected to do better, count each option's monthly cost from inputs you state with their dates, find the volume at which a machine of your own becomes the cheaper route, and name the requirements that decide the question before any cost does.

Cite this section

Format

Every format names this edition. Its link opens the current edition, where a section’s heading says when its text last changed. Accessed and retrieved dates are the day you open this.