Keep the forecast running
About 23 minutes to read, with 5 examples in 7 sections.
Written with AI. Verified by people. How we make books
This book is open to contributions, so anyone can check it or correct it. Its repository · Report an error · How to fix one
Labs
Learn this chapter by doing it
Run every example exactly as the book did, change the code to see what happens, and try your own ideas on the book's code and data. Nothing to install, and your work is saved. Opens in about 30 seconds.
Run every example exactly as the book did, change the code to see what happens, and try your own ideas on the book's code and data. Nothing to install, and your work is saved. Labs runs it for you: 40 hours a month.
Your browser can translate this page from its menu, and the code stays as the book has it.
Or read it with your AI assistant:
Once a forecast runs every day, the question changes from whether it scored well to whether it still holds on the day it's used. This chapter builds the loop that answers that question. You'll refit a forecast on a schedule, log each day's forecast before its readings exist, check that the log used only what had arrived, score the readings as they come in, flag a day whose error is out of line with the backtest, and say when a model should be retired.
The stake is the next buying round. The supplier commits to power for the next day from the forecast, so a forecast that has gone wrong without anyone noticing costs money in that purchase. This chapter does not reduce that risk to zero. It makes the risk visible on the day it appears, using only what the supplier knew at the time.
The earlier chapters checked the supplier's forecast on past data. Each method was scored after the fact on the last days of 2014, with every reading in hand. In operation the order runs the other way. The supplier buys power for Dec 29, 2014 from a forecast made at 00:00 on Dec 29, the day's cutoff, using only what it knows at that time, and it learns what happened a day later. A forecast that held up on the backtest can drift. The clients change, a holiday falls inside the window, or a meter feed stops for a few hours. The supplier needs to see that before the next buying round, not in a review at the end of the quarter.
This chapter runs the loop for the last seven days of 2014, from Dec 25 to Dec 31. Each day gets one forecast for each of the thirty clients, made from its own midnight cutoff. Every time in this chapter is a data time. A cutoff is 00:00 of its day, and a reading arrives at the end of its hour. The decision and calibration runs start on Nov 25 and Dec 9, 2014, and the refit cycle is anchored to Monday, Nov 24, 2014. All of these start after Jan 16, 2014. That is the first midnight at which the start rule admits all thirty clients into one model: MT_115's first reading is at 00:00 on Jan 14, 2014, and the rule needs 48 hours of history from it, as Chapter 6 sets out.
A forecast says what the readings will do. It does not say why they do it. The loop can tell the supplier that demand has moved away from the forecast. It cannot tell the supplier which habit, price or weather caused the move, and this chapter does not try. A forecast that holds up is not a causal claim, and no flag in this chapter turns into one.
Refit on a schedule
A refit re-estimates the model's parameters on the history known at a cutoff. Between refits, the model keeps its trees. Each forecast still reads the most recent readings to build its lag inputs, so the forecast reflects the last hours even when the trees do not.
One detail is often missed. mlforecast 1.1.0 fits each client's scaling again whenever it predicts from a new history: predict with new_df fits the target transforms on that history (forecast.py lines 1374 to 1412, and core.py lines 536 to 540 of the pinned release). Each client's mean and spread therefore come from the history known at every cutoff, and only the trees wait for a refit. That is not leakage, because that history is known at the cutoff.
The word needs one caution. In Hyndman and Athanasopoulos's Forecasting: Principles and Practice, 3rd ed. (FPP3), refit() applies a fitted model without re-estimating it (FPP3, Chapter 13). Here, "refit" means re-estimating the trees.
A refit must not use the readings it is about to forecast. A refit at cutoff c may read only readings that arrived by c. The loop enforces this by construction. The history for a refit, and for every forecast, holds only readings that had arrived by the cutoff, so the actuals of the day being forecast are never in it.
The cadence is the choice that remains. We considered two cadences, a refit every day or a refit every seven days. Both forecast every day from its own cutoff, with the same global model from Chapter 6: one scaled LightGBM over the thirty clients, with lags of 24 and 168 hours and the client identifier. The global model runs here because it is the best choosable model on the decision windows: the recorded models in Chapter 9 have no forecasts there, so they cannot be chosen. They differ only in how often the trees are re-estimated. The weekly cycle is anchored to Monday, Nov 24, 2014, so its refits fall on Mondays: Nov 24, Dec 1, Dec 8 and so on. A run's refit dates therefore do not depend on the day the run starts. Each forecast uses the model fitted on the Monday on or before its cutoff.
We choose between the cadences on the four ordinary decision windows, Nov 25, Nov 26, Dec 2 and Dec 3, 2014. None of them is a Portuguese holiday, and the held-out week is not used. The rule is the lower mean MASE on those windows. Every run in this chapter uses one scale per client, read from readings before Nov 25, 2014 under the rule in Chapter 2. The decision windows, the calibration windows and the held-out week share that yardstick. The decision and calibration histories reach back over the clock change of Oct 26, 2014. The loop fills that 01:00 hour from the hours either side, as Chapter 3 does.
Both cadences run on the four decision windows. Because the weekly model is one or two days old there, a model of each age from 0 to 6 days also forecasts those days.
1"""Choose the refit cadence on the four decision days, Nov 25 to Dec 3, 2014: a refit every day or every seven days on2Mondays, with the global model from Chapter 6. The cadence with the lower `mean MASE` line is the one the rule takes, and3the `model aged` lines show how far a model's age alone moves that score. The held-out week is not used."""4import pandas as pd56from lib.electricity import load7from lib.panel import ALL308from lib.running import DECISION_DAYS, daily_mase, feed, fit_at, forecast_at, prepare, run_days, scale_from, with_actuals9from lib.settings import show1011raw = load(ALL30)12df = prepare(raw)13scale = scale_from(raw) # one MASE scale, read before Nov 25, used by every run in the chapter14show("decision days", len(DECISION_DAYS))15# feed returns (train, known, pending count, read_by); [0] is train16show("rows read by one refit", len(feed(df, DECISION_DAYS[0])[0]))17names = {1: "daily", 7: "weekly"} # the keys are the refit interval in days18means = {}19for every in names:20 scored = with_actuals(run_days(df, DECISION_DAYS, refit_every=every), df) # refit_every: a refit every this many days21 means[every] = daily_mase(scored, scale).mean() # the mean over the four decision days of each day's MASE22 show(f"refits, {names[every]}", scored["refit"].nunique()) # the distinct refit days the decision days use23 show(f"MAE, {names[every]}", scored["ae"].mean())24 show(f"mean MASE, {names[every]}", means[every], digits=3)25show("mean MASE, daily minus weekly", means[1] - means[7], digits=3) # positive when the weekly cadence has the lower mean26show("chosen cadence", names[min(means, key=means.get)]) # the lower mean wins; the close-call tie-break is made in the prose27models = {}28for age in range(7):29 parts = []30 for day in DECISION_DAYS:31 fit_day = day - pd.Timedelta(days=age) # a model aged `age` days, fitted that many days before the decision day32 if fit_day not in models: # a model is fitted once and reused by every decision day it serves33 models[fit_day] = fit_at(df, fit_day)[0] # fit_at returns (model, read_by); [0] is the model34 # forecast_at returns (forecast, pending count, read_by); the first is the forecast35 part, _, _ = forecast_at(models[fit_day], df, day)36 parts.append(part.assign(cutoff=day)) # daily_mase groups each forecast by its cutoff37 show(f"mean MASE, model aged {age} days", daily_mase(with_actuals(pd.concat(parts), df), scale).mean(), digits=3)
python refit-cadence-decision.pydecision days: 4
rows read by one refit: 40320
refits, daily: 4
MAE, daily: 66.81
mean MASE, daily: 0.754
refits, weekly: 2
MAE, weekly: 67.64
mean MASE, weekly: 0.746
mean MASE, daily minus weekly: 0.008
chosen cadence: weekly
mean MASE, model aged 0 days: 0.754
mean MASE, model aged 1 days: 0.747
mean MASE, model aged 2 days: 0.748
mean MASE, model aged 3 days: 0.741
mean MASE, model aged 4 days: 0.748
mean MASE, model aged 5 days: 0.758
mean MASE, model aged 6 days: 0.750
On the decision windows the weekly cadence has the lower mean MASE, by 0.008, and it needs half as many fits on those four days: the daily cadence runs 4 refits, and the weekly cadence 2. Each refit reads 40320 rows, which is thirty clients times 1,344 hours of history. In operation a weekly cadence needs one seventh of the fits.
A model up to six days old scores near a fresh one on the decision windows, and that is all the age check claims. Its mean MASE runs from 0.741 at three days, the lowest of ages 0 to 6, to 0.758 at five, the highest, with a fresh model at 0.754. Age shows no trend, and the spread across all seven ages is wider than the gap between the two cadences. That gap is small, and four days cannot separate the cadences, so the example does not show that weekly is better. A reader with a larger model or a longer horizon should run the same check on their own decision windows.
The rule picks weekly, because its mean is lower. That gap sits inside the spread that model age alone produces, so the two cadences are close. When they are, the working rule takes the one with fewer refits, and the book says so here: weekly.
Working rule: Choose the refit cadence on ordinary days that come before the held-out week, scored on one scale that was read before the first of those days. Take the cadence with the lower mean MASE, and when the two are close, take the one with fewer refits and say so. Never refit on readings from the day you are forecasting.
Log every forecast before its actuals arrive
A forecast log is a table with one row for each client and hour. Each row holds the cutoff the forecast was made at, the time the row was written, the client, the hour, the forecast, its lower and upper bounds, the refit day of the model, two arrival times described below, and the count of readings filled for that forecast. The log has one job. It proves what the forecast said at the time, and what the loop could see when it said it. Without it, nobody can tell whether a later report used the forecast that was made, or a forecast rebuilt afterwards from a newer model and the actuals. A rebuilt log is leakage in disguise, because it knows what the forecast should have been.
The band in each row is built the way Chapter 8 builds one. It is the forecast plus the scaled residuals of the same hour, taken from the calibration windows, Dec 9 to Dec 20, 2014. They come before every held-out day and are fixed before the week begins. Each hour's residuals are pooled over the thirty clients, so each hour has 360 of them, not twelve. Chapter 8's client bands pool the same residuals and are offered at 95% as well. This loop logs the 80% level only, because the retirement trigger in the last section reads the 80% band, and one level keeps the log and the trigger in step. The band is computed from the logged forecast and the fixed calibration residuals, so it needs no reading from the day it covers.
A log proves less than it appears to unless its checks can fail. The rule is that a model and a forecast read only readings that had arrived by the time they were made. The log therefore records two arrival times for each row. fit_read_by is the last arrival among the readings the model was fitted on, and it must be no later than the model's refit day. read_by is the last arrival among the readings the forecast read, and it must be no later than the cutoff. A comparison of written_at with the arrival of a reading would not test this. The replay writes written_at as the cutoff, so that comparison passes for any code. In production, written_at would come from the append-only store's clock. A replay can only label it. Both checks run on every row of the held-out week, and the model's check runs again on a deliberately leaky week, which it must catch.
1"""Runs the held-out week, Dec 25 to Dec 31, 2014: one forecast a day from its own midnight cutoff, with the weekly refit2cycle and the late-data fill, each logged when it is made. The `log rows` lines show both arrival checks pass, and the3leaky run fails the first. The `MAE` and `coverage 80` lines give each day's mean absolute error and how often its 80%4band held. The 80% bands use calibration days Dec 9 to Dec 20, 2014, earlier than every scored day."""5import pandas as pd67from lib.electricity import load8from lib.panel import ALL309from lib.running import CAL_DAYS, HELD_OUT_DAYS, REFIT_EVERY, band, coverage_by_day, prepare, run_days, scale_from, with_actuals10from lib.settings import show1112raw = load(ALL30)13df = prepare(raw)14scale = scale_from(raw)15cal = with_actuals(run_days(df, CAL_DAYS), df) # the residuals that set the bands, from days earlier than the week16bands = band(with_actuals(run_days(df, HELD_OUT_DAYS), df), cal, scale)17# leak=True refits each day's model on the next day's readings18leaky = with_actuals(run_days(df, HELD_OUT_DAYS, leak=True), df)19# refit dates in the week; the Dec 22 refit is dated before it20in_week = sorted({d for d in bands["refit"] if d >= HELD_OUT_DAYS[0]})21show("refit every (days)", REFIT_EVERY)22show("refit days in the week", ", ".join(f"{d:%Y-%m-%d}" for d in in_week))23show("models used in the week", bands["refit"].nunique())24show("log rows", len(bands))25# hour 0 holds 12 calibration days times 30 clients26show("calibration residuals per hour, pooled over clients", cal.dropna(subset=["e"]).groupby("hour").size().iloc[0])27# a model may read only readings that had arrived by its refit day28show("log rows whose model read only readings in by its refit day", int((bands["fit_read_by"] <= bands["refit"]).sum()))29# a forecast may read only readings that had arrived by its cutoff30show("log rows whose forecast read only readings in by its cutoff", int((bands["read_by"] <= bands["cutoff"]).sum()))31show("leaky log rows whose model read only readings in by its refit day", int((leaky["fit_read_by"] <= leaky["refit"]).sum()))32# a reading arrives at the end of its hour, so any other arrival is late33show("readings that arrived late", int((bands["arrives"] != bands["ds"] + pd.Timedelta(hours=1)).sum()))34# every row of a cutoff has the same count, so the first row gives it35show("forecast inputs filled, Dec 29", int(bands.loc[bands["cutoff"] == "2014-12-29", "filled"].iloc[0]))36# the last of Dec 28's readings to arrive37done = bands.loc[bands["cutoff"] == "2014-12-28", "arrives"].max()38show("Dec 28 complete at", f"{done:%H:%M on %b %d, %Y}")39mae = bands.groupby("cutoff")["ae"].mean()40cov = coverage_by_day(bands)41for day in mae.index:42 show(f"MAE {day:%Y-%m-%d}", mae[day])43for day in cov.index:44 show(f"coverage 80 {day:%Y-%m-%d}", cov[day], digits=3)45show("leaky MAE 2014-12-25", leaky.loc[leaky["cutoff"] == "2014-12-25", "ae"].mean())
python rolling-refit-seven-days.pyrefit every (days): 7
refit days in the week: 2014-12-29
models used in the week: 2
log rows: 5040
calibration residuals per hour, pooled over clients: 360
log rows whose model read only readings in by its refit day: 5040
log rows whose forecast read only readings in by its cutoff: 5040
leaky log rows whose model read only readings in by its refit day: 0
readings that arrived late: 6
forecast inputs filled, Dec 29: 6
Dec 28 complete at: 09:00 on Dec 29, 2014
MAE 2014-12-25: 447.59
MAE 2014-12-26: 203.43
MAE 2014-12-27: 54.14
MAE 2014-12-28: 71.67
MAE 2014-12-29: 111.64
MAE 2014-12-30: 87.37
MAE 2014-12-31: 167.15
coverage 80 2014-12-25: 0.232
coverage 80 2014-12-26: 0.431
coverage 80 2014-12-27: 0.790
coverage 80 2014-12-28: 0.764
coverage 80 2014-12-29: 0.713
coverage 80 2014-12-30: 0.782
coverage 80 2014-12-31: 0.701
leaky MAE 2014-12-25: 351.79
The log has 5040 rows, one for each of thirty clients' 24 hours on seven days, and 5040 pass the model's arrival check, and 5040 pass the forecast's. The week uses two models. The one fitted on Monday, Dec 22, serves Dec 25 to Dec 28, and the one refitted on Monday, Dec 29, serves Dec 29 to Dec 31, so the refit falls inside the week.
A check that never fails tells you little, so the week also runs with a leak. In that run, each day's model is refitted on the next day's readings, so it has seen the day it forecasts, and the model's check fails for every row: 0 rows pass. The leak also flatters the forecast. Its MAE on Dec 25 is 351.79 kWh, against 447.59 kWh for the honest run. The check is the thing to keep. An edit that moves a refit onto the next day's readings turns the printed count to zero, and the recorded output then fails.
Working rule: Write each forecast's rows at its cutoff, from the readings that have arrived by then, with the refit day and both arrival times beside them. Never rebuild a past log from a newer model or from the actuals. If a day's rows are lost, mark that day as missing in the log, and do not fill it in afterwards.
Score the log as actuals arrive
A reading scores the forecast logged for its hour when it arrives. The day's score is its daily MASE. The daily MASE takes each client's mean absolute error over the day's 24 hours, divides it by the client's scale, and averages the results over the thirty clients:
daily MASE = mean over clients of (MAE over the day's 24 hours / scale)
The scale is the one set in the refit section, read from before Nov 25, 2014, and it does not change from day to day. A reading enters the score for its own day, and only after it has arrived, so the score never uses a value the supplier did not have at the time. That is the second half of the leakage rule in the log section. A day is scored once its last reading is in. For Dec 29, that is 00:00 on Dec 30.
Most days arrive whole. The held-out week has one day that does not. The lead client's last six readings of Dec 28, 2014, from 18:00 to 23:00, arrive at 09:00 on Dec 29. That is nine hours after the Dec 29 cutoff, so the Dec 29 forecast has to be made without them. The rule for this case is the book's own choice, and no source supports it:
- A reading that has not arrived at a cutoff is left out of that day's refit.
- For the forecast at that cutoff, the missing value is filled with the reading of the same hour a week earlier, which had arrived. The log counts how many values were filled.
- The forecast is not redone when the reading arrives.
- The reading is scored when it arrives, and its day is scored once its last reading is in.
The first option tried was to leave the missing values as they were. The library refused them. At the pinned version, 1.1.0 raised this error when the forecast was made:
ValueError: y column contains null values.
There are two ways to fill the gap. Forward fill repeats the last reading that arrived. That flattens the evening, because the lead client's 18:00 to 23:00 values all become the 17:00 value, and lag 24 carries that flat evening into the next day's evening forecast. Filling from the same hour a week earlier keeps the evening's shape. We picked the fill on the decision windows, not the held-out week. There the same six readings are made late before each decision cutoff, and both fills forecast from them.
1"""Chooses how a late reading is filled on the decision windows, Nov 25 to Dec 3, 2014: the lead client's last six readings2of the day before arrive at 09:00 on each cutoff day, as the Dec 28 readings do in the held-out week. The3`MAE, lead client hours 18 to 23` lines decide between forward fill and a fill from the same hour a week earlier; only4those hours change, because lag 24 reads them, and the held-out week is not used."""5import pandas as pd67from lib.electricity import load8from lib.panel import ALL309from lib.running import DECISION_DAYS, LATE_CLIENT, feed, late_copy, prepare, run_days, with_actuals10from lib.settings import show1112raw = load(ALL30)13df = prepare(raw)14# feed returns (train, known, pending count, read_by); [2] is the pending count15show("readings late for each decision cutoff", feed(late_copy(df, DECISION_DAYS[0]), DECISION_DAYS[0])[2])16errors = {}17for rule in ("forward", "week"):18 parts = []19 for day in DECISION_DAYS:20 # the day before's last six readings arrive at 09:00 on this day, after its cutoff, so the fill applies21 scored = with_actuals(run_days(late_copy(df, day), [day], fill=rule), df)22 # 18:00 to 23:00 of the day scored, which lag 24 reads from the day before23 parts.append(scored.loc[(scored["unique_id"] == LATE_CLIENT) & (scored["hour"] >= 18), "ae"])24 errors[rule] = pd.concat(parts).mean()25 show(f"hours scored, {rule} fill", len(pd.concat(parts)))26 show(f"MAE, lead client hours 18 to 23, {rule} fill", errors[rule])27show("chosen late fill", min(errors, key=errors.get))
python late-fill-decision.pyreadings late for each decision cutoff: 6
hours scored, forward fill: 24
MAE, lead client hours 18 to 23, forward fill: 7.92
hours scored, week fill: 24
MAE, lead client hours 18 to 23, week fill: 5.03
chosen late fill: week
The week-earlier fill is the more accurate of the two. Over the lead client's hours from 18:00 to 23:00, across 24 forecast hours, its mean absolute error is 5.03 kWh, against 7.92 kWh with forward fill. The book uses it in every run.
The held-out week shows the rule at work. 6 readings arrived late, and the Dec 29 forecast filled 6 inputs. The Dec 29 refit left the same six readings out, so the branch that leaves a reading out of a refit runs in this week too. Dec 28 was complete at 09:00 on Dec 29, 2014.
A single day's MAE does not say whether the day was bad, because the level of demand changes from day to day. That is why the flag in the next section divides by a reference.
Working rule: Score each day when its last reading arrives, against the forecast logged at its cutoff, on the chapter's one scale. When a reading arrives late, score it then, count it in the log, and leave the logged forecast alone. Never change a forecast after its cutoff, even when the change would improve the score.
Flag drift against the backtest
The daily scores from the last section need a line that says when a day is out of line. A flag is a question about a day, not a verdict. The rule asks whether a day's daily MASE is out of line with what the model did before. A day is flagged when its daily MASE is above twice a reference. The reference is the median daily MASE of the calibration windows, Dec 9 to Dec 20, 2014. Those days are earlier than every day the flag judges. None of them is a Portuguese holiday, although Dec 9's lag-24 readings reach back to Dec 8, Imaculada Conceição, a national holiday. That echo is one day's worth, and the median damps it.
The factor of two is the book's rule of thumb. We know of no published threshold for this kind of flag, so the number is ours, and we label it as one. We checked it on the decision windows, where the four ordinary days test whether the flag fires without a known cause.
1"""Flags drift against the backtest: a day is flagged when its daily MASE is above FLAG_FACTOR times the reference, the2median daily MASE of the calibration days, Dec 9 to Dec 20, 2014, which are earlier than the held-out week. The3`largest decision day` and `decision days flagged` lines show the ordinary days staying under the line, and the `MASE`4and `held-out days flagged` lines show which held-out days cross it."""5from lib.electricity import load6from lib.panel import ALL307from lib.running import (CAL_DAYS, DECISION_DAYS, FLAG_FACTOR, HELD_OUT_DAYS, daily_mase, flagged, prepare, run_days,8 scale_from, with_actuals)9from lib.settings import show1011raw = load(ALL30)12df = prepare(raw)13scale = scale_from(raw)14reference = daily_mase(with_actuals(run_days(df, CAL_DAYS), df), scale).median() # the median, not the mean, so one unusual calibration day moves it little15show("reference, median daily MASE", reference)16show("flag above, MASE", FLAG_FACTOR * reference) # FLAG_FACTOR is 2, the chapter's rule of thumb17decision = daily_mase(with_actuals(run_days(df, DECISION_DAYS), df), scale)18show("largest decision day, times the reference", decision.max() / reference)19show("decision days flagged", len(flagged(decision, reference)))20held = daily_mase(with_actuals(run_days(df, HELD_OUT_DAYS), df), scale)21for day, value in held.items():22 show(f"MASE {day:%Y-%m-%d}", value)23show("held-out days flagged", len(flagged(held, reference)))24show("flagged held-out days", ", ".join(f"{day:%Y-%m-%d}" for day in flagged(held, reference)))
python drift-flag-rolling.pyreference, median daily MASE: 0.72
flag above, MASE: 1.43
largest decision day, times the reference: 1.44
decision days flagged: 0
MASE 2014-12-25: 4.62
MASE 2014-12-26: 2.00
MASE 2014-12-27: 0.73
MASE 2014-12-28: 0.76
MASE 2014-12-29: 0.85
MASE 2014-12-30: 0.76
MASE 2014-12-31: 1.14
held-out days flagged: 2
flagged held-out days: 2014-12-25, 2014-12-26
No ordinary decision day is flagged at a factor of two (0 flagged). The largest decision day is 1.44 times the reference. A smaller factor, such as 1.5, would leave little margin over that day, so a slightly unusual ordinary day could be flagged. We kept two for its margin. Four ordinary days cannot show how often ordinary days get flagged.
2 of the seven held-out days are over the line, Dec 25 and Dec 26. Their daily MASE is 4.62 and 2.00, against a flag line of 1.43. Dec 31, New Year's Eve, comes next, below the line.
Both flags have a known cause. Dec 25 is Christmas Day, a Portuguese holiday. The model has no holiday input. Chapter 5 tried calendar and holiday columns on the subset's decision windows, the holiday ones among them, and none beat plain lags, so Chapter 6 gave the global model none. Dec 26 is not a holiday in Portugal, but its lag-24 reading is Dec 25. The forecast for each hour of Dec 26 reads the same hour on Dec 25, so the holiday's pattern carries into the next day. These are calendar effects, not drift, and the loop cannot tell the difference. It only flags the days. A person reads each flag and checks the calendar.
One day is not a verdict. A single high day can come from a holiday, a storm or a reading that was entered wrongly. Two flags in a row, with no calendar cause, are a reason to look at the model. The next section says what to do then.
Working rule: Flag a day when its daily MASE is above twice the median of the earlier calibration windows, and then ask whether a calendar cause explains it. Record the cause beside the flag when there is one. Treat two unexplained flags in a row as the start of a retirement check, never as proof that the model is broken.
Know when to retire a model
Retiring a model means stopping its use and replacing it. A refit is the lighter move, and it comes first. This chapter names three triggers for either move, and each one reads a number the loop already logs. The first retirement trigger is sustained drift: flags on consecutive days with no calendar cause. The second retirement trigger is an uncovered band: the 80% band covers well under 80% of readings on several days. The third is a change in a library the model depends on.
The second trigger needs a sense of what ordinary coverage looks like. On the ordinary days, Dec 27 to Dec 31, coverage at 80% stays between 0.701 and 0.790. Dec 29, when the late readings arrive, and Dec 31, New Year's Eve, count as ordinary days here. Dec 25 is lower, at 0.232, and Dec 26 is at 0.431, for the reason the flags gave. Dec 25 is the holiday, and Dec 26 reads it through lag 24. The bands are a little narrow even on ordinary days, as Chapter 8 found. So a day between 0.70 and 0.80 is normal here, and a day far below that is not.
To show what the flags and bands do, the chapter stages a change. It is invented for the test, not a fact about the supplier's data. From Dec 27, 2014, every reading of all thirty clients is multiplied by 0.6, as if demand had dropped by 40%. The held-out week then runs on the changed readings, with the same cadence, reference, rule and bands as before.
1"""Shows how a change appears in the flags: a stated test, not a fact about the data, in which every reading from Dec 27,22014 is multiplied by 0.6, as if demand had dropped by 40%. The calibration windows are earlier than the change, so the3reference and bands are fixed before it happens. The `held-out days flagged` and `coverage 80` lines show what the flags4and bands do, with the same cadence, reference, rule and bands as the held-out runs in drift-flag-rolling and5rolling-refit-seven-days."""6import pandas as pd78from lib.electricity import load9from lib.panel import ALL3010from lib.running import (CAL_DAYS, HELD_OUT_DAYS, band, coverage_by_day, daily_mase, flagged, prepare, run_days,11 scale_from, with_actuals)12from lib.settings import show1314raw = load(ALL30)15df = prepare(raw)16scale = scale_from(raw)17cal = with_actuals(run_days(df, CAL_DAYS), df) # calibration windows come before the change, so the reference is fixed before it18reference = daily_mase(cal, scale).median()19changed = df.copy()20after = changed["ds"] >= pd.Timestamp("2014-12-27") # the first changed day, Dec 27, from its midnight on21changed.loc[after, "y"] = changed.loc[after, "y"] * 0.6 # a stated test: every reading from Dec 27 at 60%, as if demand fell 40%22held = with_actuals(run_days(changed, HELD_OUT_DAYS), changed) # the changed readings go through the same loop, refits included2324show("reference, median daily MASE", reference)25mase = daily_mase(held, scale)26for day, value in mase.items():27 show(f"MASE {day:%Y-%m-%d}", value)28show("held-out days flagged", len(flagged(mase, reference)))29cov = coverage_by_day(band(held, cal, scale)) # the bands use the unchanged calibration days, as in the held-out run30for day, value in cov.items():31 show(f"coverage 80 {day:%Y-%m-%d}", value, digits=3)
python drift-after-a-change.pyreference, median daily MASE: 0.72
MASE 2014-12-25: 4.62
MASE 2014-12-26: 2.00
MASE 2014-12-27: 4.74
MASE 2014-12-28: 2.93
MASE 2014-12-29: 1.68
MASE 2014-12-30: 1.78
MASE 2014-12-31: 1.89
held-out days flagged: 7
coverage 80 2014-12-25: 0.232
coverage 80 2014-12-26: 0.431
coverage 80 2014-12-27: 0.078
coverage 80 2014-12-28: 0.179
coverage 80 2014-12-29: 0.328
coverage 80 2014-12-30: 0.328
coverage 80 2014-12-31: 0.333
All 7 held-out days are flagged. Dec 25 and Dec 26 are flagged for Christmas and its echo, as before, and each day from Dec 27, the first changed day, is flagged too. On Dec 27 the daily MASE is 4.74, and coverage falls to 0.078 that day, the lowest of the week.
One scheduled refit did not clear the flags. The refit on Monday, Dec 29, is the only step in this test that refits on changed readings, and it read Dec 27 and Dec 28 at the lower level. Its daily MASE fell by about 40%, from 2.93 on Dec 28 to 1.68 on Dec 29, but Dec 30 and Dec 31 stayed above the line. Coverage recovered to 0.328 on Dec 29 and 0.333 on Dec 31, still far below 80%.
A run of flags like this, with coverage that low, is a case for an early refit, and the book's rule is to refit early. We did not test an early refit on a day of its own, so that rule is a rule of thumb and not a measured result.
The third trigger is a library change. Libraries release often, and a later release can change an output even when the code stays the same. The loop depends on 1.1.0 and 4.7.0. Prophet, which Chapter 9 pins at 1.5.0, is not part of this loop, but it is another pinned library, and its release history shows how often such a library moves (PyPI release history). A release is a reason to run this chapter's checks again. It is not, by itself, a reason to retire a model.
These checks are the forecast log, the flags and the coverage of the bands. They are not a monitoring system. A monitoring system covers alerts, dashboards, data quality across many feeds and the people who respond. A later book on monitoring models in production can continue this case, and this chapter ends where that book would begin.
Working rule: Refit early when flags run on several days in a row with no calendar cause, or when the 80% band covers far below 80% of readings on several days, and record the trigger with the log. Retire the model when an early refit does not clear the flags. Re-run this chapter's checks when a pinned library is released, and keep the pins until the checks pass on the new version.
What goes wrong in production
A feed can stop in the middle of a history. The loop does not fill a reading the feed never delivered. A gap in the middle would shift every lag, so the code raises an error instead. Only the clock-change hours are filled, by Chapter 3's rule. A reading still missing at the end of a history is different, and the late-data rule handles it.
A refit can leak. If a refit reads the readings of the day it is about to forecast, the backtest looks better than the live run will be. You would notice it as live MASE well above the backtest's for the same days. The loop prevents this by its history rule, and that rule is the one thing to keep when the loop is rewritten. Keep the leak checks from Chapter 3, the shift test, and from Chapter 7, the poisoning check. The poisoning check writes wrong values over every reading after a cutoff, then confirms that the refit and the forecast at that cutoff do not move.
A flag on the mean of thirty clients can hide a change in one client. A large change in one client moves the daily MASE only by the change in that client's daily MASE divided by thirty. The stress example moves every client at once, so the flags fire. We did not test a change in one client alone. You would notice it as that client's own MASE far above its calibration median. That is the first gap we would close, perhaps with a second flag on each client's own MASE.
A pinned library can still change behaviour when the pin is updated. The null-value error in the scoring section is behaviour of mlforecast 1.1.0. A later version could fill, drop or accept the gap, and the forecast would change. Updating the pin is a change to the forecast, and it needs the checks to run again.
The last failure is the log written after the fact. Nothing in an example stops a team from rebuilding a log later, and the rebuilt output would look correct. The two arrival columns and their checks are what show it. When those checks are removed, the log stops proving anything.
Working rule: Keep three checks in every run: the history rule that keeps the future out of each refit and forecast, the two arrival checks on each log row, and the leak checks from Chapters 3 and 7. When one of them fails, stop the loop and fix the cause before the next forecast. Do not repair a history or a log by hand, because a hand repair is the kind of change the checks exist to catch.
Decisions this chapter made
- Refit cadence: every seven days, on Mondays. Weekly had the lower mean MASE on the decision windows, by 0.008, a gap inside the spread that model age alone produces, so the tie-break applies and the book takes the cadence with fewer refits.
- One MASE scale: read from the readings before Nov 25, 2014, and used for every run in this chapter, following the rule in Chapter 2. A decision day, a calibration window and a held-out day are therefore scored on one yardstick, and no scored hour helped set it.
- Flag rule: a day is flagged above twice the median daily MASE of the calibration windows, Dec 9 to Dec 20, 2014. The factor of two is a labelled rule of thumb, kept for its margin over the ordinary decision windows.
- Bands: 80% only, because the retirement trigger reads that level; Chapter 8 also offers 95%. The bands use the scaled residuals of the same hour, pooled over the thirty clients, from the twelve calibration windows, fixed before the held-out week. No total forecast, total band or purchase quantile is logged, so the log checks the client forecasts and not the purchase that Chapter 8 makes against the total. A supplier who buys against the total would add that quantile to the log.
- Late readings: left out of the refit, filled from the same hour a week earlier for the forecast, counted in the log, scored when they arrive, and never used to redo a forecast. The fill was chosen on the decision windows. The rule has no source and is the book's own.
- Scope: seven days for thirty clients. A week holds a holiday, a late reading and a weekly refit that falls inside it, so it shows each rule. A longer run would measure how often ordinary days get flagged, which this chapter does not measure.
- Monitoring boundary: this chapter's checks only. The monitoring system that would watch them over time is outside this book.
- Your own case: the book's last decision hands the work back. The supplier's case ends here: a weekly refit, a log that checks what each forecast read, scoring as the readings arrive, and a flag against the backtest. Your case needs these decisions made again on your own data: the refit cadence on your ordinary decision windows, the late-data rule on the feed you actually receive, the flag factor on your calibration windows, and the band level your buying rule needs. Make each one on days before your held-out week.
This chapter, in this edition
Edition 2026.11.11Examples passed Nov 20, 20265 of 5 examples in this chapter pass
This book is open to contributions, so anyone can check it or correct it. Found something wrong? Report it with the section's ID, the last part of the link beside each heading, or fix it yourself with a pull request.