Keep the forecast running

阅读约需 67分钟 分钟,7节中含5个示例。

版次 2026.11.11 示例于 2026年11月20日 通过 41个示例中有41个通过 变化内容

AI 翻译。由人工以英文核验。 我们如何制作图书

译自英文版图书,版次 2026.11.11。 阅读原文

这部分尚未翻译,因此以英文显示。

Once a forecast runs every day, the question changes from whether it scored well to whether it still holds on the day it's used. This chapter builds the loop that answers that question. You'll refit a forecast on a schedule, log each day's forecast before its readings exist, check that the log used only what had arrived, score the readings as they come in, flag a day whose error is out of line with the backtest, and say when a model should be retired.

The stake is the next buying round. The supplier commits to power for the next day from the forecast, so a forecast that has gone wrong without anyone noticing costs money in that purchase. This chapter does not reduce that risk to zero. It makes the risk visible on the day it appears, using only what the supplier knew at the time.

The earlier chapters checked the supplier's forecast on past data. Each method was scored after the fact on the last days of 2014, with every reading in hand. In operation the order runs the other way. The supplier buys power for Dec 29, 2014 from a forecast made at 00:00 on Dec 29, the day's cutoff, using only what it knows at that time, and it learns what happened a day later. A forecast that held up on the backtest can drift. The clients change, a holiday falls inside the window, or a meter feed stops for a few hours. The supplier needs to see that before the next buying round, not in a review at the end of the quarter.

This chapter runs the loop for the last seven days of 2014, from Dec 25 to Dec 31. Each day gets one forecast for each of the thirty clients, made from its own midnight cutoff. Every time in this chapter is a data time. A cutoff is 00:00 of its day, and a reading arrives at the end of its hour. The decision and calibration runs start on Nov 25 and Dec 9, 2014, and the refit cycle is anchored to Monday, Nov 24, 2014. All of these start after Jan 16, 2014. That is the first midnight at which the start rule admits all thirty clients into one model: MT_115's first reading is at 00:00 on Jan 14, 2014, and the rule needs 48 hours of history from it, as Chapter 6 sets out.

A forecast says what the readings will do. It does not say why they do it. The loop can tell the supplier that demand has moved away from the forecast. It cannot tell the supplier which habit, price or weather caused the move, and this chapter does not try. A forecast that holds up is not a causal claim, and no flag in this chapter turns into one.

这部分尚未翻译,因此以英文显示。

Refit on a schedule

最近一次修改见于版次 2026.11.5

A refit re-estimates the model's parameters on the history known at a cutoff. Between refits, the model keeps its trees. Each forecast still reads the most recent readings to build its lag inputs, so the forecast reflects the last hours even when the trees do not.

One detail is often missed. mlforecast 1.1.0 fits each client's scaling again whenever it predicts from a new history: predict with new_df fits the target transforms on that history (forecast.py lines 1374 to 1412, and core.py lines 536 to 540 of the pinned release). Each client's mean and spread therefore come from the history known at every cutoff, and only the trees wait for a refit. That is not leakage, because that history is known at the cutoff.

The word needs one caution. In Hyndman and Athanasopoulos's Forecasting: Principles and Practice, 3rd ed. (FPP3), refit() applies a fitted model without re-estimating it (FPP3, Chapter 13). Here, "refit" means re-estimating the trees.

A refit must not use the readings it is about to forecast. A refit at cutoff c may read only readings that arrived by c. The loop enforces this by construction. The history for a refit, and for every forecast, holds only readings that had arrived by the cutoff, so the actuals of the day being forecast are never in it.

The cadence is the choice that remains. We considered two cadences, a refit every day or a refit every seven days. Both forecast every day from its own cutoff, with the same global model from Chapter 6: one scaled LightGBM over the thirty clients, with lags of 24 and 168 hours and the client identifier. The global model runs here because it is the best choosable model on the decision windows: the recorded models in Chapter 9 have no forecasts there, so they cannot be chosen. They differ only in how often the trees are re-estimated. The weekly cycle is anchored to Monday, Nov 24, 2014, so its refits fall on Mondays: Nov 24, Dec 1, Dec 8 and so on. A run's refit dates therefore do not depend on the day the run starts. Each forecast uses the model fitted on the Monday on or before its cutoff.

We choose between the cadences on the four ordinary decision windows, Nov 25, Nov 26, Dec 2 and Dec 3, 2014. None of them is a Portuguese holiday, and the held-out week is not used. The rule is the lower mean MASE on those windows. Every run in this chapter uses one scale per client, read from readings before Nov 25, 2014 under the rule in Chapter 2. The decision windows, the calibration windows and the held-out week share that yardstick. The decision and calibration histories reach back over the clock change of Oct 26, 2014. The loop fills that 01:00 hour from the hours either side, as Chapter 3 does.

Both cadences run on the four decision windows. Because the weekly model is one or two days old there, a model of each age from 0 to 6 days also forecasts those days.

refit-cadence-decision.py Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17 通过
"""Choose the refit cadence on the four decision days, Nov 25 to Dec 3, 2014: a refit every day or every seven days onMondays, with the global model from Chapter 6. The cadence with the lower `mean MASE` line is the one the rule takes, andthe `model aged` lines show how far a model's age alone moves that score. The held-out week is not used."""import pandas as pdfrom lib.electricity import loadfrom lib.panel import ALL30from lib.running import DECISION_DAYS, daily_mase, feed, fit_at, forecast_at, prepare, run_days, scale_from, with_actualsfrom lib.settings import showraw = load(ALL30)df = prepare(raw)scale = scale_from(raw)  # one MASE scale, read before Nov 25, used by every run in the chaptershow("decision days", len(DECISION_DAYS))# feed returns (train, known, pending count, read_by); [0] is trainshow("rows read by one refit", len(feed(df, DECISION_DAYS[0])[0]))names = {1: "daily", 7: "weekly"}  # the keys are the refit interval in daysmeans = {}for every in names:    scored = with_actuals(run_days(df, DECISION_DAYS, refit_every=every), df)  # refit_every: a refit every this many days    means[every] = daily_mase(scored, scale).mean()  # the mean over the four decision days of each day's MASE    show(f"refits, {names[every]}", scored["refit"].nunique())  # the distinct refit days the decision days use    show(f"MAE, {names[every]}", scored["ae"].mean())    show(f"mean MASE, {names[every]}", means[every], digits=3)show("mean MASE, daily minus weekly", means[1] - means[7], digits=3)  # positive when the weekly cadence has the lower meanshow("chosen cadence", names[min(means, key=means.get)])  # the lower mean wins; the close-call tie-break is made in the prosemodels = {}for age in range(7):    parts = []    for day in DECISION_DAYS:        fit_day = day - pd.Timedelta(days=age)  # a model aged `age` days, fitted that many days before the decision day        if fit_day not in models:  # a model is fitted once and reused by every decision day it serves            models[fit_day] = fit_at(df, fit_day)[0]  # fit_at returns (model, read_by); [0] is the model        # forecast_at returns (forecast, pending count, read_by); the first is the forecast        part, _, _ = forecast_at(models[fit_day], df, day)        parts.append(part.assign(cutoff=day))  # daily_mase groups each forecast by its cutoff    show(f"mean MASE, model aged {age} days", daily_mase(with_actuals(pd.concat(parts), df), scale).mean(), digits=3)
输出python refit-cadence-decision.py
decision days: 4
rows read by one refit: 40320
refits, daily: 4
MAE, daily: 66.81
mean MASE, daily: 0.754
refits, weekly: 2
MAE, weekly: 67.64
mean MASE, weekly: 0.746
mean MASE, daily minus weekly: 0.008
chosen cadence: weekly
mean MASE, model aged 0 days: 0.754
mean MASE, model aged 1 days: 0.747
mean MASE, model aged 2 days: 0.748
mean MASE, model aged 3 days: 0.741
mean MASE, model aged 4 days: 0.748
mean MASE, model aged 5 days: 0.758
mean MASE, model aged 6 days: 0.750

On the decision windows the weekly cadence has the lower mean MASE, by 0.008, and it needs half as many fits on those four days: the daily cadence runs 4 refits, and the weekly cadence 2. Each refit reads 40320 rows, which is thirty clients times 1,344 hours of history. In operation a weekly cadence needs one seventh of the fits.

A model up to six days old scores near a fresh one on the decision windows, and that is all the age check claims. Its mean MASE runs from 0.741 at three days, the lowest of ages 0 to 6, to 0.758 at five, the highest, with a fresh model at 0.754. Age shows no trend, and the spread across all seven ages is wider than the gap between the two cadences. That gap is small, and four days cannot separate the cadences, so the example does not show that weekly is better. A reader with a larger model or a longer horizon should run the same check on their own decision windows.

The rule picks weekly, because its mean is lower. That gap sits inside the spread that model age alone produces, so the two cadences are close. When they are, the working rule takes the one with fewer refits, and the book says so here: weekly.

Working rule: Choose the refit cadence on ordinary days that come before the held-out week, scored on one scale that was read before the first of those days. Take the cadence with the lower mean MASE, and when the two are close, take the one with fewer refits and say so. Never refit on readings from the day you are forecasting.

在实际值到达之前记录每一次预测

最近一次修改见于版次 2026.11.5

预测日志是一张表,每个客户每小时占一行。每一行记录做出预测时的截止时间、该行的写入时间、客户、小时、预测值、预测的下界和上界、模型的重新拟合日、下文说明的两个到达时间,以及该次预测所填补的读数个数。日志只有一项任务:证明预测在当时说了什么,以及循环在做出预测时能看到什么。没有日志,就无法判断后来的报告用的是当时做出的预测,还是事后用更新的模型和实际值重建的预测。重建的日志是变相的泄漏,因为它已经知道预测本应是什么。

每一行中的区间按第 8 章的方法构建:它是预测值加上同一小时经缩放的残差,残差取自校准窗口,即 2014 年 12 月 9 日至 12 月 20 日。这些窗口在每个留出日之前,并在一周开始前就已固定。每个小时的残差在三十个客户上汇总,因此每个小时有 360 个残差,而不是十二个。第 8 章的客户区间汇总的是同样的残差,也同样提供 95% 的水平。本循环只记录 80% 的水平,因为最后一节的退役触发条件读取的是 80% 区间,只用一个水平可以让日志与触发条件保持一致。区间由记录的预测值和固定的校准残差计算得出,因此不需要读取它所覆盖那一天的任何读数。

如果检查不可能失败,日志所能证明的就比看上去的要少。规则是:模型和预测只能读取在它们生成时已经到达的读数。因此日志为每一行记录两个到达时间。fit_read_by 是模型拟合所用读数中最后到达的时间,它不得晚于模型的重新拟合日。read_by 是预测所读读数中最后到达的时间,它不得晚于截止时间。把 written_at 与读数的到达时间比较,并不能检验这一点。回放会把 written_at 写成截止时间,所以这样的比较对任何代码都会通过。在生产环境中,written_at 会来自只追加存储的时钟,而回放只能给它贴上标签。两项检查都在留出周的每一行上运行,模型的检查还会在一个刻意构造的泄漏周上再运行一次,它必须能发现泄漏。

rolling-refit-seven-days.py Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17 通过
"""Runs the held-out week, Dec 25 to Dec 31, 2014: one forecast a day from its own midnight cutoff, with the weekly refitcycle and the late-data fill, each logged when it is made. The `log rows` lines show both arrival checks pass, and theleaky run fails the first. The `MAE` and `coverage 80` lines give each day's mean absolute error and how often its 80%band held. The 80% bands use calibration days Dec 9 to Dec 20, 2014, earlier than every scored day."""import pandas as pdfrom lib.electricity import loadfrom lib.panel import ALL30from lib.running import CAL_DAYS, HELD_OUT_DAYS, REFIT_EVERY, band, coverage_by_day, prepare, run_days, scale_from, with_actualsfrom lib.settings import showraw = load(ALL30)df = prepare(raw)scale = scale_from(raw)cal = with_actuals(run_days(df, CAL_DAYS), df)  # the residuals that set the bands, from days earlier than the weekbands = band(with_actuals(run_days(df, HELD_OUT_DAYS), df), cal, scale)# leak=True refits each day's model on the next day's readingsleaky = with_actuals(run_days(df, HELD_OUT_DAYS, leak=True), df)# refit dates in the week; the Dec 22 refit is dated before itin_week = sorted({d for d in bands["refit"] if d >= HELD_OUT_DAYS[0]})show("refit every (days)", REFIT_EVERY)show("refit days in the week", ", ".join(f"{d:%Y-%m-%d}" for d in in_week))show("models used in the week", bands["refit"].nunique())show("log rows", len(bands))# hour 0 holds 12 calibration days times 30 clientsshow("calibration residuals per hour, pooled over clients", cal.dropna(subset=["e"]).groupby("hour").size().iloc[0])# a model may read only readings that had arrived by its refit dayshow("log rows whose model read only readings in by its refit day", int((bands["fit_read_by"] <= bands["refit"]).sum()))# a forecast may read only readings that had arrived by its cutoffshow("log rows whose forecast read only readings in by its cutoff", int((bands["read_by"] <= bands["cutoff"]).sum()))show("leaky log rows whose model read only readings in by its refit day", int((leaky["fit_read_by"] <= leaky["refit"]).sum()))# a reading arrives at the end of its hour, so any other arrival is lateshow("readings that arrived late", int((bands["arrives"] != bands["ds"] + pd.Timedelta(hours=1)).sum()))# every row of a cutoff has the same count, so the first row gives itshow("forecast inputs filled, Dec 29", int(bands.loc[bands["cutoff"] == "2014-12-29", "filled"].iloc[0]))# the last of Dec 28's readings to arrivedone = bands.loc[bands["cutoff"] == "2014-12-28", "arrives"].max()show("Dec 28 complete at", f"{done:%H:%M on %b %d, %Y}")mae = bands.groupby("cutoff")["ae"].mean()cov = coverage_by_day(bands)for day in mae.index:    show(f"MAE {day:%Y-%m-%d}", mae[day])for day in cov.index:    show(f"coverage 80 {day:%Y-%m-%d}", cov[day], digits=3)show("leaky MAE 2014-12-25", leaky.loc[leaky["cutoff"] == "2014-12-25", "ae"].mean())
输出python rolling-refit-seven-days.py
refit every (days): 7
refit days in the week: 2014-12-29
models used in the week: 2
log rows: 5040
calibration residuals per hour, pooled over clients: 360
log rows whose model read only readings in by its refit day: 5040
log rows whose forecast read only readings in by its cutoff: 5040
leaky log rows whose model read only readings in by its refit day: 0
readings that arrived late: 6
forecast inputs filled, Dec 29: 6
Dec 28 complete at: 09:00 on Dec 29, 2014
MAE 2014-12-25: 447.59
MAE 2014-12-26: 203.43
MAE 2014-12-27: 54.14
MAE 2014-12-28: 71.67
MAE 2014-12-29: 111.64
MAE 2014-12-30: 87.37
MAE 2014-12-31: 167.15
coverage 80 2014-12-25: 0.232
coverage 80 2014-12-26: 0.431
coverage 80 2014-12-27: 0.790
coverage 80 2014-12-28: 0.764
coverage 80 2014-12-29: 0.713
coverage 80 2014-12-30: 0.782
coverage 80 2014-12-31: 0.701
leaky MAE 2014-12-25: 351.79

日志共有 5040 行,即三十个客户在七天中每天 24 个小时各一行,其中 5040 行通过了模型的到达检查,5040 行通过了预测的到达检查。这一周使用两个模型:12 月 22 日(星期一)拟合的模型服务于 12 月 25 日至 12 月 28 日,12 月 29 日(星期一)重新拟合的模型服务于 12 月 29 日至 12 月 31 日,因此重新拟合发生在这一周之内。

从不失败的检查提供的信息很少,所以这一周也带着泄漏运行一次。在这次运行中,每一天的模型都用次日的读数重新拟合,因而已经见过它所预测的那一天,模型的检查对每一行都失败:通过的行数为 0。泄漏还会让预测看起来更好。它在 12 月 25 日的 MAE 为 351.79 kWh,而如实运行的 MAE 为 447.59 kWh。应当保留的是这项检查。如果有改动把某次重新拟合移到次日的读数上,打印出的计数就会变为零,记录的输出随即失败。

工作准则: 在每次预测的截止时间写入它的各行,所用读数仅限于届时已经到达的读数,并在旁边记录重新拟合日和两个到达时间。绝不要用更新的模型或实际值重建过去的日志。如果某一天的行丢失,应在日志中把这一天标记为缺失,事后不要补填。

这部分尚未翻译,因此以英文显示。

Score the log as actuals arrive

最近一次修改见于版次 2026.11.11

A reading scores the forecast logged for its hour when it arrives. The day's score is its daily MASE. The daily MASE takes each client's mean absolute error over the day's 24 hours, divides it by the client's scale, and averages the results over the thirty clients:

daily MASE = mean over clients of (MAE over the day's 24 hours / scale)

The scale is the one set in the refit section, read from before Nov 25, 2014, and it does not change from day to day. A reading enters the score for its own day, and only after it has arrived, so the score never uses a value the supplier did not have at the time. That is the second half of the leakage rule in the log section. A day is scored once its last reading is in. For Dec 29, that is 00:00 on Dec 30.

Most days arrive whole. The held-out week has one day that does not. The lead client's last six readings of Dec 28, 2014, from 18:00 to 23:00, arrive at 09:00 on Dec 29. That is nine hours after the Dec 29 cutoff, so the Dec 29 forecast has to be made without them. The rule for this case is the book's own choice, and no source supports it:

  • A reading that has not arrived at a cutoff is left out of that day's refit.
  • For the forecast at that cutoff, the missing value is filled with the reading of the same hour a week earlier, which had arrived. The log counts how many values were filled.
  • The forecast is not redone when the reading arrives.
  • The reading is scored when it arrives, and its day is scored once its last reading is in.

The first option tried was to leave the missing values as they were. The library refused them. At the pinned version, 1.1.0 raised this error when the forecast was made:

ValueError: y column contains null values.

There are two ways to fill the gap. Forward fill repeats the last reading that arrived. That flattens the evening, because the lead client's 18:00 to 23:00 values all become the 17:00 value, and lag 24 carries that flat evening into the next day's evening forecast. Filling from the same hour a week earlier keeps the evening's shape. We picked the fill on the decision windows, not the held-out week. There the same six readings are made late before each decision cutoff, and both fills forecast from them.

late-fill-decision.py Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17 通过
"""Chooses how a late reading is filled on the decision windows, Nov 25 to Dec 3, 2014: the lead client's last six readingsof the day before arrive at 09:00 on each cutoff day, as the Dec 28 readings do in the held-out week. The`MAE, lead client hours 18 to 23` lines decide between forward fill and a fill from the same hour a week earlier; onlythose hours change, because lag 24 reads them, and the held-out week is not used."""import pandas as pdfrom lib.electricity import loadfrom lib.panel import ALL30from lib.running import DECISION_DAYS, LATE_CLIENT, feed, late_copy, prepare, run_days, with_actualsfrom lib.settings import showraw = load(ALL30)df = prepare(raw)# feed returns (train, known, pending count, read_by); [2] is the pending countshow("readings late for each decision cutoff", feed(late_copy(df, DECISION_DAYS[0]), DECISION_DAYS[0])[2])errors = {}for rule in ("forward", "week"):    parts = []    for day in DECISION_DAYS:        # the day before's last six readings arrive at 09:00 on this day, after its cutoff, so the fill applies        scored = with_actuals(run_days(late_copy(df, day), [day], fill=rule), df)        # 18:00 to 23:00 of the day scored, which lag 24 reads from the day before        parts.append(scored.loc[(scored["unique_id"] == LATE_CLIENT) & (scored["hour"] >= 18), "ae"])    errors[rule] = pd.concat(parts).mean()    show(f"hours scored, {rule} fill", len(pd.concat(parts)))    show(f"MAE, lead client hours 18 to 23, {rule} fill", errors[rule])show("chosen late fill", min(errors, key=errors.get))
输出python late-fill-decision.py
readings late for each decision cutoff: 6
hours scored, forward fill: 24
MAE, lead client hours 18 to 23, forward fill: 7.92
hours scored, week fill: 24
MAE, lead client hours 18 to 23, week fill: 5.03
chosen late fill: week

The week-earlier fill is the more accurate of the two. Over the lead client's hours from 18:00 to 23:00, across 24 forecast hours, its mean absolute error is 5.03 kWh, against 7.92 kWh with forward fill. The book uses it in every run.

The held-out week shows the rule at work. 6 readings arrived late, and the Dec 29 forecast filled 6 inputs. The Dec 29 refit left the same six readings out, so the branch that leaves a reading out of a refit runs in this week too. Dec 28 was complete at 09:00 on Dec 29, 2014.

A single day's MAE does not say whether the day was bad, because the level of demand changes from day to day. That is why the flag in the next section divides by a reference.

Working rule: Score each day when its last reading arrives, against the forecast logged at its cutoff, on the chapter's one scale. When a reading arrives late, score it then, count it in the log, and leave the logged forecast alone. Never change a forecast after its cutoff, even when the change would improve the score.

对照回测标记漂移

最近一次修改见于版次 2026.11.5

上一节中的每日得分,还需要一条界线,用来说明某一天何时偏离常态。标记是对某一天提出的问题,而不是裁决。规则要问的是:某一天的每日 MASE 是否偏离了模型此前的表现。当某一天的每日 MASE 高于参考值的两倍时,该日被标记。参考值是校准窗口(2014 年 12 月 9 日至 12 月 20 日)每日 MASE 的中位数。这些日期都早于标记所要判断的每一天。其中没有一天是葡萄牙的节假日,不过 12 月 9 日的滞后 24 读数会回溯到 12 月 8 日,即圣母无原罪节,这是一个全国性假日。这种回响只有一天的量,而中位数会削弱它的影响。

系数 2 是本书的经验法则。我们不知道有哪个已发表的阈值适用于这类标记,所以这个数字是我们自己定的,并且如实标明。我们在决策窗口上检验了它,其中的四个普通日用来检验:在没有已知原因时,标记是否会触发。

drift-flag-rolling.py Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17 通过
"""Flags drift against the backtest: a day is flagged when its daily MASE is above FLAG_FACTOR times the reference, themedian daily MASE of the calibration days, Dec 9 to Dec 20, 2014, which are earlier than the held-out week. The`largest decision day` and `decision days flagged` lines show the ordinary days staying under the line, and the `MASE`and `held-out days flagged` lines show which held-out days cross it."""from lib.electricity import loadfrom lib.panel import ALL30from lib.running import (CAL_DAYS, DECISION_DAYS, FLAG_FACTOR, HELD_OUT_DAYS, daily_mase, flagged, prepare, run_days,                         scale_from, with_actuals)from lib.settings import showraw = load(ALL30)df = prepare(raw)scale = scale_from(raw)reference = daily_mase(with_actuals(run_days(df, CAL_DAYS), df), scale).median()  # the median, not the mean, so one unusual calibration day moves it littleshow("reference, median daily MASE", reference)show("flag above, MASE", FLAG_FACTOR * reference)  # FLAG_FACTOR is 2, the chapter's rule of thumbdecision = daily_mase(with_actuals(run_days(df, DECISION_DAYS), df), scale)show("largest decision day, times the reference", decision.max() / reference)show("decision days flagged", len(flagged(decision, reference)))held = daily_mase(with_actuals(run_days(df, HELD_OUT_DAYS), df), scale)for day, value in held.items():    show(f"MASE {day:%Y-%m-%d}", value)show("held-out days flagged", len(flagged(held, reference)))show("flagged held-out days", ", ".join(f"{day:%Y-%m-%d}" for day in flagged(held, reference)))
输出python drift-flag-rolling.py
reference, median daily MASE: 0.72
flag above, MASE: 1.43
largest decision day, times the reference: 1.44
decision days flagged: 0
MASE 2014-12-25: 4.62
MASE 2014-12-26: 2.00
MASE 2014-12-27: 0.73
MASE 2014-12-28: 0.76
MASE 2014-12-29: 0.85
MASE 2014-12-30: 0.76
MASE 2014-12-31: 1.14
held-out days flagged: 2
flagged held-out days: 2014-12-25, 2014-12-26

在系数为 2 时,没有任何普通决策日被标记(被标记 0 天)。最大的决策日是参考值的 1.44 倍。若取更小的系数,例如 1.5,该日与界线之间的余量就很小,稍有异常的普通日就可能被标记。我们保留 2,是因为它留有余量。四个普通日无法说明普通日被标记的频率有多高。

在七个留出日中,有 2 天越过界线,即 12 月 25 日和 12 月 26 日。它们的每日 MASE 分别为 4.62 和 2.00,而标记线为 1.43。其后是 12 月 31 日,即新年前夜,低于界线。

两个标记都有已知原因。12 月 25 日是圣诞节,是葡萄牙的节假日。模型没有节假日输入。第 5 章曾在该子集的决策窗口上尝试日历列和节假日列,其中包括节假日列,没有一种胜过单纯的滞后特征,因此第 6 章没有给全局模型加入任何此类列。12 月 26 日在葡萄牙不是节假日,但它的滞后 24 读数是 12 月 25 日。12 月 26 日每个小时的预测都读取 12 月 25 日的同一小时,所以节假日的模式延续到了第二天。这些是日历效应,不是漂移,而这个循环无法分辨二者的差别。它只负责标记日期。由人来查看每个标记,并核对日历。

一天不构成裁决。单独一个偏高的日子,可能源于节假日、一场风暴,或一个录入错误的读数。连续两个标记,且没有日历上的原因,就是检查模型的理由。下一节说明届时该怎么做。

工作准则: 当某一天的每日 MASE 高于此前校准窗口中位数的两倍时,标记该日,然后询问是否有日历原因可以解释它。若有,就把原因记在标记旁边。连续两个无法解释的标记,应视为退役检查的开始,绝不能当作模型已坏的证据。

这部分尚未翻译,因此以英文显示。

Know when to retire a model

最近一次修改见于版次 2026.11.11

Retiring a model means stopping its use and replacing it. A refit is the lighter move, and it comes first. This chapter names three triggers for either move, and each one reads a number the loop already logs. The first retirement trigger is sustained drift: flags on consecutive days with no calendar cause. The second retirement trigger is an uncovered band: the 80% band covers well under 80% of readings on several days. The third is a change in a library the model depends on.

The second trigger needs a sense of what ordinary coverage looks like. On the ordinary days, Dec 27 to Dec 31, coverage at 80% stays between 0.701 and 0.790. Dec 29, when the late readings arrive, and Dec 31, New Year's Eve, count as ordinary days here. Dec 25 is lower, at 0.232, and Dec 26 is at 0.431, for the reason the flags gave. Dec 25 is the holiday, and Dec 26 reads it through lag 24. The bands are a little narrow even on ordinary days, as Chapter 8 found. So a day between 0.70 and 0.80 is normal here, and a day far below that is not.

To show what the flags and bands do, the chapter stages a change. It is invented for the test, not a fact about the supplier's data. From Dec 27, 2014, every reading of all thirty clients is multiplied by 0.6, as if demand had dropped by 40%. The held-out week then runs on the changed readings, with the same cadence, reference, rule and bands as before.

drift-after-a-change.py Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17 通过
"""Shows how a change appears in the flags: a stated test, not a fact about the data, in which every reading from Dec 27,2014 is multiplied by 0.6, as if demand had dropped by 40%. The calibration windows are earlier than the change, so thereference and bands are fixed before it happens. The `held-out days flagged` and `coverage 80` lines show what the flagsand bands do, with the same cadence, reference, rule and bands as the held-out runs in drift-flag-rolling androlling-refit-seven-days."""import pandas as pdfrom lib.electricity import loadfrom lib.panel import ALL30from lib.running import (CAL_DAYS, HELD_OUT_DAYS, band, coverage_by_day, daily_mase, flagged, prepare, run_days,                         scale_from, with_actuals)from lib.settings import showraw = load(ALL30)df = prepare(raw)scale = scale_from(raw)cal = with_actuals(run_days(df, CAL_DAYS), df)  # calibration windows come before the change, so the reference is fixed before itreference = daily_mase(cal, scale).median()changed = df.copy()after = changed["ds"] >= pd.Timestamp("2014-12-27")  # the first changed day, Dec 27, from its midnight onchanged.loc[after, "y"] = changed.loc[after, "y"] * 0.6  # a stated test: every reading from Dec 27 at 60%, as if demand fell 40%held = with_actuals(run_days(changed, HELD_OUT_DAYS), changed)  # the changed readings go through the same loop, refits includedshow("reference, median daily MASE", reference)mase = daily_mase(held, scale)for day, value in mase.items():    show(f"MASE {day:%Y-%m-%d}", value)show("held-out days flagged", len(flagged(mase, reference)))cov = coverage_by_day(band(held, cal, scale))  # the bands use the unchanged calibration days, as in the held-out runfor day, value in cov.items():    show(f"coverage 80 {day:%Y-%m-%d}", value, digits=3)
输出python drift-after-a-change.py
reference, median daily MASE: 0.72
MASE 2014-12-25: 4.62
MASE 2014-12-26: 2.00
MASE 2014-12-27: 4.74
MASE 2014-12-28: 2.93
MASE 2014-12-29: 1.68
MASE 2014-12-30: 1.78
MASE 2014-12-31: 1.89
held-out days flagged: 7
coverage 80 2014-12-25: 0.232
coverage 80 2014-12-26: 0.431
coverage 80 2014-12-27: 0.078
coverage 80 2014-12-28: 0.179
coverage 80 2014-12-29: 0.328
coverage 80 2014-12-30: 0.328
coverage 80 2014-12-31: 0.333

All 7 held-out days are flagged. Dec 25 and Dec 26 are flagged for Christmas and its echo, as before, and each day from Dec 27, the first changed day, is flagged too. On Dec 27 the daily MASE is 4.74, and coverage falls to 0.078 that day, the lowest of the week.

One scheduled refit did not clear the flags. The refit on Monday, Dec 29, is the only step in this test that refits on changed readings, and it read Dec 27 and Dec 28 at the lower level. Its daily MASE fell by about 40%, from 2.93 on Dec 28 to 1.68 on Dec 29, but Dec 30 and Dec 31 stayed above the line. Coverage recovered to 0.328 on Dec 29 and 0.333 on Dec 31, still far below 80%.

A run of flags like this, with coverage that low, is a case for an early refit, and the book's rule is to refit early. We did not test an early refit on a day of its own, so that rule is a rule of thumb and not a measured result.

The third trigger is a library change. Libraries release often, and a later release can change an output even when the code stays the same. The loop depends on 1.1.0 and 4.7.0. Prophet, which Chapter 9 pins at 1.5.0, is not part of this loop, but it is another pinned library, and its release history shows how often such a library moves (PyPI release history). A release is a reason to run this chapter's checks again. It is not, by itself, a reason to retire a model.

These checks are the forecast log, the flags and the coverage of the bands. They are not a monitoring system. A monitoring system covers alerts, dashboards, data quality across many feeds and the people who respond. A later book on monitoring models in production can continue this case, and this chapter ends where that book would begin.

Working rule: Refit early when flags run on several days in a row with no calendar cause, or when the 80% band covers far below 80% of readings on several days, and record the trigger with the log. Retire the model when an early refit does not clear the flags. Re-run this chapter's checks when a pinned library is released, and keep the pins until the checks pass on the new version.

生产中会出什么问题

数据源可能在某段历史的中间中断。循环不会填补数据源从未送达的读数。中间出现缺口会使每一个滞后值都发生偏移,因此代码会抛出错误。只有时钟调整所涉及的那些小时才会被填补,依据是第 3 章的规则。历史末尾仍然缺失的读数则不同,由迟到数据规则处理。

重新拟合可能造成泄漏。如果重新拟合读取了即将预测的那一天的读数,回测的结果就会好于实际运行的结果。你会发现,同样这些天的实际 MASE 明显高于回测的 MASE。循环通过它的历史规则防止这种情况,重写循环时,这条规则是唯一必须保留的。请保留第 3 章的泄漏检查(即移位测试),以及第 7 章的泄漏检查(即污染检查)。污染检查会把截止点之后的每一个读数都改写为错误的值,然后确认该截止点处的重新拟合和预测都没有变化。

对三十个客户的均值设置的标志,可能掩盖某一个客户的变化。单个客户的大幅变化,只会使每日 MASE 移动该客户每日 MASE 的变化量除以三十。压力示例让所有客户同时变化,所以标志会触发。我们没有测试只有一个客户发生变化的情况。你会发现,该客户自己的 MASE 远高于其校准中位数。这是我们首先要弥补的缺口,也许会为每个客户自己的 MASE 再设置一个标志。

固定了版本的库在更新固定版本后,行为仍可能改变。评分一节中的空值错误是 mlforecast 1.1.0 的行为。更高的版本可能会填补、丢弃或接受这个缺口,预测也会随之改变。更新固定版本就是对预测的一次改动,需要重新运行这些检查。

最后一种失败是事后补写的日志。示例中没有任何机制阻止团队事后重建日志,而且重建出来的输出看上去是正确的。两个到达列及其检查可以显示出这一点。一旦去掉这些检查,日志就不再能证明任何事情。

**工作准则:**每次运行都要保留三项检查:使未来数据不进入每次重新拟合和预测的历史规则,日志每一行上的两项到达检查,以及第 3 章和第 7 章的泄漏检查。其中任何一项失败时,先停止循环,并在下一次预测之前修复原因。不要手工修补历史或日志,因为手工修补正是这些检查要捕捉的那类改动。

本章做出的决定

最近一次修改见于版次 2026.11.5
  • **重拟合频率:**每七天一次,在周一进行。在决策窗口上,每周重拟合的平均 MASE 更低,差值为 0.008。这一差距处于仅由模型陈旧程度就会产生的波动范围之内,因此适用平局规则,本书选取重拟合次数更少的频率。
  • **统一的 MASE 尺度:**取自 2014 年 11 月 25 日之前的读数,并用于本章的每一次运行,遵循第 2 章中的规则。因此,决策日、校准窗口和留出日都用同一把尺子计分,而且没有任何一个计分小时参与确定这个尺度。
  • **标记规则:**某一天的 MASE 高于校准窗口(2014 年 12 月 9 日至 12 月 20 日)每日 MASE 中位数的两倍时,该日被标记。系数 2 是明确标注的经验法则,保留它是因为它相对于普通决策窗口留有余量。
  • **区间:**只用 80%,因为停用触发条件读取的就是这一水平;第 8 章还给出了 95% 的区间。区间使用同一小时的缩放残差,这些残差来自十二个校准窗口,在三十个客户之间合并,并在留出周之前固定。日志中没有记录总量预测、总量区间或采购分位数,因此日志检查的是各客户的预测,而不是第 8 章针对总量所做的采购。若供应商按总量采购,则应把该分位数加入日志。
  • **迟到的读数:**不纳入重拟合;预测时用一周前同一小时的值填补;在日志中计数;到达时计分;绝不用来重做预测。这一填补方式是在决策窗口上选定的。该规则没有外部来源,是本书自己的规则。
  • **范围:**三十个客户,七天。一周内包含一个假日、一个迟到的读数,以及一次落在其中的每周重拟合,因此每条规则都能体现出来。更长的运行可以衡量普通日被标记的频率,而本章并不衡量这一点。
  • **监控边界:**仅限本章的检查。长期观察这些检查的监控系统不在本书范围之内。
  • **你自己的情形:**本书的最后一个决定把工作交还给你。供应商的情形到此为止:每周重拟合、检查每次预测所读取内容的日志、随读数到达而计分,以及对照回测设定的标记。你的情形需要在你自己的数据上重新做出这些决定:依据你的普通决策窗口确定重拟合频率,依据你实际收到的数据流确定迟到数据规则,依据你的校准窗口确定标记系数,并依据你的采购规则所需确定区间水平。每一项决定都应基于留出周之前的日子做出。

本章,在本版次中

版次 2026.11.11示例于 2026年11月20日 通过本章 5个示例 个示例中有 5 个通过

译自英文版图书,版次 2026.11.11。

引用本节

格式

每种格式均引用英文原书,本译本以其为准。

每种格式都会注明该版次。其链接会打开当前版次,节标题处会说明该节文本最后一次修改的时间。访问日期和检索日期即为您打开此页的当天。

本页内容

第 10 章,共 10 章: Keep the forecast running

  1. Refit on a schedule
  2. 在实际值到达之前记录每一次预测
  3. Score the log as actuals arrive
  4. 对照回测标记漂移
  5. Know when to retire a model
  6. 生产中会出什么问题
  7. 本章做出的决定

返回顶部全部章节