Keep the forecast running
閱讀約需 67 分鐘 分鐘,7 節 中共有 5 個範例。
此部分尚未翻譯,因此以英文呈現。
Once a forecast runs every day, the question changes from whether it scored well to whether it still holds on the day it's used. This chapter builds the loop that answers that question. You'll refit a forecast on a schedule, log each day's forecast before its readings exist, check that the log used only what had arrived, score the readings as they come in, flag a day whose error is out of line with the backtest, and say when a model should be retired.
The stake is the next buying round. The supplier commits to power for the next day from the forecast, so a forecast that has gone wrong without anyone noticing costs money in that purchase. This chapter does not reduce that risk to zero. It makes the risk visible on the day it appears, using only what the supplier knew at the time.
The earlier chapters checked the supplier's forecast on past data. Each method was scored after the fact on the last days of 2014, with every reading in hand. In operation the order runs the other way. The supplier buys power for Dec 29, 2014 from a forecast made at 00:00 on Dec 29, the day's cutoff, using only what it knows at that time, and it learns what happened a day later. A forecast that held up on the backtest can drift. The clients change, a holiday falls inside the window, or a meter feed stops for a few hours. The supplier needs to see that before the next buying round, not in a review at the end of the quarter.
This chapter runs the loop for the last seven days of 2014, from Dec 25 to Dec 31. Each day gets one forecast for each of the thirty clients, made from its own midnight cutoff. Every time in this chapter is a data time. A cutoff is 00:00 of its day, and a reading arrives at the end of its hour. The decision and calibration runs start on Nov 25 and Dec 9, 2014, and the refit cycle is anchored to Monday, Nov 24, 2014. All of these start after Jan 16, 2014. That is the first midnight at which the start rule admits all thirty clients into one model: MT_115's first reading is at 00:00 on Jan 14, 2014, and the rule needs 48 hours of history from it, as Chapter 6 sets out.
A forecast says what the readings will do. It does not say why they do it. The loop can tell the supplier that demand has moved away from the forecast. It cannot tell the supplier which habit, price or weather caused the move, and this chapter does not try. A forecast that holds up is not a causal claim, and no flag in this chapter turns into one.
此部分尚未翻譯,因此以英文呈現。
Refit on a schedule
A refit re-estimates the model's parameters on the history known at a cutoff. Between refits, the model keeps its trees. Each forecast still reads the most recent readings to build its lag inputs, so the forecast reflects the last hours even when the trees do not.
One detail is often missed. mlforecast 1.1.0 fits each client's scaling again whenever it predicts from a new history: predict with new_df fits the target transforms on that history (forecast.py lines 1374 to 1412, and core.py lines 536 to 540 of the pinned release). Each client's mean and spread therefore come from the history known at every cutoff, and only the trees wait for a refit. That is not leakage, because that history is known at the cutoff.
The word needs one caution. In Hyndman and Athanasopoulos's Forecasting: Principles and Practice, 3rd ed. (FPP3), refit() applies a fitted model without re-estimating it (FPP3, Chapter 13). Here, "refit" means re-estimating the trees.
A refit must not use the readings it is about to forecast. A refit at cutoff c may read only readings that arrived by c. The loop enforces this by construction. The history for a refit, and for every forecast, holds only readings that had arrived by the cutoff, so the actuals of the day being forecast are never in it.
The cadence is the choice that remains. We considered two cadences, a refit every day or a refit every seven days. Both forecast every day from its own cutoff, with the same global model from Chapter 6: one scaled LightGBM over the thirty clients, with lags of 24 and 168 hours and the client identifier. The global model runs here because it is the best choosable model on the decision windows: the recorded models in Chapter 9 have no forecasts there, so they cannot be chosen. They differ only in how often the trees are re-estimated. The weekly cycle is anchored to Monday, Nov 24, 2014, so its refits fall on Mondays: Nov 24, Dec 1, Dec 8 and so on. A run's refit dates therefore do not depend on the day the run starts. Each forecast uses the model fitted on the Monday on or before its cutoff.
We choose between the cadences on the four ordinary decision windows, Nov 25, Nov 26, Dec 2 and Dec 3, 2014. None of them is a Portuguese holiday, and the held-out week is not used. The rule is the lower mean MASE on those windows. Every run in this chapter uses one scale per client, read from readings before Nov 25, 2014 under the rule in Chapter 2. The decision windows, the calibration windows and the held-out week share that yardstick. The decision and calibration histories reach back over the clock change of Oct 26, 2014. The loop fills that 01:00 hour from the hours either side, as Chapter 3 does.
Both cadences run on the four decision windows. Because the weekly model is one or two days old there, a model of each age from 0 to 6 days also forecasts those days.
1"""Choose the refit cadence on the four decision days, Nov 25 to Dec 3, 2014: a refit every day or every seven days on2Mondays, with the global model from Chapter 6. The cadence with the lower `mean MASE` line is the one the rule takes, and3the `model aged` lines show how far a model's age alone moves that score. The held-out week is not used."""4import pandas as pd56from lib.electricity import load7from lib.panel import ALL308from lib.running import DECISION_DAYS, daily_mase, feed, fit_at, forecast_at, prepare, run_days, scale_from, with_actuals9from lib.settings import show1011raw = load(ALL30)12df = prepare(raw)13scale = scale_from(raw) # one MASE scale, read before Nov 25, used by every run in the chapter14show("decision days", len(DECISION_DAYS))15# feed returns (train, known, pending count, read_by); [0] is train16show("rows read by one refit", len(feed(df, DECISION_DAYS[0])[0]))17names = {1: "daily", 7: "weekly"} # the keys are the refit interval in days18means = {}19for every in names:20 scored = with_actuals(run_days(df, DECISION_DAYS, refit_every=every), df) # refit_every: a refit every this many days21 means[every] = daily_mase(scored, scale).mean() # the mean over the four decision days of each day's MASE22 show(f"refits, {names[every]}", scored["refit"].nunique()) # the distinct refit days the decision days use23 show(f"MAE, {names[every]}", scored["ae"].mean())24 show(f"mean MASE, {names[every]}", means[every], digits=3)25show("mean MASE, daily minus weekly", means[1] - means[7], digits=3) # positive when the weekly cadence has the lower mean26show("chosen cadence", names[min(means, key=means.get)]) # the lower mean wins; the close-call tie-break is made in the prose27models = {}28for age in range(7):29 parts = []30 for day in DECISION_DAYS:31 fit_day = day - pd.Timedelta(days=age) # a model aged `age` days, fitted that many days before the decision day32 if fit_day not in models: # a model is fitted once and reused by every decision day it serves33 models[fit_day] = fit_at(df, fit_day)[0] # fit_at returns (model, read_by); [0] is the model34 # forecast_at returns (forecast, pending count, read_by); the first is the forecast35 part, _, _ = forecast_at(models[fit_day], df, day)36 parts.append(part.assign(cutoff=day)) # daily_mase groups each forecast by its cutoff37 show(f"mean MASE, model aged {age} days", daily_mase(with_actuals(pd.concat(parts), df), scale).mean(), digits=3)
python refit-cadence-decision.pydecision days: 4
rows read by one refit: 40320
refits, daily: 4
MAE, daily: 66.81
mean MASE, daily: 0.754
refits, weekly: 2
MAE, weekly: 67.64
mean MASE, weekly: 0.746
mean MASE, daily minus weekly: 0.008
chosen cadence: weekly
mean MASE, model aged 0 days: 0.754
mean MASE, model aged 1 days: 0.747
mean MASE, model aged 2 days: 0.748
mean MASE, model aged 3 days: 0.741
mean MASE, model aged 4 days: 0.748
mean MASE, model aged 5 days: 0.758
mean MASE, model aged 6 days: 0.750
On the decision windows the weekly cadence has the lower mean MASE, by 0.008, and it needs half as many fits on those four days: the daily cadence runs 4 refits, and the weekly cadence 2. Each refit reads 40320 rows, which is thirty clients times 1,344 hours of history. In operation a weekly cadence needs one seventh of the fits.
A model up to six days old scores near a fresh one on the decision windows, and that is all the age check claims. Its mean MASE runs from 0.741 at three days, the lowest of ages 0 to 6, to 0.758 at five, the highest, with a fresh model at 0.754. Age shows no trend, and the spread across all seven ages is wider than the gap between the two cadences. That gap is small, and four days cannot separate the cadences, so the example does not show that weekly is better. A reader with a larger model or a longer horizon should run the same check on their own decision windows.
The rule picks weekly, because its mean is lower. That gap sits inside the spread that model age alone produces, so the two cadences are close. When they are, the working rule takes the one with fewer refits, and the book says so here: weekly.
Working rule: Choose the refit cadence on ordinary days that come before the held-out week, scored on one scale that was read before the first of those days. Take the cadence with the lower mean MASE, and when the two are close, take the one with fewer refits and say so. Never refit on readings from the day you are forecasting.
在實際值到達之前記錄每一筆預測
預測記錄是一張表,每個客戶、每個小時各有一列。每一列包含:預測當時的截止時間、該列的寫入時間、客戶、小時、預測值、其下界與上界、模型的重新擬合日、下文所述的兩個到達時間,以及該筆預測所填補的讀數筆數。這份記錄只有一項工作:證明當時的預測是怎麼說的,以及迴圈在作出預測時能看到什麼。沒有它,就沒有人能判斷後來的報告用的是當時作出的預測,還是事後以較新的模型與實際值重建的預測。重建的記錄是偽裝起來的資料洩漏,因為它已經知道預測應該是什麼。
每一列中的區間,建構方式與第 8 章相同。它是預測值加上同一小時經過縮放的殘差,這些殘差取自校準視窗,即 2014 年 12 月 9 日至 12 月 20 日。校準視窗位於每個保留日之前,並在該週開始前就已固定。每個小時的殘差是在三十個客戶之間合併的,因此每個小時有 360 筆殘差,而不是十二筆。第 8 章的客戶區間合併的是相同的殘差,也提供 95% 的水準。這個迴圈只記錄 80% 的水準,因為最後一節的淘汰觸發條件讀取的是 80% 區間,單一水準能讓記錄與觸發條件保持一致。區間由記錄下來的預測與固定的校準殘差計算而得,因此不需要讀取它所涵蓋當天的任何讀數。
如果檢查無法失敗,記錄能證明的就比表面上少。規則是:模型與預測只能讀取在它們作出時已經到達的讀數。因此記錄為每一列保存兩個到達時間。fit_read_by 是模型擬合所用讀數中最後到達的時間,它不得晚於模型的重新擬合日。read_by 是預測所讀取讀數中最後到達的時間,它不得晚於截止時間。把 written_at 與某筆讀數的到達時間相比,並不能測試這一點。重播時會把 written_at 寫成截止時間,因此這樣的比較對任何程式碼都會通過。在正式環境中,written_at 會來自只能附加的儲存系統的時鐘;重播只能為它貼上標記。這兩項檢查會在保留週的每一列上執行,而模型的檢查還會在一個刻意洩漏的週上再執行一次,它必須能抓出洩漏。
1"""Runs the held-out week, Dec 25 to Dec 31, 2014: one forecast a day from its own midnight cutoff, with the weekly refit2cycle and the late-data fill, each logged when it is made. The `log rows` lines show both arrival checks pass, and the3leaky run fails the first. The `MAE` and `coverage 80` lines give each day's mean absolute error and how often its 80%4band held. The 80% bands use calibration days Dec 9 to Dec 20, 2014, earlier than every scored day."""5import pandas as pd67from lib.electricity import load8from lib.panel import ALL309from lib.running import CAL_DAYS, HELD_OUT_DAYS, REFIT_EVERY, band, coverage_by_day, prepare, run_days, scale_from, with_actuals10from lib.settings import show1112raw = load(ALL30)13df = prepare(raw)14scale = scale_from(raw)15cal = with_actuals(run_days(df, CAL_DAYS), df) # the residuals that set the bands, from days earlier than the week16bands = band(with_actuals(run_days(df, HELD_OUT_DAYS), df), cal, scale)17# leak=True refits each day's model on the next day's readings18leaky = with_actuals(run_days(df, HELD_OUT_DAYS, leak=True), df)19# refit dates in the week; the Dec 22 refit is dated before it20in_week = sorted({d for d in bands["refit"] if d >= HELD_OUT_DAYS[0]})21show("refit every (days)", REFIT_EVERY)22show("refit days in the week", ", ".join(f"{d:%Y-%m-%d}" for d in in_week))23show("models used in the week", bands["refit"].nunique())24show("log rows", len(bands))25# hour 0 holds 12 calibration days times 30 clients26show("calibration residuals per hour, pooled over clients", cal.dropna(subset=["e"]).groupby("hour").size().iloc[0])27# a model may read only readings that had arrived by its refit day28show("log rows whose model read only readings in by its refit day", int((bands["fit_read_by"] <= bands["refit"]).sum()))29# a forecast may read only readings that had arrived by its cutoff30show("log rows whose forecast read only readings in by its cutoff", int((bands["read_by"] <= bands["cutoff"]).sum()))31show("leaky log rows whose model read only readings in by its refit day", int((leaky["fit_read_by"] <= leaky["refit"]).sum()))32# a reading arrives at the end of its hour, so any other arrival is late33show("readings that arrived late", int((bands["arrives"] != bands["ds"] + pd.Timedelta(hours=1)).sum()))34# every row of a cutoff has the same count, so the first row gives it35show("forecast inputs filled, Dec 29", int(bands.loc[bands["cutoff"] == "2014-12-29", "filled"].iloc[0]))36# the last of Dec 28's readings to arrive37done = bands.loc[bands["cutoff"] == "2014-12-28", "arrives"].max()38show("Dec 28 complete at", f"{done:%H:%M on %b %d, %Y}")39mae = bands.groupby("cutoff")["ae"].mean()40cov = coverage_by_day(bands)41for day in mae.index:42 show(f"MAE {day:%Y-%m-%d}", mae[day])43for day in cov.index:44 show(f"coverage 80 {day:%Y-%m-%d}", cov[day], digits=3)45show("leaky MAE 2014-12-25", leaky.loc[leaky["cutoff"] == "2014-12-25", "ae"].mean())
python rolling-refit-seven-days.pyrefit every (days): 7
refit days in the week: 2014-12-29
models used in the week: 2
log rows: 5040
calibration residuals per hour, pooled over clients: 360
log rows whose model read only readings in by its refit day: 5040
log rows whose forecast read only readings in by its cutoff: 5040
leaky log rows whose model read only readings in by its refit day: 0
readings that arrived late: 6
forecast inputs filled, Dec 29: 6
Dec 28 complete at: 09:00 on Dec 29, 2014
MAE 2014-12-25: 447.59
MAE 2014-12-26: 203.43
MAE 2014-12-27: 54.14
MAE 2014-12-28: 71.67
MAE 2014-12-29: 111.64
MAE 2014-12-30: 87.37
MAE 2014-12-31: 167.15
coverage 80 2014-12-25: 0.232
coverage 80 2014-12-26: 0.431
coverage 80 2014-12-27: 0.790
coverage 80 2014-12-28: 0.764
coverage 80 2014-12-29: 0.713
coverage 80 2014-12-30: 0.782
coverage 80 2014-12-31: 0.701
leaky MAE 2014-12-25: 351.79
記錄共有 5040 列,即三十個客戶在七天內每天 24 個小時各一列,其中 5040 列通過模型的到達檢查,5040 列通過預測的到達檢查。這一週使用兩個模型。12 月 22 日星期一擬合的模型負責 12 月 25 日至 12 月 28 日,12 月 29 日星期一重新擬合的模型負責 12 月 29 日至 12 月 31 日,因此重新擬合發生在這一週之內。
一項從不失敗的檢查說明不了什麼,所以這一週也會在有洩漏的情況下執行一次。在該次執行中,每天的模型都以隔天的讀數重新擬合,因此它已經看過自己要預測的那一天,模型的檢查對每一列都會失敗:通過的有 0 列。洩漏也會讓預測看起來更好。它在 12 月 25 日的 MAE 為 351.79 kWh,而誠實執行的 MAE 為 447.59 kWh。真正該保留的是這項檢查。只要有任何修改把重新擬合移到隔天的讀數上,印出的筆數就會變成零,記錄的輸出隨即失敗。
實用原則: 在每筆預測的截止時間,以當時已到達的讀數寫入它的各列,並在旁邊附上重新擬合日與兩個到達時間。切勿以較新的模型或實際值重建過去的記錄。如果某一天的列遺失,就在記錄中將那一天標示為缺漏,事後不要補填。
此部分尚未翻譯,因此以英文呈現。
Score the log as actuals arrive
A reading scores the forecast logged for its hour when it arrives. The day's score is its daily MASE. The daily MASE takes each client's mean absolute error over the day's 24 hours, divides it by the client's scale, and averages the results over the thirty clients:
daily MASE = mean over clients of (MAE over the day's 24 hours / scale)
The scale is the one set in the refit section, read from before Nov 25, 2014, and it does not change from day to day. A reading enters the score for its own day, and only after it has arrived, so the score never uses a value the supplier did not have at the time. That is the second half of the leakage rule in the log section. A day is scored once its last reading is in. For Dec 29, that is 00:00 on Dec 30.
Most days arrive whole. The held-out week has one day that does not. The lead client's last six readings of Dec 28, 2014, from 18:00 to 23:00, arrive at 09:00 on Dec 29. That is nine hours after the Dec 29 cutoff, so the Dec 29 forecast has to be made without them. The rule for this case is the book's own choice, and no source supports it:
- A reading that has not arrived at a cutoff is left out of that day's refit.
- For the forecast at that cutoff, the missing value is filled with the reading of the same hour a week earlier, which had arrived. The log counts how many values were filled.
- The forecast is not redone when the reading arrives.
- The reading is scored when it arrives, and its day is scored once its last reading is in.
The first option tried was to leave the missing values as they were. The library refused them. At the pinned version, 1.1.0 raised this error when the forecast was made:
ValueError: y column contains null values.
There are two ways to fill the gap. Forward fill repeats the last reading that arrived. That flattens the evening, because the lead client's 18:00 to 23:00 values all become the 17:00 value, and lag 24 carries that flat evening into the next day's evening forecast. Filling from the same hour a week earlier keeps the evening's shape. We picked the fill on the decision windows, not the held-out week. There the same six readings are made late before each decision cutoff, and both fills forecast from them.
1"""Chooses how a late reading is filled on the decision windows, Nov 25 to Dec 3, 2014: the lead client's last six readings2of the day before arrive at 09:00 on each cutoff day, as the Dec 28 readings do in the held-out week. The3`MAE, lead client hours 18 to 23` lines decide between forward fill and a fill from the same hour a week earlier; only4those hours change, because lag 24 reads them, and the held-out week is not used."""5import pandas as pd67from lib.electricity import load8from lib.panel import ALL309from lib.running import DECISION_DAYS, LATE_CLIENT, feed, late_copy, prepare, run_days, with_actuals10from lib.settings import show1112raw = load(ALL30)13df = prepare(raw)14# feed returns (train, known, pending count, read_by); [2] is the pending count15show("readings late for each decision cutoff", feed(late_copy(df, DECISION_DAYS[0]), DECISION_DAYS[0])[2])16errors = {}17for rule in ("forward", "week"):18 parts = []19 for day in DECISION_DAYS:20 # the day before's last six readings arrive at 09:00 on this day, after its cutoff, so the fill applies21 scored = with_actuals(run_days(late_copy(df, day), [day], fill=rule), df)22 # 18:00 to 23:00 of the day scored, which lag 24 reads from the day before23 parts.append(scored.loc[(scored["unique_id"] == LATE_CLIENT) & (scored["hour"] >= 18), "ae"])24 errors[rule] = pd.concat(parts).mean()25 show(f"hours scored, {rule} fill", len(pd.concat(parts)))26 show(f"MAE, lead client hours 18 to 23, {rule} fill", errors[rule])27show("chosen late fill", min(errors, key=errors.get))
python late-fill-decision.pyreadings late for each decision cutoff: 6
hours scored, forward fill: 24
MAE, lead client hours 18 to 23, forward fill: 7.92
hours scored, week fill: 24
MAE, lead client hours 18 to 23, week fill: 5.03
chosen late fill: week
The week-earlier fill is the more accurate of the two. Over the lead client's hours from 18:00 to 23:00, across 24 forecast hours, its mean absolute error is 5.03 kWh, against 7.92 kWh with forward fill. The book uses it in every run.
The held-out week shows the rule at work. 6 readings arrived late, and the Dec 29 forecast filled 6 inputs. The Dec 29 refit left the same six readings out, so the branch that leaves a reading out of a refit runs in this week too. Dec 28 was complete at 09:00 on Dec 29, 2014.
A single day's MAE does not say whether the day was bad, because the level of demand changes from day to day. That is why the flag in the next section divides by a reference.
Working rule: Score each day when its last reading arrives, against the forecast logged at its cutoff, on the chapter's one scale. When a reading arrives late, score it then, count it in the log, and leave the logged forecast alone. Never change a forecast after its cutoff, even when the change would improve the score.
對照回測標示漂移
上一節的每日分數需要一條界線,用來判斷某一天是否偏離常態。標示是針對某一天提出的問題,不是定論。這條規則要問的是:某一天的每日 MASE 是否偏離模型先前的表現。當某一天的每日 MASE 高於參考值的兩倍時,該日就會被標示。參考值是校準視窗的每日 MASE 中位數,涵蓋 2014 年 12 月 9 日至 12 月 20 日。這些日子都早於標示所要判斷的每一天。其中沒有任何一天是葡萄牙的假日,不過 12 月 9 日的 lag-24 讀值會回溯到 12 月 8 日,也就是全國假日 Imaculada Conceição(聖母無原罪節)。這個回聲只有一天的分量,中位數會將它的影響減弱。
「兩倍」這個係數是本書的經驗法則。我們不知道有哪個已發表的門檻適用於這類標示,所以這個數字是我們自訂的,並且明確標註為自訂。我們在決策視窗上檢驗了它,其中四個平常日用來測試:在沒有已知原因時,標示是否會觸發。
1"""Flags drift against the backtest: a day is flagged when its daily MASE is above FLAG_FACTOR times the reference, the2median daily MASE of the calibration days, Dec 9 to Dec 20, 2014, which are earlier than the held-out week. The3`largest decision day` and `decision days flagged` lines show the ordinary days staying under the line, and the `MASE`4and `held-out days flagged` lines show which held-out days cross it."""5from lib.electricity import load6from lib.panel import ALL307from lib.running import (CAL_DAYS, DECISION_DAYS, FLAG_FACTOR, HELD_OUT_DAYS, daily_mase, flagged, prepare, run_days,8 scale_from, with_actuals)9from lib.settings import show1011raw = load(ALL30)12df = prepare(raw)13scale = scale_from(raw)14reference = daily_mase(with_actuals(run_days(df, CAL_DAYS), df), scale).median() # the median, not the mean, so one unusual calibration day moves it little15show("reference, median daily MASE", reference)16show("flag above, MASE", FLAG_FACTOR * reference) # FLAG_FACTOR is 2, the chapter's rule of thumb17decision = daily_mase(with_actuals(run_days(df, DECISION_DAYS), df), scale)18show("largest decision day, times the reference", decision.max() / reference)19show("decision days flagged", len(flagged(decision, reference)))20held = daily_mase(with_actuals(run_days(df, HELD_OUT_DAYS), df), scale)21for day, value in held.items():22 show(f"MASE {day:%Y-%m-%d}", value)23show("held-out days flagged", len(flagged(held, reference)))24show("flagged held-out days", ", ".join(f"{day:%Y-%m-%d}" for day in flagged(held, reference)))
python drift-flag-rolling.pyreference, median daily MASE: 0.72
flag above, MASE: 1.43
largest decision day, times the reference: 1.44
decision days flagged: 0
MASE 2014-12-25: 4.62
MASE 2014-12-26: 2.00
MASE 2014-12-27: 0.73
MASE 2014-12-28: 0.76
MASE 2014-12-29: 0.85
MASE 2014-12-30: 0.76
MASE 2014-12-31: 1.14
held-out days flagged: 2
flagged held-out days: 2014-12-25, 2014-12-26
在係數為二時,沒有任何平常的決策日被標示(標示了 0 天)。最大的決策日是參考值的 1.44 倍。若採用較小的係數,例如 1.5,這一天的餘裕就很小,稍微異常的平常日便可能被標示。我們保留 2,是因為它有較多餘裕。四個平常日無法顯示平常日被標示的頻率。
七個保留日中有 2 天超過界線,分別是 12 月 25 日與 12 月 26 日。它們的每日 MASE 為 4.62 與 2.00,而標示線為 1.43。接下來是 12 月 31 日跨年夜,它低於界線。
這兩個標示都有已知原因。12 月 25 日是聖誕節,是葡萄牙的假日。模型沒有假日輸入。第 5 章曾在子集的決策視窗上嘗試日曆與假日欄位,包括假日欄位,但沒有一個勝過單純的 lag,所以第 6 章沒有給全域模型任何這類欄位。12 月 26 日在葡萄牙不是假日,但它的 lag-24 讀值是 12 月 25 日。12 月 26 日每個小時的預測,讀取的都是 12 月 25 日的同一小時,因此假日的型態延續到了隔天。這些是日曆效應,不是漂移,而迴圈無法分辨兩者的差別。它只負責標示日期,由人來檢視每個標示並對照日曆。
單單一天並不構成定論。一個偏高的日子可能來自假日、暴風雨,或是輸入錯誤的讀值。連續兩個標示而且沒有日曆上的原因,就有理由檢視模型。下一節會說明屆時該怎麼做。
實用原則: 當某一天的每日 MASE 高於先前校準視窗中位數的兩倍時,標示該日,接著詢問是否有日曆上的原因可以解釋。若有原因,就把原因記錄在標示旁。連續兩個無法解釋的標示,應視為汰換檢查的開端,絕不能當作模型已經損壞的證明。
此部分尚未翻譯,因此以英文呈現。
Know when to retire a model
Retiring a model means stopping its use and replacing it. A refit is the lighter move, and it comes first. This chapter names three triggers for either move, and each one reads a number the loop already logs. The first retirement trigger is sustained drift: flags on consecutive days with no calendar cause. The second retirement trigger is an uncovered band: the 80% band covers well under 80% of readings on several days. The third is a change in a library the model depends on.
The second trigger needs a sense of what ordinary coverage looks like. On the ordinary days, Dec 27 to Dec 31, coverage at 80% stays between 0.701 and 0.790. Dec 29, when the late readings arrive, and Dec 31, New Year's Eve, count as ordinary days here. Dec 25 is lower, at 0.232, and Dec 26 is at 0.431, for the reason the flags gave. Dec 25 is the holiday, and Dec 26 reads it through lag 24. The bands are a little narrow even on ordinary days, as Chapter 8 found. So a day between 0.70 and 0.80 is normal here, and a day far below that is not.
To show what the flags and bands do, the chapter stages a change. It is invented for the test, not a fact about the supplier's data. From Dec 27, 2014, every reading of all thirty clients is multiplied by 0.6, as if demand had dropped by 40%. The held-out week then runs on the changed readings, with the same cadence, reference, rule and bands as before.
1"""Shows how a change appears in the flags: a stated test, not a fact about the data, in which every reading from Dec 27,22014 is multiplied by 0.6, as if demand had dropped by 40%. The calibration windows are earlier than the change, so the3reference and bands are fixed before it happens. The `held-out days flagged` and `coverage 80` lines show what the flags4and bands do, with the same cadence, reference, rule and bands as the held-out runs in drift-flag-rolling and5rolling-refit-seven-days."""6import pandas as pd78from lib.electricity import load9from lib.panel import ALL3010from lib.running import (CAL_DAYS, HELD_OUT_DAYS, band, coverage_by_day, daily_mase, flagged, prepare, run_days,11 scale_from, with_actuals)12from lib.settings import show1314raw = load(ALL30)15df = prepare(raw)16scale = scale_from(raw)17cal = with_actuals(run_days(df, CAL_DAYS), df) # calibration windows come before the change, so the reference is fixed before it18reference = daily_mase(cal, scale).median()19changed = df.copy()20after = changed["ds"] >= pd.Timestamp("2014-12-27") # the first changed day, Dec 27, from its midnight on21changed.loc[after, "y"] = changed.loc[after, "y"] * 0.6 # a stated test: every reading from Dec 27 at 60%, as if demand fell 40%22held = with_actuals(run_days(changed, HELD_OUT_DAYS), changed) # the changed readings go through the same loop, refits included2324show("reference, median daily MASE", reference)25mase = daily_mase(held, scale)26for day, value in mase.items():27 show(f"MASE {day:%Y-%m-%d}", value)28show("held-out days flagged", len(flagged(mase, reference)))29cov = coverage_by_day(band(held, cal, scale)) # the bands use the unchanged calibration days, as in the held-out run30for day, value in cov.items():31 show(f"coverage 80 {day:%Y-%m-%d}", value, digits=3)
python drift-after-a-change.pyreference, median daily MASE: 0.72
MASE 2014-12-25: 4.62
MASE 2014-12-26: 2.00
MASE 2014-12-27: 4.74
MASE 2014-12-28: 2.93
MASE 2014-12-29: 1.68
MASE 2014-12-30: 1.78
MASE 2014-12-31: 1.89
held-out days flagged: 7
coverage 80 2014-12-25: 0.232
coverage 80 2014-12-26: 0.431
coverage 80 2014-12-27: 0.078
coverage 80 2014-12-28: 0.179
coverage 80 2014-12-29: 0.328
coverage 80 2014-12-30: 0.328
coverage 80 2014-12-31: 0.333
All 7 held-out days are flagged. Dec 25 and Dec 26 are flagged for Christmas and its echo, as before, and each day from Dec 27, the first changed day, is flagged too. On Dec 27 the daily MASE is 4.74, and coverage falls to 0.078 that day, the lowest of the week.
One scheduled refit did not clear the flags. The refit on Monday, Dec 29, is the only step in this test that refits on changed readings, and it read Dec 27 and Dec 28 at the lower level. Its daily MASE fell by about 40%, from 2.93 on Dec 28 to 1.68 on Dec 29, but Dec 30 and Dec 31 stayed above the line. Coverage recovered to 0.328 on Dec 29 and 0.333 on Dec 31, still far below 80%.
A run of flags like this, with coverage that low, is a case for an early refit, and the book's rule is to refit early. We did not test an early refit on a day of its own, so that rule is a rule of thumb and not a measured result.
The third trigger is a library change. Libraries release often, and a later release can change an output even when the code stays the same. The loop depends on 1.1.0 and 4.7.0. Prophet, which Chapter 9 pins at 1.5.0, is not part of this loop, but it is another pinned library, and its release history shows how often such a library moves (PyPI release history). A release is a reason to run this chapter's checks again. It is not, by itself, a reason to retire a model.
These checks are the forecast log, the flags and the coverage of the bands. They are not a monitoring system. A monitoring system covers alerts, dashboards, data quality across many feeds and the people who respond. A later book on monitoring models in production can continue this case, and this chapter ends where that book would begin.
Working rule: Refit early when flags run on several days in a row with no calendar cause, or when the 80% band covers far below 80% of readings on several days, and record the trigger with the log. Retire the model when an early refit does not clear the flags. Re-run this chapter's checks when a pinned library is released, and keep the pins until the checks pass on the new version.
正式環境中會出什麼問題
資料來源可能在某段歷史資料的中途停止。迴圈不會填補資料來源從未送達的讀數。中間若有缺口,所有的落後值(lag)都會位移,因此程式碼會改為引發錯誤。只有時鐘調整所造成的那幾個小時會被填補,依據的是第 3 章的規則。歷史資料結尾仍然缺少的讀數則不同,由延遲資料規則處理。
重新擬合可能造成洩漏。如果重新擬合讀取了即將預測的那一天的讀數,回測的結果就會比實際運行時更好。你會看到同一些日子的實際 MASE 遠高於回測的 MASE。迴圈以其歷史資料規則防止這種情況,重寫迴圈時,這條規則是唯一必須保留的。請保留第 3 章的洩漏檢查(位移測試),以及第 7 章的汙染檢查。汙染檢查會在截止點之後,把所有讀數都改寫成錯誤的值,然後確認該截止點的重新擬合與預測不會改變。
針對三十個客戶平均值所設的旗標,可能掩蓋單一客戶的變化。單一客戶的大幅變化,只會讓每日 MASE 移動該客戶每日 MASE 變化量除以三十的幅度。壓力測試範例會同時改變所有客戶,所以旗標會觸發。我們沒有測試只有單一客戶改變的情況。你會看到該客戶自身的 MASE 遠高於其校準中位數。這是我們最先會補上的缺口,做法可能是針對每個客戶自身的 MASE 另設第二個旗標。
固定版本的函式庫在更新固定版本後,行為仍可能改變。評分一節中的空值錯誤是 mlforecast 1.1.0 的行為。後續版本可能會填補、捨棄或接受該缺口,預測也就會隨之改變。更新固定版本等於更改預測,需要重新執行檢查。
最後一種失敗是事後才寫成的日誌。範例中沒有任何機制能阻止團隊日後重建日誌,而重建出來的輸出看起來會是正確的。兩個到達時間欄位及其檢查,正是用來揭露這種情況的。一旦移除這些檢查,日誌就不再能證明任何事。
實用原則: 每次執行都要保留三項檢查:讓未來資料不進入每次重新擬合與預測的歷史資料規則、針對每一列日誌的兩項到達時間檢查,以及第 3 章與第 7 章的洩漏檢查。其中任何一項失敗時,請停止迴圈,並在下一次預測之前修正原因。不要以手動方式修補歷史資料或日誌,因為手動修補正是這些檢查要攔截的那類變更。
本章做出的決定
- **重新擬合頻率:**每七天一次,於星期一進行。在決策視窗上,每週重新擬合的平均 MASE 較低,差距為 0.008,這個差距落在僅由模型老化就會產生的波動範圍之內,因此採用平手時的取捨規則,本書選擇重新擬合次數較少的頻率。
- **單一 MASE 尺度:**取自 2014 年 11 月 25 日之前的讀數,並用於本章的每一次執行,遵循第 2 章的規則。因此,決策日、校準視窗與保留日都以同一把尺來評分,而且沒有任何被評分的小時參與尺度的設定。
- **標記規則:**當某一天的 MASE 高於校準視窗(2014 年 12 月 9 日至 12 月 20 日)每日 MASE 中位數的兩倍時,該日即被標記。兩倍這個係數是有標示的經驗法則,保留它是因為它與一般決策視窗之間有足夠的餘裕。
- **預測區間:**只採用 80%,因為停用觸發條件讀取的是這個水準;第 8 章也提供 95%。區間使用同一小時的縮放後殘差,由三十位客戶彙整,取自十二個校準視窗,並在保留週之前固定。紀錄中沒有記錄總量預測、總量區間或採購分位數,所以紀錄檢查的是各客戶的預測,而不是第 8 章針對總量所做的採購。若供應商依總量採購,就要把該分位數加入紀錄。
- **延遲讀數:**不納入重新擬合;預測時以一週前同一小時的值填補;在紀錄中計數;到達時才評分;絕不用來重做預測。填補方式是在決策視窗上選定的。這條規則沒有外部來源,是本書自己訂的。
- **範圍:**三十位客戶、七天。一週內包含一個假日、一筆延遲讀數,以及落在其中的一次每週重新擬合,因此每條規則都能呈現出來。若執行更長的時間,就能衡量一般日子被標記的頻率,而本章並未衡量這一點。
- **監控邊界:**僅限本章的檢查。長期監看這些檢查的監控系統不在本書範圍內。
- **你自己的情況:**本書最後一項決定是把工作交還給你。供應商的案例到此為止:每週重新擬合、檢查每次預測所讀內容的紀錄、隨讀數到達而評分,以及對照回測的標記。你的情況需要在自己的資料上重新做出這些決定:依你的一般決策視窗決定重新擬合頻率,依你實際收到的資料流決定延遲資料規則,依你的校準視窗決定標記係數,並依你的採購規則所需決定區間水準。每一項決定都要根據保留週之前的日子來做。
本章,於本版次中
版次 2026.11.11範例於 2026年11 月 20 通過測試本章 5 個範例 個範例中有 5 個通過
譯自英文書,版次 2026.11.11。