为每位客户拟合统计模型
阅读约需 60分钟 分钟,10节中含6个示例。
本章在同一回测上为每位客户拟合 AutoETS 和固定的季节性 ARIMA,将两者与季节性朴素法对比评分,并学习如何让起步较晚的客户从其第一个读数开始建模。本章假定读者已掌握前两章的基线与回测。
供应商的每位客户都需要各自的日前预测,而第 2 章中的季节性朴素法对所有客户给出同样的答案:昨天的同一小时。这条规则不保留对客户水平或趋势的任何估计,而各客户在这两方面都不相同。因此,本章为每位客户拟合一个统计模型,并考察这些额外的工作是否值得。
这部分尚未翻译,因此以英文显示。
Why a statistical model comes next
Chapter 2 set the bar. Seasonal naive forecasts each hour as the same hour yesterday, and the scaled error, MASE, compares clients with different loads on one scale. Chapter 3 backtested that forecast over many cutoffs. A cutoff is the time a forecast is made, and its history ends there. A window is one cutoff and the 24 hours after it, which is the horizon. Each window's history ends at its cutoff, so no hour is forecast with data from its own day.
A statistical model adds to that loop. A series has a state. The level of a series is its typical reading, the trend is the slow drift of the level, the season is the daily shape of the load, and what none of these explain is error. A statistical model holds an estimate of each part of the state of a series, updates the estimates as each reading arrives, and projects them over the horizon. Seasonal naive has no such state. It copies yesterday, so it gets a client's level only as well as yesterday did.
A per-client model is one fit for each client, on that client's history alone, so the cost is one fit per client per window. The subset this chapter uses, SUBSET8, has eight clients. Over the book's three standard cutoffs, Dec 29 to Dec 31, 2014, the last three days of the held-out week and the windows every later chapter reports, that is 24 fits for each model. The code runs the fits in one worker (n_jobs=1), which keeps the output identical from run to run. Every model in this chapter gets the same three cutoffs and the same 1,344 hours of history before each one. The cutoffs fall in the held-out week, and the subset holds the one late starter a later section needs.
Working rule: Fit one model per client only after a baseline and a backtest are in place. Count the fits before you choose the number of windows: clients, times windows, times refits if you run the model every day. Use the same cutoffs and history for every model in a comparison. Otherwise the comparison measures the windows, not the models.
这部分尚未翻译,因此以英文显示。
Fit exponential smoothing
Seasonal naive keeps only yesterday, so the first candidate keeps a level instead. Exponential smoothing moves its level toward each new reading. The simplest form updates the level as level_t = alpha * y_t + (1 - alpha) * level_(t-1). The constant alpha, between 0 and 1, sets how fast the level follows the data. The forecast for every hour of the horizon is the last level. Recent readings count for more than older ones, and the weights fall off with age.
ETS, for error, trend and season, extends the simplest form of exponential smoothing. Each part is chosen from a short list. The trend can be none, additive or damped. The error can be additive or multiplicative, and the season can be none, additive or multiplicative. A name such as ETS(A,N,A) reads in that order: additive error, no trend, additive season. ETS(M,Ad,M) has a multiplicative error, a damped trend and a multiplicative season.
AutoETS chooses the ETS form for you. Its default criterion is AICc, the corrected Akaike information criterion, which charges a model for each parameter and corrects for small samples, as the AutoETS docstring in statsforecast 2.1.1 (statsforecast/models.py) says. Two of its defaults matter for hourly demand. The season length defaults to 1, which means no season at all, and the model gives no warning when you leave it that way. The model argument defaults to "ZZZ", which lets the model choose all three parts. For hourly demand with a daily pattern, the call sets season_length=24.
Which form does AutoETS pick for each client, and do its forecasts beat seasonal naive? The lines it prints answer both.
1"""Backtest AutoETS and seasonal naive for the eight clients of SUBSET8 over the three standard cutoffs, and score both.2The fits lines count the forms AutoETS chose, and the MASE and coverage lines are the scores the comparison table quotes."""3from collections import Counter, defaultdict45import numpy as np6import pandas as pd7from statsforecast import StatsForecast8from statsforecast.models import AutoETS, SeasonalNaive910from lib.scoring import book_scale11from lib.electricity import load12from lib.quirks import filled_history, mask_clock_changes13from lib.settings import EVAL_CUTOFFS, HORIZON, SEED, SUBSET8, show14from lib.subsets import score_clients, show_client_lines15np.random.seed(SEED) # the seed from lib/settings.py, set before any fit16masked = mask_clock_changes(load(SUBSET8)) # clock-change hours become NaN, so they are never scored; filled_history fills them from their neighbours before each fit17scale = book_scale(load(SUBSET8), EVAL_CUTOFFS[0]) # MASE scale: rows before the first cutoff, from each client's first reading18# season_length=24 for hourly demand; the default of 1 would mean no season at all. n_jobs=1 keeps the output the same from run to run.19sf = StatsForecast(models=[AutoETS(season_length=24), SeasonalNaive(season_length=24)], freq="h", n_jobs=1)20parts, forms, by_client = [], Counter(), defaultdict(list)21for c in pd.to_datetime(EVAL_CUTOFFS):22 sf.fit(df=filled_history(masked, c)[["unique_id", "ds", "y"]]) # the eight weeks before c, clock hours filled from them23 part = sf.predict(h=HORIZON, level=[95]) # level=[95] asks for the 95% interval the coverage line scores24 # the first forecast hour must be the cutoff itself, or the window is misaligned25 if part["ds"].min() != c:26 raise ValueError(f"forecast for {c:%Y-%m-%d} does not start at the cutoff")27 parts.append(part)28 for i, uid in enumerate(sf.uids):29 form = sf.fitted_[i, 0].model_["method"] # model 0 is AutoETS; its method name is the form, such as ETS(A,N,A)30 forms[form] += 131 by_client[uid].append(form)32forecasts = pd.concat(parts, ignore_index=True)33summary, per_client = score_clients(forecasts, masked, ["AutoETS", "SeasonalNaive"], scale, interval_model="AutoETS")34show("horizon hours", HORIZON)35show("windows", len(parts))36for form, n in sorted(forms.items()):37 show(f"fits that chose {form}", n)38# the form reads ETS(error,trend,season): a multiplicative part is an M in any of the three slots39show("fits with a multiplicative part", sum(n for form, n in forms.items() if "M" in form[4:-1].split(",")))40for label, value in summary.items():41 show(label, value, digits=3 if label.startswith("coverage") else 2) # coverage is a share, to three decimals; the rest are kWh or ratios42show("clients where AutoETS has lower MASE", int((per_client["AutoETS"] < per_client["SeasonalNaive"]).sum()))43show_client_lines(by_client, per_client, ["AutoETS", "SeasonalNaive"])
python autoets-subset-backtest.pyhorizon hours: 24
windows: 3
fits that chose ETS(A,Ad,A): 1
fits that chose ETS(A,N,A): 8
fits that chose ETS(A,N,N): 9
fits that chose ETS(M,Ad,M): 2
fits that chose ETS(M,N,A): 3
fits that chose ETS(M,N,M): 1
fits with a multiplicative part: 6
clients: 8
scored hours per client: 72
MAE AutoETS: 362.11
RMSE AutoETS: 537.20
MASE AutoETS: 1.30
MAE SeasonalNaive: 423.68
RMSE SeasonalNaive: 591.25
MASE SeasonalNaive: 1.16
coverage 95 AutoETS: 0.910
clients where AutoETS has lower MASE: 4
forms MT_014, one per cutoff: ETS(A,N,N) ETS(A,N,N) ETS(A,N,N)
MASE MT_014 AutoETS: 1.18
MASE MT_014 SeasonalNaive: 0.97
forms MT_064, one per cutoff: ETS(A,N,N) ETS(A,N,N) ETS(A,N,N)
MASE MT_064 AutoETS: 2.32
MASE MT_064 SeasonalNaive: 1.08
forms MT_115, one per cutoff: ETS(M,N,A) ETS(M,N,A) ETS(M,N,A)
MASE MT_115 AutoETS: 0.95
MASE MT_115 SeasonalNaive: 1.31
forms MT_166, one per cutoff: ETS(A,Ad,A) ETS(A,N,A) ETS(A,N,A)
MASE MT_166 AutoETS: 1.18
MASE MT_166 SeasonalNaive: 1.29
forms MT_217, one per cutoff: ETS(M,N,M) ETS(M,Ad,M) ETS(M,Ad,M)
MASE MT_217 AutoETS: 1.76
MASE MT_217 SeasonalNaive: 1.25
forms MT_268, one per cutoff: ETS(A,N,A) ETS(A,N,A) ETS(A,N,A)
MASE MT_268 AutoETS: 0.94
MASE MT_268 SeasonalNaive: 0.89
forms MT_319, one per cutoff: ETS(A,N,A) ETS(A,N,A) ETS(A,N,A)
MASE MT_319 AutoETS: 1.11
MASE MT_319 SeasonalNaive: 1.26
forms MT_370, one per cutoff: ETS(A,N,N) ETS(A,N,N) ETS(A,N,N)
MASE MT_370 AutoETS: 0.96
MASE MT_370 SeasonalNaive: 1.25
The chosen form changes from client to client, and for some clients from one cutoff to the next. Across the 24 fits, AutoETS chose ETS(A,N,N), with no season, for 9 of them, and ETS(A,N,A), with an additive season, for 8. ETS(A,Ad,A), a damped trend with an additive season, came up in 1 of the fits. The other fits used a multiplicative part, in the error, the season or both. The lines under the totals give each client's three forms. Three clients, MT_014, MT_064 and MT_370, take the flat form at all three cutoffs. MT_014 is the lead client, the one Chapter 1 met.
A flat form winning looks odd when the seasonal form contains it. ETS(A,N,N) is ETS(A,N,A) with the seasonal part switched off, so the best seasonal fit cannot have a lower likelihood than the flat fit on the same history. Fitting both forms on each of the 24 windows shows whether that happens.
1"""For each subset window, fit the flat form ETS(A,N,N) and the additive-season form ETS(A,N,A) on the same history, and count where the seasonal fit has the lower likelihood.2The two counts printed last are the ones the prose quotes."""3import numpy as np4import pandas as pd5from statsforecast import StatsForecast6from statsforecast.models import AutoETS78from lib.electricity import load9from lib.quirks import filled_history, mask_clock_changes10from lib.settings import EVAL_CUTOFFS, SEED, SUBSET8, show1112np.random.seed(SEED) # the seed from lib/settings.py, set before any fit13masked = mask_clock_changes(load(SUBSET8))14windows = seasonal_lower_loglik = seasonal_higher_aicc = 015for c in pd.to_datetime(EVAL_CUTOFFS):16 hist = filled_history(masked, c)[["unique_id", "ds", "y"]]17 # ANN is additive error, no trend, no season: the flat form. ANA is the same with an additive season.18 sf = StatsForecast(models=[AutoETS(model="ANN", season_length=24, alias="flat"),19 AutoETS(model="ANA", season_length=24, alias="seasonal")], freq="h", n_jobs=1)20 sf.fit(df=hist)21 for i in range(len(sf.uids)):22 flat = sf.fitted_[i, 0].model_ # models are fitted in the order listed: 0 is the flat fit23 seasonal = sf.fitted_[i, 1].model_ # 1 is the seasonal fit24 windows += 125 # ANA contains ANN, so a lower seasonal likelihood points to a fit that stopped short of its best26 seasonal_lower_loglik += int(seasonal["loglik"] < flat["loglik"])27 # AICc charges each parameter, so this count can differ from the likelihood count28 seasonal_higher_aicc += int(seasonal["aicc"] > flat["aicc"])29show("windows", windows)30show("seasonal fit has the lower log-likelihood", seasonal_lower_loglik)31show("seasonal fit has the higher AICc", seasonal_higher_aicc)
python flat-form-check.pywindows: 24
seasonal fit has the lower log-likelihood: 9
seasonal fit has the higher AICc: 9
The seasonal fit loses on likelihood, not only on AICc. Its log-likelihood is lower in 9 of the 24 windows, and its AICc is higher in 9. Those are the same nine windows in which the default run chose the flat form. AICc charges a model for each parameter, and the seasonal form has 24 more. That charge adds to the gap; it does not cause it, because the seasonal fit's likelihood is already lower before any charge. The seasonal fit most likely stopped short of its best value, and the flat form won by default. The book does not chase the optimiser; it tests the choice with a forced season.
Chapter 1 forced an additive season on the lead client, with model="ZZA", and seasonal naive still had the lower error there. Does that hold across all eight? The forced run covers the same eight clients, beside the default.
1"""Backtest AutoETS with the season forced to additive (model ZZA) and seasonal naive, for the eight clients of SUBSET8 over the three standard cutoffs.2The ZZA lines are the forced-season scores that the comparison table and the forced-season paragraph quote."""3from collections import Counter, defaultdict45import numpy as np6import pandas as pd7from statsforecast import StatsForecast8from statsforecast.models import AutoETS, SeasonalNaive910from lib.scoring import book_scale11from lib.electricity import load12from lib.quirks import filled_history, mask_clock_changes13from lib.settings import EVAL_CUTOFFS, HORIZON, SEED, SUBSET8, show14from lib.subsets import score_clients, show_client_lines15np.random.seed(SEED) # the seed from lib/settings.py, set before any fit16masked = mask_clock_changes(load(SUBSET8)) # clock-change hours become NaN, so they are never scored; filled_history fills them from their neighbours before each fit17scale = book_scale(load(SUBSET8), EVAL_CUTOFFS[0]) # MASE scale: rows before the first cutoff, from each client's first reading18# ZZA: Z lets the model choose the error and the trend, A forces an additive season. alias names the output columns.19sf = StatsForecast(models=[AutoETS(model="ZZA", season_length=24, alias="ZZA"), SeasonalNaive(season_length=24)],20 freq="h", n_jobs=1)21parts, forms, by_client = [], Counter(), defaultdict(list)22for c in pd.to_datetime(EVAL_CUTOFFS):23 sf.fit(df=filled_history(masked, c)[["unique_id", "ds", "y"]]) # the eight weeks before c, clock hours filled from them24 part = sf.predict(h=HORIZON, level=[95]) # level=[95] asks for the 95% interval the coverage line scores25 # the first forecast hour must be the cutoff itself, or the window is misaligned26 if part["ds"].min() != c:27 raise ValueError(f"forecast for {c:%Y-%m-%d} does not start at the cutoff")28 parts.append(part)29 for i, uid in enumerate(sf.uids):30 form = sf.fitted_[i, 0].model_["method"] # model 0 is the forced AutoETS; its method name is the form31 forms[form] += 132 by_client[uid].append(form)33forecasts = pd.concat(parts, ignore_index=True)34summary, per_client = score_clients(forecasts, masked, ["ZZA", "SeasonalNaive"], scale, interval_model="ZZA")35show("horizon hours", HORIZON)36show("windows", len(parts))37for form, n in sorted(forms.items()):38 show(f"fits that chose {form}", n)39for label, value in summary.items():40 show(label, value, digits=3 if label.startswith("coverage") else 2)41show("clients where ZZA has lower MASE", int((per_client["ZZA"] < per_client["SeasonalNaive"]).sum()))42show_client_lines(by_client, per_client, ["ZZA", "SeasonalNaive"])
python autoets-forced-season-backtest.pyhorizon hours: 24
windows: 3
fits that chose ETS(A,Ad,A): 1
fits that chose ETS(A,N,A): 17
fits that chose ETS(M,N,A): 6
clients: 8
scored hours per client: 72
MAE ZZA: 365.24
RMSE ZZA: 553.11
MASE ZZA: 1.17
MAE SeasonalNaive: 423.68
RMSE SeasonalNaive: 591.25
MASE SeasonalNaive: 1.16
coverage 95 ZZA: 0.925
clients where ZZA has lower MASE: 4
forms MT_014, one per cutoff: ETS(A,N,A) ETS(A,N,A) ETS(A,N,A)
MASE MT_014 ZZA: 1.08
MASE MT_014 SeasonalNaive: 0.97
forms MT_064, one per cutoff: ETS(A,N,A) ETS(A,N,A) ETS(A,N,A)
MASE MT_064 ZZA: 1.27
MASE MT_064 SeasonalNaive: 1.08
forms MT_115, one per cutoff: ETS(M,N,A) ETS(M,N,A) ETS(M,N,A)
MASE MT_115 ZZA: 0.95
MASE MT_115 SeasonalNaive: 1.31
forms MT_166, one per cutoff: ETS(A,Ad,A) ETS(A,N,A) ETS(A,N,A)
MASE MT_166 ZZA: 1.18
MASE MT_166 SeasonalNaive: 1.29
forms MT_217, one per cutoff: ETS(M,N,A) ETS(M,N,A) ETS(M,N,A)
MASE MT_217 ZZA: 1.78
MASE MT_217 SeasonalNaive: 1.25
forms MT_268, one per cutoff: ETS(A,N,A) ETS(A,N,A) ETS(A,N,A)
MASE MT_268 ZZA: 0.94
MASE MT_268 SeasonalNaive: 0.89
forms MT_319, one per cutoff: ETS(A,N,A) ETS(A,N,A) ETS(A,N,A)
MASE MT_319 ZZA: 1.11
MASE MT_319 SeasonalNaive: 1.26
forms MT_370, one per cutoff: ETS(A,N,A) ETS(A,N,A) ETS(A,N,A)
MASE MT_370 ZZA: 1.05
MASE MT_370 SeasonalNaive: 1.25
On mean MASE, forcing the season brings AutoETS within a hair of seasonal naive, though not below it: 1.17 against 1.30 for the default and 1.16 for seasonal naive.
The forced season helps some clients and hurts others. It lowers the lead client's MASE, from 1.18 to 1.08, and MT_064's too, but it raises MT_370's, from 0.96 to 1.05. So forcing the season is a change to test on each client, not a fix to apply to all of them.
Working rule: Start with AutoETS for hourly demand with season_length=24, and print the chosen forms in every backtest. When a client takes a flat form, check that its seasonal form is not failing. If the seasonal fit has the lower likelihood, force the season with model="ZZA", and score both forms on earlier windows before you keep either: cutoffs before the held-out week that the report does not score, such as the four ordinary days in late November and early December 2014, which Chapter 5 sets aside for choices. The standard cutoffs show the comparison here, but they do not decide it. When a forecast gets worse, look first for a form that changed between cutoffs.
拟合固定的季节性 ARIMA
第二个候选模型针对季节性朴素法遗留下来的部分建模。第 3 章的回测则用来检验它新增了什么。ARIMA 模型在差分之后,把下一个读数写成先前读数与先前误差的加权和;差分即用一个读数减去前一个季节的读数。该模型记作 ARIMA(p,d,q)(P,D,Q)[m]。非季节部分有 p 个自回归项、d 次差分和 q 个移动平均项。季节部分有同样的三类项,只是滞后阶数为季节长度 m 的倍数。
本书使用的模型是 ARIMA(1,0,0)(1,1,0)[24]。它有一个自回归项,没有普通差分,没有移动平均项,有一次滞后为 24 的季节差分,以及一个滞后为 24 的季节自回归项。设 B 为后移算子,即 B y_t 等于 y_(t-1),则该模型写成:
(1 - phi1 B)(1 - Phi1 B^24)(1 - B^24) y_t = e_t
(1 - B^24) 一项把每个读数替换为它相对于昨天同一小时的变化量。对小时级需求而言,滞后 24 正是这个模型的关键。每个小时都与昨天的同一小时比较,这正是季节性朴素法所做的比较,随后自回归项再对这一比较遗留下来的部分建模。季节差分消除了水平,因此模型没有均值项。statsforecast 的 ARIMA 默认对差分后的序列不设均值项(见 statsforecast/models.py 中 ARIMA 的文档字符串)。
我们固定阶数,而不去搜索它,原因见自动搜索一节。在把这个阶数放进回测之前,先用 statsmodels 0.15.0 在主客户第一个标准截断点之前的历史数据上拟合一次,并打印拟合参数的名称。这样可以看出模型是否含有你所写下的那些项。
1"""Fit the fixed seasonal ARIMA with statsmodels on the lead client's history before the first standard cutoff, and print its parameter names. The parameter lines are what the prose reads."""2import pandas as pd3from statsmodels.tsa.statespace.sarimax import SARIMAX45from lib.backtest import history_before6from lib.electricity import load7from lib.settings import EVAL_CUTOFFS, LEAD, show89hist = history_before(load([LEAD]), pd.Timestamp(EVAL_CUTOFFS[0]))10y = pd.Series(hist["y"].to_numpy(), index=pd.DatetimeIndex(hist["ds"], freq="h"))11res = SARIMAX(y, order=(1, 0, 0), seasonal_order=(1, 1, 0, 24)).fit(disp=False)12show("hours of history", len(y))13for name in res.params.index:14 show("parameter", name)
python arima-summary-look.pyhours of history: 1344
parameter: ar.L1
parameter: ar.S.L24
parameter: sigma2
列表中有三个名称:ar.L1、ar.S.L24 和 sigma2。第一个是自回归项,表示某一小时相对于昨天同一小时的变化有多少会延续到下一小时。第二个是滞后为 24 的季节项,表示某一天在某个小时的变化有多少会延续到次日的同一小时。第三个是误差的方差。列表中没有截距。只有设置了 trend 时,statsmodels 才会给季节性 ARIMA 加上截距,所以无论序列是否经过差分,列表都不会显示截距;而经过季节差分的模型无论如何都不需要截距。完整的 statsmodels 摘要还会给出各项的取值。本书不引用这些取值。它们会随窗口和库的版本而变化,因此读者更需要了解每一项的含义,而不是它的数值。
于是,这个固定模型成为回测中的又一个候选模型,与 AutoETS 和季节性朴素法并列。它的得分见下面的比较一节。
工作准则: 当搜索慢到无法在每个截断点都运行时,就固定阶数,并把它写在调用中。只打印一次摘要,并检查其中列出的项是否正是你所指定的。在每个截断点都重新拟合这个固定的阶数,就像重新拟合任何模型一样。阶数保持固定,系数则不然。
这部分尚未翻译,因此以英文显示。
Compare models on the same cutoffs
Four forecasts now cover the same 24 hours at the same three cutoffs, from the same eight weeks of history, for the same eight clients: seasonal naive, AutoETS, AutoETS with the season forced, and ARIMA. Each backtest is a separate script, so each finishes well inside the 45 seconds the book allows an example on two CPUs.
One score is still missing: the fixed ARIMA's, on the same windows and measures.
1"""Backtest the fixed seasonal ARIMA(1,0,0)(1,1,0)[24] and seasonal naive for the eight clients of SUBSET8 over the three standard cutoffs.2The MASE gap and the count of clients where ARIMA wins, printed last, are the figures the comparison section quotes."""3import numpy as np4import pandas as pd5from statsforecast import StatsForecast6from statsforecast.models import ARIMA, SeasonalNaive78from lib.scoring import book_scale9from lib.electricity import load10from lib.quirks import filled_history, mask_clock_changes11from lib.settings import EVAL_CUTOFFS, HORIZON, SEED, SUBSET8, show12from lib.subsets import score_clients1314np.random.seed(SEED) # the seed from lib/settings.py, set before any fit15masked = mask_clock_changes(load(SUBSET8)) # clock-change hours become NaN, so they are never scored; filled_history fills them from their neighbours before each fit16scale = book_scale(load(SUBSET8), EVAL_CUTOFFS[0]) # MASE scale: rows before the first cutoff, from each client's first reading17# The order is fixed, not searched: one AR term, one seasonal difference and one seasonal AR term, all at lag 24.18# n_jobs=1 keeps the output the same from run to run.19sf = StatsForecast(models=[ARIMA(order=(1, 0, 0), season_length=24, seasonal_order=(1, 1, 0)),20 SeasonalNaive(season_length=24)], freq="h", n_jobs=1)21parts = []22for c in pd.to_datetime(EVAL_CUTOFFS):23 sf.fit(df=filled_history(masked, c)[["unique_id", "ds", "y"]]) # the eight weeks before c, clock hours filled from them24 part = sf.predict(h=HORIZON)25 # the first forecast hour must be the cutoff itself, or the window is misaligned26 if part["ds"].min() != c:27 raise ValueError(f"forecast for {c:%Y-%m-%d} does not start at the cutoff")28 parts.append(part)29forecasts = pd.concat(parts, ignore_index=True)3031summary, per_client = score_clients(forecasts, masked, ["ARIMA", "SeasonalNaive"], scale)32show("horizon hours", HORIZON)33show("windows", len(parts))34for label, value in summary.items():35 show(label, value)36show("MASE gap, SeasonalNaive minus ARIMA", summary["MASE SeasonalNaive"] - summary["MASE ARIMA"])37show("clients where ARIMA has lower MASE", int((per_client["ARIMA"] < per_client["SeasonalNaive"]).sum()))
python arima-subset-backtest.pyhorizon hours: 24
windows: 3
clients: 8
scored hours per client: 72
MAE ARIMA: 386.13
RMSE ARIMA: 576.30
MASE ARIMA: 1.09
MAE SeasonalNaive: 423.68
RMSE SeasonalNaive: 591.25
MASE SeasonalNaive: 1.16
MASE gap, SeasonalNaive minus ARIMA: 0.07
clients where ARIMA has lower MASE: 5
MAE is in kilowatt hours, and RMSE, the root mean squared error, punishes large misses more. Both are computed for each client over its 72 scored hours, then averaged over the eight clients, so each client counts once. MASE divides each client's MAE by the MAE that seasonal naive makes in its own history, over every reading before Dec 29, 2014, from each client's first reading and with the clock-change hours left out, then averages over clients. A MASE of 1 means the model is as accurate as seasonal naive was on its own past. The coverage of AutoETS is the share of scored hours that fall inside its 95% prediction interval, the range the model gives for each hour.
| Model | MAE (kWh) | RMSE (kWh) | MASE | Clients beating seasonal naive on MASE |
|---|---|---|---|---|
| SeasonalNaive | 423.68 | 591.25 | 1.16 | not applicable |
| AutoETS | 362.11 | 537.20 | 1.30 | 4 |
| AutoETS, season forced (ZZA) | 365.24 | 553.11 | 1.17 | 4 |
| ARIMA | 386.13 | 576.30 | 1.09 | 5 |
Read the last column as a count out of the eight clients, not as a score. The AutoETS intervals cover 0.910 of the scored hours, and the forced-season intervals cover 0.925. The nominal share is 0.95, so the intervals are a little narrow on this sample. Coverage checks the intervals, not the point forecast. An interval that covers too little is too narrow, and a planner who buys to it will be short more often than its label says. Chapter 8 returns to intervals and shows how to calibrate them on earlier windows.
The two measures disagree about AutoETS. Its mean MAE is the lowest of the four rows, and its mean MASE is the highest. MAE in kilowatt hours gives more weight to the clients with large loads, because their errors are large. MASE scales each client's error by that client's own history, so every client counts about the same. The per-client lines in the AutoETS run show where the MASE comes from. MT_064 has a MASE of 2.32, against 1.08 for seasonal naive, and it is one of the flat-form clients. MT_217 has 1.76, against 1.25, and its form changes from one cutoff to the next. The ranking depends on the measure, so the choice of measure is part of the decision.
Three windows are a small sample, as Chapter 3 explains. AutoETS beats seasonal naive on 4 of the eight clients, and ARIMA on 5. ARIMA is better than seasonal naive on both measures, and on MASE by 0.07. Three windows can't settle a gap that small.
Working rule: Score every candidate on the same cutoffs, with the same history and the same scale, and report MAE and MASE side by side. When the two disagree, say which one the decision needs. We recommend more windows before choosing between two models that differ by less than about a tenth of MASE. That threshold is a judgement from this sample, not a measurement.
认识晚起步的客户
子集中有一个客户起步较晚。MT_115 的第一个非零读数出现在 2014 年 1 月 14 日 00:00。在此之前,电表报告的全是零,数据把这些小时标记为 before_first_reading。这些零并不代表低用电量。模型读到它们,就会学到一个该客户实际并不具有的用电水平。晚起步的客户,是指其第一个读数落在其模型所读取的历史之内的客户。解决办法是起始规则:序列从它的第一个读数开始。这关系到供应商的决策。用电水平被拉低,意味着为该客户购买的电量偏少,而这是供应商两类误差中代价更高的一类。
在 12 月的截止点上,每个截止点之前的八周全部是有效读数,因此本节改用 2 月的截止点:2014 年 2 月 1 日、2014 年 2 月 15 日和 2014 年 2 月 28 日。这些截止点对应的八周历史仍然会向前延伸到第一个读数之前,所以这种情形就包含在其中。第 6 章会把同一规则应用于全部三十个客户。
以零开头的历史
先看原始历史,即每个 2 月截止日之前的八周,其中包含零值。这段历史中零值小时所占的比例,在 2014 年 2 月 1 日为 68%,在 2014 年 2 月 15 日为 43%,在 2014 年 2 月 28 日为 20%。随着截止日越过首次读数,这一比例逐渐下降。
让历史从首次读数开始,或许可以解决这个问题。在每个 2 月截止日上对 AutoETS 拟合两次:一次用原始的八周数据,一次用起始历史,即从 MT_115 的首次读数开始的历史。以各截止日之后的 24 小时评分,可以看出这样做是否有效。
1"""Backtest AutoETS for MT_115, a late starter, on three February cutoffs: once on its raw eight weeks and once on its history from its first reading.2The start-rule counts, printed first, and the three-window MAE lines are the figures the point rests on."""3import numpy as np4import pandas as pd5from statsforecast import StatsForecast6from statsforecast.models import AutoETS78from lib.backtest import history_before9from lib.electricity import load10from lib.quirks import fill_clock_change_hours, mask_clock_changes11from lib.settings import HORIZON, SEED, SUBSET8, show12from lib.starts import check_start_rule, history_from_start1314LATE = "MT_115" # the one late starter in SUBSET8, first non-zero reading on Jan 14, 201415# February cutoffs: at the December cutoffs every hour of the history is a reading, so the rule changes nothing there16CUTOFFS = pd.to_datetime(["2014-02-01", "2014-02-15", "2014-02-28"])1718np.random.seed(SEED) # the seed from lib/settings.py, set before any fit19dropped, flagged = check_start_rule(load(SUBSET8)) # raises unless the rule drops exactly the flagged rows, so the counts agree20show("start rule drops, SUBSET8", dropped)21show("flagged before_first_reading, SUBSET8", flagged)22masked = mask_clock_changes(load([LATE]))[["unique_id", "ds", "y", "quirk"]]232425def fit_and_score(hist, c):26 """Fit AutoETS on one history, forecast the 24 hours from c, and score it on those hours."""27 sf = StatsForecast(models=[AutoETS(season_length=24)], freq="h", n_jobs=1)28 sf.fit(df=fill_clock_change_hours(hist)[["unique_id", "ds", "y"]]) # clock-change hours filled from their neighbouring hours, as elsewhere29 fc = sf.predict(h=HORIZON)30 actual = masked[(masked["ds"] >= c) & (masked["ds"] < c + pd.Timedelta(hours=HORIZON))] # the day's 24 readings, to score against31 joined = fc.merge(actual[["ds", "y"]], on="ds", how="left")32 # the form's method name, the mean forecast, the day's MAE, and the mean of the readings33 return sf.fitted_[0, 0].model_["method"], float(joined["AutoETS"].mean()), float((joined["y"] - joined["AutoETS"]).abs().mean()), float(joined["y"].mean())343536maes = {"raw": [], "start": []}37for c in CUTOFFS:38 raw_hist = history_before(masked, c) # the eight weeks before c, zeros included; it insists on exactly 1,344 rows39 # the share of hours that read zero, including the leading zeros before the first reading40 show(f"zero percent before {c:%Y-%m-%d}", round(100 * (raw_hist["y"] == 0).mean()))41 # the start history begins at the first reading, so it is shorter wherever the client starts late42 for label, hist in [("raw", raw_hist), ("start", history_from_start(masked, c))]:43 form, fc_mean, window_mae, actual_mean = fit_and_score(hist, c)44 show(f"{label} history hours {c:%Y-%m-%d}", len(hist))45 show(f"{label} form {c:%Y-%m-%d}", form)46 show(f"{label} mean forecast {c:%Y-%m-%d}", fc_mean)47 show(f"{label} mean actual {c:%Y-%m-%d}", actual_mean)48 show(f"MAE {label} {c:%Y-%m-%d}", window_mae)49 maes[label].append(window_mae)50for label in maes:51 show(f"MAE {label}, three windows", np.mean(maes[label]))52 show(f"MAE {label}, 2014-02-15 and 2014-02-28", np.mean(maes[label][1:])) # [1:] skips Feb 1, the first cutoff
python late-starter-trim.pystart rule drops, SUBSET8: 9072
flagged before_first_reading, SUBSET8: 9072
zero percent before 2014-02-01: 68
raw history hours 2014-02-01: 1344
raw form 2014-02-01: ETS(A,N,N)
raw mean forecast 2014-02-01: 16.43
raw mean actual 2014-02-01: 24.06
MAE raw 2014-02-01: 8.81
start history hours 2014-02-01: 432
start form 2014-02-01: ETS(M,Ad,A)
start mean forecast 2014-02-01: 23.88
start mean actual 2014-02-01: 24.06
MAE start 2014-02-01: 3.72
zero percent before 2014-02-15: 43
raw history hours 2014-02-15: 1344
raw form 2014-02-15: ETS(A,N,A)
raw mean forecast 2014-02-15: 24.54
raw mean actual 2014-02-15: 27.11
MAE raw 2014-02-15: 4.29
start history hours 2014-02-15: 768
start form 2014-02-15: ETS(M,Ad,A)
start mean forecast 2014-02-15: 23.84
start mean actual 2014-02-15: 27.11
MAE start 2014-02-15: 4.77
zero percent before 2014-02-28: 20
raw history hours 2014-02-28: 1344
raw form 2014-02-28: ETS(A,N,A)
raw mean forecast 2014-02-28: 23.73
raw mean actual 2014-02-28: 23.54
MAE raw 2014-02-28: 2.93
start history hours 2014-02-28: 1080
start form 2014-02-28: ETS(M,Ad,A)
start mean forecast 2014-02-28: 22.48
start mean actual 2014-02-28: 23.54
MAE start 2014-02-28: 2.73
MAE raw, three windows: 5.34
MAE raw, 2014-02-15 and 2014-02-28: 3.61
MAE start, three windows: 3.74
MAE start, 2014-02-15 and 2014-02-28: 3.75
损害出在水平上,并且在 2 月 1 日最为严重。AutoETS 拟合原始历史时没有报错,预测时也没有警告。在 2014 年 2 月 1 日,它选择了平坦形式 ETS(A,N,N),其预测在当天平均为每小时 16.43 kWh。随后的读数平均为 24.06 kWh。当天的 MAE 为 8.81 kWh。可能的原因是这些零值:它们拉低了水平,而模型无法分辨它们并不是用电量。
在另外两天,原始历史的误差要小得多,2014 年 2 月 15 日为 4.29 kWh,2014 年 2 月 28 日为 2.93 kWh。因此,损害的大小取决于窗口中包含多少零值。无论零值占比多少,这条规则都是正确的,因为开头的零值并不是用电量。
让每条序列从首次读数开始
起点规则规定,一条序列从它的首次读数开始。该规则会丢弃客户首个非零读数之前的所有读数,并保留此后的全部读数,其中包括零值,因为起点之后的零是真实的低用电量,模型需要这些数据。若窗口在首次读数之后才开始,它的所有小时都会保留,因为该规则只去除开头的零。规则写在 examples/lib/starts.py 中,因此后续各章调用的是同一份代码。其函数 history_from_start(df, cutoff) 返回在该规则下某个截止点的历史:即截止点之前八周内、从首次读数起的读数。它只根据截止点之前的行来确定首次读数,所以截止点之后的读数不会进入模型。如果某个客户在截止点之前没有任何读数,或历史不足 48 小时,函数会报错。
在使用该规则之前,先检查它是否做到了标记所说的事。check_start_rule 统计该规则在整条序列上会丢弃的行,除非这些行恰好就是标记为 before_first_reading 的行,否则会报错。在子集上,该规则丢弃 9072 小时,而标记所指的是 9072。两者一致。子集中唯一的晚启动客户是 MT_115。
该规则在第一个截止点处缩短历史最多,为 432 小时,而原本是 1,344 小时;随着截止点越过首次读数,缩短的幅度变小:2 月 15 日为 768 小时,2 月 28 日为 1080 小时。
起点历史在三个截止点都选出 ETS(M,Ad,A),这是一种误差为乘性、趋势为阻尼、季节为加性的形式。原始历史在不同日期之间会改变形式,从 2 月 1 日的平坦形式,变为 2014 年 2 月 15 日的 ETS(A,N,A) 和 2014 年 2 月 28 日的 ETS(A,N,A)。
三个窗口的 MAE 从 5.34 kWh 降到 3.74 kWh。各窗口的改善并不均匀,全部改善都来自 2 月 1 日,该日的 MAE 从 8.81 降到 3.72 kWh。
在 2 月 15 日,该规则的表现更差,在 2 月 28 日则略好一些。把这两天合起来看,原始历史的表现略好,平均 MAE 为 3.61 kWh,而起点历史为 3.75。
拟合结果说明了该规则在 2 月 15 日为何吃亏。起点历史仍然很短,只有 768 小时,它的阻尼趋势给出的预测比原始拟合更低:平均每小时 23.84 kWh,而原始拟合为 24.54,此时读数的平均值为 27.11 kWh。这个偏低的预测正是该案例所警示的代价。一个客户、三天只是很小的样本,所以这些数字用于说明该规则,而不能证明它。
工作准则: 当客户的历史中含有零值时,检查其首个非零读数是否落在历史之内。若是,则无论零值占比多少,都要在任何模型看到历史之前应用起点规则,因为开头的零并不是用电。先在整条序列上运行 check_start_rule,若未通过,先查明原因,再做任何裁剪。打印每个截止点处的历史长度,因为历史过短是规则取走过多数据的第一个迹象。
为什么本书在季节周期为 24 时避免使用自动 ARIMA
AutoARIMA 通过搜索来确定各阶数,而不是事先固定阶数。它对候选阶数进行逐步搜索,按 AICc(其默认准则)选出最优模型,最多评估 94 个候选模型后停止(statsforecast 2.1.1、statsforecast/models.py,以及 statsforecast/arima.py 中的停止信息)。每个候选模型都要做一次完整的状态空间拟合。季节周期为 24 时,每个含滞后 24 季节项的候选模型都拟合得很慢,而搜索会尝试几十个这样的模型,因此耗时远远超过本章中的各次拟合。
在本书固定的环境中,即两个 CPU、statsforecast 2.1.1,为一个客户拟合一个窗口大约用了 75 秒。这是在 MT_014 上、于第一个标准截断点处进行的一次运行。该子集需要 24 个这样的窗口,因此对它做一次回测大约需要半小时,而本书允许一个示例运行的时间是 45 秒。这是本书自己的测量结果,而不是书中的示例,并且它取决于具体环境,所以在围绕它安排调度之前,请先在你自己的机器上计时。固定阶数时每个窗口只需拟合一次,这正是这里的关键差别。它的弱点在于:某个客户的模式与该阶数的假设不一致。回测就是对这一点的检验,应查看的位置是对比一节中的得分。
固定阶数是一种判断,而不是一项发现。我们选择季节周期为 24 的 (1,0,0)(1,1,0),是因为数据有日内模式,也因为本章的回测需要一个能够跑完的模型。阶数写在调用之中,读者可以看到并加以修改。后续各章在相同的窗口上与它做比较,因此阶数的改动会体现在得分上。
工作准则: 不要在每天重新拟合的调度中,对小时数据以季节周期 24 运行自动搜索。在采用之前,先在一个客户上为搜索计时。如果太慢,就固定阶数,并在回测中检验该阶数。
生产中会出什么问题
- 缺失的季节项。 季节长度保持为 1 时,得到的模型没有日内模式,也不会报错。其迹象是每个客户选出的都是不含季节项的形式,例如 ETS(A,N,N)。只有少数客户如此,则是另一种情况:季节性拟合失败,这由指数平滑一节中的平坦形式检查来检验。每次重新拟合时,都要记录所选的形式。
- 后起步客户的零值。 如果某个客户在历史区间内才开始产生读数,那么在其第一个读数移出窗口之前,它的水平项会被拉低。其迹象是起步之后的若干天里预测值偏低。每次重新拟合之前,先统计每个客户历史中零值所占的比例,并对第一个读数落在历史区间内的客户应用起步规则。
- 形式的变化。 同一客户所选的形式可能在两次重新拟合之间发生变化。这本身并不算错,但从季节形式切换为不含季节项的形式,值得对照读数加以核查。请把这些形式保存在运行日志中。
- 不断累积的成本。 拟合次数随客户数、窗口数和重新拟合次数的乘积增长。在向调度中增加客户之前,先把次数算出来。
- 失准的区间。 在这份样本上,AutoETS 区间覆盖了 0.910 的小时,低于名义上的 0.95。第 8 章说明如何在更早的窗口上校准区间带;在区间带以这种方式校准之前,不要把它表述为 95% 的范围。
- 如何察觉。 在每天的窗口上,让季节性朴素法与每个统计模型并行运行,并记录 MASE 差值。差值的符号逐日变化,是需要检查模型的信号,而不是可以报告的结果。
本章做出的决定
- 季节长度为 24 的 AutoETS。 默认的
model="ZZZ"与强制季节的运行model="ZZA"都给出了结果,因为平坦的选择可能是季节性拟合失败。本章不在两种形式之间做选择:标准截止点的比较只作报告,不用于选择,选择应当放在更早的窗口上进行。第 1 章曾对主要客户强制使用ZZA,本章则展示了对全部八个客户的强制运行结果。 - 固定的 ARIMA(1,0,0)(1,1,0)[24],不做搜索。 这是一项明确陈述的判断,回测正是用来检验它的。
- 八个客户,而不是三十个。 第 6 章用一个模型拟合全部三十个客户。在固定的环境下,对三十个客户逐一拟合 AutoETS 并跑三个窗口,约需 97 秒,而本书允许每个示例使用 45 秒,因此本章使用子集。
- 子集、截止点与尺度。
SUBSET8、三个标准截止点、2014 年 12 月 29 日至 12 月 31 日、1,344 小时的历史数据,以及在每次比较中都作为基准的季节性朴素法。MASE 的尺度从每个客户的首个读数算起,并去掉时钟变更的小时。 - 时钟变更的小时。 子集中的每个示例都会将这些小时屏蔽,并用历史数据填补,详见第 3 章。三个窗口都不含时钟变更的小时,因此屏蔽不会改变任何预测,也不会改变任何被评分的小时。但它确实会改变 MASE 的尺度,因为该尺度读取 2014 年 12 月 29 日之前的全部历史,并且从 2014 年 1 月 14 日起统计
MT_115。 - 晚开始的客户:二月截止点与起始规则。 十二月的截止点遇不上
MT_115的晚开始,二月的截止点则可以。第 6 章将该规则应用于全部三十个客户。 - 仅对两次 AutoETS 运行给出覆盖率。 这里的 ARIMA 回测不带区间评分。
- 不引用拟合值。 ARIMA 脚本只打印参数名称,平坦形式的检查只打印计数。误差四舍五入到两位小数,覆盖率保留三位小数。
- 为第 5 章提供统计基线。 这些窗口上逐客户的 AutoETS 与 ARIMA 得分,是梯度提升模型必须超越的基准。
- 需要带走的要点。 平坦形式可能是季节性拟合失败,因此在查看每个得分时都要同时看所选的形式。MAE 与 MASE 对模型的排序可能不同,因此要说明决策需要的是哪一个。让每个序列从其首个读数开始,并在选择差距很小的模型之前增加窗口。
本章,在本版次中
版次 2026.11.11示例于 2026年11月20日 通过本章 6个示例 个示例中有 6 个通过
译自英文版图书,版次 2026.11.11。