无前视回测

阅读约需 71分钟 分钟,10节中含7个示例。

版次 2026.11.11 示例于 2026年11月20日 通过 41个示例中有41个通过 变化内容

AI 翻译。由人工以英文核验。 我们如何制作图书

译自英文版图书,版次 2026.11.11。 阅读原文

本章教你在一年的预测截止时间点上,为日前预测打分,同时不让未来的数据进入它的历史;教你处理数据中两个时钟调整所涉及的小时;还教你运行一项平移测试,用来找出读取了截止时间之后数据的特征。你需要用到第 2 章给出的内容:季节性朴素基线、MAE,以及取自历史的尺度。方法本身保持不变。变化的是打分的次数,以及每次切分数据时需要多么谨慎。

这部分尚未翻译,因此以英文显示。

Why one test week misleads

最近一次修改见于版次 2026.11.11

So far the evidence is the held-out week of Chapter 1: seven day-ahead forecasts for one client, 168 scored hours in all. A sample that size can be lucky. A forecast can match a calm week and miss the next, and one week cannot tell the two apart. Before the supplier trusts a method with each day's purchase, it needs to know how a forecast does across many days, the unusual ones included.

A backtest answers that question with the data you already have. It makes a forecast at many past moments, each with only the readings available at that moment, and scores each one against the readings that followed. Each such moment is a cutoff. Each cutoff, with the hours it forecasts, is a window. The scores of all the windows together say how the method behaves across the year, not on one week.

This matters most for the clock changes. A year of cutoffs contains the March change, when the 01:00 hour does not occur on the clock, and the October change, when one 01:00 label holds two hours of energy. A single week, such as the last week of December, contains neither. A backtest that does not reach those dates cannot show what the method does on them, so the sample has to be chosen with them in mind.

Working rule: Prefer a year of cutoffs before you trust a difference between two methods. When cost forces fewer, say so beside each score, as Chapters 4 to 9 do. Make sure the sample reaches the odd days of the year. Report how many windows and how many scored hours stand behind each number. A figure from one week is one sample, and a reader should be able to see that.

这部分尚未翻译,因此以英文显示。

Roll the cutoff forward

最近一次修改见于版次 2026.11.11

A window has three parts: the cutoff, the horizon and the history. The cutoff is the moment the forecast is made. The horizon is how far ahead it reaches, which is 24 hours here, so the forecast covers the 24 hours that start at the cutoff. The history is the readings before the cutoff that the model may use, which the book sets at 1,344 hours, or eight weeks. Move the cutoff by a step and you have the next window. The step is one week for the Sunday sample, and one day for the daily samples in the two clock-change subsections, The March hour that reads zero and The October hour that holds two.

In words, each window's targets are the 24 hours from its cutoff, and its history is the readings from 1,344 hours before the cutoff up to, but not including, the cutoff. The next window moves both by one step. A reading can therefore be a target in one window and history in a later one. That is correct, because by the later cutoff the reading lies in the past, and it is the same reading in both places.

The Sunday sample runs seasonal naive over every Sunday of 2014, from Jan 5 to Dec 28, 2014. The EU puts every clock change on the last Sunday of March and of October (Directive 2000/84/EC, Articles 2 and 3), and both changes of 2014, on Mar 30 and Oct 26, are in the sample. The sample uses SUBSET8, the book's eight-client subset. Here the year is scored on those eight clients, window by window:

backtest-sundays-2014.py Python 3.13.16 · holidays 0.110.dev20261118063242 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17 通过
"""Seasonal naive over the 52 Sunday cutoffs of 2014, for eight clients, with no clock-change rule: how much history eachwindow gets, then the year's scores. The prose's headline rests on the mean per-client MASE line."""import pandas as pdfrom holidays import country_holidaysfrom statsforecast import StatsForecastfrom statsforecast.models import SeasonalNaivefrom lib.backtest import forecast_windows, history_beforefrom lib.electricity import loadfrom lib.quirks import year_scoresfrom lib.scoring import book_scale, clients_left_outfrom lib.settings import HISTORY_HOURS, HORIZON, SUBSET8, showCOLS = ["unique_id", "ds", "y"]cutoffs = pd.date_range("2014-01-05", "2014-12-28", freq="W-SUN")  # 52 Sundays, both clock changes among themdf = load(SUBSET8)[COLS + ["quirk"]]sf = StatsForecast(models=[SeasonalNaive(season_length=24)], freq="h", n_jobs=1)  # 24 hourly readings a day: each hour copies the same hour of the day beforeforecasts = forecast_windows(sf, df[COLS], cutoffs, h=HORIZON)scored = forecasts.merge(df, on=["unique_id", "ds"])  # attaches each forecast hour's reading and its quirk flagscale = book_scale(df, cutoffs[0])  # the MASE scale: every reading before the first cutoff, from 2013 onsizes = pd.concat([history_before(df, c).groupby("unique_id").size() for c in cutoffs])  # history rows per client in each windowassert sizes.min() == sizes.max() == HISTORY_HOURS  # every window's history is the same length, the prose's claimholidays = country_holidays("PT", years=[2014])show("windows", len(cutoffs))show("history hours per series, min over windows", sizes.min())show("history hours per series, max over windows", sizes.max())show("Sunday windows on a Portuguese holiday", sum(c.date() in holidays for c in cutoffs))show("client left out of MASE", ", ".join(clients_left_out(df, cutoffs[0])))for label, value in year_scores(scored, scale).items():  # pooled MAE and RMSE, per-client MASE, the largest client's share    show(label, value)
输出python backtest-sundays-2014.py
windows: 52
history hours per series, min over windows: 1344
history hours per series, max over windows: 1344
Sunday windows on a Portuguese holiday: 1
client left out of MASE: MT_115
scored hours: 9936
scored hours before a first reading: 48
MAE: 382.02
RMSE: 1369.57
largest client by absolute error: MT_370
largest client's share of absolute error, percent: 88.69
clients in MASE: 7
mean MASE: 0.97
lowest client MASE: 0.77
highest client MASE: 1.26

Every window's history holds the same 1344 hours for each client, the eight weeks the book specifies, and the example asserts that the length is the same in every window. The 52 windows score 9936 hours: eight clients, 24 hours each, times the windows, less the hours of one late starter before its first reading, which the next paragraphs explain.

The headline is the mean per-client MASE, not the pooled error. The MASE scale is one number per client, and scoring.book_scale computes it for the whole book: every reading before the first cutoff, with the clock-change hours masked and each client's leading zeros removed. The first cutoff is Jan 5, 2014, so this sample's scale is read from Jan 1, 2013 to Jan 4, 2014. The mean of the per-client MASE values is 0.97, with the lowest at 0.77 and the highest at 1.26, over 7 clients.

The pooled MAE, 382.02 kWh over the scored hours, compares methods on the same clients and windows, but it cannot compare clients with each other. Its largest client, MT_370, carries 88.69 percent of the absolute error, because that client uses thousands of kilowatt hours an hour, so its errors are in the thousands too. The RMSE is 1369.57 kWh.

Client MT_115 has no reading before Jan 14, 2014. Until then the meter reports zeros, and the data card says those zeros are not demand. The code leaves the 48 hours before its first reading out of every score, which are the first two windows. Its history still holds those zeros, because the start rule for histories belongs to Chapter 4. Its MASE is left out for a different reason: its scale is read from the readings before Jan 5, 2014, and it has none. The output lists that client as left out of MASE, and the start rule is left to that chapter.

The statsforecast library has a function for this loop, cross_validation, and it is possible to call it without the window you wanted. At version 2.1.1, the training slice of each window starts at the first reading of the series unless you pass input_size. The history then grows from one window to the next, and the eight-week rule is not in force. Hyndman and Athanasopoulos state the principle that the training set consists only of observations before the test set (Forecasting: Principles and Practice, 3rd ed., chapter 5). The code behind cross_validation decides the length of that set (statsforecast 2.1.1, statsforecast/core.py). Its cutoff is the last hour a window trains on, where this chapter's cutoff is the first hour it forecasts, so the two names differ by one hour. This chapter builds each window's history itself, from the cutoff backwards. That gives the same 1,344 hours at every window and makes the rule visible in the code.

Working rule: Set the history length explicitly, in hours, and check that it is the same for every window. When a library loops over windows for you, read how it sets the training slice before you trust its output, because its default may be an expanding window rather than the length you chose.

这部分尚未翻译,因此以英文显示。

Keep the future out of the history

最近一次修改见于版次 2026.11.5

A backtest leaks when a window's model sees a reading from the hours it is asked to forecast. The reading can reach the model in several ways: through the history, through a feature, or through a scale computed from data that includes the target. Hewamalage, Ackermann and Bergmeir describe rolling-origin evaluation as prone to exactly this, because the test data of one window becomes training data of the next. In forecasting, they note, the leak is easier to make and harder to avoid than in other machine learning tasks (2022, section 3.4).

One rule closes the history path. Each window's history is the readings whose timestamps fall in the eight weeks before its cutoff, and nothing at or after the cutoff. In the book's code, history_before keeps the rows with ds from the cutoff minus 1,344 hours up to, but not including, the cutoff. The reading at the cutoff itself belongs to the target day, so the same comparison excludes it. Membership is decided by timestamps, not by row positions, so a missing row cannot shift a window by an hour. history_before also raises an error unless every client has exactly 1,344 rows in the window.

The window's own first hour is the cutoff, so a cut that is one hour too generous passes the first target reading into the history, and nothing in a forecast shows it. A row count can catch it, and the example below counts the rows.

history-cut-check.py Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · utilsforecast 0.2.17 通过
"""One window's history, cut by its cutoff: its first and last timestamps, and what a cut that keeps the cutoff hour adds.The prose's timestamps rest on the first two lines, and its off-by-one point on the row countswith the cutoff hour in and out."""import pandas as pdfrom lib.backtest import history_beforefrom lib.electricity import loadfrom lib.settings import HISTORY_HOURS, LEAD, showdf = load([LEAD])  # the lead client's readings only: one series is enough to show the cutcutoff = pd.Timestamp("2014-12-29")  # the window's cutoff at midnight, which is also its first forecast hourhist = history_before(df, cutoff)  # the 1,344 hours before the cutoff; the cutoff hour itself is not in itshow("history rows", len(hist))show("first history timestamp", hist["ds"].min().strftime("%Y-%m-%d %H:%M"))show("last history timestamp", hist["ds"].max().strftime("%Y-%m-%d %H:%M"))wrong = df[(df["ds"] >= cutoff - pd.Timedelta(hours=HISTORY_HOURS)) & (df["ds"] <= cutoff)]  # the off-by-one cut: <= keeps the cutoff hourshow("history rows with ds at or before the cutoff", len(wrong))show("target-day readings in that cut", int((wrong["ds"] == cutoff).sum()))
输出python history-cut-check.py
history rows: 1344
first history timestamp: 2014-11-03 00:00
last history timestamp: 2014-12-28 23:00
history rows with ds at or before the cutoff: 1345
target-day readings in that cut: 1

Keeping the cutoff hour adds one row, and the count shows it. The history for the cutoff of Dec 29, 2014 runs from 2014-11-03 00:00 to 2014-12-28 23:00 and holds 1344 rows. A cut with ds <= cutoff holds 1345 rows, because the target day's first reading, the cutoff hour, has entered the history. A count one row too high is the sign of that error.

The same care applies to the scale. evaluate in utilsforecast takes a training frame for its scale, and it cuts that frame by time only when the forecast frame carries a cutoff column. Then it keeps the training rows with ds at or before the cutoff, which includes the cutoff hour itself (utilsforecast 0.2.17, evaluation.py). This chapter does not use that option. Its scale comes from scoring.book_scale, which passes only the rows before the first cutoff, and the frame it scores has no cutoff column.

Working rule: Slice each window's history by its cutoff, never by the start of its test window, and pass a library only the rows it may use. Check the first and last timestamps of every history, and make sure the last one is an hour before the cutoff.

认识时钟变更

最近一次修改见于版次 2026.11.5

数据中有两个小时,打破了“每个时钟小时对应一次读数”的简单图景。在 3 月的变更日,本地时钟从 00:59 直接跳到 02:00,因此 01:00 这个小时在时钟上并不存在。葡萄牙本土在冬季使用格林尼治标准时间,欧盟在格林尼治标准时间 01:00 调整时钟,即本地时间 02:00(Directive 2000/84/EC,第 2 条)。在 10 月的变更日,时钟从 02:00 回拨到 01:00,因此 01:00 这个小时会出现两次。文件中每天只有一行对应 01:00,在本书的数据中被标记为 clock_change_gap 或 clock_change_repeat。

本书在一个明确陈述的假设下读取该文件,之后的每一章都依赖这一假设。读数的时间戳 ds 是该小时在本地时钟时间下的起点,索引是一个不带时区的朴素小时索引。3 月的证据支持这种读法。数据集自身的说明见 Dataset 321 of the UCI Machine Learning Repository,其中指出,3 月变更日 01:00 的值对所有客户都为零。文件在大体上与此一致,第一小节会说明这一点,第二小节则展示 10 月标签中的这两个小时。

这部分尚未翻译,因此以英文显示。

The March hour that reads zero

最近一次修改见于版次 2026.11.5

On Mar 30, 2014, the 01:00 label reads zero for 5 of the eight clients in the subset. The others read a small amount, not an hour of demand. The census covers all thirty clients.

clock-change-census.py Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17 通过
"""The clock-change hours of all thirty clients, each against its neighbours. The point that the March hours read near zerorests on the gap rows below 5 percent of their neighbours, with the zero count and the median ratio beside them."""from lib.electricity import loadfrom lib.quirks import clock_ratiosfrom lib.settings import showr = clock_ratios(load())  # each clock-change hour divided by the mean of the hour before and the hour aftergap = r[r["quirk"] == "clock_change_gap"]  # the file's flag: 01:00 on the March change day, an hour the clock skipsrepeat = r[r["quirk"] == "clock_change_repeat"]  # the file's flag: 01:00 holds two hours on the October change dayshow("gap rows", len(gap))show("gap rows that read zero", int((gap["y"] == 0).sum()))show("gap rows that read above zero", int((gap["y"] > 0).sum()))show("gap rows below 5 percent of their neighbours", int((gap["ratio"] < 0.05).sum()))  # a NaN ratio, where both neighbours are zero, counts in neither lineshow("gap rows at or above 5 percent of their neighbours", int((gap["ratio"] >= 0.05).sum()))show("median gap ratio to neighbours", gap["ratio"].median(), digits=3)show("repeat rows", len(repeat))show("median repeat ratio to neighbours", repeat["ratio"].median(), digits=3)
输出python clock-change-census.py
gap rows: 57
gap rows that read zero: 40
gap rows that read above zero: 17
gap rows below 5 percent of their neighbours: 55
gap rows at or above 5 percent of their neighbours: 2
median gap ratio to neighbours: 0.000
repeat rows: 58
median repeat ratio to neighbours: 1.948

Most flagged hours read near zero: 55 of the 57 flagged client-hours on the two March change days sit below 5 percent of their neighbouring hours, which is what this chapter means by near zero. The median ratio to the neighbours is 0.000.

The file mostly agrees with the dataset's note: 40 of the 57 read exactly zero, and 17 read above zero.

The data say zero or near zero, enough for the rule. The meter recorded nothing for an hour that did not occur on the clock, so the reading is not demand.

Seasonal naive copies the last day's values forward, so the forecast for 01:00 on Mar 30 is the demand of Mar 29 at 01:00. That is an ordinary hour, and where the reading is zero, the error is the whole forecast.

Make a guess before the next run. The cutoff of Mar 31 looks back over 24 hours, and those hours include the zero. How large is the error at 01:00 on Mar 31 compared with an ordinary hour? The run covers the 15 daily cutoffs from Mar 23 to Apr 6, 2014, first with no rule and then with the rule from the next section:

clock-gap-daily-cutoffs.py Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17 通过
"""Seasonal naive around the March clock change, for eight clients: the raw failure, then the rule. The prose's claims reston the gap-hour and next-day 1am lines, with and without the rule, the median client ratios, and the count of missingforecasts without the fill."""import pandas as pdfrom lib.backtest import history_beforefrom lib.electricity import loadfrom lib.quirks import filled_history, mae_of, mask_clock_changes, median_client_ratio, seasonal_naive_atfrom lib.settings import SUBSET8, showcutoffs = pd.date_range("2014-03-23", "2014-04-06", freq="D")  # 15 daily cutoffs around Mar 30, 2014next_day = pd.Timestamp("2014-03-31 01:00")  # 01:00 the day after the change: the forecast that copies the zerodf = load(SUBSET8)[["unique_id", "ds", "y", "quirk"]]masked = mask_clock_changes(df)  # a copy with the gap and repeat hours set to NaNraw = seasonal_naive_at(df, cutoffs, lambda c: history_before(df, c))  # no rule: the history keeps the zeros as the file has themgap = raw["quirk"] == "clock_change_gap"  # the file's flag marks the gap hoursat_next_day = raw["ds"] == next_dayshow("gap hours in actuals", int(gap.sum()))show("gap hours that read zero", int((raw[gap]["y"] == 0).sum()))show("gap-hour MAE with no rule", mae_of(raw[gap]))show("other-hour MAE with no rule", mae_of(raw[~gap]))# per client: the mean error at gap hours over the mean at other hours, then the median across clientsshow("median client ratio, gap hour to other hours, no rule", median_client_ratio(raw, gap, ~gap))show("next-day MAE at 1am with no rule", mae_of(raw[at_next_day]))# a forecast of exactly zero is the March zero copied forward from the day beforeshow("next-day 1am forecasts that are zero, no rule", int((raw[at_next_day]["SeasonalNaive"] == 0).sum()))show("median client ratio, next-day 1am to other hours, no rule", median_client_ratio(raw, at_next_day, ~gap))no_fill = seasonal_naive_at(masked, cutoffs, lambda c: history_before(masked, c))  # masked history, no fill: a NaN copies forward and stays NaNshow("NaN forecasts without the fill", int(no_fill["SeasonalNaive"].isna().sum()))fixed = seasonal_naive_at(masked, cutoffs, lambda c: filled_history(masked, c))  # the rule: masked hours filled from the history before each cutoffshow("next-day MAE at 1am with the rule", mae_of(fixed[fixed["ds"] == next_day]))show("other-hour MAE with the rule", mae_of(fixed[fixed["quirk"] != "clock_change_gap"]))
输出python clock-gap-daily-cutoffs.py
gap hours in actuals: 8
gap hours that read zero: 5
gap-hour MAE with no rule: 2322.69
other-hour MAE with no rule: 234.21
median client ratio, gap hour to other hours, no rule: 7.93
next-day MAE at 1am with no rule: 2393.01
next-day 1am forecasts that are zero, no rule: 5
median client ratio, next-day 1am to other hours, no rule: 9.92
NaN forecasts without the fill: 8
next-day MAE at 1am with the rule: 274.41
other-hour MAE with the rule: 228.31

For the median client, the error at 01:00 on Mar 31 is 9.92 times its error at other hours. The forecast for that hour copies the zero for 5 clients.

The gap hour itself fails as well. Its pooled error is 2322.69 kWh, against 234.21 kWh at other hours, and the median client's ratio is 7.93. Both pooled figures are set mostly by the largest client, while the per-client median counts each client once.

Pooled, the next day's error at 01:00 is 2393.01 kWh, about as much as the gap hour's. The method is wrong on the change hour, and wrong by about as much the day after, because it copied the zero.

Working rule: Treat a 01:00 reading of zero on a March clock-change day as a missing hour, not as zero demand. Keep it out of a model's history and out of the scores, and do not let a seasonal method copy it into the next day.

包含两个小时的十月那一小时

最近一次修改见于版次 2026.11.5

2014 年 10 月 26 日,01:00 这个标签约含两个小时的能耗。普查给出该小时与相邻小时之比的中位数为 1.948,统计范围是文件中 58 个被标记的客户小时。UCI 的说明与此一致:01:00 的数值汇总了两个小时的用电量。

这种失败与三月的情形正好相反。在换钟当天,季节性朴素预测用前一天的值预测 01:00 标签,而前一天的这个小时是普通小时,实际值却约为其两倍。10 月 27 日的截止点会复制这个加倍的值,因此 10 月 27 日 01:00 的预测约为普通小时的两倍,而实际值是普通水平。

同样的比较在 2014 年 10 月 19 日至 11 月 2 日的 15 个每日截止点上进行:

clock-repeat-daily-cutoffs.py Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17 通过
"""Seasonal naive around the October clock change, for eight clients: the raw failure, then the rule. The prose's claims reston the repeat-hour and next-day 1am lines, the median client ratios, and the count of missing forecasts without the fill."""import pandas as pdfrom lib.backtest import history_beforefrom lib.electricity import loadfrom lib.quirks import filled_history, mae_of, mask_clock_changes, median_client_ratio, seasonal_naive_atfrom lib.settings import SUBSET8, showcutoffs = pd.date_range("2014-10-19", "2014-11-02", freq="D")  # 15 daily cutoffs around Oct 26, 2014next_day = pd.Timestamp("2014-10-27 01:00")  # 01:00 the day after the change: the forecast that copies the doubled valuedf = load(SUBSET8)[["unique_id", "ds", "y", "quirk"]]masked = mask_clock_changes(df)  # a copy with the gap and repeat hours set to NaNraw = seasonal_naive_at(df, cutoffs, lambda c: history_before(df, c))  # no rule: the history keeps the doubled 01:00 valuerepeat = raw["quirk"] == "clock_change_repeat"  # the file's flag: two hours of energy in one 01:00 labelat_next_day = raw["ds"] == next_dayshow("repeat hours in actuals", int(repeat.sum()))show("repeat-hour MAE with no rule", mae_of(raw[repeat]))show("other-hour MAE with no rule", mae_of(raw[~repeat]))# per client: the mean error at the repeat hour over the mean at other hours, then the median across clientsshow("median client ratio, repeat hour to other hours, no rule", median_client_ratio(raw, repeat, ~repeat))show("next-day MAE at 1am with no rule", mae_of(raw[at_next_day]))show("median client ratio, next-day 1am to other hours, no rule", median_client_ratio(raw, at_next_day, ~repeat))no_fill = seasonal_naive_at(masked, cutoffs, lambda c: history_before(masked, c))  # masked history, no fill: a NaN copies forward and stays NaNshow("NaN forecasts without the fill", int(no_fill["SeasonalNaive"].isna().sum()))fixed = seasonal_naive_at(masked, cutoffs, lambda c: filled_history(masked, c))  # the rule: masked hours filled from the history before each cutoffshow("next-day MAE at 1am with the rule", mae_of(fixed[fixed["ds"] == next_day]))show("other-hour MAE with the rule", mae_of(fixed[fixed["quirk"] != "clock_change_repeat"]))
输出python clock-repeat-daily-cutoffs.py
repeat hours in actuals: 8
repeat-hour MAE with no rule: 637.18
other-hour MAE with no rule: 397.96
median client ratio, repeat hour to other hours, no rule: 6.98
next-day MAE at 1am with no rule: 1524.26
median client ratio, next-day 1am to other hours, no rule: 7.53
NaN forecasts without the fill: 8
next-day MAE at 1am with the rule: 184.22
other-hour MAE with the rule: 394.23

汇总后的数字低估了换钟小时的失败程度。在换钟小时,预测值是普通小时,实际值却加倍,因此汇总误差为 637.18 kWh,而其他小时为 397.96 kWh。这一差距看起来比实际要小,因为最大客户的误差在每个小时都很大。对于中位数客户,换钟小时的误差是其他小时误差的 6.98 倍。

加倍的值在第二天造成的损害最大。10 月 27 日 01:00 的预测复制了该值,因此那里的汇总误差为 1524.26 kWh,是三者中最大的,中位数客户的比值为 7.53。如果读者把十月的原始标签当作普通小时处理,就会在一个小时低估预测,并在下一个小时高估预测。

工作准则: 应把十月的 01:00 标签视为一个时段内包含两个小时的能耗,绝不能当作普通小时。不要为了让它看起来像一个小时而将其减半,原因见下一节。

将换钟时段标记为缺失

最近一次修改见于版次 2026.11.5

本书采用的规则是一种选择,本节把它集中写在一处,以便后续每一章都能调用同一段代码。两个 01:00 时段,即三月的缺失时段和十月的重复时段,在历史数据和评分中都被设为缺失,因此被屏蔽的时段不会参与评分。能够处理缺失值的模型会跳过该时段。不能处理缺失值的模型,例如季节性朴素模型,则读取填充值,因为缺失值会被复制到次日的预测中。无论哪种方式,该时段都不会被评分,而预测仍会为其标签给出一个值,因为输出网格必须保持完整。填充值是前一小时与后一小时的均值,且只取自截止时间之前的历史数据。它是针对夹在两个普通时段之间的时段而设的经验规则,本书也将其标明为经验规则。

另一种做法是把十月的那个标签减半,使每个实际小时只取其读数的一半,但这假定两个小时用电量相同。数据说明并未显示这一点,所以本书不采用这种做法。

若没有填充,在八个客户中,每次换钟后次日 01:00 的预测在三月有 8 个缺失,在十月有 8 个缺失。

采用填充后,该时段的次日误差在三月从 2393.01 降至 274.41 kWh,在十月从 1524.26 降至 184.22 kWh,而其他时段的误差变化很小。

全年的周日截止时间是另一种测试,因为它把屏蔽用于评分,填充不起作用。把全年分别按未屏蔽和已屏蔽各评分一次,就能看出这两个小时使总体结果移动了多少。

clock-masked-backtest.py Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17 通过
"""The Sunday backtest of 2014 for eight clients, scored twice: as the readings stand, and with the clock-change rule, whichleaves the masked hours out of the scores and fills the history. The prose's comparison rests on the masked-hour count,the two pooled MAE lines, the two mean MASE lines and the two scored-hour lines."""import pandas as pdfrom lib.backtest import history_beforefrom lib.electricity import loadfrom lib.quirks import CLOCK_CHANGE_QUIRKS, filled_history, mask_clock_changes, seasonal_naive_at, year_scoresfrom lib.scoring import book_scalefrom lib.settings import SUBSET8, showcutoffs = pd.date_range("2014-01-05", "2014-12-28", freq="W-SUN")  # the same 52 Sunday cutoffs as the plain backtestdf = load(SUBSET8)[["unique_id", "ds", "y", "quirk"]]masked = mask_clock_changes(df)  # a copy with the gap and repeat hours set to NaN; the rule's scores skip themscale = book_scale(df, cutoffs[0])  # book_scale masks the clock-change hours itself, so both runs share this one scaleplain = seasonal_naive_at(df, cutoffs, lambda c: history_before(df, c))  # as the readings stand: the zeros and the doubled hour stay inruled = seasonal_naive_at(masked, cutoffs, lambda c: filled_history(masked, c))  # the rule: history filled from before each cutoffshow("masked hours in scored windows", int(plain["quirk"].isin(CLOCK_CHANGE_QUIRKS).sum()))  # counted in the plain run, where the flags are still presentfor tag, scored in [("without the mask", plain), ("with the mask", ruled)]:    result = year_scores(scored, scale)    show(f"scored hours {tag}", result["scored hours"])    show(f"MAE {tag}", result["MAE"])    show(f"mean MASE {tag}", result["mean MASE"])
输出python clock-masked-backtest.py
masked hours in scored windows: 16
scored hours without the mask: 9936
MAE without the mask: 382.02
mean MASE without the mask: 0.97
scored hours with the mask: 9920
MAE with the mask: 380.25
mean MASE with the mask: 0.96

屏蔽从评分窗口中去掉了 16 个被屏蔽的时段,即子集中三月和十月的那些时段,汇总的 MAE 只有小幅变化,从 382.02 kWh 变为 380.25 kWh。

平均 MASE 从 0.97 变为 0.96,评分小时数为 9920,而不是 9936。这一差异来自被去掉的时段。

周日样本完全无法显示填充的作用,因为季节性朴素模型只读取每个截止时间之前的一天,而每个周日截止时间的前一天都是周六,没有哪个周六含有换钟。

工作准则: 在每一份历史数据和每一次评分中,都把两个换钟时段标记为缺失,在报告中给出被屏蔽时段的数量,并且只有当模型不能处理缺失值时,才用截止时间之前的历史数据来填充它们。如果被屏蔽时段的数量在两次运行之间发生变化,说明样本或文件的标记已经改变,重新运行评分之前应先查明原因。

这部分尚未翻译,因此以英文显示。

Check a backtest for leakage

最近一次修改见于版次 2026.11.11

Every window in a backtest should be built from data available at its cutoff, so the check for a leak is simple to state. A feature for a window must have the same value whether it is computed from the history cut at the cutoff or from the whole series. If the two values differ, the feature read a reading after the cutoff, and the backtest flatters the method. The shift test compares a feature computed from the history cut at the cutoff with the same feature computed from the whole series. It is cheap to run, and it catches a mistake that the backtest cannot catch on its own, because a leaking feature scores well by design.

The test has two halves, and each catches what the other misses. Take the lead client, MT_014, and three features over its hourly readings. The first is a trailing mean: the average of the 24 hours up to and including each hour, which reads nothing after its hour. The second is the same mean shifted one hour forward, so each value is the average of a window that ends one hour later, and it reads the first hour after the cutoff. The third is a lag of one hour, the most common day-ahead leak. The history half compares each feature on the rows of the eight weeks before each of the three evaluation cutoffs, Dec 29 to Dec 31, 2014. The target half compares the 24 hours that each cutoff forecasts, computed from the readings before the cutoff only. Both versions start from the same first reading, so the warm-up of a rolling mean matches on both sides. Computed from different starts, the warm-up and floating-point rounding would show as false differences, which is why the check compares with a tolerance:

leakage-shift-check.py Python 3.13.16 · numpy 2.5.5 · pandas 2.3.3 通过
"""Shift test: a feature must read nothing from its cutoff on, on the history rows and on the target hours. The prose'spoint rests on the counts of differing rows: an honest feature shows zero, and a leaking one shows its leak."""from lib.electricity import loadfrom lib.leakage import history_test, lag, peeking_mean, target_test, trailing_meanfrom lib.settings import EVAL_CUTOFFS, LEAD, showy = load([LEAD]).set_index("ds")["y"]  # indexed by timestamp, so each feature can be cut at a cutoff by timecompared, honest = history_test(y, trailing_mean)  # rows compared, and the rows where cutting at the cutoff changes the feature_, peeking = history_test(y, peeking_mean)  # the leaking feature: its last value before each cutoff reads the cutoff hour_, lag1_history = history_test(y, lag(1))  # lag 1 reads only the past on history rows, so this half passes itshow("cutoffs", len(EVAL_CUTOFFS))show("rows compared", compared)show("rows differing for the trailing mean", honest)show("rows differing for the peeking mean", peeking)show("rows differing for lag 1", lag1_history)targets, lag1_target = target_test(y, lag(1))  # target hours compared, and the hours that differ, over the 24 hours of each cutoff_, lag24_target = target_test(y, lag(24))show("target hours compared", targets)show("target hours differing for lag 1", lag1_target)show("target hours differing for lag 24", lag24_target)
输出python leakage-shift-check.py
cutoffs: 3
rows compared: 4032
rows differing for the trailing mean: 0
rows differing for the peeking mean: 3
rows differing for lag 1: 0
target hours compared: 72
target hours differing for lag 1: 69
target hours differing for lag 24: 0

On the history rows, the shifted mean differs on 3 of the 4032 compared, one at each cutoff: the last hour before each cutoff, whose value depends on the reading at the cutoff itself. The trailing mean differs on 0 of them.

The lag of one hour is the case the history half cannot catch. It differs on 0 history rows, because every row it reads lies in the past, so the history half alone passes it.

On the 72 target hours it differs on 69, since for every target hour after the first, the reading it uses is in the day being forecast. The lag of 24 hours differs on 0 of them, because it reads the same hour of the day before, which was known at each cutoff.

The shift test checks features and nothing else. It does not check the scale, the fill or the model's own code, which is why the earlier sections set those rules separately. Run it on every feature a model reads, at several cutoffs spread through the year, and include a cutoff on a clock-change day if the feature reads the hour before it.

When a feature fails the test, do not fix the score. Move the feature back until it reads only what was known at its cutoff, then run the test again until the count is zero.

Working rule: Keep every lag at least the horizon, 24 hours, so that the reading a lag uses for any target hour was known at the cutoff. Before you trust a backtest with features, run both halves of the shift test on every feature, at several cutoffs spread through the year, and require zero differing rows for every feature the model reads. A feature that differs is reading the future, so fix or remove it before you score anything.

这部分尚未翻译,因此以英文显示。

What goes wrong in production

最近一次修改见于版次 2026.11.5

A backtest is a sample of the past, and the sample is chosen. Every clock change falls on the last Sunday of March or October (Directive 2000/84/EC, Articles 2 and 3), so a Sunday sample reaches every change. A sample of Mondays would reach the day after each change without reaching the change hour itself, so the choice of days changes what the backtest can see. Record which dates the sample covers, and rerun the clock check when the sample's dates change.

Calendar effects are a separate sample problem. Easter Sunday, Apr 20, 2014, is a Portuguese holiday, which the holidays package lists for Portugal, and the run counts it: 1 of 52 windows. The held-out week contains Dec 25, 2014, a national holiday in Portugal, and a week of ordinary days and one holiday gives a different score from a week of ordinary days alone. A backtest that uses holiday features must count the holidays in its windows, and say how many fall in the sample.

A pooled error hides the windows that went wrong. A year of windows can average to a respectable number while one month, or one client, carries the error. Break the scores down by month and by client before you report one figure, and look at the worst windows, not only the average.

The masked count follows the calendar flag, not the readings. It changes when the sample's dates or the file's flags change, and it cannot see drift in the readings. Count the masked hours on every run, and alarm on a change in the count, because it is the cheapest signal that the sample or the file has changed. Drift in the readings needs the census: the gap hours that read zero, and the repeat hours against their neighbours. Chapter 10 returns to this in the running log.

Finally, a backtest that passes the shift test can still leak through the scale or the fill, if either is taken from data after the cutoff. The checks in this chapter cover those paths in the examples and the rules, not in every library a reader might use. A reader who adds a library should read how it cuts its training data, as the default of cross_validation shows.

Working rule: Run the clock and shift checks again on every data refresh, and before any change to the sample's dates, the history length or the scale. Keep the masked-hour count, the census counts and the shift-test counts in the run log next to the scores, so that a later change in either can be traced to its cause.

本章做出的决定

最近一次修改见于版次 2026.11.5
  • 回测使用 2014 年各个星期日的截止点。每次时钟调整都发生在星期日,因此样本覆盖每一次调整,2014 年的两次调整都在其中。
  • 每个窗口的历史是其截止点之前的 1,344 小时,并按截止点截断。长度在代码中设定,不依赖库的默认值。
  • 时间戳 ds 是该小时在本地时钟时间下的起点,不带时区,本书按此假设读取文件。
  • 两个时钟调整所在的小时,在每一份历史和每一次评分中都被屏蔽。对于不能处理缺失值的模型,该小时取截止点之前历史中相邻小时的均值。
  • 将十月的那个小时减半的做法被否决,因为它假定两个实际存在的小时用电量相同,而数据并未显示这一点。
  • 主要指标是各客户 MASE 的均值,采用第 2 章起通过 scoring.book_scale 沿用的同一标尺。合并 MAE 是第二项指标,并同时打印最大客户所占的份额。
  • 正文引用的每个数字,都像第 1 章所规定的那样,与其示例输出中带标签的一行绑定,因此库更新会改变本章的数字,而无需任何人重新键入。
  • 在第一个截止点之前没有读数的客户,不计入 MASE。起步较晚的客户在其第一个读数之前的小时,不计入任何评分。历史起点的规则由第 4 章负责。
  • 平移检验对每个特征的两半都要运行。每个滞后至少等于预测步长,在任何截止点上出现差异的特征都不使用。

本章,在本版次中

版次 2026.11.11示例于 2026年11月20日 通过本章 7个示例 个示例中有 7 个通过

译自英文版图书,版次 2026.11.11。

引用本节

格式

每种格式均引用英文原书,本译本以其为准。

每种格式都会注明该版次。其链接会打开当前版次,节标题处会说明该节文本最后一次修改的时间。访问日期和检索日期即为您打开此页的当天。

本页内容

第 3 章,共 10 章: 无前视回测

  1. Why one test week misleads
  2. Roll the cutoff forward
  3. Keep the future out of the history
  4. 认识时钟变更
  5. The March hour that reads zero
  6. 包含两个小时的十月那一小时
  7. 将换钟时段标记为缺失
  8. Check a backtest for leakage
  9. What goes wrong in production
  10. 本章做出的决定

返回顶部全部章节