量化预测的不确定性
阅读约需 72分钟 分钟,7节中含4个示例。
本章为每一个次日预测给出一个取值范围,使你能够判断某次采购是否可以依赖该预测。你将在早于评分日的那些日期上校准预测区间,度量这些区间成立的频率,并根据一次误差的两种成本来选择采购所依据的置信水平。
这部分尚未翻译,因此以英文显示。
Why one number is not enough
The forecasts so far give one number for each client and hour, and a single number is not enough to buy on. The supplier cannot buy an expected value. It buys an amount of power a day ahead for each hour, so it needs a range around each forecast. Each kWh it buys is either used or not. Buying short means buying the rest later at a premium, and buying long means paying for power that nobody uses. The two errors cost different amounts, so the right purchase is usually not the forecast itself. Take one hour. Suppose the forecast is 100 kWh. If the reading turns out 10 kWh higher, the supplier is short and pays the short cost on those 10 kWh. If it turns out 10 kWh lower, the supplier has 10 kWh it pays for and does not use. When the short cost is three times the unused cost, a purchase above the forecast costs less on average, because it makes the expensive error less likely. How far above the forecast the purchase should sit depends on that ratio, and on how wide the errors are.
This chapter states two costs as assumptions. A kWh bought short costs 3.00 cost units, and a kWh bought and not used costs 1.00. These numbers are illustrative. They are not prices, and the book does not claim that any real supplier faces them. A later section shows how they set the purchase.
A prediction interval gives a lower and an upper bound for each hour. A prediction interval's level, such as 80 or 95, is the share of readings the bounds should contain. The share of readings the bounds contain is the coverage, and coverage belongs to many forecasts made the same way, not to one. One day's band either contains the readings or it does not. Across the 72 hours of three days, the share inside the band is a measurement. Across a year of days, it becomes a track record. A 95 percent band whose hours fall inside it 90 percent of the time has undercovered, but no single day shows it. Only the average over many hours shows it, and that average is what we measure.
Quantiles connect the bands to the costs. The lower bound of a central 80 percent interval is the 0.1 quantile of the forecast's distribution, the value that the reading falls below one time in ten. The upper bound is the 0.9 quantile. The purchase that minimises the expected cost is also a quantile, and Choose the level from the cost of error derives which one.
In the case, the range that matters is the total's, because the total is what the supplier buys. Chapter 6 forecast the thirty clients with one global model, and the total is the hourly sum of those forecasts. Each client has its own range, but the total's range cannot be read off the clients' ranges by adding them. Choose the level from the cost of error compares the two.
Working rule: Ask for a range wherever the costs of the two errors differ, and choose its level from those costs rather than from a number that looks reassuring. Report coverage as the share of hours inside the band over many windows, never as the chance that one day falls inside.
由残差构建区间
区间需要上下界,而最简单的界来自过去的误差。残差是先前一次预测的误差:读数减去预测值,即 e = y - f。共形预测(conformal prediction)用先前各日以同样方式作出的预测的残差来构建区间带。取一组校准窗口的残差,按预测步长分组,再读出所需的分位数。在 00:00 作出的次日预测,一天中的每个小时各有一个预测步长,因此分组仅按小时划分。对于以百分数表示的水平 L,在给定步长上作出的预测 f 的上下界为:
lower = f + Q((1 - L/100) / 2)
upper = f + Q((1 + L/100) / 2)
其中 Q(p) 是该步长上校准残差的经验 p 分位数。该方法不对误差假定任何模型,只假定校准残差与将来的残差相似,后文一节会对此进行检验。校准窗口是指一个截止点,其残差用来确定区间带的宽度。它总是位于被预测的截止点之前,并且绝不是参与评分的窗口之一。
为本书的模型完成这项工作的库是 mlforecast,版本为 1.1.0。它的类 PredictionIntervals 接受窗口数、预测范围和方法三个参数,提供四种共形方法:conformal_error、conformal_distribution,以及它们的加权版本。两种不加权的方法对绝对残差 |y - f| 评分,所以其上下界围绕预测值对称(mlforecast 1.1.0,mlforecast/conformal_prediction.py,函数 compute_conformity_scores)。默认方法 conformal_distribution 每次处理一个客户和一个预测步长。对每个客户和步长,它取校准窗口的绝对残差,构造预测值加上和减去每个残差所得的路径,再读出这些路径的分位数。各客户之间不做汇总。统计库 statsforecast 的版本为 2.1.1,其区间类名为 ConformalIntervals,而不是 PredictionIntervals。
要了解该库如何处理较短的校准历史,可用三个校准窗口运行它的类。它对三个标准截止点评分,即 2014 年 12 月 29 日至 12 月 31 日,每个都在 00:00。这是留出一周的最后三天,也是本章评分的三天。mlforecast 在每个截止点重新拟合全局模型,并用其之前的三个窗口对每个截止点进行校准。
1"""mlforecast's own conformal intervals, with three calibration windows, refitted at each of the three standard cutoffs.2Compare the coverage at 80 and 95 with their labels, and with the highest coverage three windows allow."""3import pandas as pd45from lib.electricity import load6from lib.intervals import builtin_intervals, design_coverage, symmetric_ceiling7from lib.panel import ALL308from lib.quirks import mask_clock_changes9from lib.settings import EVAL_CUTOFFS, show1011raw = load(ALL30)12# Every clock-change hour (four changes, 2013 and 2014) is masked as NaN. builtin_intervals keeps only rows from Oct 27, 2014 on, so no masked hour is in a window.13df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]14# One row per client and hour. The bounds are columns named by model and level, such as LGBMRegressor-lo-80.15cv = builtin_intervals(df, pd.to_datetime(EVAL_CUTOFFS))1617show("scored hours", len(cv))18show("calibration windows per cutoff", 3)19show("residuals per client and hour", 3)20# Ceiling of a band of plus and minus n absolute residuals, n / (n + 1), with n = 3: at most 3 of the 4 places a new one can take.21show("highest coverage three windows allow, any level", symmetric_ceiling(3), 3)22for level in (80, 95):23 # Simulated under exchangeable normal errors, with mlforecast's construction, to set against the measured coverage.24 show(f"simulated coverage at {level}, exchangeable errors", design_coverage(level, 3, "builtin"), 3)25for level in (80, 95):26 lo, hi = cv[f"LGBMRegressor-lo-{level}"], cv[f"LGBMRegressor-hi-{level}"]27 inside = (cv["y"] >= lo) & (cv["y"] <= hi)28 show(f"coverage at {level}", inside.mean(), 3)29 show(f"mean width at {level}", (hi - lo).mean())30 if level == 80:31 # The by-day lines are printed for the 80 level only. Each hour is grouped by its calendar day.32 day = cv["ds"].dt.floor("D")33 for d, share in inside.groupby(day).mean().items():34 show(f"coverage at 80 on {d:%Y-%m-%d}", share, 3)35 for d, err in (cv["y"] - cv["LGBMRegressor"]).abs().groupby(day).mean().items():36 show(f"mean absolute error on {d:%Y-%m-%d}", err)
python conformal-three-windows.pyscored hours: 2160
calibration windows per cutoff: 3
residuals per client and hour: 3
highest coverage three windows allow, any level: 0.750
simulated coverage at 80, exchangeable errors: 0.656
simulated coverage at 95, exchangeable errors: 0.732
coverage at 80: 0.690
mean width at 80: 281.07
coverage at 80 on 2014-12-29: 0.756
coverage at 80 on 2014-12-30: 0.676
coverage at 80 on 2014-12-31: 0.637
mean absolute error on 2014-12-29: 114.62
mean absolute error on 2014-12-30: 94.42
mean absolute error on 2014-12-31: 163.37
coverage at 95: 0.750
mean width at 95: 355.39
输出在任何修正之前就显示了问题:在 2160 个客户小时上,80 水平的区间带覆盖率为 0.690,95 水平为 0.750。两者都低于其标称水平,而 80 水平的区间带平均宽度为 281.07 kWh。
该设计存在上限,95 水平的结果正落在这一上限上。只有三个窗口时,每个客户和小时只有三个绝对残差,区间带最远只能到达其中最大的一个。对于可交换的误差,测试误差有四分之一的机会是四个绝对误差中最大的一个,因此区间带有同样比例的机会漏掉它。三个窗口所能达到的最高覆盖率为 0.750,与水平无关。在同样的可交换假设下,区间带的期望覆盖率在 80 水平为 0.656,在 95 水平为 0.732。模拟抽取正态误差,构造方式相同。实测数值略高于这些期望值,所以校准窗口的误差并不比评分日的小。若校准窗口较为平静,覆盖率会低于期望值,而不会高于它。覆盖不足源于这一构造本身。
按日划分的覆盖率显示了不足出现的位置。在 80 水平,12 月 29 日为 0.756,12 月 30 日为 0.676,12 月 31 日为 0.637。这几天的预测误差平均分别为 114.62、94.42 和 163.37 kWh。12 月 31 日误差最大、覆盖率最低,但仅凭三天无法把逐日的波动与设计本身的影响区分开。
第二个问题来自校准。每个截止点只用其之前的日子校准:12 月 31 日的区间带使用 12 月 28、29 和 30 日。但 12 月 29 日和 12 月 30 日也是评分日。因此三个评分窗口并不独立,因为一个窗口的校准数据包含另一个评分窗口的读数。没有任何区间带使用其自身当天的读数,这一规则对实际运行是正确的,因为那时这些读数已知。但对检验而言,它削弱了结果。要修正它,需要为每个截止点单独做一次校准拟合,这不止是改动一个参数,所以本章自己的区间带改用固定的校准集,如在较早的窗口上校准所示。
工作准则: 每个区间带都应由其截止点之前的窗口的残差构建,并在信任它之前,检查它在未曾使用的窗口上达到的覆盖率。先看设计所能达到的水平,再看覆盖率。每个客户和小时只有三个残差,达不到任何一个标称水平,所以应当增大校准集,而不是换一种方式解读输出。
这部分尚未翻译,因此以英文显示。
Calibrate on earlier windows
The calibration windows must come before the windows you score, and they must not share a target hour with them. The book adds one more rule: a calibration window must not fall on a Portuguese public holiday, because a holiday's load follows the calendar rather than the week before it. The rule applies to the decision windows as well. The held-out evaluation calibrates on twelve ordinary days, Dec 9 to 20, 2014, which end four days before the held-out week. Both examples in this section print the holiday hours and the shared hours in their calibration, and both counts are zero.
Twelve windows with thirty clients give 360 residuals for each hour of the day. They come from twelve days, though, and the thirty clients share each day's weather and calendar. So a day-level error is still a sample of twelve. The pooled residuals have a second problem. The clients differ enormously in size. In the scored week the largest client, MT_370, averages 17499.81 kWh an hour over the three scored days. Pooled residuals in kWh are dominated by such clients, so the bands fit the large clients and misfit the small ones. The fix is to divide each client's residuals by that client's scale before pooling, and multiply the bound back by the scale of the client being forecast. The scale is the seasonal naive error on the history before the first calibration cutoff, the same kind of scale the book uses for MASE. With the scale, the thirty clients count alike.
Two choices in this design were made on the decision windows, not on the held-out evaluation. The decision windows are the four cutoffs from Nov 25 to Dec 3, 2014. Their calibration is Nov 9 to 20, 2014, twelve holiday-free days that end four days before the first decision cutoff, the same shape as the held-out design. The windows are scored on their own, and the held-out evaluation is not read. The choices are judged by the interval score and the lowest client's coverage. The interval score adds a band's width to a penalty for each reading that misses the band, the penalty growing with the distance of the miss and scaled by 2 over the miss rate (Gneiting and Raftery (2007), eq. 43). Those windows settle both choices, as the measurements below show.
1"""The two calibration choices, scaling each client's residuals and the finite-sample levels, decided on the four decision2windows (Nov 25 to Dec 3, 2014), with calibration on Nov 9 to 20. Compare the lowest client's coverage and the interval scores printed for each variant;3the standard cutoffs are not read."""4from lib.baselines import seasonal_scale5from lib.electricity import load6from lib.intervals import DECISION_CAL, bounds, calibration_facts, coverage, interval_score, paired, total_of7from lib.panel import ALL30, DECISION_CUTOFFS8from lib.quirks import mask_clock_changes9from lib.scoring import book_scale10from lib.settings import show1112raw = load(ALL30)13# Every clock-change hour (four changes, 2013 and 2014) is masked as NaN, as in every example of this chapter.14df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]15# One row per client and hour, with the forecast f, the reading y and the residual e.16cal, sc = paired(df, DECISION_CAL), paired(df, DECISION_CUTOFFS)17# Each client's MASE-style scale, from the history before the first calibration cutoff.18scale = seasonal_scale(book_scale(raw, DECISION_CAL[0]))1920for label, value in calibration_facts(df, DECISION_CUTOFFS, DECISION_CAL).items():21 show(label, value)22show("residuals per hour of the day", len(cal) // 24)23variants = (("unscaled", None, False), ("scaled", scale, False),24 ("unscaled, finite", None, True), ("scaled, finite", scale, True))25for name, s, fin in variants:26 for level in (80, 95):27 # scale=None pools the raw kWh residuals; finite=True applies the (n + 1) quantile correction.28 lo, hi = bounds(sc, cal, level, scale=s, finite=fin)29 show(f"{name}, coverage at {level}", coverage(sc["y"], lo, hi), 3)30 show(f"{name}, interval score at {level}", interval_score(sc["y"], lo, hi, level))31 lo, hi = bounds(sc, cal, 80, scale=s, finite=fin)32 # Coverage per client, then the lowest of the thirty clients.33 show(f"{name}, lowest client coverage at 80", ((sc["y"] >= lo) & (sc["y"] <= hi)).groupby(sc["unique_id"]).mean().min(), 3)34# One series, so no scale: the scale exists only to pool clients of different size.35tcal, tsc = total_of(cal), total_of(sc)36for name, fin in (("total", False), ("total, finite", True)):37 lo, hi = bounds(tsc, tcal, 80, finite=fin)38 show(f"{name}, coverage at 80", coverage(tsc["y"], lo, hi), 3)39 show(f"{name}, interval score at 80", interval_score(tsc["y"], lo, hi, 80))
python conformal-decision-windows.pycalibration windows: 12
holiday hours in calibration: 0
calibration hours shared with scored: 0
residuals per hour of the day: 360
unscaled, coverage at 80: 0.809
unscaled, interval score at 80: 603.88
unscaled, coverage at 95: 0.957
unscaled, interval score at 95: 1950.32
unscaled, lowest client coverage at 80: 0.021
scaled, coverage at 80: 0.834
scaled, interval score at 80: 336.72
scaled, coverage at 95: 0.954
scaled, interval score at 95: 581.28
scaled, lowest client coverage at 80: 0.365
unscaled, finite, coverage at 80: 0.811
unscaled, finite, interval score at 80: 604.19
unscaled, finite, coverage at 95: 0.959
unscaled, finite, interval score at 95: 1961.48
unscaled, finite, lowest client coverage at 80: 0.021
scaled, finite, coverage at 80: 0.835
scaled, finite, interval score at 80: 337.31
scaled, finite, coverage at 95: 0.956
scaled, finite, interval score at 95: 592.35
scaled, finite, lowest client coverage at 80: 0.375
total, coverage at 80: 0.948
total, interval score at 80: 9268.36
total, finite, coverage at 80: 0.990
total, finite, interval score at 80: 10724.99
Scaling wins on the two measures the variants were judged by, and loses only on the pooled coverage at 80. The worst-covered client's coverage at 80 rises from 0.021 to 0.365, and the interval score at 80 falls from 603.88 to 336.72. It falls at 95 as well, as the output shows.
We judged the variants by these two measures because the pooled figure hides clients. We chose them after seeing the decision numbers, so the pooled figure is reported beside them. At 80 the pooled coverage is closer to its label without the scale, at 0.809 against 0.834. The scale stays.
The finite-sample levels stay out of the bands. Each new residual is one more among n + 1, so the finite-sample levels move outward a little: each quantile level moves to floor((n + 1) * a) / n for the lower bound and ceil((n + 1) * (1 - a)) / n for the upper, where n is the number of residuals and a the tail share. Their effect on the scaled coverage at 80 is small, from 0.834 to 0.835, and the interval score at 80 gets worse, from 336.72 to 337.31. It gets worse at 95 too, as the output shows. The correction widens the bands and does not help, so the bands do not use it.
The held-out evaluation applies the same design to the standard cutoffs, which are the days the rest of the chapter scores.
1"""Conformal bands for every client, calibrated on twelve holiday-free days before the scored week, with each client's2residuals scaled first. Read the coverage and interval score at 80 and 95 first; the unscaled, three-day and3finite-sample lines are printed for comparison."""4import pandas as pd56from lib.baselines import seasonal_scale7from lib.electricity import load8from lib.intervals import CAL_CUTOFFS, bounds, calibration_facts, coverage, interval_score, paired9from lib.panel import ALL3010from lib.quirks import mask_clock_changes11from lib.scoring import book_scale12from lib.settings import EVAL_CUTOFFS, show1314raw = load(ALL30)15df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]16scored_cutoffs = pd.to_datetime(EVAL_CUTOFFS)17cal, sc = paired(df, CAL_CUTOFFS), paired(df, scored_cutoffs)18# The scale is read from the history before the first calibration cutoff, as the book's MASE scale is.19scale = seasonal_scale(book_scale(raw, CAL_CUTOFFS[0]))20# The largest client is the one with the highest mean reading over the scored windows.21big = sc.groupby("unique_id")["y"].mean().idxmax()2223for label, value in calibration_facts(df, scored_cutoffs).items():24 show(label, value)25show("calibration residuals per hour of the day", len(cal) // 24)26show("largest client", big)27show("mean hourly load of the largest client", sc.groupby("unique_id")["y"].mean().max())28for level in (80, 95):29 lo, hi = bounds(sc, cal, level, scale=scale)30 show(f"coverage at {level}", coverage(sc["y"], lo, hi), 3)31 show(f"mean width at {level}", (hi - lo).mean())32 show(f"interval score at {level}", interval_score(sc["y"], lo, hi, level))33lo, hi = bounds(sc, cal, 80, scale=scale)34per_client = ((sc["y"] >= lo) & (sc["y"] <= hi)).groupby(sc["unique_id"]).mean()35show("lowest client coverage at 80", per_client.min(), 3)36# No scale here: the pooled residuals stay in kWh, which is what lets the large clients set the bands.37lo, hi = bounds(sc, cal, 80)38unscaled = ((sc["y"] >= lo) & (sc["y"] <= hi)).groupby(sc["unique_id"]).mean()39show("largest client coverage at 80, unscaled residuals", unscaled[big], 3)40# The three calibration days nearest the scored week: Dec 18, 19 and 20, 2014.41recent = cal[cal["cutoff"] >= "2014-12-18"]42lo, hi = bounds(sc, recent, 80, scale=scale)43show("coverage at 80 from three calibration days", coverage(sc["y"], lo, hi), 3)44lo, hi = bounds(sc, cal, 80, scale=scale, finite=True)45show("coverage at 80 with finite-sample levels", coverage(sc["y"], lo, hi), 3)
python conformal-earlier-windows.pycalibration windows: 12
holiday hours in calibration: 0
calibration hours shared with scored: 0
calibration residuals per hour of the day: 360
largest client: MT_370
mean hourly load of the largest client: 17499.81
coverage at 80: 0.739
mean width at 80: 267.62
interval score at 80: 673.89
coverage at 95: 0.902
mean width at 95: 532.00
interval score at 95: 1326.80
lowest client coverage at 80: 0.472
largest client coverage at 80, unscaled residuals: 0.000
coverage at 80 from three calibration days: 0.711
coverage at 80 with finite-sample levels: 0.741
The scaled bands still fall short of both labels on the standard cutoffs: they cover 0.739 of client-hours at 80 and 0.902 at 95. Their mean widths are 267.62 kWh and 532.00 kWh.
The scores are 673.89 at 80 and 1326.80 at 95. They are in kWh, so the large clients dominate them. Lower is better, and the score rewards a band that is both narrow and right.
The pooled bands still have to be read per client. The lowest client coverage at 80 is 0.472, for the client the scaled bands fit least well. With unscaled residuals, the largest client is covered in 0.000 of its hours at 80. The pooled kWh bands were set by the large clients, and they miss the largest client entirely.
Two more checks on the standard cutoffs test the design. Calibrating on only the three days nearest the scored week, Dec 18 to 20, gives 0.711 at 80, against 0.739 with all twelve. Three days are a smaller sample, and the coverage is a little lower. The finite-sample levels give 0.741 at 80, a small change that agrees with the decision windows: the correction does little for the client bands.
Working rule: Calibrate on windows that come before the scored ones, with no holiday in them, and scale each client before pooling its residuals. Make each design choice on windows that are not the ones you report. Report how many windows and residuals went into the bands and the coverage they gave on the scored days, even when that coverage falls short of the level.
这部分尚未翻译,因此以英文显示。
Measure coverage on held-out days
Coverage is measured on days that the calibration never used. The standard cutoffs give three days, and each day gives 24 hours for each client, so the 2160 client-hours are not that many independent observations. The thirty clients share the calendar and the weather, so the hours of one day agree with one another. For day-level errors the sample is three days, which cannot measure a coverage level. Three days can show that a band fell short on those days, but they cannot say by how much a band falls short over a longer period.
The supplier buys the total, so the total's coverage matters most. Its bands are built from its own residuals over the same twelve days, with the quantiles read directly, and are offered at 80 only, because twelve residuals cannot support 95. With signed bounds, the band from the smallest residual to the largest covers at most 0.846 of the time, if the residuals are exchangeable, and a 95 band, or a 90 band, asks for more.
The total's bands cover 0.667 of its 72 scored hours at 80. For exchangeable errors, the same construction covers 0.682, so most of the gap to 80 comes from the twelve-residual sample. The section on cost puts these bands to use.
The total's errors vary by day. The mean absolute error of the total was 2415.51 kWh on the calibration windows and 2985.88 kWh on the scored days. Dec 31 accounts for most of the gap between the scored days and the calibration windows.
| Scored day | Error of the total, kWh | Coverage at 80 |
|---|---|---|
| Dec 29, 2014 | 2448.53 | 0.625 |
| Dec 30, 2014 | 2149.78 | 0.708 |
| Dec 31, 2014 | 4359.32 | 0.667 |
The coverage does not follow the errors. Dec 29 has the lowest coverage with a middle-sized error, and Dec 31 has the largest error with a middle-sized coverage. Each day has 24 hours, and three days are too few to tell the two apart.
The finite-sample levels would change the total's figure. On the scored days they give 0.736 at 80. The choice was made on the decision windows, where the correction widened the total's bands and worsened their interval score, so the bands do not use it. The decision windows also show that the direction of the gap is not fixed. There the total's 80 bands covered 0.948 of hours, more than their label, and the finite levels pushed that to 0.990.
Angelopoulos and Bates (2022) show that split conformal prediction covers the test point with probability at least 1 - alpha when the calibration and test points are independent and identically distributed, or more generally exchangeable. They also note that adjacent points in a time series are not exchangeable, using a temperature series as the example. The same holds for hourly load: each hour carries the one before it, and a day's hours share the weather and the calendar. Barber and co-authors (2023) call the loss in coverage when exchangeability fails a coverage gap. Their bound does not assume a particular joint distribution. It is small when recent calibration points are weighted in a way that matches the test point's distribution, as in slow drift. That result is theoretical, and this chapter does not test it. A 2026 preprint, a survey of conformal methods for dependent series, states that temporal dependence violates the exchangeability assumption. It is not peer reviewed, and we cite it only for that point.
So the book promises nothing about coverage that it does not measure. The coverage in this chapter is what three days showed, under the calibration described in Calibrate on earlier windows.
Working rule: Report coverage on windows that the calibration never used, with the number of windows and the coverage of each. Keep the same measure in a running log once the forecast runs every day, as Chapter 10 does. Read three scored days as a warning, not as a measurement of the level you were promised.
这部分尚未翻译,因此以英文显示。
Choose the level from the cost of error
The supplier buys one amount for each hour, and the costs of the two errors set that amount. Let cu be the cost of one kWh bought short and co the cost of one kWh bought and not used. Let F be the distribution of demand. Differentiating the expected cost with respect to the purchase q gives co F(q) - cu (1 - F(q)), which is zero where F(q) = cu / (cu + co), the smallest expected cost. The level cu / (cu + co) is the critical fractile of the newsvendor model. The HKUST IEEM 101 notes on the newsvendor model give the same proof, writing the cost of excess as c0. This chapter writes it co. The purchase is therefore the quantile at level tau = cu / (cu + co).
With the illustrative costs from Why one number is not enough, tau is 0.75. The rule says to buy the quantile of the demand at that level, for each hour. The quantile at level tau is the purchase that minimises the quantile loss at that level, as Gneiting and Raftery (2007) show. The quantile loss is tau * (y - q) when the reading y is at or above the purchase q, and (1 - tau) * (q - y) when it is below. The quantile loss is not a cost in kWh or in money. It is the rule that the quantile minimises, and it has the same shape as the supplier's cost. Multiplying it by cu + co gives the cost exactly, which the example prints beside the mean cost.
Does buying at the critical fractile cost less than buying at the median or higher, on days the bands never saw? Below, the purchase is set for each hour from these costs, and the scored days answer the question. The level 0.95 is not offered, as Measure coverage on held-out days explains.
1"""The buy level for the total, set from the two costs: the critical fractile, cu / (cu + co), is the quantile that minimises2the expected cost. Compare the mean cost and the hours short at each level; the costs are illustrative, not prices."""3import pandas as pd45from lib.electricity import load6from lib.intervals import CAL_CUTOFFS, COST_EXCESS, COST_SHORT, buy_report, critical_fractile, paired7from lib.panel import ALL308from lib.quirks import mask_clock_changes9from lib.settings import EVAL_CUTOFFS, show1011raw = load(ALL30)12# Every clock-change hour (four changes, 2013 and 2014) is masked as NaN, as in every example of this chapter.13df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]14# Calibration on Dec 9 to 20, 2014; scored on the three standard cutoffs.15pairs_cal, pairs_sc = paired(df, CAL_CUTOFFS), paired(df, pd.to_datetime(EVAL_CUTOFFS))1617show("cost of a kWh short", COST_SHORT)18show("cost of a kWh unused", COST_EXCESS)19# The quantile level cu / (cu + co), from the two costs printed above.20show("critical fractile", critical_fractile())21# buy_report returns the printed rows as (label, value, decimal places).22for label, value, digits in buy_report(pairs_cal, pairs_sc):23 show(label, value, digits)
python buy-ahead-level-by-cost.pycost of a kWh short: 3.00
cost of a kWh unused: 1.00
critical fractile: 0.75
scored hours: 72
residuals per hour for the total: 12
highest coverage the total's residuals allow: 0.846
simulated coverage at 80, exchangeable errors: 0.682
mean absolute error of the total, calibration days: 2415.51
mean absolute error of the total, scored days: 2985.88
total coverage at 80: 0.667
total coverage at 80 with finite-sample levels, not used: 0.736
total coverage at 80 on 2014-12-29: 0.625
total coverage at 80 on 2014-12-30: 0.708
total coverage at 80 on 2014-12-31: 0.667
total mean absolute error on 2014-12-29: 2448.53
total mean absolute error on 2014-12-30: 2149.78
total mean absolute error on 2014-12-31: 4359.32
mean cost at 0.50: 5087.95
hours short at 0.50: 0.528
mean cost at 0.75: 3843.61
hours short at 0.75: 0.403
mean cost at 0.90: 3667.43
hours short at 0.90: 0.222
quantile loss at critical fractile, times both costs: 3843.61
mean hourly total bound from the total: 31983.95
mean hourly total bound from the sum of client bounds: 32388.77
hours short at critical fractile, sum of client bounds: 0.306
mean cost at critical fractile, sum of client bounds: 3729.17
The identity holds: multiplied by the sum of the two costs, the quantile loss at the critical fractile is 3843.61, the same as the mean cost at that level, 3843.61. The rule is correct as arithmetic.
| Level | Mean cost | Hours short |
|---|---|---|
| 0.50 | 5087.95 | 0.528 |
| 0.75, the critical fractile | 3843.61 | 0.403 |
| 0.90 | 3667.43 | 0.222 |
The result on the scored days is less comfortable. The rule chose the critical fractile, and on these three days 0.90 cost less, at 3667.43 against 3843.61. At the critical fractile the purchase was short in 0.403 of the hours, where a calibrated level would fall short in one hour in four. That is the coverage shortfall from Measure coverage on held-out days seen as money. We do not move the level to 0.90. The three scored days are the test, and a level chosen because it wins on them has been chosen on the test. The honest reading is that the bands were too narrow here, so the purchase was too low, and that the level is right only when the bands are.
Should the purchase follow the clients' bounds instead? Summing each client's bound at the critical fractile gives a bound of 32388.77 kWh an hour, against 31983.95 from the total's own residuals. Summed quantiles equal the total's quantile when the errors move together perfectly. For an upper quantile, with errors that partly cancel, the sum usually sits above it, as it does here.
On the scored days the summed bound did better, with hours short at 0.306 against 0.403, and a mean cost of 3729.17 against 3843.61. That advantage comes from its higher level, not from a better calibration, and the scored days cannot show that it holds in general. We keep the total's own bound, because the purchase is made against the total's residuals, and we do not switch to the bound that did better here.
Working rule: Set the buy level from the two costs, as tau = cu / (cu + co), fix it before you score the days, and calibrate the total's bands on the total's own residuals. Show the realised cost at other levels as a check, but never move the level to whichever one would have won on the days you score.
生产环境中会出什么问题
- 校准所在的季节不是被评分的季节。 校准时排除了节假日,这使测试保持公正,但总量的误差在不同窗口之间会有变化,被评分的那几天里误差可能比校准窗口中的更大。在平稳时期校准出的区间,到了更难的时期就会失准。要发现这一点,可以每天把覆盖率与名义水平相比较,并观察其滚动平均值。覆盖率一周内低于其标称值就是征兆,应首先查看那一周的误差。
- 汇总的区间会掩盖个别客户。 以 kWh 为单位汇总的区间由最大的客户决定。使用未缩放的残差时,在 80 水平上,最大客户的区间覆盖了它 0.000 的小时。使用缩放后的残差时,80 水平上客户覆盖率的最低值为 0.472,所以即使是缩放后的汇总,也会掩盖未达标的客户。应当逐个客户查看覆盖率,而不能只看汇总数字。
- 三天说明不了什么。 三个被评分日的覆盖率数字看上去可能像一个结论,但它只是日级误差的三次观测。应使用多个窗口上的覆盖率,并把短期的结果当作需要查看的理由,而不是当作结论。
- 相关性会破坏简单的保证。 逐小时负荷存在自相关,相邻小时的校准残差并不相互独立。这些区间不具备教科书定理所给出的保证。天气或日历发生变化时,它们的覆盖率可能漂移,只有实测的覆盖率才能告诉你何时发生了这种情况。需要留意的模式是若干天的误差符号相同,而滚动日志正是它显现的地方。
- 总量每个小时的残差很少。 一个校准窗口为总量在每个小时给出一个残差,所以十二个窗口只有十二个残差。使用带符号的残差时,十二个残差所允许的最高覆盖率为 0.846,因此总量的区间止于 80,在给出任何区间之前,应先做水平检查。增加校准窗口可以让总量得到更大的样本,但每个窗口都必须是被评分那一周之前的普通日。
- 滚动校准可能用到你要评分的日子。 由残差构建区间中的内置规则用每个截止点之前的三个截止点来校准该截止点。在实际运行中这没有问题,因为那些读数是已知的。在测试中,它会使各个被评分窗口相互依赖。应让校准集保持固定,并且早于每一个被评分窗口。
本章做出的决定
- 校准窗口。 取留出周之前的十二个普通日,即 2014 年 12 月 9 日至 20 日。节假日被排除在外,校准时段内的任何一小时都不计分。决策窗口同样使用十二天,即 2014 年 11 月 9 日至 20 日。这一长度是一种选择,而不是结果:数据并未表明十二天是最佳长度。
- 先缩放再合并。 每个客户的残差,除以该客户在首个校准截断点之前的历史数据上的季节性朴素误差,所用尺度与本书计算 MASE 时相同。这一做法由决策窗口确定,而总量不需要缩放。
- 区间带不做有限样本修正。 该修正已在决策窗口上针对各客户和总量做过检验,并没有改善区间得分。它仅用于判断一组残差能够支持哪些水平。
- 带符号的残差。 区间带和购电分位数使用带符号的残差,因此下界与上界不必对称。mlforecast 的共形方法使用绝对残差,给出对称的区间带。Build intervals from residuals 中的内置运行结果用于对照。
- 购电针对总量。 购电水平及其分位数依据总量的残差确定。各客户的区间带用于解读预测,而不用于购电。
- 成本属于假设。 3.00 与 1.00 这两项成本已在 Why one number is not enough 中说明,它们是判断而非实测值。
- 统计库的区间类。 statsforecast 的类名为
ConformalIntervals。它与 mlforecast 的PredictionIntervals名称不同,本章对各个库分别使用其各自的名称。 - 第三个示例。 购电决策需要单独的度量,因此新增了
buy-ahead-level-by-cost,同时新增conformal-decision-windows,后者在留出评估不使用的窗口上做出设计选择。 - 不声称覆盖节假日。 2014 年 12 月 25 日位于留出周内,但既不在校准窗口内,也不在计分窗口内。关于区间带对节假日当天的覆盖情况,本书不作任何报告。
- 先有区间带,再有水平。 购电水平的好坏取决于其下的区间带,而区间带的好坏取决于确定它的窗口。
本章,在本版次中
版次 2026.11.11示例于 2026年11月20日 通过本章 4个示例 个示例中有 4 个通过
译自英文版图书,版次 2026.11.11。