量化預測的不確定性

閱讀約需 72 分鐘 分鐘,7 節 中共有 4 個範例。

版次 2026.11.11 範例於 2026年11 月 20 通過測試 41 個範例中有 41 個通過 變更內容

由 AI 翻譯。經人工以英文驗證。 我們如何製作書籍

譯自英文書,版次 2026.11.11。 閱讀原文

本章會為每一個隔日預測加上一個區間,讓你能判斷採購是否可以依賴該預測。你會使用早於評分日期的資料來校準預測區間,衡量這些區間成立的頻率,並根據錯誤的兩種成本來選擇採購所採用的水準。

此部分尚未翻譯,因此以英文呈現。

Why one number is not enough

最近一次變動於第 2026.11.11 版

The forecasts so far give one number for each client and hour, and a single number is not enough to buy on. The supplier cannot buy an expected value. It buys an amount of power a day ahead for each hour, so it needs a range around each forecast. Each kWh it buys is either used or not. Buying short means buying the rest later at a premium, and buying long means paying for power that nobody uses. The two errors cost different amounts, so the right purchase is usually not the forecast itself. Take one hour. Suppose the forecast is 100 kWh. If the reading turns out 10 kWh higher, the supplier is short and pays the short cost on those 10 kWh. If it turns out 10 kWh lower, the supplier has 10 kWh it pays for and does not use. When the short cost is three times the unused cost, a purchase above the forecast costs less on average, because it makes the expensive error less likely. How far above the forecast the purchase should sit depends on that ratio, and on how wide the errors are.

This chapter states two costs as assumptions. A kWh bought short costs 3.00 cost units, and a kWh bought and not used costs 1.00. These numbers are illustrative. They are not prices, and the book does not claim that any real supplier faces them. A later section shows how they set the purchase.

A prediction interval gives a lower and an upper bound for each hour. A prediction interval's level, such as 80 or 95, is the share of readings the bounds should contain. The share of readings the bounds contain is the coverage, and coverage belongs to many forecasts made the same way, not to one. One day's band either contains the readings or it does not. Across the 72 hours of three days, the share inside the band is a measurement. Across a year of days, it becomes a track record. A 95 percent band whose hours fall inside it 90 percent of the time has undercovered, but no single day shows it. Only the average over many hours shows it, and that average is what we measure.

Quantiles connect the bands to the costs. The lower bound of a central 80 percent interval is the 0.1 quantile of the forecast's distribution, the value that the reading falls below one time in ten. The upper bound is the 0.9 quantile. The purchase that minimises the expected cost is also a quantile, and Choose the level from the cost of error derives which one.

In the case, the range that matters is the total's, because the total is what the supplier buys. Chapter 6 forecast the thirty clients with one global model, and the total is the hourly sum of those forecasts. Each client has its own range, but the total's range cannot be read off the clients' ranges by adding them. Choose the level from the cost of error compares the two.

Working rule: Ask for a range wherever the costs of the two errors differ, and choose its level from those costs rather than from a number that looks reassuring. Report coverage as the share of hours inside the band over many windows, never as the chance that one day falls inside.

以殘差建立區間

最近一次變動於第 2026.11.6 版

範圍需要界限,而最簡單的界限來自過去的誤差。殘差是先前預測的誤差:讀數減去預測值,即 e = y - f。保形預測(conformal prediction)以先前各日以相同方式所做預測的殘差建立區間帶。取一組校準窗口的殘差,依預測步長分組,再讀出所需的分位數。於 00:00 做出的隔日預測,一天中每個小時各有一個預測步長,因此分組只依小時而定。對於以百分比表示的水準 L,在某一步長所做預測 f 的界限為:

lower = f + Q((1 - L/100) / 2)
upper = f + Q((1 + L/100) / 2)

其中 Q(p) 是該步長校準殘差的經驗 p 分位數。此方法不對誤差假設任何模型,只假設校準殘差與未來的殘差相似,後續一節會檢驗這項假設。校準窗口是一個截止點,其殘差用來決定區間帶的寬度。它一定早於被預測的截止點,也絕不會是用來評分的窗口之一。

為本書的模型完成這項工作的函式庫是 mlforecast,版本為 1.1.0。它的類別 PredictionIntervals 接受窗口數、預測期長與方法,並提供四種保形方法:conformal_error、conformal_distribution,以及它們的加權版本。兩種未加權的方法以絕對殘差 |y - f| 評分,因此其界限對預測值對稱(mlforecast 1.1.0,mlforecast/conformal_prediction.py,函式 compute_conformity_scores)。預設的 conformal_distribution 一次處理一個客戶與一個預測步長。對每個客戶與步長,它取校準窗口的絕對殘差,建立預測值加上與減去每個殘差的路徑,再讀出這些路徑的分位數。不同客戶之間不做任何合併。統計函式庫 statsforecast 的版本為 2.1.1,其區間類別名為 ConformalIntervals,而非 PredictionIntervals。

若要了解函式庫在校準歷史很短時的表現,可用三個校準窗口執行其類別。它對三個標準截止點評分,即 2014 年 12 月 29 日至 12 月 31 日,每個皆在 00:00。這是保留一週的最後三天,也是本章評分的日子。mlforecast 在每個截止點重新擬合全域模型,並以其前三個截止點校準。

conformal-three-windows.py Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17 通過
"""mlforecast's own conformal intervals, with three calibration windows, refitted at each of the three standard cutoffs.Compare the coverage at 80 and 95 with their labels, and with the highest coverage three windows allow."""import pandas as pdfrom lib.electricity import loadfrom lib.intervals import builtin_intervals, design_coverage, symmetric_ceilingfrom lib.panel import ALL30from lib.quirks import mask_clock_changesfrom lib.settings import EVAL_CUTOFFS, showraw = load(ALL30)# Every clock-change hour (four changes, 2013 and 2014) is masked as NaN. builtin_intervals keeps only rows from Oct 27, 2014 on, so no masked hour is in a window.df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]# One row per client and hour. The bounds are columns named by model and level, such as LGBMRegressor-lo-80.cv = builtin_intervals(df, pd.to_datetime(EVAL_CUTOFFS))show("scored hours", len(cv))show("calibration windows per cutoff", 3)show("residuals per client and hour", 3)# Ceiling of a band of plus and minus n absolute residuals, n / (n + 1), with n = 3: at most 3 of the 4 places a new one can take.show("highest coverage three windows allow, any level", symmetric_ceiling(3), 3)for level in (80, 95):    # Simulated under exchangeable normal errors, with mlforecast's construction, to set against the measured coverage.    show(f"simulated coverage at {level}, exchangeable errors", design_coverage(level, 3, "builtin"), 3)for level in (80, 95):    lo, hi = cv[f"LGBMRegressor-lo-{level}"], cv[f"LGBMRegressor-hi-{level}"]    inside = (cv["y"] >= lo) & (cv["y"] <= hi)    show(f"coverage at {level}", inside.mean(), 3)    show(f"mean width at {level}", (hi - lo).mean())    if level == 80:        # The by-day lines are printed for the 80 level only. Each hour is grouped by its calendar day.        day = cv["ds"].dt.floor("D")        for d, share in inside.groupby(day).mean().items():            show(f"coverage at 80 on {d:%Y-%m-%d}", share, 3)        for d, err in (cv["y"] - cv["LGBMRegressor"]).abs().groupby(day).mean().items():            show(f"mean absolute error on {d:%Y-%m-%d}", err)
輸出python conformal-three-windows.py
scored hours: 2160
calibration windows per cutoff: 3
residuals per client and hour: 3
highest coverage three windows allow, any level: 0.750
simulated coverage at 80, exchangeable errors: 0.656
simulated coverage at 95, exchangeable errors: 0.732
coverage at 80: 0.690
mean width at 80: 281.07
coverage at 80 on 2014-12-29: 0.756
coverage at 80 on 2014-12-30: 0.676
coverage at 80 on 2014-12-31: 0.637
mean absolute error on 2014-12-29: 114.62
mean absolute error on 2014-12-30: 94.42
mean absolute error on 2014-12-31: 163.37
coverage at 95: 0.750
mean width at 95: 355.39

輸出在任何修正之前就顯示出失敗:在 2160 個客戶小時中,80 水準的區間帶涵蓋 0.690,95 水準則涵蓋 0.750。兩者都低於其標示的水準,而 80 水準的區間帶平均寬 281.07 kWh。

此設計設下了上限,而 95 的結果正落在這個上限上。只有三個窗口時,每個客戶與小時只有三個絕對殘差,區間帶最遠只到其中最大者。對於可交換的誤差,測試誤差有四分之一的機會是四個絕對誤差中最大的,因此區間帶有四分之一的次數會漏掉它。三個窗口所能達到的最高涵蓋率為 0.750,與水準無關。在同樣的可交換假設下,區間帶的預期涵蓋率在 80 為 0.656,在 95 為 0.732。模擬以相同的建構方式抽取常態誤差。實測數字略高於這些預期值,所以校準窗口的誤差並未比評分日更小。平靜的校準窗口會使涵蓋率低於預期值,而非高於預期值。不足之處出在此建構本身。

依日期查看涵蓋率,可知不足出現在哪裡。在 80 水準,12 月 29 日為 0.756,12 月 30 日為 0.676,12 月 31 日為 0.637。這幾天的預測誤差平均分別為 114.62、94.42 與 163.37 kWh。12 月 31 日的誤差最大,涵蓋率最低,而三天的資料無法把逐日變動與設計本身的影響區分開來。

第二個問題來自校準。每個截止點只以其之前的日子校準:12 月 31 日的區間帶使用 12 月 28、29 與 30 日。但 12 月 29 日與 12 月 30 日也是評分日。因此這三個評分窗口並不獨立,因為某一窗口的校準包含了另一個評分窗口的讀數。沒有任何區間帶使用自己那一天的讀數,而這項規則適用於實際運作,因為運作時那些讀數是已知的。對測試而言,這會削弱結果。要修正這點,需為每個截止點另做一次校準擬合,這不只是更改一個引數而已,因此本章自己的區間帶改用固定的校準集,如以較早的窗口校準所示。

實用原則: 每個區間帶都以早於其截止點的窗口殘差建立,並在信任它之前,先檢查它在未使用過的窗口上達到的涵蓋率。先看設計能達到的水準,再看涵蓋率。每個客戶與小時只有三個殘差,無法達到任一標示水準,因此解法是擴大校準集,而不是換一種方式解讀輸出。

此部分尚未翻譯,因此以英文呈現。

Calibrate on earlier windows

最近一次變動於第 2026.11.11 版

The calibration windows must come before the windows you score, and they must not share a target hour with them. The book adds one more rule: a calibration window must not fall on a Portuguese public holiday, because a holiday's load follows the calendar rather than the week before it. The rule applies to the decision windows as well. The held-out evaluation calibrates on twelve ordinary days, Dec 9 to 20, 2014, which end four days before the held-out week. Both examples in this section print the holiday hours and the shared hours in their calibration, and both counts are zero.

Twelve windows with thirty clients give 360 residuals for each hour of the day. They come from twelve days, though, and the thirty clients share each day's weather and calendar. So a day-level error is still a sample of twelve. The pooled residuals have a second problem. The clients differ enormously in size. In the scored week the largest client, MT_370, averages 17499.81 kWh an hour over the three scored days. Pooled residuals in kWh are dominated by such clients, so the bands fit the large clients and misfit the small ones. The fix is to divide each client's residuals by that client's scale before pooling, and multiply the bound back by the scale of the client being forecast. The scale is the seasonal naive error on the history before the first calibration cutoff, the same kind of scale the book uses for MASE. With the scale, the thirty clients count alike.

Two choices in this design were made on the decision windows, not on the held-out evaluation. The decision windows are the four cutoffs from Nov 25 to Dec 3, 2014. Their calibration is Nov 9 to 20, 2014, twelve holiday-free days that end four days before the first decision cutoff, the same shape as the held-out design. The windows are scored on their own, and the held-out evaluation is not read. The choices are judged by the interval score and the lowest client's coverage. The interval score adds a band's width to a penalty for each reading that misses the band, the penalty growing with the distance of the miss and scaled by 2 over the miss rate (Gneiting and Raftery (2007), eq. 43). Those windows settle both choices, as the measurements below show.

conformal-decision-windows.py Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17 通過
"""The two calibration choices, scaling each client's residuals and the finite-sample levels, decided on the four decisionwindows (Nov 25 to Dec 3, 2014), with calibration on Nov 9 to 20. Compare the lowest client's coverage and the interval scores printed for each variant;the standard cutoffs are not read."""from lib.baselines import seasonal_scalefrom lib.electricity import loadfrom lib.intervals import DECISION_CAL, bounds, calibration_facts, coverage, interval_score, paired, total_offrom lib.panel import ALL30, DECISION_CUTOFFSfrom lib.quirks import mask_clock_changesfrom lib.scoring import book_scalefrom lib.settings import showraw = load(ALL30)# Every clock-change hour (four changes, 2013 and 2014) is masked as NaN, as in every example of this chapter.df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]# One row per client and hour, with the forecast f, the reading y and the residual e.cal, sc = paired(df, DECISION_CAL), paired(df, DECISION_CUTOFFS)# Each client's MASE-style scale, from the history before the first calibration cutoff.scale = seasonal_scale(book_scale(raw, DECISION_CAL[0]))for label, value in calibration_facts(df, DECISION_CUTOFFS, DECISION_CAL).items():    show(label, value)show("residuals per hour of the day", len(cal) // 24)variants = (("unscaled", None, False), ("scaled", scale, False),            ("unscaled, finite", None, True), ("scaled, finite", scale, True))for name, s, fin in variants:    for level in (80, 95):        # scale=None pools the raw kWh residuals; finite=True applies the (n + 1) quantile correction.        lo, hi = bounds(sc, cal, level, scale=s, finite=fin)        show(f"{name}, coverage at {level}", coverage(sc["y"], lo, hi), 3)        show(f"{name}, interval score at {level}", interval_score(sc["y"], lo, hi, level))    lo, hi = bounds(sc, cal, 80, scale=s, finite=fin)    # Coverage per client, then the lowest of the thirty clients.    show(f"{name}, lowest client coverage at 80", ((sc["y"] >= lo) & (sc["y"] <= hi)).groupby(sc["unique_id"]).mean().min(), 3)# One series, so no scale: the scale exists only to pool clients of different size.tcal, tsc = total_of(cal), total_of(sc)for name, fin in (("total", False), ("total, finite", True)):    lo, hi = bounds(tsc, tcal, 80, finite=fin)    show(f"{name}, coverage at 80", coverage(tsc["y"], lo, hi), 3)    show(f"{name}, interval score at 80", interval_score(tsc["y"], lo, hi, 80))
輸出python conformal-decision-windows.py
calibration windows: 12
holiday hours in calibration: 0
calibration hours shared with scored: 0
residuals per hour of the day: 360
unscaled, coverage at 80: 0.809
unscaled, interval score at 80: 603.88
unscaled, coverage at 95: 0.957
unscaled, interval score at 95: 1950.32
unscaled, lowest client coverage at 80: 0.021
scaled, coverage at 80: 0.834
scaled, interval score at 80: 336.72
scaled, coverage at 95: 0.954
scaled, interval score at 95: 581.28
scaled, lowest client coverage at 80: 0.365
unscaled, finite, coverage at 80: 0.811
unscaled, finite, interval score at 80: 604.19
unscaled, finite, coverage at 95: 0.959
unscaled, finite, interval score at 95: 1961.48
unscaled, finite, lowest client coverage at 80: 0.021
scaled, finite, coverage at 80: 0.835
scaled, finite, interval score at 80: 337.31
scaled, finite, coverage at 95: 0.956
scaled, finite, interval score at 95: 592.35
scaled, finite, lowest client coverage at 80: 0.375
total, coverage at 80: 0.948
total, interval score at 80: 9268.36
total, finite, coverage at 80: 0.990
total, finite, interval score at 80: 10724.99

Scaling wins on the two measures the variants were judged by, and loses only on the pooled coverage at 80. The worst-covered client's coverage at 80 rises from 0.021 to 0.365, and the interval score at 80 falls from 603.88 to 336.72. It falls at 95 as well, as the output shows.

We judged the variants by these two measures because the pooled figure hides clients. We chose them after seeing the decision numbers, so the pooled figure is reported beside them. At 80 the pooled coverage is closer to its label without the scale, at 0.809 against 0.834. The scale stays.

The finite-sample levels stay out of the bands. Each new residual is one more among n + 1, so the finite-sample levels move outward a little: each quantile level moves to floor((n + 1) * a) / n for the lower bound and ceil((n + 1) * (1 - a)) / n for the upper, where n is the number of residuals and a the tail share. Their effect on the scaled coverage at 80 is small, from 0.834 to 0.835, and the interval score at 80 gets worse, from 336.72 to 337.31. It gets worse at 95 too, as the output shows. The correction widens the bands and does not help, so the bands do not use it.

The held-out evaluation applies the same design to the standard cutoffs, which are the days the rest of the chapter scores.

conformal-earlier-windows.py Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17 通過
"""Conformal bands for every client, calibrated on twelve holiday-free days before the scored week, with each client'sresiduals scaled first. Read the coverage and interval score at 80 and 95 first; the unscaled, three-day andfinite-sample lines are printed for comparison."""import pandas as pdfrom lib.baselines import seasonal_scalefrom lib.electricity import loadfrom lib.intervals import CAL_CUTOFFS, bounds, calibration_facts, coverage, interval_score, pairedfrom lib.panel import ALL30from lib.quirks import mask_clock_changesfrom lib.scoring import book_scalefrom lib.settings import EVAL_CUTOFFS, showraw = load(ALL30)df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]scored_cutoffs = pd.to_datetime(EVAL_CUTOFFS)cal, sc = paired(df, CAL_CUTOFFS), paired(df, scored_cutoffs)# The scale is read from the history before the first calibration cutoff, as the book's MASE scale is.scale = seasonal_scale(book_scale(raw, CAL_CUTOFFS[0]))# The largest client is the one with the highest mean reading over the scored windows.big = sc.groupby("unique_id")["y"].mean().idxmax()for label, value in calibration_facts(df, scored_cutoffs).items():    show(label, value)show("calibration residuals per hour of the day", len(cal) // 24)show("largest client", big)show("mean hourly load of the largest client", sc.groupby("unique_id")["y"].mean().max())for level in (80, 95):    lo, hi = bounds(sc, cal, level, scale=scale)    show(f"coverage at {level}", coverage(sc["y"], lo, hi), 3)    show(f"mean width at {level}", (hi - lo).mean())    show(f"interval score at {level}", interval_score(sc["y"], lo, hi, level))lo, hi = bounds(sc, cal, 80, scale=scale)per_client = ((sc["y"] >= lo) & (sc["y"] <= hi)).groupby(sc["unique_id"]).mean()show("lowest client coverage at 80", per_client.min(), 3)# No scale here: the pooled residuals stay in kWh, which is what lets the large clients set the bands.lo, hi = bounds(sc, cal, 80)unscaled = ((sc["y"] >= lo) & (sc["y"] <= hi)).groupby(sc["unique_id"]).mean()show("largest client coverage at 80, unscaled residuals", unscaled[big], 3)# The three calibration days nearest the scored week: Dec 18, 19 and 20, 2014.recent = cal[cal["cutoff"] >= "2014-12-18"]lo, hi = bounds(sc, recent, 80, scale=scale)show("coverage at 80 from three calibration days", coverage(sc["y"], lo, hi), 3)lo, hi = bounds(sc, cal, 80, scale=scale, finite=True)show("coverage at 80 with finite-sample levels", coverage(sc["y"], lo, hi), 3)
輸出python conformal-earlier-windows.py
calibration windows: 12
holiday hours in calibration: 0
calibration hours shared with scored: 0
calibration residuals per hour of the day: 360
largest client: MT_370
mean hourly load of the largest client: 17499.81
coverage at 80: 0.739
mean width at 80: 267.62
interval score at 80: 673.89
coverage at 95: 0.902
mean width at 95: 532.00
interval score at 95: 1326.80
lowest client coverage at 80: 0.472
largest client coverage at 80, unscaled residuals: 0.000
coverage at 80 from three calibration days: 0.711
coverage at 80 with finite-sample levels: 0.741

The scaled bands still fall short of both labels on the standard cutoffs: they cover 0.739 of client-hours at 80 and 0.902 at 95. Their mean widths are 267.62 kWh and 532.00 kWh.

The scores are 673.89 at 80 and 1326.80 at 95. They are in kWh, so the large clients dominate them. Lower is better, and the score rewards a band that is both narrow and right.

The pooled bands still have to be read per client. The lowest client coverage at 80 is 0.472, for the client the scaled bands fit least well. With unscaled residuals, the largest client is covered in 0.000 of its hours at 80. The pooled kWh bands were set by the large clients, and they miss the largest client entirely.

Two more checks on the standard cutoffs test the design. Calibrating on only the three days nearest the scored week, Dec 18 to 20, gives 0.711 at 80, against 0.739 with all twelve. Three days are a smaller sample, and the coverage is a little lower. The finite-sample levels give 0.741 at 80, a small change that agrees with the decision windows: the correction does little for the client bands.

Working rule: Calibrate on windows that come before the scored ones, with no holiday in them, and scale each client before pooling its residuals. Make each design choice on windows that are not the ones you report. Report how many windows and residuals went into the bands and the coverage they gave on the scored days, even when that coverage falls short of the level.

此部分尚未翻譯,因此以英文呈現。

Measure coverage on held-out days

最近一次變動於第 2026.11.6 版

Coverage is measured on days that the calibration never used. The standard cutoffs give three days, and each day gives 24 hours for each client, so the 2160 client-hours are not that many independent observations. The thirty clients share the calendar and the weather, so the hours of one day agree with one another. For day-level errors the sample is three days, which cannot measure a coverage level. Three days can show that a band fell short on those days, but they cannot say by how much a band falls short over a longer period.

The supplier buys the total, so the total's coverage matters most. Its bands are built from its own residuals over the same twelve days, with the quantiles read directly, and are offered at 80 only, because twelve residuals cannot support 95. With signed bounds, the band from the smallest residual to the largest covers at most 0.846 of the time, if the residuals are exchangeable, and a 95 band, or a 90 band, asks for more.

The total's bands cover 0.667 of its 72 scored hours at 80. For exchangeable errors, the same construction covers 0.682, so most of the gap to 80 comes from the twelve-residual sample. The section on cost puts these bands to use.

The total's errors vary by day. The mean absolute error of the total was 2415.51 kWh on the calibration windows and 2985.88 kWh on the scored days. Dec 31 accounts for most of the gap between the scored days and the calibration windows.

Scored day Error of the total, kWh Coverage at 80
Dec 29, 2014 2448.53 0.625
Dec 30, 2014 2149.78 0.708
Dec 31, 2014 4359.32 0.667

The coverage does not follow the errors. Dec 29 has the lowest coverage with a middle-sized error, and Dec 31 has the largest error with a middle-sized coverage. Each day has 24 hours, and three days are too few to tell the two apart.

The finite-sample levels would change the total's figure. On the scored days they give 0.736 at 80. The choice was made on the decision windows, where the correction widened the total's bands and worsened their interval score, so the bands do not use it. The decision windows also show that the direction of the gap is not fixed. There the total's 80 bands covered 0.948 of hours, more than their label, and the finite levels pushed that to 0.990.

Angelopoulos and Bates (2022) show that split conformal prediction covers the test point with probability at least 1 - alpha when the calibration and test points are independent and identically distributed, or more generally exchangeable. They also note that adjacent points in a time series are not exchangeable, using a temperature series as the example. The same holds for hourly load: each hour carries the one before it, and a day's hours share the weather and the calendar. Barber and co-authors (2023) call the loss in coverage when exchangeability fails a coverage gap. Their bound does not assume a particular joint distribution. It is small when recent calibration points are weighted in a way that matches the test point's distribution, as in slow drift. That result is theoretical, and this chapter does not test it. A 2026 preprint, a survey of conformal methods for dependent series, states that temporal dependence violates the exchangeability assumption. It is not peer reviewed, and we cite it only for that point.

So the book promises nothing about coverage that it does not measure. The coverage in this chapter is what three days showed, under the calibration described in Calibrate on earlier windows.

Working rule: Report coverage on windows that the calibration never used, with the number of windows and the coverage of each. Keep the same measure in a running log once the forecast runs every day, as Chapter 10 does. Read three scored days as a warning, not as a measurement of the level you were promised.

此部分尚未翻譯,因此以英文呈現。

Choose the level from the cost of error

最近一次變動於第 2026.11.11 版

The supplier buys one amount for each hour, and the costs of the two errors set that amount. Let cu be the cost of one kWh bought short and co the cost of one kWh bought and not used. Let F be the distribution of demand. Differentiating the expected cost with respect to the purchase q gives co F(q) - cu (1 - F(q)), which is zero where F(q) = cu / (cu + co), the smallest expected cost. The level cu / (cu + co) is the critical fractile of the newsvendor model. The HKUST IEEM 101 notes on the newsvendor model give the same proof, writing the cost of excess as c0. This chapter writes it co. The purchase is therefore the quantile at level tau = cu / (cu + co).

With the illustrative costs from Why one number is not enough, tau is 0.75. The rule says to buy the quantile of the demand at that level, for each hour. The quantile at level tau is the purchase that minimises the quantile loss at that level, as Gneiting and Raftery (2007) show. The quantile loss is tau * (y - q) when the reading y is at or above the purchase q, and (1 - tau) * (q - y) when it is below. The quantile loss is not a cost in kWh or in money. It is the rule that the quantile minimises, and it has the same shape as the supplier's cost. Multiplying it by cu + co gives the cost exactly, which the example prints beside the mean cost.

Does buying at the critical fractile cost less than buying at the median or higher, on days the bands never saw? Below, the purchase is set for each hour from these costs, and the scored days answer the question. The level 0.95 is not offered, as Measure coverage on held-out days explains.

buy-ahead-level-by-cost.py Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17 通過
"""The buy level for the total, set from the two costs: the critical fractile, cu / (cu + co), is the quantile that minimisesthe expected cost. Compare the mean cost and the hours short at each level; the costs are illustrative, not prices."""import pandas as pdfrom lib.electricity import loadfrom lib.intervals import CAL_CUTOFFS, COST_EXCESS, COST_SHORT, buy_report, critical_fractile, pairedfrom lib.panel import ALL30from lib.quirks import mask_clock_changesfrom lib.settings import EVAL_CUTOFFS, showraw = load(ALL30)# Every clock-change hour (four changes, 2013 and 2014) is masked as NaN, as in every example of this chapter.df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]# Calibration on Dec 9 to 20, 2014; scored on the three standard cutoffs.pairs_cal, pairs_sc = paired(df, CAL_CUTOFFS), paired(df, pd.to_datetime(EVAL_CUTOFFS))show("cost of a kWh short", COST_SHORT)show("cost of a kWh unused", COST_EXCESS)# The quantile level cu / (cu + co), from the two costs printed above.show("critical fractile", critical_fractile())# buy_report returns the printed rows as (label, value, decimal places).for label, value, digits in buy_report(pairs_cal, pairs_sc):    show(label, value, digits)
輸出python buy-ahead-level-by-cost.py
cost of a kWh short: 3.00
cost of a kWh unused: 1.00
critical fractile: 0.75
scored hours: 72
residuals per hour for the total: 12
highest coverage the total's residuals allow: 0.846
simulated coverage at 80, exchangeable errors: 0.682
mean absolute error of the total, calibration days: 2415.51
mean absolute error of the total, scored days: 2985.88
total coverage at 80: 0.667
total coverage at 80 with finite-sample levels, not used: 0.736
total coverage at 80 on 2014-12-29: 0.625
total coverage at 80 on 2014-12-30: 0.708
total coverage at 80 on 2014-12-31: 0.667
total mean absolute error on 2014-12-29: 2448.53
total mean absolute error on 2014-12-30: 2149.78
total mean absolute error on 2014-12-31: 4359.32
mean cost at 0.50: 5087.95
hours short at 0.50: 0.528
mean cost at 0.75: 3843.61
hours short at 0.75: 0.403
mean cost at 0.90: 3667.43
hours short at 0.90: 0.222
quantile loss at critical fractile, times both costs: 3843.61
mean hourly total bound from the total: 31983.95
mean hourly total bound from the sum of client bounds: 32388.77
hours short at critical fractile, sum of client bounds: 0.306
mean cost at critical fractile, sum of client bounds: 3729.17

The identity holds: multiplied by the sum of the two costs, the quantile loss at the critical fractile is 3843.61, the same as the mean cost at that level, 3843.61. The rule is correct as arithmetic.

Level Mean cost Hours short
0.50 5087.95 0.528
0.75, the critical fractile 3843.61 0.403
0.90 3667.43 0.222

The result on the scored days is less comfortable. The rule chose the critical fractile, and on these three days 0.90 cost less, at 3667.43 against 3843.61. At the critical fractile the purchase was short in 0.403 of the hours, where a calibrated level would fall short in one hour in four. That is the coverage shortfall from Measure coverage on held-out days seen as money. We do not move the level to 0.90. The three scored days are the test, and a level chosen because it wins on them has been chosen on the test. The honest reading is that the bands were too narrow here, so the purchase was too low, and that the level is right only when the bands are.

Should the purchase follow the clients' bounds instead? Summing each client's bound at the critical fractile gives a bound of 32388.77 kWh an hour, against 31983.95 from the total's own residuals. Summed quantiles equal the total's quantile when the errors move together perfectly. For an upper quantile, with errors that partly cancel, the sum usually sits above it, as it does here.

On the scored days the summed bound did better, with hours short at 0.306 against 0.403, and a mean cost of 3729.17 against 3843.61. That advantage comes from its higher level, not from a better calibration, and the scored days cannot show that it holds in general. We keep the total's own bound, because the purchase is made against the total's residuals, and we do not switch to the bound that did better here.

Working rule: Set the buy level from the two costs, as tau = cu / (cu + co), fix it before you score the days, and calibrate the total's bands on the total's own residuals. Show the realised cost at other levels as a check, but never move the level to whichever one would have won on the days you score.

正式環境中會出什麼問題

最近一次變動於第 2026.11.5 版
  • 校準所用的季節不是被評分的季節。 假日不納入校準,這讓測試保持誠實,但總量的誤差會隨視窗而異,被評分的那幾天,誤差可能比校準視窗中的更大。在平穩時期校準的區間,遇到較難的時期就會失準。要察覺這一點,可以每天將覆蓋率與名目水準比較,並觀察移動平均。覆蓋率連續一週低於其標示的水準,就是徵兆,而那一週的誤差是最先該檢視的地方。
  • 合併的區間會掩蓋個別客戶。 以 kWh 合併的區間,是由最大的客戶決定的。在未縮放殘差的情況下,最大客戶的區間在 80 水準涵蓋其 0.000 的時數。在縮放殘差的情況下,80 水準時最低的客戶覆蓋率是 0.472,所以即使是縮放後的合併,仍會掩蓋未達標的客戶。要逐一客戶讀取覆蓋率,不能只看合併後的數字。
  • 三天能說明的很少。 三個被評分日的覆蓋率數字看起來像是一項發現,但它只是日層級誤差的三次觀測。應使用多個視窗的覆蓋率,並把短期的結果視為值得檢視的理由,而不是結論。
  • 相依性破壞了簡單的保證。 每小時負載具有自相關,相鄰小時的校準殘差並不獨立。這些區間不具有教科書定理所給的保證。天氣或日曆改變時,其覆蓋率可能漂移,只有實測的覆蓋率能告訴你何時發生。誤差正負號相同的連續數日,是需要留意的模式,而它會顯現在持續記錄的日誌中。
  • 總量每小時的殘差很少。 每個校準視窗在每個小時只給出總量的一個殘差,所以十二個視窗只有十二個。使用帶正負號的殘差時,十二個殘差所能允許的最高覆蓋率是 0.846,因此總量的區間止於 80,而且在提供任何區間之前,應先進行水準檢查。更多的校準視窗可以讓總量有較大的樣本,但每個視窗都必須是被評分那一週之前的普通日子。
  • 滾動式校準可能用到你要評分的日子。 Build intervals from residuals 中內建的規則,以每個截止點之前的三個截止點來校準該截止點。在實際運作中這沒有問題,因為那些讀數是已知的。在測試中,這會使被評分的視窗彼此相依。應讓校準集固定,並且早於每一個被評分的視窗。

本章所做的決定

最近一次變動於第 2026.11.5 版
  • 校準視窗。 取保留週之前的十二個一般日,即 2014 年 12 月 9 日至 20 日。假日不納入,校準時段也不計分。決策視窗同樣使用十二天,即 2014 年 11 月 9 日至 20 日。這個長度是選擇,不是結果:資料並未顯示十二天是最佳長度。
  • 彙整前先縮放。 每個客戶的殘差,會除以該客戶在第一個校準截止點之前歷史資料上的季節性樸素誤差,這與本書計算 MASE 所用的尺度相同。這是由決策視窗決定的,而總量不需要縮放。
  • 區間帶不做有限樣本修正。 這項修正已在決策視窗上測試過,對客戶與總量皆然,它並未改善區間分數。它僅用來判讀一組殘差能支撐哪些水準。
  • 帶正負號的殘差。 區間帶與購買分位數使用帶正負號的殘差,因此下界與上界不必對稱。mlforecast 的保形方法使用絕對殘差,得到對稱的區間帶。Build intervals from residuals 中的內建執行結果,是用來做比較的。
  • 購買針對總量。 購買水準及其分位數,是依據總量的殘差設定。客戶的區間帶用於解讀預測,不用於購買。
  • 成本屬於假設。 3.00 與 1.00 這兩項成本,已在 Why one number is not enough 中說明,屬於判斷,並非實測。
  • 統計函式庫的區間類別。 statsforecast 的類別是 ConformalIntervals,名稱與 mlforecast 的 PredictionIntervals 不同,本章對各個函式庫分別使用其各自的名稱。
  • 第三個範例。 購買決策需要自己的量測,因此新增了 buy-ahead-level-by-cost,並一併新增 conformal-decision-windows,後者在保留評估不使用的視窗上做出設計選擇。
  • 不宣稱涵蓋假日。 2014 年 12 月 25 日位於保留週內,但它不在校準視窗,也不在計分視窗之內。本書未報告區間帶對假日本身的涵蓋情形。
  • 先有區間帶,再有水準。 購買水準的好壞取決於其下的區間帶,而區間帶的好壞取決於設定它的視窗。

本章,於本版次中

版次 2026.11.11範例於 2026年11 月 20 通過測試本章 4 個範例 個範例中有 4 個通過

譯自英文書,版次 2026.11.11。

引用本節

格式

各種格式皆引用英文原書,本譯本即依循該原書。

每種格式都會標示本版。其連結會開啟目前版次,節標題會說明該節內文最後一次變動的時間。存取日期與擷取日期即為你開啟此頁的當日。

本頁內容

第 8 章,共 10 章: 量化預測的不確定性

  1. Why one number is not enough
  2. 以殘差建立區間
  3. Calibrate on earlier windows
  4. Measure coverage on held-out days
  5. Choose the level from the cost of error
  6. 正式環境中會出什麼問題
  7. 本章所做的決定

返回頂端所有章節