予測の不確実性を定量化する

読了まで約57 分分、7 節 節の 4 例 例。

第2026.11.11版 2026年11月20日時点で例が通過 41 例中 41 例が合格 変更点

AIによる翻訳。英語版は人による検証済みである。 当社の書籍の作り方

英語版の書籍(版2026.11.11)から翻訳した。 原文を読む

本章では、前日予測のそれぞれに区間を設け、その予測を購入の判断に用いてよいかどうかを述べられるようにする。評価対象の日より前の日々を用いて予測区間を較正し、その区間がどの程度の頻度で成り立つかを測定し、誤りに伴う二つのコストから、購入に用いる水準を選ぶ。

なぜ1つの数値では足りないのか

最終変更:版2026.11.11

ここまでの予測は、顧客と時間帯ごとに1つの数値を与える。しかし、1つの数値だけでは購入を決められない。供給事業者は期待値を購入することはできない。事業者が購入するのは、翌日分の各時間帯の電力量であり、したがって各予測の周りに幅が必要になる。購入した1 kWhは、使われるか、使われないかのどちらかである。不足して購入すれば、残りを後から割増価格で買うことになり、過剰に購入すれば、誰も使わない電力に対価を支払うことになる。この2つの誤りは費用が異なるため、適切な購入量は予測値そのものではないことが多い。1つの時間帯で考える。予測が100 kWhであるとする。実測値が10 kWh高く出れば、事業者は不足となり、その10 kWhに対して不足の費用を支払う。実測値が10 kWh低く出れば、事業者は対価を支払ったのに使わない電力を10 kWh抱えることになる。不足の費用が未使用の費用の3倍であるとき、予測を上回る購入量のほうが平均的には費用が小さくなる。高くつくほうの誤りが起こりにくくなるからである。購入量を予測よりどれだけ上に置くべきかは、この比と、誤差の幅の広さによって決まる。

この章では、2つの費用を仮定として置く。不足して購入した1 kWhの費用は3.00費用単位であり、購入したが使われなかった1 kWhの費用は1.00である。これらの数値は例示であり、価格ではない。また、現実の供給事業者がこの値に直面していると主張するものでもない。後の節で、これらの数値が購入量をどう決めるかを示す。

予測区間は、各時間帯に下限と上限を与える。予測区間の水準は80や95などの値で、区間が含むべき実測値の割合を表す。区間が実際に含む実測値の割合がカバレッジであり、カバレッジは1回の予測ではなく、同じ方法で行われた多数の予測に属する。ある1日の区間が実測値を含むか含まないかは、そのどちらかでしかない。3日間の72時間帯にわたって区間の内側に入った割合は、1つの測定値である。1年分の日数にわたれば、それは実績となる。95パーセントの区間であっても、区間内に入る時間帯が90パーセントにとどまっていれば、カバレッジは不足しているが、1日だけを見てもそれはわからない。それを示すのは多数の時間帯にわたる平均だけであり、この平均こそが測定の対象である。

分位点が、区間と費用を結びつける。中央80パーセント区間の下限は、予測分布の0.1分位点であり、実測値が10回に1回の割合で下回る値である。上限は0.9分位点である。期待費用を最小にする購入量もまた分位点であり、それがどの分位点かは誤りの費用から水準を選ぶで導出する。

このケースで重要になるのは、合計の幅である。供給事業者が購入するのは合計だからである。第6章では、30の顧客を1つのグローバルモデルで予測したが、合計はそれらの予測の時間ごとの和である。各顧客にはそれぞれの幅があるが、合計の幅を、顧客ごとの幅を足し合わせて求めることはできない。誤りの費用から水準を選ぶで、両者を比較する。

実践ルール: 2つの誤りの費用が異なる場面では幅を求め、その水準は、安心できそうに見える数値ではなく、それらの費用から選ぶ。カバレッジは、多数のウィンドウにわたる区間内の時間帯の割合として報告し、ある1日が区間に入る確率として報告してはならない。

この部分はまだ翻訳されていないため、英語で表示している。

Build intervals from residuals

最終変更:版2026.11.6

A range needs bounds, and the simplest bounds come from past errors. A residual is the error of an earlier forecast: the reading minus the forecast, e = y - f. Conformal prediction builds a band from the residuals of forecasts made the same way on earlier days. Take the residuals of a set of calibration windows, group them by horizon step, and read off the quantiles you need. A day-ahead forecast made at 00:00 has one horizon step for each hour of the day, so the group is the hour alone. For a level L in percent, the bounds for a forecast f made at a given step are:

lower = f + Q((1 - L/100) / 2)
upper = f + Q((1 + L/100) / 2)

where Q(p) is the empirical p-quantile of the calibration residuals at that step. The method assumes no model of the errors. It only assumes that the calibration residuals resemble the residuals to come, which a later section tests. A calibration window is a cutoff whose residuals size the bands. It always comes before the cutoffs being forecast, and it is never one of the windows that are scored.

The library that does this for the book's model is mlforecast, at version 1.1.0. Its class PredictionIntervals takes a number of windows, a horizon and a method, and it has four conformal methods: conformal_error, conformal_distribution, and their weighted versions. The two unweighted methods score the absolute residuals, |y - f|, so their bounds are symmetric around the forecast (mlforecast 1.1.0, mlforecast/conformal_prediction.py, function compute_conformity_scores). The default, conformal_distribution, works one client and one horizon step at a time. For each client and step it takes the calibration windows' absolute residuals, makes the paths of the forecast plus and minus each one, and reads the quantiles of those paths. Nothing is pooled across clients. The statistical library, statsforecast at version 2.1.1, names its interval class ConformalIntervals, not PredictionIntervals.

To see what the library does with a short calibration history, run its class with three calibration windows. It scores the three standard cutoffs, Dec 29 to Dec 31, 2014, each at 00:00. Those are the held-out week's last three days, and the days this chapter scores. mlforecast refits the global model at each cutoff and calibrates each one on the three before it.

conformal-three-windows.py Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17 合格
"""mlforecast's own conformal intervals, with three calibration windows, refitted at each of the three standard cutoffs.Compare the coverage at 80 and 95 with their labels, and with the highest coverage three windows allow."""import pandas as pdfrom lib.electricity import loadfrom lib.intervals import builtin_intervals, design_coverage, symmetric_ceilingfrom lib.panel import ALL30from lib.quirks import mask_clock_changesfrom lib.settings import EVAL_CUTOFFS, showraw = load(ALL30)# Every clock-change hour (four changes, 2013 and 2014) is masked as NaN. builtin_intervals keeps only rows from Oct 27, 2014 on, so no masked hour is in a window.df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]# One row per client and hour. The bounds are columns named by model and level, such as LGBMRegressor-lo-80.cv = builtin_intervals(df, pd.to_datetime(EVAL_CUTOFFS))show("scored hours", len(cv))show("calibration windows per cutoff", 3)show("residuals per client and hour", 3)# Ceiling of a band of plus and minus n absolute residuals, n / (n + 1), with n = 3: at most 3 of the 4 places a new one can take.show("highest coverage three windows allow, any level", symmetric_ceiling(3), 3)for level in (80, 95):    # Simulated under exchangeable normal errors, with mlforecast's construction, to set against the measured coverage.    show(f"simulated coverage at {level}, exchangeable errors", design_coverage(level, 3, "builtin"), 3)for level in (80, 95):    lo, hi = cv[f"LGBMRegressor-lo-{level}"], cv[f"LGBMRegressor-hi-{level}"]    inside = (cv["y"] >= lo) & (cv["y"] <= hi)    show(f"coverage at {level}", inside.mean(), 3)    show(f"mean width at {level}", (hi - lo).mean())    if level == 80:        # The by-day lines are printed for the 80 level only. Each hour is grouped by its calendar day.        day = cv["ds"].dt.floor("D")        for d, share in inside.groupby(day).mean().items():            show(f"coverage at 80 on {d:%Y-%m-%d}", share, 3)        for d, err in (cv["y"] - cv["LGBMRegressor"]).abs().groupby(day).mean().items():            show(f"mean absolute error on {d:%Y-%m-%d}", err)
出力python conformal-three-windows.py
scored hours: 2160
calibration windows per cutoff: 3
residuals per client and hour: 3
highest coverage three windows allow, any level: 0.750
simulated coverage at 80, exchangeable errors: 0.656
simulated coverage at 95, exchangeable errors: 0.732
coverage at 80: 0.690
mean width at 80: 281.07
coverage at 80 on 2014-12-29: 0.756
coverage at 80 on 2014-12-30: 0.676
coverage at 80 on 2014-12-31: 0.637
mean absolute error on 2014-12-29: 114.62
mean absolute error on 2014-12-30: 94.42
mean absolute error on 2014-12-31: 163.37
coverage at 95: 0.750
mean width at 95: 355.39

The output shows the failure before any fix: across the 2160 client-hours, the bands at the 80 level cover 0.690 and at the 95 level 0.750. Both fall short of their labels, and the bands at 80 are 281.07 kWh wide on average.

The design sets a ceiling, and the 95 result sits on it. With three windows, each client and hour has three absolute residuals, and the band reaches no further than the largest of them. For exchangeable errors, the test error is the largest of four absolute errors one time in four, so the band misses it that often. The highest coverage three windows allow is 0.750, at any level. Under the same exchangeable assumption, the band's expected coverage is 0.656 at 80 and 0.732 at 95. The simulation draws normal errors, with the same construction. The measured figures are a little above those expected values, so the calibration windows did not have smaller errors than the scored days. Calm calibration windows would push coverage below the expected value, not above it. The shortfall is the construction's.

The coverage by day shows where the shortfall sits. At the 80 level it is 0.756 on Dec 29, 0.676 on Dec 30 and 0.637 on Dec 31. The forecast errors on those days were 114.62, 94.42 and 163.37 kWh on average. Dec 31 has the largest errors and the lowest coverage, and three days cannot separate day-to-day variation from the design.

A second problem comes from the calibration. Each cutoff calibrates only on days before it: the bands for Dec 31 use Dec 28, 29 and 30. But Dec 29 and Dec 30 are scored days too. So the three scored windows are not independent, because one window's calibration contains the readings of another scored window. No band uses its own day's readings, and the rule is right for operation, where those readings are known. For a test it weakens the result. Fixing it would need a separate calibration fit for each cutoff, which is more than a one-argument change, so the chapter's own bands use a fixed calibration set instead, as Calibrate on earlier windows shows.

Working rule: Build each band from the residuals of windows that come before its cutoff, and check the coverage it reaches on windows it did not use before you trust it. Read the level a design can reach before you read the coverage. Three residuals per client and hour cannot reach either label, so the fix is a larger calibration set, not a different reading of the output.

この部分はまだ翻訳されていないため、英語で表示している。

Calibrate on earlier windows

最終変更:版2026.11.11

The calibration windows must come before the windows you score, and they must not share a target hour with them. The book adds one more rule: a calibration window must not fall on a Portuguese public holiday, because a holiday's load follows the calendar rather than the week before it. The rule applies to the decision windows as well. The held-out evaluation calibrates on twelve ordinary days, Dec 9 to 20, 2014, which end four days before the held-out week. Both examples in this section print the holiday hours and the shared hours in their calibration, and both counts are zero.

Twelve windows with thirty clients give 360 residuals for each hour of the day. They come from twelve days, though, and the thirty clients share each day's weather and calendar. So a day-level error is still a sample of twelve. The pooled residuals have a second problem. The clients differ enormously in size. In the scored week the largest client, MT_370, averages 17499.81 kWh an hour over the three scored days. Pooled residuals in kWh are dominated by such clients, so the bands fit the large clients and misfit the small ones. The fix is to divide each client's residuals by that client's scale before pooling, and multiply the bound back by the scale of the client being forecast. The scale is the seasonal naive error on the history before the first calibration cutoff, the same kind of scale the book uses for MASE. With the scale, the thirty clients count alike.

Two choices in this design were made on the decision windows, not on the held-out evaluation. The decision windows are the four cutoffs from Nov 25 to Dec 3, 2014. Their calibration is Nov 9 to 20, 2014, twelve holiday-free days that end four days before the first decision cutoff, the same shape as the held-out design. The windows are scored on their own, and the held-out evaluation is not read. The choices are judged by the interval score and the lowest client's coverage. The interval score adds a band's width to a penalty for each reading that misses the band, the penalty growing with the distance of the miss and scaled by 2 over the miss rate (Gneiting and Raftery (2007), eq. 43). Those windows settle both choices, as the measurements below show.

conformal-decision-windows.py Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17 合格
"""The two calibration choices, scaling each client's residuals and the finite-sample levels, decided on the four decisionwindows (Nov 25 to Dec 3, 2014), with calibration on Nov 9 to 20. Compare the lowest client's coverage and the interval scores printed for each variant;the standard cutoffs are not read."""from lib.baselines import seasonal_scalefrom lib.electricity import loadfrom lib.intervals import DECISION_CAL, bounds, calibration_facts, coverage, interval_score, paired, total_offrom lib.panel import ALL30, DECISION_CUTOFFSfrom lib.quirks import mask_clock_changesfrom lib.scoring import book_scalefrom lib.settings import showraw = load(ALL30)# Every clock-change hour (four changes, 2013 and 2014) is masked as NaN, as in every example of this chapter.df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]# One row per client and hour, with the forecast f, the reading y and the residual e.cal, sc = paired(df, DECISION_CAL), paired(df, DECISION_CUTOFFS)# Each client's MASE-style scale, from the history before the first calibration cutoff.scale = seasonal_scale(book_scale(raw, DECISION_CAL[0]))for label, value in calibration_facts(df, DECISION_CUTOFFS, DECISION_CAL).items():    show(label, value)show("residuals per hour of the day", len(cal) // 24)variants = (("unscaled", None, False), ("scaled", scale, False),            ("unscaled, finite", None, True), ("scaled, finite", scale, True))for name, s, fin in variants:    for level in (80, 95):        # scale=None pools the raw kWh residuals; finite=True applies the (n + 1) quantile correction.        lo, hi = bounds(sc, cal, level, scale=s, finite=fin)        show(f"{name}, coverage at {level}", coverage(sc["y"], lo, hi), 3)        show(f"{name}, interval score at {level}", interval_score(sc["y"], lo, hi, level))    lo, hi = bounds(sc, cal, 80, scale=s, finite=fin)    # Coverage per client, then the lowest of the thirty clients.    show(f"{name}, lowest client coverage at 80", ((sc["y"] >= lo) & (sc["y"] <= hi)).groupby(sc["unique_id"]).mean().min(), 3)# One series, so no scale: the scale exists only to pool clients of different size.tcal, tsc = total_of(cal), total_of(sc)for name, fin in (("total", False), ("total, finite", True)):    lo, hi = bounds(tsc, tcal, 80, finite=fin)    show(f"{name}, coverage at 80", coverage(tsc["y"], lo, hi), 3)    show(f"{name}, interval score at 80", interval_score(tsc["y"], lo, hi, 80))
出力python conformal-decision-windows.py
calibration windows: 12
holiday hours in calibration: 0
calibration hours shared with scored: 0
residuals per hour of the day: 360
unscaled, coverage at 80: 0.809
unscaled, interval score at 80: 603.88
unscaled, coverage at 95: 0.957
unscaled, interval score at 95: 1950.32
unscaled, lowest client coverage at 80: 0.021
scaled, coverage at 80: 0.834
scaled, interval score at 80: 336.72
scaled, coverage at 95: 0.954
scaled, interval score at 95: 581.28
scaled, lowest client coverage at 80: 0.365
unscaled, finite, coverage at 80: 0.811
unscaled, finite, interval score at 80: 604.19
unscaled, finite, coverage at 95: 0.959
unscaled, finite, interval score at 95: 1961.48
unscaled, finite, lowest client coverage at 80: 0.021
scaled, finite, coverage at 80: 0.835
scaled, finite, interval score at 80: 337.31
scaled, finite, coverage at 95: 0.956
scaled, finite, interval score at 95: 592.35
scaled, finite, lowest client coverage at 80: 0.375
total, coverage at 80: 0.948
total, interval score at 80: 9268.36
total, finite, coverage at 80: 0.990
total, finite, interval score at 80: 10724.99

Scaling wins on the two measures the variants were judged by, and loses only on the pooled coverage at 80. The worst-covered client's coverage at 80 rises from 0.021 to 0.365, and the interval score at 80 falls from 603.88 to 336.72. It falls at 95 as well, as the output shows.

We judged the variants by these two measures because the pooled figure hides clients. We chose them after seeing the decision numbers, so the pooled figure is reported beside them. At 80 the pooled coverage is closer to its label without the scale, at 0.809 against 0.834. The scale stays.

The finite-sample levels stay out of the bands. Each new residual is one more among n + 1, so the finite-sample levels move outward a little: each quantile level moves to floor((n + 1) * a) / n for the lower bound and ceil((n + 1) * (1 - a)) / n for the upper, where n is the number of residuals and a the tail share. Their effect on the scaled coverage at 80 is small, from 0.834 to 0.835, and the interval score at 80 gets worse, from 336.72 to 337.31. It gets worse at 95 too, as the output shows. The correction widens the bands and does not help, so the bands do not use it.

The held-out evaluation applies the same design to the standard cutoffs, which are the days the rest of the chapter scores.

conformal-earlier-windows.py Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17 合格
"""Conformal bands for every client, calibrated on twelve holiday-free days before the scored week, with each client'sresiduals scaled first. Read the coverage and interval score at 80 and 95 first; the unscaled, three-day andfinite-sample lines are printed for comparison."""import pandas as pdfrom lib.baselines import seasonal_scalefrom lib.electricity import loadfrom lib.intervals import CAL_CUTOFFS, bounds, calibration_facts, coverage, interval_score, pairedfrom lib.panel import ALL30from lib.quirks import mask_clock_changesfrom lib.scoring import book_scalefrom lib.settings import EVAL_CUTOFFS, showraw = load(ALL30)df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]scored_cutoffs = pd.to_datetime(EVAL_CUTOFFS)cal, sc = paired(df, CAL_CUTOFFS), paired(df, scored_cutoffs)# The scale is read from the history before the first calibration cutoff, as the book's MASE scale is.scale = seasonal_scale(book_scale(raw, CAL_CUTOFFS[0]))# The largest client is the one with the highest mean reading over the scored windows.big = sc.groupby("unique_id")["y"].mean().idxmax()for label, value in calibration_facts(df, scored_cutoffs).items():    show(label, value)show("calibration residuals per hour of the day", len(cal) // 24)show("largest client", big)show("mean hourly load of the largest client", sc.groupby("unique_id")["y"].mean().max())for level in (80, 95):    lo, hi = bounds(sc, cal, level, scale=scale)    show(f"coverage at {level}", coverage(sc["y"], lo, hi), 3)    show(f"mean width at {level}", (hi - lo).mean())    show(f"interval score at {level}", interval_score(sc["y"], lo, hi, level))lo, hi = bounds(sc, cal, 80, scale=scale)per_client = ((sc["y"] >= lo) & (sc["y"] <= hi)).groupby(sc["unique_id"]).mean()show("lowest client coverage at 80", per_client.min(), 3)# No scale here: the pooled residuals stay in kWh, which is what lets the large clients set the bands.lo, hi = bounds(sc, cal, 80)unscaled = ((sc["y"] >= lo) & (sc["y"] <= hi)).groupby(sc["unique_id"]).mean()show("largest client coverage at 80, unscaled residuals", unscaled[big], 3)# The three calibration days nearest the scored week: Dec 18, 19 and 20, 2014.recent = cal[cal["cutoff"] >= "2014-12-18"]lo, hi = bounds(sc, recent, 80, scale=scale)show("coverage at 80 from three calibration days", coverage(sc["y"], lo, hi), 3)lo, hi = bounds(sc, cal, 80, scale=scale, finite=True)show("coverage at 80 with finite-sample levels", coverage(sc["y"], lo, hi), 3)
出力python conformal-earlier-windows.py
calibration windows: 12
holiday hours in calibration: 0
calibration hours shared with scored: 0
calibration residuals per hour of the day: 360
largest client: MT_370
mean hourly load of the largest client: 17499.81
coverage at 80: 0.739
mean width at 80: 267.62
interval score at 80: 673.89
coverage at 95: 0.902
mean width at 95: 532.00
interval score at 95: 1326.80
lowest client coverage at 80: 0.472
largest client coverage at 80, unscaled residuals: 0.000
coverage at 80 from three calibration days: 0.711
coverage at 80 with finite-sample levels: 0.741

The scaled bands still fall short of both labels on the standard cutoffs: they cover 0.739 of client-hours at 80 and 0.902 at 95. Their mean widths are 267.62 kWh and 532.00 kWh.

The scores are 673.89 at 80 and 1326.80 at 95. They are in kWh, so the large clients dominate them. Lower is better, and the score rewards a band that is both narrow and right.

The pooled bands still have to be read per client. The lowest client coverage at 80 is 0.472, for the client the scaled bands fit least well. With unscaled residuals, the largest client is covered in 0.000 of its hours at 80. The pooled kWh bands were set by the large clients, and they miss the largest client entirely.

Two more checks on the standard cutoffs test the design. Calibrating on only the three days nearest the scored week, Dec 18 to 20, gives 0.711 at 80, against 0.739 with all twelve. Three days are a smaller sample, and the coverage is a little lower. The finite-sample levels give 0.741 at 80, a small change that agrees with the decision windows: the correction does little for the client bands.

Working rule: Calibrate on windows that come before the scored ones, with no holiday in them, and scale each client before pooling its residuals. Make each design choice on windows that are not the ones you report. Report how many windows and residuals went into the bands and the coverage they gave on the scored days, even when that coverage falls short of the level.

この部分はまだ翻訳されていないため、英語で表示している。

Measure coverage on held-out days

最終変更:版2026.11.6

Coverage is measured on days that the calibration never used. The standard cutoffs give three days, and each day gives 24 hours for each client, so the 2160 client-hours are not that many independent observations. The thirty clients share the calendar and the weather, so the hours of one day agree with one another. For day-level errors the sample is three days, which cannot measure a coverage level. Three days can show that a band fell short on those days, but they cannot say by how much a band falls short over a longer period.

The supplier buys the total, so the total's coverage matters most. Its bands are built from its own residuals over the same twelve days, with the quantiles read directly, and are offered at 80 only, because twelve residuals cannot support 95. With signed bounds, the band from the smallest residual to the largest covers at most 0.846 of the time, if the residuals are exchangeable, and a 95 band, or a 90 band, asks for more.

The total's bands cover 0.667 of its 72 scored hours at 80. For exchangeable errors, the same construction covers 0.682, so most of the gap to 80 comes from the twelve-residual sample. The section on cost puts these bands to use.

The total's errors vary by day. The mean absolute error of the total was 2415.51 kWh on the calibration windows and 2985.88 kWh on the scored days. Dec 31 accounts for most of the gap between the scored days and the calibration windows.

Scored day Error of the total, kWh Coverage at 80
Dec 29, 2014 2448.53 0.625
Dec 30, 2014 2149.78 0.708
Dec 31, 2014 4359.32 0.667

The coverage does not follow the errors. Dec 29 has the lowest coverage with a middle-sized error, and Dec 31 has the largest error with a middle-sized coverage. Each day has 24 hours, and three days are too few to tell the two apart.

The finite-sample levels would change the total's figure. On the scored days they give 0.736 at 80. The choice was made on the decision windows, where the correction widened the total's bands and worsened their interval score, so the bands do not use it. The decision windows also show that the direction of the gap is not fixed. There the total's 80 bands covered 0.948 of hours, more than their label, and the finite levels pushed that to 0.990.

Angelopoulos and Bates (2022) show that split conformal prediction covers the test point with probability at least 1 - alpha when the calibration and test points are independent and identically distributed, or more generally exchangeable. They also note that adjacent points in a time series are not exchangeable, using a temperature series as the example. The same holds for hourly load: each hour carries the one before it, and a day's hours share the weather and the calendar. Barber and co-authors (2023) call the loss in coverage when exchangeability fails a coverage gap. Their bound does not assume a particular joint distribution. It is small when recent calibration points are weighted in a way that matches the test point's distribution, as in slow drift. That result is theoretical, and this chapter does not test it. A 2026 preprint, a survey of conformal methods for dependent series, states that temporal dependence violates the exchangeability assumption. It is not peer reviewed, and we cite it only for that point.

So the book promises nothing about coverage that it does not measure. The coverage in this chapter is what three days showed, under the calibration described in Calibrate on earlier windows.

Working rule: Report coverage on windows that the calibration never used, with the number of windows and the coverage of each. Keep the same measure in a running log once the forecast runs every day, as Chapter 10 does. Read three scored days as a warning, not as a measurement of the level you were promised.

この部分はまだ翻訳されていないため、英語で表示している。

Choose the level from the cost of error

最終変更:版2026.11.11

The supplier buys one amount for each hour, and the costs of the two errors set that amount. Let cu be the cost of one kWh bought short and co the cost of one kWh bought and not used. Let F be the distribution of demand. Differentiating the expected cost with respect to the purchase q gives co F(q) - cu (1 - F(q)), which is zero where F(q) = cu / (cu + co), the smallest expected cost. The level cu / (cu + co) is the critical fractile of the newsvendor model. The HKUST IEEM 101 notes on the newsvendor model give the same proof, writing the cost of excess as c0. This chapter writes it co. The purchase is therefore the quantile at level tau = cu / (cu + co).

With the illustrative costs from Why one number is not enough, tau is 0.75. The rule says to buy the quantile of the demand at that level, for each hour. The quantile at level tau is the purchase that minimises the quantile loss at that level, as Gneiting and Raftery (2007) show. The quantile loss is tau * (y - q) when the reading y is at or above the purchase q, and (1 - tau) * (q - y) when it is below. The quantile loss is not a cost in kWh or in money. It is the rule that the quantile minimises, and it has the same shape as the supplier's cost. Multiplying it by cu + co gives the cost exactly, which the example prints beside the mean cost.

Does buying at the critical fractile cost less than buying at the median or higher, on days the bands never saw? Below, the purchase is set for each hour from these costs, and the scored days answer the question. The level 0.95 is not offered, as Measure coverage on held-out days explains.

buy-ahead-level-by-cost.py Python 3.13.16 · holidays 0.110.dev20261118063242 · lightgbm 4.7.0 · mlforecast 1.1.0 · numpy 2.5.5 · pandas 2.3.3 · statsforecast 2.1.1 · utilsforecast 0.2.17 合格
"""The buy level for the total, set from the two costs: the critical fractile, cu / (cu + co), is the quantile that minimisesthe expected cost. Compare the mean cost and the hours short at each level; the costs are illustrative, not prices."""import pandas as pdfrom lib.electricity import loadfrom lib.intervals import CAL_CUTOFFS, COST_EXCESS, COST_SHORT, buy_report, critical_fractile, pairedfrom lib.panel import ALL30from lib.quirks import mask_clock_changesfrom lib.settings import EVAL_CUTOFFS, showraw = load(ALL30)# Every clock-change hour (four changes, 2013 and 2014) is masked as NaN, as in every example of this chapter.df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]# Calibration on Dec 9 to 20, 2014; scored on the three standard cutoffs.pairs_cal, pairs_sc = paired(df, CAL_CUTOFFS), paired(df, pd.to_datetime(EVAL_CUTOFFS))show("cost of a kWh short", COST_SHORT)show("cost of a kWh unused", COST_EXCESS)# The quantile level cu / (cu + co), from the two costs printed above.show("critical fractile", critical_fractile())# buy_report returns the printed rows as (label, value, decimal places).for label, value, digits in buy_report(pairs_cal, pairs_sc):    show(label, value, digits)
出力python buy-ahead-level-by-cost.py
cost of a kWh short: 3.00
cost of a kWh unused: 1.00
critical fractile: 0.75
scored hours: 72
residuals per hour for the total: 12
highest coverage the total's residuals allow: 0.846
simulated coverage at 80, exchangeable errors: 0.682
mean absolute error of the total, calibration days: 2415.51
mean absolute error of the total, scored days: 2985.88
total coverage at 80: 0.667
total coverage at 80 with finite-sample levels, not used: 0.736
total coverage at 80 on 2014-12-29: 0.625
total coverage at 80 on 2014-12-30: 0.708
total coverage at 80 on 2014-12-31: 0.667
total mean absolute error on 2014-12-29: 2448.53
total mean absolute error on 2014-12-30: 2149.78
total mean absolute error on 2014-12-31: 4359.32
mean cost at 0.50: 5087.95
hours short at 0.50: 0.528
mean cost at 0.75: 3843.61
hours short at 0.75: 0.403
mean cost at 0.90: 3667.43
hours short at 0.90: 0.222
quantile loss at critical fractile, times both costs: 3843.61
mean hourly total bound from the total: 31983.95
mean hourly total bound from the sum of client bounds: 32388.77
hours short at critical fractile, sum of client bounds: 0.306
mean cost at critical fractile, sum of client bounds: 3729.17

The identity holds: multiplied by the sum of the two costs, the quantile loss at the critical fractile is 3843.61, the same as the mean cost at that level, 3843.61. The rule is correct as arithmetic.

Level Mean cost Hours short
0.50 5087.95 0.528
0.75, the critical fractile 3843.61 0.403
0.90 3667.43 0.222

The result on the scored days is less comfortable. The rule chose the critical fractile, and on these three days 0.90 cost less, at 3667.43 against 3843.61. At the critical fractile the purchase was short in 0.403 of the hours, where a calibrated level would fall short in one hour in four. That is the coverage shortfall from Measure coverage on held-out days seen as money. We do not move the level to 0.90. The three scored days are the test, and a level chosen because it wins on them has been chosen on the test. The honest reading is that the bands were too narrow here, so the purchase was too low, and that the level is right only when the bands are.

Should the purchase follow the clients' bounds instead? Summing each client's bound at the critical fractile gives a bound of 32388.77 kWh an hour, against 31983.95 from the total's own residuals. Summed quantiles equal the total's quantile when the errors move together perfectly. For an upper quantile, with errors that partly cancel, the sum usually sits above it, as it does here.

On the scored days the summed bound did better, with hours short at 0.306 against 0.403, and a mean cost of 3729.17 against 3843.61. That advantage comes from its higher level, not from a better calibration, and the scored days cannot show that it holds in general. We keep the total's own bound, because the purchase is made against the total's residuals, and we do not switch to the bound that did better here.

Working rule: Set the buy level from the two costs, as tau = cu / (cu + co), fix it before you score the days, and calibrate the total's bands on the total's own residuals. Show the realised cost at other levels as a check, but never move the level to whichever one would have won on the days you score.

本番で起こる問題

最終変更:版2026.11.5
  • 較正の期間と評価の期間は同じではない。 祝日は較正から除外してある。これにより検証は公正に保たれるが、合計値の誤差は窓ごとに異なり、評価対象の日では較正の窓より大きくなることがある。穏やかな時期に較正した区間は、より厳しい時期では外れる。これに気づくには、毎日、被覆率を名目水準と比べ、移動平均を見る。被覆率がラベルの値を下回る状態が1週間続けば、それが兆候であり、最初に調べるべきはその週の誤差である。
  • プールした区間は特定の需要家を隠す。 kWh でプールした区間は、最大の需要家によって決まる。スケーリングしていない残差を用いると、最大の需要家の区間は、80 の水準でその需要家の時間の 0.000 を覆う。スケーリングした残差を用いた場合、80 での需要家別被覆率の最低値は 0.472 である。したがって、スケーリングしたプールでも、外れる需要家は隠れてしまう。プールした値だけでなく、需要家ごとに被覆率を読むこと。
  • 3日分ではほとんど何も言えない。 評価した3日分の被覆率は、一つの知見のように見えることがあるが、実際には日単位の誤差を3回観測したにすぎない。多くの窓にわたる被覆率を用い、短い期間の結果は、結果ではなく調べるきっかけとして扱う。
  • 従属性は単純な保証を崩す。 時間別の負荷は自己相関をもち、隣り合う時間の較正残差は独立ではない。この区間には、教科書の定理による保証はない。天候や暦が変わると被覆率はずれることがあり、そのことを知らせるのは実測した被覆率だけである。誤差の符号がそろう日が続くパターンに注意すべきであり、それが見つかるのは日々記録している継続的なログの中である。
  • 合計値は時間ごとの残差が少ない。 較正の窓1つにつき、合計値の残差は各時間に1つ得られるため、12個の窓では12個になる。符号付き残差の場合、12個の残差が許す最高の被覆率は 0.846 であり、合計値の区間は 80 で頭打ちになる。そのため、区間を提示する前に水準の確認を行うべきである。較正の窓を増やせば合計値の標本は大きくなるが、各窓は評価する週より前の通常の日でなければならない。
  • ローリング較正は評価対象の日を使うことがある。 Build intervals from residuals の組み込みの規則は、各カットオフを、その直前の3つのカットオフで較正する。運用ではそれで問題ない。それらの値はすでに分かっているからである。しかしテストでは、評価する窓どうしが互いに依存することになる。較正集合は固定し、評価するどの窓よりも前のものにすること。

この章で行った判断

最終変更:版2026.11.5
  • 較正窓。 保持した週より前の、通常の12日間、2014年12月9日から20日までを用いた。祝日は除外し、較正に用いた時間はどれも採点しない。判断窓も同じく12日間で、2014年11月9日から20日までである。この長さは選択であり、結果ではない。12日間が最良の長さであることを、データは示していない。
  • プールする前の尺度化。 各クライアントの残差は、最初の較正カットオフより前の履歴における、そのクライアントの季節ナイーブ誤差で割る。これは本書が MASE に用いるものと同じ尺度である。これは判断窓で決めた。合計には尺度は不要である。
  • 区間に有限標本補正を用いない。 この補正は、クライアントと合計のそれぞれについて判断窓で検証したが、区間スコアを改善しなかった。補正は、ある残差の集合がどの水準まで支えうるかを読み取るためにのみ用いる。
  • 符号付き残差。 区間と購入の分位点には符号付き残差を用いるため、下限と上限は対称である必要がない。mlforecast の conformal 手法は絶対残差を用い、対称な区間を与える。Build intervals from residuals の組み込みの実行は、比較のために置いてある。
  • 購入の対象は合計である。 購入水準とその分位点は、合計の残差で定める。クライアントごとの区間は予測を読み取るためのものであり、購入のためのものではない。
  • 費用は仮定である。 3.00 と 1.00 という費用は、Why one number is not enough に、判断として示したものであり、測定したものではない。
  • 統計ライブラリの区間クラス。 statsforecast のクラスは ConformalIntervals である。この名前は mlforecast の PredictionIntervals とは異なり、本章ではそれぞれのライブラリに対して、そのライブラリの名前を用いる。
  • 3つ目の例。 購入の判断には専用の測定が必要だったため、buy-ahead-level-by-cost を追加した。あわせて conformal-decision-windows も追加した。これは、保持した評価では用いない窓で、設計上の選択を行うためのものである。
  • 祝日の被覆は主張しない。 2014年12月25日は保持した週に含まれるが、較正窓と採点窓のどちらにも含まれない。区間が祝日そのものをどの程度被覆するかについて、本書は何も報告しない。
  • 区間が先、水準はその次。 購入水準の良さは、その下にある区間の良さを超えない。区間の良さは、それを定める窓の良さを超えない。

この版における本章

第2026.11.11版2026年11月20日時点で例が通過この章の4 例件のうち4件が合格

英語版の書籍(版2026.11.11)から翻訳した。

この節を引用する

形式

いずれの形式も、本訳書が従う英語版の書籍を出典として示す。

いずれの形式もこの版を明示している。そのリンクは現行版を開き、節の見出しには本文が最後に変更された日付が示される。アクセス日と取得日は、これを開いた日である。

このページの内容

全10章中の第8章: 予測の不確実性を定量化する

  1. なぜ1つの数値では足りないのか
  2. Build intervals from residuals
  3. Calibrate on earlier windows
  4. Measure coverage on held-out days
  5. Choose the level from the cost of error
  6. 本番で起こる問題
  7. この章で行った判断

ページの先頭に戻るすべての章