比較 Prophet 與基礎模型
閱讀約需 84 分鐘 分鐘,7 節 中共有 6 個範例。
讀完本章後,你可以在本書的三個標準截止點上擬合 Prophet,載入一個已記錄的基礎模型(Chronos-2)與一個已記錄的神經網路模型(NHITS),並在同一個 MASE 尺度上,針對相同的讀數為所有模型評分。
讀者會在本書之外接觸到 Prophet 與神經網路預測模型,而且各有各的主張。本章要問的是:在供應商的客戶資料上、在標準截止點(2014 年 12 月 29 日至 31 日)上,這些模型各自表現如何。供應商對任何模型的需求更為單純:為每位客戶提供隔日預測,以及一個可據以採購的區間。第 8 章曾在較早的時間窗上校準這類區間。本章的區間並未以那種方式校準,因此本章只檢查其涵蓋率,而不加以修正。每個模型都與第 2 章的季節性樸素預測,以及第 6 章的全域模型比較,採用相同的時段與同一個 MASE 尺度。其中兩個模型是以記錄結果呈現,而非實際執行。
此部分尚未翻譯,因此以英文呈現。
Where Prophet fits
Prophet is the first model a reader tries beyond the baselines, so we test it first, and its assumptions decide where it can help.
Prophet writes a series as a trend, plus Fourier seasonal terms, plus holiday effects, plus an error term. The authors describe it as a modular regression model whose parameters an analyst can read and adjust (Taylor and Letham (2018)). By default Prophet's fit is a maximum a posteriori estimate, an optimisation rather than a set of sampled draws. The code of version 1.5.0 sets that default in its optimisation call (models.py, lines 165 and 172), and its mcmc_samples argument defaults to 0 (forecaster.py, line 144). Both are in the Prophet source distribution for the pinned version. Each client gets its own fit, so Prophet does not learn from the other clients.
Prophet does several things well. It needs no interpolation of missing observations: the code drops rows whose y is missing before the fit (forecaster.py, line 1335). The clock-change hours are missing in the history by the rule in Chapter 3, so Prophet leaves them out without any extra code. Prophet takes holidays as a table of dates and effects, and the table this chapter tests holds the Portuguese national holidays for 2013 and 2014. The authors report that, on their own series with default settings, Prophet had lower error than auto.arima, ETS, seasonal naive and TBATS (Taylor and Letham (2018)). That result rests on their data, so we cite it as the authors' result and do not use it as evidence about this case.
Several defaults get in the way here. Yearly seasonality turns on only with at least 730 days of history (forecaster.py, line 1187), so it stays off with eight weeks. The default interval is 80%, not 95% (forecaster.py, line 145). It covers trend uncertainty and observation noise, not the seasonal pattern, unless the model is fitted with full sampling (Prophet documentation). Its coverage therefore makes a narrower claim than the word "interval" suggests. Prophet also assumes that the future trend changes as often, and by as much, as it did in the history (Prophet documentation). With eight weeks of history there is little to learn trend changes from, so in our reading that assumption carries most of the forecast.
Working rule: Use Prophet per client when a reader needs its trend and seasonal settings as parameters they can read, and when its scored forecasts beat seasonal naive on the decision windows you chose before scoring. Set its interval width in code, and read its default 80% interval as an 80% interval, never as a 95% range. Check its coverage against the level it claims.
此部分尚未翻譯,因此以英文呈現。
Run Prophet on the same cutoffs
The last section named Prophet's assumptions. This one fixes its settings and runs it on the same hours as every other model.
The comparison uses the three standard cutoffs, at 00:00 on Dec 29, Dec 30 and Dec 31, 2014. These are the windows Chapters 4 to 8 score. Each window's history is the 1,344 hours, eight weeks, before its cutoff, with the start rule applied, so a late starter's history begins at its first reading. Prophet is fitted once for each client and cutoff, which makes 90 fits.
We chose the holiday table on decision windows, not on the standard cutoffs. The ordinary decision windows hold no holiday in their scored hours, so they cannot test a holiday table at all, while the holiday decision windows can. So the choice uses both kinds. A run with the table and one without it shows what the table does on those windows for SUBSET8: the four ordinary days and seven holidays, Apr 18, Apr 20, Apr 25, May 1, Jun 10, Aug 15 and Dec 8, 2014. All of them come before the held-out week, and none has Christmas Day in its history.
1"""Prophet with and without the Portuguese holiday table, on the eleven decision windows for SUBSET8 (B-9, B-10),2before any standard cutoff is scored. The point that the table stays off rests on the two mean MASE lines and on3holiday table kept, which says whether the table lowers the mean per-client MASE."""4from lib.compare import prophet_windows, seasonal_naive, score5from lib.electricity import load6from lib.features import is_holiday7from lib.panel import DECISION_CUTOFFS, HOLIDAY_DECISION_CUTOFFS8from lib.quirks import mask_clock_changes9from lib.settings import SUBSET8, show1011# the four ordinary and seven holiday windows, in date order12DECISION = sorted([*DECISION_CUTOFFS, *HOLIDAY_DECISION_CUTOFFS])13raw = load(SUBSET8)14# clock-change hours set to missing, so Prophet fits without them15df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]16keys = ["unique_id", "ds"]17# one fit per client and window, with the Portuguese holidays as Prophet's holiday table; two processes, and the same18# fits at any count19with_holidays = prophet_windows(df, DECISION, use_holidays=True, workers=2)[keys + ["Prophet"]].rename(20 columns={"Prophet": "WithHolidays"})21without = prophet_windows(df, DECISION, use_holidays=False, workers=2)[keys + ["Prophet"]].rename(22 columns={"Prophet": "NoHolidays"})23forecasts = with_holidays.merge(without, on=keys, validate="one_to_one").merge(24 seasonal_naive(df, DECISION), on=keys, validate="one_to_one")25models = ["WithHolidays", "NoHolidays", "SeasonalNaive"]26# per_client: one MASE per client and model; scored: forecasts beside readings27per_client, scored = score(raw, forecasts, models, SUBSET8, DECISION)2829show("decision windows", len(DECISION))30show("holiday decision windows", len(HOLIDAY_DECISION_CUTOFFS))31show("clients", len(SUBSET8))32show("scored hours", len(scored))33# 1 for each hour on a Portuguese national holiday34show("holiday hours in the decision windows", int(is_holiday(scored["ds"]).sum()))35for model in models:36 # the mean of the per-client MASEs, not a pooled error37 show(f"mean MASE {model}", per_client[model].mean(), digits=3)38show("hours where the forecasts differ", int((scored["WithHolidays"] != scored["NoHolidays"]).sum()))39show("clients where holidays help", int((per_client["WithHolidays"] < per_client["NoHolidays"]).sum()))40# the table is kept only on a strictly lower mean; a tie keeps the simpler model, with the table off41show("holiday table kept", "yes" if per_client["WithHolidays"].mean() < per_client["NoHolidays"].mean() else "no")
python prophet-holiday-decision.pydecision windows: 11
holiday decision windows: 7
clients: 8
scored hours: 2112
holiday hours in the decision windows: 1344
mean MASE WithHolidays: 1.070
mean MASE NoHolidays: 1.061
mean MASE SeasonalNaive: 0.934
hours where the forecasts differ: 768
clients where holidays help: 2
holiday table kept: no
On the decision windows the table makes Prophet worse. Over the eleven of them, its mean per-client MASE is 1.070 with the table and 1.061 without it, against 0.934 for seasonal naive.
A setting is kept only if it lowers the decision-window MASE, and a tie keeps the simpler model. The example's verdict on keeping the table is no, so the table stays off.
The holiday windows hold 1344 client-hours of holiday, and they are the only place the table can change a forecast. 768 of the 2112 scored hours differ between the two variants.
The standard cutoffs tell a different story, which we report without using it to choose. Their histories include Christmas Day, and the holiday windows' histories do not. That is a likely reason for the gap, but we did not test which holiday causes it. With the table on, Prophet's mean per-client MASE on the standard cutoffs would be 1.27, against 1.35 with it off. A team that wants the table should test it on its own decision windows before it scores anything.
Daily and weekly seasonality are on, and yearly is off. These are the choices Prophet makes for itself on eight weeks of hourly data: yearly needs at least 730 days, weekly at least two weeks, and daily at least two days of sub-daily data (forecaster.py, lines 1187, 1209 and 1223). We set them explicitly and did not tune them. The interval is 95%, set in code and printed as Prophet interval level, so coverage sits beside the recordings' 95% ranges. Prophet draws its interval samples from numpy's global generator, so the code sets the seed (SEED) before each predict. The fit itself is deterministic.
Every model is scored on the book's one MASE scale: the seasonal naive error in sample, over every reading before Dec 29, 2014, with the clock-change hours masked. A MASE below 1 means the model's error on the scored hours is smaller than seasonal naive's in-sample error. Each client has its own scale, so a large client does not outweigh a small one. The next run fits Prophet for all thirty clients and scores it beside seasonal naive and the global model on this scale, which is the comparison every later result in the chapter uses.
1"""Prophet for each of the thirty clients on the three standard cutoffs, with a 95% interval, scored on the book's MASE2scale beside seasonal naive and the Chapter 6 global model. The point that Prophet's mean MASE is higher rests on3the mean MASE lines, and the coverage line says whether its 95% interval holds."""4import pandas as pd56from lib.compare import (PROPHET_HOLIDAYS, PROPHET_LEVEL, global_forecasts, prophet_windows, report_range,7 report_subset, score, seasonal_naive)8from lib.electricity import load9from lib.panel import ALL3010from lib.quirks import mask_clock_changes11from lib.scoring import report_scores12from lib.settings import EVAL_CUTOFFS, show1314raw = load(ALL30)15# clock-change hours set to missing, as chapter 3 sets them16df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]17cutoffs = pd.DatetimeIndex(EVAL_CUTOFFS)18# Prophet at the book's settings from lib.compare: holiday table off, 95% interval, one fit per client and cutoff19forecasts = prophet_windows(df, cutoffs).merge(20 seasonal_naive(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one").merge(21 global_forecasts(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one")22models = ["Prophet", "SeasonalNaive", "Global"]23# per_client: one MASE per client and model; scored: forecasts beside readings24per_client, scored = score(raw, forecasts, models, ALL30)2526show("Prophet interval level", PROPHET_LEVEL)27show("Prophet holiday table", "on" if PROPHET_HOLIDAYS else "off")28show("clients scored", scored["unique_id"].nunique())29show("scored hours", len(scored))30show("clients where Prophet beats seasonal naive", int((per_client["Prophet"] < per_client["SeasonalNaive"]).sum()))31show("clients where Prophet beats the global model", int((per_client["Prophet"] < per_client["Global"]).sum()))32report_scores(scored, per_client, models)33report_subset(per_client, models)34# the 95% range from Prophet's lower and upper interval columns35report_range(scored, "Prophet", "ProphetLo", "ProphetHi", PROPHET_LEVEL)36# Sensitivity only, not used to choose: the holiday table on the same cutoffs, which the decision windows37# did not favour.38# two processes; each fit is the same at any worker count39on = prophet_windows(df, cutoffs, use_holidays=True, workers=2)[["unique_id", "ds", "Prophet"]].rename(40 columns={"Prophet": "ProphetHolidays"})41per_holiday, _ = score(raw, forecasts.merge(on, on=["unique_id", "ds"], validate="one_to_one"),42 ["ProphetHolidays"], ALL30)43show("mean MASE Prophet with holidays, not chosen", per_holiday["ProphetHolidays"].mean())
python prophet-thirty-clients-backtest.pyProphet interval level: 95
Prophet holiday table: off
clients scored: 30
scored hours: 2160
clients where Prophet beats seasonal naive: 9
clients where Prophet beats the global model: 6
MAE Prophet: 182.44
MAE SeasonalNaive: 148.26
MAE Global: 122.81
RMSE Prophet: 764.88
RMSE SeasonalNaive: 780.48
RMSE Global: 678.81
mean MASE Prophet: 1.35
mean MASE SeasonalNaive: 1.18
mean MASE Global: 0.91
mean MASE Prophet SUBSET8: 1.39
mean MASE SeasonalNaive SUBSET8: 1.16
mean MASE Global SUBSET8: 1.03
coverage 95 Prophet: 0.918
interval score 95 Prophet: 932.66
mean MASE Prophet with holidays, not chosen: 1.27
On these cutoffs Prophet has a higher mean per-client MASE than seasonal naive: 1.35 over the thirty clients, against 1.18 for seasonal naive and 0.91 for the global model.
The loss is on most clients too, not only on average: Prophet beats seasonal naive on only 9 of the thirty clients, and the global model on 6.
Pooled over every scored hour, Prophet's RMSE is 764.88 kWh, against 780.48 for seasonal naive, so it makes fewer very large misses. It loses to seasonal naive on MAE and MASE, and to the global model on every measure. Its 95% interval covers 0.918 of the readings, and its interval score is 932.66. Coverage below 0.95 means the interval is too narrow for its stated level on these hours.
Working rule: Set Prophet's seasonality, holidays and interval width in code before you score it, and choose any of them on decision windows, never on the standard cutoffs. Keep the holiday table only if it lowers the MASE on the holiday decision windows. Compare Prophet with seasonal naive and the global model on the same windows and the same scale, and list the clients it loses. Set the seed before each predict, and check that a second run prints the same lines.
此部分尚未翻譯,因此以英文呈現。
Load a recorded foundation model's forecasts
Prophet fits each client from its own eight weeks. A foundation model brings what it learned from other series, and this section shows how the book holds its forecasts.
A foundation model is trained once on many series and then applied to a context, with no fitting on that series. Chronos-2 is one such model. Chronos-2 is an encoder-only transformer that takes a context of past values and outputs quantiles for the next steps (Ansari et al. (2025); the earlier Chronos model is described in Ansari et al. (2024)). Its weights are published as amazon/chronos-2 on Hugging Face under the Apache-2.0 licence, not gated (model card).
Its pretraining covers this dataset, which limits how far its score can be read. The Chronos-2 report lists the electricity dataset in its pretraining corpus. Appendix A, Table 6 gives "Electricity", with 370 series in the energy domain (Ansari et al. (2025)). Those series match the UCI ElectricityLoadDiagrams20112014 dataset the book reads, and the book's thirty clients are drawn from them. The report notes partial overlap with some GIFT-Eval training portions. For strictly zero-shot results it points to a variant trained only on synthetic data, in its Section 5.4. We have not run that variant. The report does not show which readings of these clients the model saw, so the scores below are upper bounds, not out-of-sample results.
We do not run Chronos-2 in an example. Its forecasts are recorded: the model ran once, outside the examples, on a CPU, with the revision pinned to 29ec3766d36d6f73f0696f85560a422f50e8498c, and the examples read the saved output. The recording is dated Nov 10, 2026. Examples must run offline and give the same output every time, and a model call does neither. A downloaded model can change, and a reader's machine may not hold the files at all.
The input for each client and cutoff is the 1,344 hours before the cutoff, with the start rule applied. None of the standard cutoffs needs that step. Their histories begin on Nov 3, 2014 or later, after the autumn clock change, and the three late starters had their first readings before Nov 3, 2014. The manifest counts no missing values in the contexts, so the clock-change handling changes no scored number here. The package builds a mask from missing values and scales around them (chronos-forecasting source, chronos2/model.py, line 398). The 90 contexts went in one batch, with each series kept separate. Cross-learning, which would let one series inform another, is off by default, and the recording does not turn it on (chronos2/pipeline.py, line 833). So the Dec 30 context does not inform the Dec 29 forecast.
The output is, for each client, cutoff and hour, five quantiles for the next 24 hours: 0.025, 0.1, 0.5, 0.9 and 0.975. That is 2160 rows in all. The manifest lists the SHA-256 of the output and of inputs.csv.gz, which keeps the exact context each window was given, along with the package versions and the machine, described only as a 4-vCPU Linux container with no GPU.
Not every requested quantile is trained. The package is trained on 21 levels, from 0.01 to 0.99 (Ansari et al. (2025)). The levels 0.1, 0.5 and 0.9 are among them, so they are read directly. The levels 0.025 and 0.975 are not, so the package interpolates each one linearly between the two nearest trained levels: 0.01 and 0.05 for the lower, and 0.95 and 0.99 for the upper. The code is interpolate_quantiles in chronos/utils.py, called from predict_quantiles in chronos2/pipeline.py (chronos-forecasting source). The manifest names the interpolated levels, and a 95% range built from them is partly the package's interpolation.
Before any score reads the recording, run the check. It hashes each output and compares it with manifest.json, counts the rows, clients and cutoffs in the forecast file, and looks for quantiles that run out of order.
1"""Checks the recorded Chronos-2 outputs before any score reads them: each file's SHA-256 against manifest.json, the2shape of the forecast file, and any quantiles out of order, without running the model. The point that the recording3is intact rests on the hash mismatches line and the rows with quantiles out of order line."""4from lib.compare import check_recording5from lib.settings import show67for label, value in check_recording().items(): # the default name is the chronos-2 folder in examples/data/recorded/8 show(label, value)
python recorded-chronos2-check.pyoutputs checked: 2
hash mismatches: 0
rows: 2160
clients: 30
windows: 3
interpolated levels: 2
rows with quantiles out of order: 0
largest quantile crossing, kWh: 0.00
inverted 80 or 95 ranges: 0
The recording is intact: 0 of its 2 outputs fail their hashes, and 0 rows have quantiles out of order. The check verifies the output files, not the prompt hash, which covers the input and the configuration.
The licence decides which weights the book can run or record: this book uses only Apache-2.0 or MIT weights. Chronos-2's weights and the chronos-forecasting package are Apache-2.0 (model card; package metadata). Two other families were considered and are not used. The TimesFM 3.0 weights carry a non-commercial licence, which allows non-commercial and non-production use only (TimesFM 3.0 licence). TimesFM 2.5 is Apache-2.0 (TimesFM 2.5 card), so the licence depends on the version named. The Moirai R-series checkpoints are tagged CC-BY-NC-4.0, which is also non-commercial (Moirai 2.0 card).
A later edition can run Chronos-2 live once its weights and package fit the release service's limits. Until then the recording stands in for the live run.
Working rule: Record a foundation model only with its script, its pinned revision, its package versions and a manifest that hashes every output. Read the recorded outputs in every example, and check the hashes before any score reads them. Check the licence of the weights, not only the licence of the package, and name the model version in any licence sentence.
此部分尚未翻譯,因此以英文呈現。
Score the recordings on the same cutoffs
The recordings mean something only when they are scored on the same hours as Prophet and the baselines.
A forecast with five quantiles can be scored in two ways. The median, the 0.5 quantile, is a point forecast, scored with MASE on the book's scale. The 0.1 and 0.9 quantiles form an 80% range, and the 0.025 and 0.975 quantiles form a 95% range. Each range is scored by coverage, the share of readings inside it, and by the interval score, which adds the width to a penalty for each reading outside it. The penalty is 2 divided by the nominal miss rate, 0.05 for a 95% range, times the distance outside. Chapter 8 defines the interval score with its source. Each quantile is also scored by its quantile loss, which charges tau for each kWh a reading falls above the quantile and one minus tau for each kWh below it (Chapter 8).
Two examples score the recordings, one for each model, on the same thirty clients, cutoffs and MASE scale as the Prophet run. Each rebuilds the global model and seasonal naive instead of copying them from Chapter 6, so every model goes through one scoring path.
1"""Scores the recorded Chronos-2 forecasts on the three standard cutoffs and the MASE scale of the other models: the2median is the point forecast, the 0.1 and 0.9 quantiles the 80% range, and the 0.025 and 0.975 quantiles the 95%3range. The point about the lead rests on the mean MASE lines, and the code reads the recording only, never the4model."""5import pandas as pd67from lib.compare import (global_forecasts, recorded_forecasts, report_pinball, report_range, report_subset, score,8 seasonal_naive)9from lib.electricity import load10from lib.panel import ALL3011from lib.quirks import mask_clock_changes12from lib.scoring import report_scores13from lib.settings import EVAL_CUTOFFS, show1415raw = load(ALL30)16# clock-change hours set to missing, as chapter 3 sets them17df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]18cutoffs = pd.DatetimeIndex(EVAL_CUTOFFS)19# the recording, read only after each output matches its manifest hash; its median is the Chronos column20forecasts = recorded_forecasts().merge(21 seasonal_naive(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one").merge(22 global_forecasts(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one")23models = ["Chronos", "SeasonalNaive", "Global"]24# per_client: one MASE per client and model; scored: forecasts beside readings25per_client, scored = score(raw, forecasts, models, ALL30)2627show("clients scored", scored["unique_id"].nunique())28show("scored hours", len(scored))29show("clients where Chronos beats seasonal naive", int((per_client["Chronos"] < per_client["SeasonalNaive"]).sum()))30report_scores(scored, per_client, models)31report_subset(per_client, models)32report_pinball(scored, "Chronos", "q10", 0.1) # quantile loss at tau 0.1: the 0.1 quantile column33report_pinball(scored, "Chronos", "q50", 0.5) # at tau 0.5 the loss is half the absolute error34report_pinball(scored, "Chronos", "q90", 0.9)35report_range(scored, "Chronos", "q10", "q90", 80)36report_range(scored, "Chronos", "q025", "q975", 95)
python recorded-chronos2-score.pyclients scored: 30
scored hours: 2160
clients where Chronos beats seasonal naive: 27
MAE Chronos: 100.13
MAE SeasonalNaive: 148.26
MAE Global: 122.81
RMSE Chronos: 655.39
RMSE SeasonalNaive: 780.48
RMSE Global: 678.81
mean MASE Chronos: 0.81
mean MASE SeasonalNaive: 1.18
mean MASE Global: 0.91
mean MASE Chronos SUBSET8: 0.87
mean MASE SeasonalNaive SUBSET8: 1.16
mean MASE Global SUBSET8: 1.03
quantile loss 0.1 Chronos: 39.44
quantile loss 0.5 Chronos: 50.07
quantile loss 0.9 Chronos: 17.85
coverage 80 Chronos: 0.749
interval score 80 Chronos: 572.89
coverage 95 Chronos: 0.944
interval score 95 Chronos: 958.19
1"""Scores the recorded NHITS forecasts on the same three cutoffs and MASE scale as Chronos-2 and the other models: the2median is the point forecast, and each range comes from levels trained directly, so none is interpolated. The point3about its ranges rests on the coverage and interval score lines, and the code reads the recording only, never the4model."""5import pandas as pd67from lib.compare import (global_forecasts, recorded_forecasts, report_pinball, report_range, report_subset, score,8 seasonal_naive)9from lib.electricity import load10from lib.panel import ALL3011from lib.quirks import mask_clock_changes12from lib.scoring import report_scores13from lib.settings import EVAL_CUTOFFS, show1415raw = load(ALL30)16# clock-change hours set to missing, as chapter 3 sets them17df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]18cutoffs = pd.DatetimeIndex(EVAL_CUTOFFS)19# the NHITS recording, read only after each output matches its manifest hash; its median is the NHITS column20forecasts = recorded_forecasts("nhits", "NHITS").merge(21 seasonal_naive(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one").merge(22 global_forecasts(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one")23models = ["NHITS", "SeasonalNaive", "Global"]24# per_client: one MASE per client and model; scored: forecasts beside readings25per_client, scored = score(raw, forecasts, models, ALL30)2627show("clients scored", scored["unique_id"].nunique())28show("scored hours", len(scored))29show("clients where NHITS beats seasonal naive", int((per_client["NHITS"] < per_client["SeasonalNaive"]).sum()))30report_scores(scored, per_client, models)31report_subset(per_client, models)32report_pinball(scored, "NHITS", "q10", 0.1) # quantile loss at tau 0.1: the 0.1 quantile column33report_pinball(scored, "NHITS", "q50", 0.5) # at tau 0.5 the loss is half the absolute error34report_pinball(scored, "NHITS", "q90", 0.9)35report_range(scored, "NHITS", "q10", "q90", 80)36report_range(scored, "NHITS", "q025", "q975", 95)
python recorded-nhits-score.pyclients scored: 30
scored hours: 2160
clients where NHITS beats seasonal naive: 24
MAE NHITS: 136.61
MAE SeasonalNaive: 148.26
MAE Global: 122.81
RMSE NHITS: 649.41
RMSE SeasonalNaive: 780.48
RMSE Global: 678.81
mean MASE NHITS: 1.03
mean MASE SeasonalNaive: 1.18
mean MASE Global: 0.91
mean MASE NHITS SUBSET8: 1.17
mean MASE SeasonalNaive SUBSET8: 1.16
mean MASE Global SUBSET8: 1.03
quantile loss 0.1 NHITS: 27.66
quantile loss 0.5 NHITS: 68.31
quantile loss 0.9 NHITS: 40.58
coverage 80 NHITS: 0.700
interval score 80 NHITS: 682.37
coverage 95 NHITS: 0.877
interval score 95 NHITS: 1250.94
| Model | Mean MASE, ALL30 | Mean MASE, SUBSET8 | MAE, ALL30 (kWh) |
|---|---|---|---|
| Seasonal naive | 1.18 | 1.16 | 148.26 |
| Global model (Chapter 6) | 0.91 | 1.03 | 122.81 |
| Chronos-2, median | 0.81 | 0.87 | 100.13 |
| NHITS, median | 1.03 | 1.17 | 136.61 |
| Prophet, point forecast | 1.35 | 1.39 | 182.44 |
Chronos-2 has the lowest mean per-client MASE of the five models on both sets of clients: 0.81 over the thirty clients and 0.87 on SUBSET8. It beats seasonal naive on 27 of the thirty clients. The global model from Chapter 6 is second on both sets. Chapter 4's fixed seasonal ARIMA, on the same eight clients, cutoffs and scale, scores 1.09 on SUBSET8, between the global model and seasonal naive.
NHITS is third on the thirty clients and fourth on SUBSET8, where it is narrowly behind seasonal naive, 1.17 against 1.16. Prophet has the highest mean on both sets.
The lead needs its caveat. Chronos-2's pretraining corpus includes this dataset (the recording section), so these readings may have been part of its training, and its score here is an upper bound on what to expect on readings it has not seen. NHITS has no such advantage, because each of its cutoffs was trained only on that cutoff's own context. NHITS trails Chronos-2 on the same hours, but the two are not on equal footing, and this chapter does not rank them on this comparison.
The median is also a check on the quantile arithmetic. The pinball loss at 0.5 is half the absolute error, so the median's loss should be half its MAE. For Chronos-2 the loss at 0.5 is 50.07, against the MAE of 100.13 kWh divided by two. For NHITS the loss at 0.5 is 68.31, against half its MAE of 136.61 kWh.
Chronos-2's 95% range covers 0.944 of the readings, close to its level, and its 80% range covers 0.749, a little under its level of 0.80. The Chronos-2 95% range is partly interpolated, so its score is for the model and the package together.
Prophet's 95% range has an interval score a little lower than Chronos-2's, 932.66 against 958.19, but its coverage of 0.918 sits further from its level.
NHITS's ranges are too narrow for their levels on these hours: its 80% range covers 0.700 of the readings, and its 95% range covers 0.877.
Its 95% interval score is the highest of the three 95% ranges, and its quantiles are trained directly, so no range is interpolated. The recording does not put its quantiles in order, though. The check counts 22 rows with an adjacent pair out of order, by at most 21.11 kWh, and 0 inverted 80% or 95% ranges. The scores use the values as recorded.
Three limits apply. These are three cutoffs in one week of one dataset, so the results are not a general claim about any model. Neither range was calibrated on the calibration windows that Chapter 8 used, so the coverage here is a check, not a calibration. The comparison makes no choice, because decisions in this book are made on decision windows, and the recordings have no forecasts there, so nothing is picked with these scores.
Working rule: Score a recorded forecast on the same windows, the same scale and the same readings as every other model, and name any interpolated quantile wherever its range is reported. Before you read a lead, check what the model was trained on, because a model that saw the scored readings cannot show skill on them. Do not choose a model or a setting on these windows. A recorded forecast with no forecasts on the decision windows cannot be chosen for production until it has them.
為什麼神經網路模型採用錄製方式
本版次不執行任何神經網路模型。其中兩項神經網路預測採用錄製的結果,原因在於執行環境的成本,並不是對神經網路模型下的定論。
NHITS 是 neuralforecast 函式庫中的神經網路預測器,而該函式庫將適用於多條時間序列面板資料的神經網路預測器整合在一起。此函式庫採用 Apache-2.0 授權(neuralforecast 中繼資料)。我們以錄製 Chronos-2 的方式錄製 NHITS:它有自己的指令碼、鎖定版本與清單檔,並與 Chronos-2 在相同的情境視窗、截止點、尺度與指標下評分。它的錄製針對每個截止點各訓練一次,使用的是與 Chronos-2 相同的三十個 1,344 小時情境視窗。輸入視窗為 168 小時,即一週,預測時距為 24 小時。五個分位數水準以 MQLoss 損失函數直接訓練,每個水準各有一個輸出。縮放器採用 neuralforecast 的標準縮放器,隨機種子固定,並在 CPU 上執行。每個截止點的模型只看到該截止點的情境視窗,因此截止點之後的任何讀數都不會進入模型。模型權重不隨書提供。
NHITS 同樣要通過檢查,比對的是它自己清單檔中的雜湊值。
1"""Checks the recorded NHITS outputs before any score reads them: each file's SHA-256 against manifest.json, the shape2of the forecast file, and any quantiles out of order, without running the model. The point that the recording is3intact rests on the hash mismatches line; the interpolated levels line shows that every level is trained directly, and4the rows with quantiles out of order line is the one the chapter reports."""5from lib.compare import check_recording6from lib.settings import show78for label, value in check_recording("nhits").items(): # "nhits" is the NHITS folder in examples/data/recorded/9 show(label, value)
python recorded-nhits-check.pyoutputs checked: 1
hash mismatches: 0
rows: 2160
clients: 30
windows: 3
interpolated levels: 0
rows with quantiles out of order: 22
largest quantile crossing, kWh: 21.11
inverted 80 or 95 ranges: 0
NHITS 的檔案與其清單檔相符,而且它的各個水準都沒有經過內插,因為每個水準都是直接訓練的。
有兩項成本使 NHITS 不納入範例。它的函式庫會帶入 torch、ray 與 pytorch-lightning,使範例的環境從 687 MB 增加到 1.7 GB;而且在發行服務的 2 vCPU 與 4 GiB 記憶體上,CPU 訓練所需的時間超過範例應有的長度。這裡不涉及版本衝突。NHITS 的需求檔 recordings/requirements-nhits.txt 所安裝的 neuralforecast,使用的 pandas 版本與範例鎖定的相同(2.3.3)。
Chronos-2 有它自己的套件組合。它的需求檔 recordings/requirements.txt 會連同 transformers 與 torch 安裝 chronos-forecasting,並為了錄製而鎖定 pandas 3。這個套件本身並不需要 pandas 3。它的中繼資料接受 2.0 以上、4 以下(不含 4)的 pandas(chronos-forecasting 原始碼,pyproject.toml)。範例從不匯入這兩個套件,只讀取已儲存的輸出。
若日後有符合發行服務限制的執行環境,或讓 NHITS 擁有自己的執行環境,後續版次便可即時執行 NHITS。無論如何,本章的視窗、尺度與評分都維持不變,因此後來加入的模型會在相同的時段上與其他模型並列評分。
實用原則: 當某個模型的相依套件會超出執行環境的預算時,就不要把它放進範例的執行環境,改為錄製其結果,並將它的鎖定版本放在獨立的需求檔中。在加入相依套件之前,先在乾淨的環境中安裝,並量測它在大小與套件數量上增加了多少。若相依性解析會使某個已鎖定的套件變動,這就是錄製該模型的理由,而不是更動鎖定版本的理由。
此部分尚未翻譯,因此以英文呈現。
What goes wrong in production
Prophet's holiday table covers 2013 and 2014 only. The book's runs leave it off, because the holiday decision windows gave no reason to turn it on, but the standard cutoffs show that it moves the result. A forecast made in 2015 with the table unchanged gets no holiday effect on later holidays, and the run reports nothing wrong. A team that turns the table on must extend it to the forecast's years. Holidays are known in advance, so the fix is to extend the table. Count the holiday hours in each window as well. A window that contains a holiday should show a count above zero, and a count of zero where a holiday falls is the sign to look for.
Interval width fails differently: a Prophet interval left at its default of 80% gets read as a 95% range by a reader who sees only the word "interval" in a report. The coverage line is the check. Printed beside its level, a coverage near 0.8 is correct for an 80% interval and a failure for a 95% one. Print the level with the coverage every time.
A recorded forecast goes stale in a different way. The recording covers three cutoffs. A day-ahead forecast in production needs one cutoff every day, and the release service does not run the model. Covering another cutoff means running the model, saving its output and hashing it, which is work outside the examples. The sign that the recording is out of date is a window with no rows in the file, which the check's window count shows.
A later model revision changes every number that depends on the recording, even when the code does not change. Treat the revision as an update trigger. Record it, score it on the same windows as the old one, and compare the two before the replacement recording is used. A package that changes its interpolation between versions changes the 95% range without touching the median, so the manifest's list of interpolated levels should be compared as well.
A recorded forecast can also break the order of its own quantiles. The NHITS recording does in 22 of its 2160 rows, by at most 21.11 kWh. A range built from a crossed pair would be read as inverted. The check counts the crossings, and a consumer should sort or reject such rows before it uses them.
Prophet's trend assumption fails when a client's demand moves to a different level, as it does after a change of contract or the loss of a large load. Prophet expects the trend to change as often as it did in the history, and its default interval does not cover the seasonal pattern. The sign is residuals that keep one sign across successive windows, and a coverage line that falls over them. The fix is a refit on a shorter history for that client, not a wider interval.
Working rule: Before any Prophet or recorded forecast is reported, check that the holiday table covers the forecast's years when the table is on, that the interval level is printed beside its coverage, that the recording has a window for every cutoff in use, that its quantiles are in order, and that the model revision matches the one scored. A failed check stops the report, and the report says which check failed.
本章做出的決定
- Prophet 在全部 30 個客戶端上執行,而不只在
SUBSET8上執行。 全部 30 個都執行,才能讓 Prophet 與全域模型在相同的客戶端上並列比較。SUBSET8的平均值取自同一次執行的輸出。 - Prophet 的區間設為 95%,在程式碼中設定並印出。 如此一來,它的涵蓋率與區間分數就能與錄製資料的 95% 範圍並列。預設的 80% 不使用。
- Prophet 的季節性旗標在程式碼中固定。 每日與每週季節性開啟,每年季節性關閉。這是 Prophet 對八週逐時資料會做的選擇,這裡只是明確地設定出來。
- 假日表關閉。 在假日決策窗口上,這張表使平均 MASE 升高,因此保留較簡單的模型。在標準截止點上,這張表會讓 Prophet 的平均 MASE 降低,我們會報告該數字,但不以它作為選擇的依據。
- Chronos-2 與 NHITS 採用記錄的結果,不實際執行。 範例讀取已儲存的輸出、檢查其雜湊值,而且從不呼叫模型,因為範例必須能離線執行,且每次都產生相同的輸出。
- NHITS 記錄下來,是因為它的執行階段成本。 它的相依套件會使範例的執行環境從 687 MB 增加到 1.7 GB,超過發布服務的 2 vCPU 與 4 GiB,而且在 CPU 上訓練所需的時間比一個範例應有的時間更長。
- Chronos-2 的 95% 範圍以它的兩個內插分位數評分,本章也會明確說明。 NHITS 的分位數是直接訓練出來的,因此它的範圍都不是內插而得。
- Chronos-2 的領先幅度視為上界。 它的預訓練語料包含這個資料集,所以它在此處的分數並非樣本外的結果。
- 選擇只來自決策窗口。 錄製資料在那些窗口中沒有預測,因此本章不以它們做任何選擇。
本章,於本版次中
版次 2026.11.11範例於 2026年11 月 20 通過測試本章 6 個範例 個範例中有 6 個通過
譯自英文書,版次 2026.11.11。