比较 Prophet 与基础模型
阅读约需 83分钟 分钟,7节中含6个示例。
读完本章后,你将能够在本书的三个标准截止点上拟合 Prophet,加载一个已记录的基础模型(Chronos-2)和一个已记录的神经网络模型(NHITS),并在同一个 MASE 尺度上,对照相同的读数为它们全部打分。
读者会在本书之外接触到 Prophet 和神经网络预测模型,它们各自都有自己的主张。本章要问的是:在供应商的客户数据上,在 2014 年 12 月 29 日至 31 日这三个标准截止点上,它们各自表现如何。供应商对任何模型的需求都更为具体:为每个客户给出日前预测,并给出一个可据以购买的区间。第 8 章曾在更早的窗口上校准过这类区间。这里的区间并未以那种方式校准,因此本章只检查它们的覆盖率,并不加以修正。每个模型都与第 2 章的季节性朴素预测以及第 6 章的全局模型比较,使用相同的小时数据,并采用同一个 MASE 尺度。其中两个模型是记录下来的结果,而不是实际运行得到的。
这部分尚未翻译,因此以英文显示。
Where Prophet fits
Prophet is the first model a reader tries beyond the baselines, so we test it first, and its assumptions decide where it can help.
Prophet writes a series as a trend, plus Fourier seasonal terms, plus holiday effects, plus an error term. The authors describe it as a modular regression model whose parameters an analyst can read and adjust (Taylor and Letham (2018)). By default Prophet's fit is a maximum a posteriori estimate, an optimisation rather than a set of sampled draws. The code of version 1.5.0 sets that default in its optimisation call (models.py, lines 165 and 172), and its mcmc_samples argument defaults to 0 (forecaster.py, line 144). Both are in the Prophet source distribution for the pinned version. Each client gets its own fit, so Prophet does not learn from the other clients.
Prophet does several things well. It needs no interpolation of missing observations: the code drops rows whose y is missing before the fit (forecaster.py, line 1335). The clock-change hours are missing in the history by the rule in Chapter 3, so Prophet leaves them out without any extra code. Prophet takes holidays as a table of dates and effects, and the table this chapter tests holds the Portuguese national holidays for 2013 and 2014. The authors report that, on their own series with default settings, Prophet had lower error than auto.arima, ETS, seasonal naive and TBATS (Taylor and Letham (2018)). That result rests on their data, so we cite it as the authors' result and do not use it as evidence about this case.
Several defaults get in the way here. Yearly seasonality turns on only with at least 730 days of history (forecaster.py, line 1187), so it stays off with eight weeks. The default interval is 80%, not 95% (forecaster.py, line 145). It covers trend uncertainty and observation noise, not the seasonal pattern, unless the model is fitted with full sampling (Prophet documentation). Its coverage therefore makes a narrower claim than the word "interval" suggests. Prophet also assumes that the future trend changes as often, and by as much, as it did in the history (Prophet documentation). With eight weeks of history there is little to learn trend changes from, so in our reading that assumption carries most of the forecast.
Working rule: Use Prophet per client when a reader needs its trend and seasonal settings as parameters they can read, and when its scored forecasts beat seasonal naive on the decision windows you chose before scoring. Set its interval width in code, and read its default 80% interval as an 80% interval, never as a 95% range. Check its coverage against the level it claims.
这部分尚未翻译,因此以英文显示。
Run Prophet on the same cutoffs
The last section named Prophet's assumptions. This one fixes its settings and runs it on the same hours as every other model.
The comparison uses the three standard cutoffs, at 00:00 on Dec 29, Dec 30 and Dec 31, 2014. These are the windows Chapters 4 to 8 score. Each window's history is the 1,344 hours, eight weeks, before its cutoff, with the start rule applied, so a late starter's history begins at its first reading. Prophet is fitted once for each client and cutoff, which makes 90 fits.
We chose the holiday table on decision windows, not on the standard cutoffs. The ordinary decision windows hold no holiday in their scored hours, so they cannot test a holiday table at all, while the holiday decision windows can. So the choice uses both kinds. A run with the table and one without it shows what the table does on those windows for SUBSET8: the four ordinary days and seven holidays, Apr 18, Apr 20, Apr 25, May 1, Jun 10, Aug 15 and Dec 8, 2014. All of them come before the held-out week, and none has Christmas Day in its history.
1"""Prophet with and without the Portuguese holiday table, on the eleven decision windows for SUBSET8 (B-9, B-10),2before any standard cutoff is scored. The point that the table stays off rests on the two mean MASE lines and on3holiday table kept, which says whether the table lowers the mean per-client MASE."""4from lib.compare import prophet_windows, seasonal_naive, score5from lib.electricity import load6from lib.features import is_holiday7from lib.panel import DECISION_CUTOFFS, HOLIDAY_DECISION_CUTOFFS8from lib.quirks import mask_clock_changes9from lib.settings import SUBSET8, show1011# the four ordinary and seven holiday windows, in date order12DECISION = sorted([*DECISION_CUTOFFS, *HOLIDAY_DECISION_CUTOFFS])13raw = load(SUBSET8)14# clock-change hours set to missing, so Prophet fits without them15df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]16keys = ["unique_id", "ds"]17# one fit per client and window, with the Portuguese holidays as Prophet's holiday table; two processes, and the same18# fits at any count19with_holidays = prophet_windows(df, DECISION, use_holidays=True, workers=2)[keys + ["Prophet"]].rename(20 columns={"Prophet": "WithHolidays"})21without = prophet_windows(df, DECISION, use_holidays=False, workers=2)[keys + ["Prophet"]].rename(22 columns={"Prophet": "NoHolidays"})23forecasts = with_holidays.merge(without, on=keys, validate="one_to_one").merge(24 seasonal_naive(df, DECISION), on=keys, validate="one_to_one")25models = ["WithHolidays", "NoHolidays", "SeasonalNaive"]26# per_client: one MASE per client and model; scored: forecasts beside readings27per_client, scored = score(raw, forecasts, models, SUBSET8, DECISION)2829show("decision windows", len(DECISION))30show("holiday decision windows", len(HOLIDAY_DECISION_CUTOFFS))31show("clients", len(SUBSET8))32show("scored hours", len(scored))33# 1 for each hour on a Portuguese national holiday34show("holiday hours in the decision windows", int(is_holiday(scored["ds"]).sum()))35for model in models:36 # the mean of the per-client MASEs, not a pooled error37 show(f"mean MASE {model}", per_client[model].mean(), digits=3)38show("hours where the forecasts differ", int((scored["WithHolidays"] != scored["NoHolidays"]).sum()))39show("clients where holidays help", int((per_client["WithHolidays"] < per_client["NoHolidays"]).sum()))40# the table is kept only on a strictly lower mean; a tie keeps the simpler model, with the table off41show("holiday table kept", "yes" if per_client["WithHolidays"].mean() < per_client["NoHolidays"].mean() else "no")
python prophet-holiday-decision.pydecision windows: 11
holiday decision windows: 7
clients: 8
scored hours: 2112
holiday hours in the decision windows: 1344
mean MASE WithHolidays: 1.070
mean MASE NoHolidays: 1.061
mean MASE SeasonalNaive: 0.934
hours where the forecasts differ: 768
clients where holidays help: 2
holiday table kept: no
On the decision windows the table makes Prophet worse. Over the eleven of them, its mean per-client MASE is 1.070 with the table and 1.061 without it, against 0.934 for seasonal naive.
A setting is kept only if it lowers the decision-window MASE, and a tie keeps the simpler model. The example's verdict on keeping the table is no, so the table stays off.
The holiday windows hold 1344 client-hours of holiday, and they are the only place the table can change a forecast. 768 of the 2112 scored hours differ between the two variants.
The standard cutoffs tell a different story, which we report without using it to choose. Their histories include Christmas Day, and the holiday windows' histories do not. That is a likely reason for the gap, but we did not test which holiday causes it. With the table on, Prophet's mean per-client MASE on the standard cutoffs would be 1.27, against 1.35 with it off. A team that wants the table should test it on its own decision windows before it scores anything.
Daily and weekly seasonality are on, and yearly is off. These are the choices Prophet makes for itself on eight weeks of hourly data: yearly needs at least 730 days, weekly at least two weeks, and daily at least two days of sub-daily data (forecaster.py, lines 1187, 1209 and 1223). We set them explicitly and did not tune them. The interval is 95%, set in code and printed as Prophet interval level, so coverage sits beside the recordings' 95% ranges. Prophet draws its interval samples from numpy's global generator, so the code sets the seed (SEED) before each predict. The fit itself is deterministic.
Every model is scored on the book's one MASE scale: the seasonal naive error in sample, over every reading before Dec 29, 2014, with the clock-change hours masked. A MASE below 1 means the model's error on the scored hours is smaller than seasonal naive's in-sample error. Each client has its own scale, so a large client does not outweigh a small one. The next run fits Prophet for all thirty clients and scores it beside seasonal naive and the global model on this scale, which is the comparison every later result in the chapter uses.
1"""Prophet for each of the thirty clients on the three standard cutoffs, with a 95% interval, scored on the book's MASE2scale beside seasonal naive and the Chapter 6 global model. The point that Prophet's mean MASE is higher rests on3the mean MASE lines, and the coverage line says whether its 95% interval holds."""4import pandas as pd56from lib.compare import (PROPHET_HOLIDAYS, PROPHET_LEVEL, global_forecasts, prophet_windows, report_range,7 report_subset, score, seasonal_naive)8from lib.electricity import load9from lib.panel import ALL3010from lib.quirks import mask_clock_changes11from lib.scoring import report_scores12from lib.settings import EVAL_CUTOFFS, show1314raw = load(ALL30)15# clock-change hours set to missing, as chapter 3 sets them16df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]17cutoffs = pd.DatetimeIndex(EVAL_CUTOFFS)18# Prophet at the book's settings from lib.compare: holiday table off, 95% interval, one fit per client and cutoff19forecasts = prophet_windows(df, cutoffs).merge(20 seasonal_naive(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one").merge(21 global_forecasts(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one")22models = ["Prophet", "SeasonalNaive", "Global"]23# per_client: one MASE per client and model; scored: forecasts beside readings24per_client, scored = score(raw, forecasts, models, ALL30)2526show("Prophet interval level", PROPHET_LEVEL)27show("Prophet holiday table", "on" if PROPHET_HOLIDAYS else "off")28show("clients scored", scored["unique_id"].nunique())29show("scored hours", len(scored))30show("clients where Prophet beats seasonal naive", int((per_client["Prophet"] < per_client["SeasonalNaive"]).sum()))31show("clients where Prophet beats the global model", int((per_client["Prophet"] < per_client["Global"]).sum()))32report_scores(scored, per_client, models)33report_subset(per_client, models)34# the 95% range from Prophet's lower and upper interval columns35report_range(scored, "Prophet", "ProphetLo", "ProphetHi", PROPHET_LEVEL)36# Sensitivity only, not used to choose: the holiday table on the same cutoffs, which the decision windows37# did not favour.38# two processes; each fit is the same at any worker count39on = prophet_windows(df, cutoffs, use_holidays=True, workers=2)[["unique_id", "ds", "Prophet"]].rename(40 columns={"Prophet": "ProphetHolidays"})41per_holiday, _ = score(raw, forecasts.merge(on, on=["unique_id", "ds"], validate="one_to_one"),42 ["ProphetHolidays"], ALL30)43show("mean MASE Prophet with holidays, not chosen", per_holiday["ProphetHolidays"].mean())
python prophet-thirty-clients-backtest.pyProphet interval level: 95
Prophet holiday table: off
clients scored: 30
scored hours: 2160
clients where Prophet beats seasonal naive: 9
clients where Prophet beats the global model: 6
MAE Prophet: 182.44
MAE SeasonalNaive: 148.26
MAE Global: 122.81
RMSE Prophet: 764.88
RMSE SeasonalNaive: 780.48
RMSE Global: 678.81
mean MASE Prophet: 1.35
mean MASE SeasonalNaive: 1.18
mean MASE Global: 0.91
mean MASE Prophet SUBSET8: 1.39
mean MASE SeasonalNaive SUBSET8: 1.16
mean MASE Global SUBSET8: 1.03
coverage 95 Prophet: 0.918
interval score 95 Prophet: 932.66
mean MASE Prophet with holidays, not chosen: 1.27
On these cutoffs Prophet has a higher mean per-client MASE than seasonal naive: 1.35 over the thirty clients, against 1.18 for seasonal naive and 0.91 for the global model.
The loss is on most clients too, not only on average: Prophet beats seasonal naive on only 9 of the thirty clients, and the global model on 6.
Pooled over every scored hour, Prophet's RMSE is 764.88 kWh, against 780.48 for seasonal naive, so it makes fewer very large misses. It loses to seasonal naive on MAE and MASE, and to the global model on every measure. Its 95% interval covers 0.918 of the readings, and its interval score is 932.66. Coverage below 0.95 means the interval is too narrow for its stated level on these hours.
Working rule: Set Prophet's seasonality, holidays and interval width in code before you score it, and choose any of them on decision windows, never on the standard cutoffs. Keep the holiday table only if it lowers the MASE on the holiday decision windows. Compare Prophet with seasonal naive and the global model on the same windows and the same scale, and list the clients it loses. Set the seed before each predict, and check that a second run prints the same lines.
这部分尚未翻译,因此以英文显示。
Load a recorded foundation model's forecasts
Prophet fits each client from its own eight weeks. A foundation model brings what it learned from other series, and this section shows how the book holds its forecasts.
A foundation model is trained once on many series and then applied to a context, with no fitting on that series. Chronos-2 is one such model. Chronos-2 is an encoder-only transformer that takes a context of past values and outputs quantiles for the next steps (Ansari et al. (2025); the earlier Chronos model is described in Ansari et al. (2024)). Its weights are published as amazon/chronos-2 on Hugging Face under the Apache-2.0 licence, not gated (model card).
Its pretraining covers this dataset, which limits how far its score can be read. The Chronos-2 report lists the electricity dataset in its pretraining corpus. Appendix A, Table 6 gives "Electricity", with 370 series in the energy domain (Ansari et al. (2025)). Those series match the UCI ElectricityLoadDiagrams20112014 dataset the book reads, and the book's thirty clients are drawn from them. The report notes partial overlap with some GIFT-Eval training portions. For strictly zero-shot results it points to a variant trained only on synthetic data, in its Section 5.4. We have not run that variant. The report does not show which readings of these clients the model saw, so the scores below are upper bounds, not out-of-sample results.
We do not run Chronos-2 in an example. Its forecasts are recorded: the model ran once, outside the examples, on a CPU, with the revision pinned to 29ec3766d36d6f73f0696f85560a422f50e8498c, and the examples read the saved output. The recording is dated Nov 10, 2026. Examples must run offline and give the same output every time, and a model call does neither. A downloaded model can change, and a reader's machine may not hold the files at all.
The input for each client and cutoff is the 1,344 hours before the cutoff, with the start rule applied. None of the standard cutoffs needs that step. Their histories begin on Nov 3, 2014 or later, after the autumn clock change, and the three late starters had their first readings before Nov 3, 2014. The manifest counts no missing values in the contexts, so the clock-change handling changes no scored number here. The package builds a mask from missing values and scales around them (chronos-forecasting source, chronos2/model.py, line 398). The 90 contexts went in one batch, with each series kept separate. Cross-learning, which would let one series inform another, is off by default, and the recording does not turn it on (chronos2/pipeline.py, line 833). So the Dec 30 context does not inform the Dec 29 forecast.
The output is, for each client, cutoff and hour, five quantiles for the next 24 hours: 0.025, 0.1, 0.5, 0.9 and 0.975. That is 2160 rows in all. The manifest lists the SHA-256 of the output and of inputs.csv.gz, which keeps the exact context each window was given, along with the package versions and the machine, described only as a 4-vCPU Linux container with no GPU.
Not every requested quantile is trained. The package is trained on 21 levels, from 0.01 to 0.99 (Ansari et al. (2025)). The levels 0.1, 0.5 and 0.9 are among them, so they are read directly. The levels 0.025 and 0.975 are not, so the package interpolates each one linearly between the two nearest trained levels: 0.01 and 0.05 for the lower, and 0.95 and 0.99 for the upper. The code is interpolate_quantiles in chronos/utils.py, called from predict_quantiles in chronos2/pipeline.py (chronos-forecasting source). The manifest names the interpolated levels, and a 95% range built from them is partly the package's interpolation.
Before any score reads the recording, run the check. It hashes each output and compares it with manifest.json, counts the rows, clients and cutoffs in the forecast file, and looks for quantiles that run out of order.
1"""Checks the recorded Chronos-2 outputs before any score reads them: each file's SHA-256 against manifest.json, the2shape of the forecast file, and any quantiles out of order, without running the model. The point that the recording3is intact rests on the hash mismatches line and the rows with quantiles out of order line."""4from lib.compare import check_recording5from lib.settings import show67for label, value in check_recording().items(): # the default name is the chronos-2 folder in examples/data/recorded/8 show(label, value)
python recorded-chronos2-check.pyoutputs checked: 2
hash mismatches: 0
rows: 2160
clients: 30
windows: 3
interpolated levels: 2
rows with quantiles out of order: 0
largest quantile crossing, kWh: 0.00
inverted 80 or 95 ranges: 0
The recording is intact: 0 of its 2 outputs fail their hashes, and 0 rows have quantiles out of order. The check verifies the output files, not the prompt hash, which covers the input and the configuration.
The licence decides which weights the book can run or record: this book uses only Apache-2.0 or MIT weights. Chronos-2's weights and the chronos-forecasting package are Apache-2.0 (model card; package metadata). Two other families were considered and are not used. The TimesFM 3.0 weights carry a non-commercial licence, which allows non-commercial and non-production use only (TimesFM 3.0 licence). TimesFM 2.5 is Apache-2.0 (TimesFM 2.5 card), so the licence depends on the version named. The Moirai R-series checkpoints are tagged CC-BY-NC-4.0, which is also non-commercial (Moirai 2.0 card).
A later edition can run Chronos-2 live once its weights and package fit the release service's limits. Until then the recording stands in for the live run.
Working rule: Record a foundation model only with its script, its pinned revision, its package versions and a manifest that hashes every output. Read the recorded outputs in every example, and check the hashes before any score reads them. Check the licence of the weights, not only the licence of the package, and name the model version in any licence sentence.
这部分尚未翻译,因此以英文显示。
Score the recordings on the same cutoffs
The recordings mean something only when they are scored on the same hours as Prophet and the baselines.
A forecast with five quantiles can be scored in two ways. The median, the 0.5 quantile, is a point forecast, scored with MASE on the book's scale. The 0.1 and 0.9 quantiles form an 80% range, and the 0.025 and 0.975 quantiles form a 95% range. Each range is scored by coverage, the share of readings inside it, and by the interval score, which adds the width to a penalty for each reading outside it. The penalty is 2 divided by the nominal miss rate, 0.05 for a 95% range, times the distance outside. Chapter 8 defines the interval score with its source. Each quantile is also scored by its quantile loss, which charges tau for each kWh a reading falls above the quantile and one minus tau for each kWh below it (Chapter 8).
Two examples score the recordings, one for each model, on the same thirty clients, cutoffs and MASE scale as the Prophet run. Each rebuilds the global model and seasonal naive instead of copying them from Chapter 6, so every model goes through one scoring path.
1"""Scores the recorded Chronos-2 forecasts on the three standard cutoffs and the MASE scale of the other models: the2median is the point forecast, the 0.1 and 0.9 quantiles the 80% range, and the 0.025 and 0.975 quantiles the 95%3range. The point about the lead rests on the mean MASE lines, and the code reads the recording only, never the4model."""5import pandas as pd67from lib.compare import (global_forecasts, recorded_forecasts, report_pinball, report_range, report_subset, score,8 seasonal_naive)9from lib.electricity import load10from lib.panel import ALL3011from lib.quirks import mask_clock_changes12from lib.scoring import report_scores13from lib.settings import EVAL_CUTOFFS, show1415raw = load(ALL30)16# clock-change hours set to missing, as chapter 3 sets them17df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]18cutoffs = pd.DatetimeIndex(EVAL_CUTOFFS)19# the recording, read only after each output matches its manifest hash; its median is the Chronos column20forecasts = recorded_forecasts().merge(21 seasonal_naive(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one").merge(22 global_forecasts(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one")23models = ["Chronos", "SeasonalNaive", "Global"]24# per_client: one MASE per client and model; scored: forecasts beside readings25per_client, scored = score(raw, forecasts, models, ALL30)2627show("clients scored", scored["unique_id"].nunique())28show("scored hours", len(scored))29show("clients where Chronos beats seasonal naive", int((per_client["Chronos"] < per_client["SeasonalNaive"]).sum()))30report_scores(scored, per_client, models)31report_subset(per_client, models)32report_pinball(scored, "Chronos", "q10", 0.1) # quantile loss at tau 0.1: the 0.1 quantile column33report_pinball(scored, "Chronos", "q50", 0.5) # at tau 0.5 the loss is half the absolute error34report_pinball(scored, "Chronos", "q90", 0.9)35report_range(scored, "Chronos", "q10", "q90", 80)36report_range(scored, "Chronos", "q025", "q975", 95)
python recorded-chronos2-score.pyclients scored: 30
scored hours: 2160
clients where Chronos beats seasonal naive: 27
MAE Chronos: 100.13
MAE SeasonalNaive: 148.26
MAE Global: 122.81
RMSE Chronos: 655.39
RMSE SeasonalNaive: 780.48
RMSE Global: 678.81
mean MASE Chronos: 0.81
mean MASE SeasonalNaive: 1.18
mean MASE Global: 0.91
mean MASE Chronos SUBSET8: 0.87
mean MASE SeasonalNaive SUBSET8: 1.16
mean MASE Global SUBSET8: 1.03
quantile loss 0.1 Chronos: 39.44
quantile loss 0.5 Chronos: 50.07
quantile loss 0.9 Chronos: 17.85
coverage 80 Chronos: 0.749
interval score 80 Chronos: 572.89
coverage 95 Chronos: 0.944
interval score 95 Chronos: 958.19
1"""Scores the recorded NHITS forecasts on the same three cutoffs and MASE scale as Chronos-2 and the other models: the2median is the point forecast, and each range comes from levels trained directly, so none is interpolated. The point3about its ranges rests on the coverage and interval score lines, and the code reads the recording only, never the4model."""5import pandas as pd67from lib.compare import (global_forecasts, recorded_forecasts, report_pinball, report_range, report_subset, score,8 seasonal_naive)9from lib.electricity import load10from lib.panel import ALL3011from lib.quirks import mask_clock_changes12from lib.scoring import report_scores13from lib.settings import EVAL_CUTOFFS, show1415raw = load(ALL30)16# clock-change hours set to missing, as chapter 3 sets them17df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]18cutoffs = pd.DatetimeIndex(EVAL_CUTOFFS)19# the NHITS recording, read only after each output matches its manifest hash; its median is the NHITS column20forecasts = recorded_forecasts("nhits", "NHITS").merge(21 seasonal_naive(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one").merge(22 global_forecasts(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one")23models = ["NHITS", "SeasonalNaive", "Global"]24# per_client: one MASE per client and model; scored: forecasts beside readings25per_client, scored = score(raw, forecasts, models, ALL30)2627show("clients scored", scored["unique_id"].nunique())28show("scored hours", len(scored))29show("clients where NHITS beats seasonal naive", int((per_client["NHITS"] < per_client["SeasonalNaive"]).sum()))30report_scores(scored, per_client, models)31report_subset(per_client, models)32report_pinball(scored, "NHITS", "q10", 0.1) # quantile loss at tau 0.1: the 0.1 quantile column33report_pinball(scored, "NHITS", "q50", 0.5) # at tau 0.5 the loss is half the absolute error34report_pinball(scored, "NHITS", "q90", 0.9)35report_range(scored, "NHITS", "q10", "q90", 80)36report_range(scored, "NHITS", "q025", "q975", 95)
python recorded-nhits-score.pyclients scored: 30
scored hours: 2160
clients where NHITS beats seasonal naive: 24
MAE NHITS: 136.61
MAE SeasonalNaive: 148.26
MAE Global: 122.81
RMSE NHITS: 649.41
RMSE SeasonalNaive: 780.48
RMSE Global: 678.81
mean MASE NHITS: 1.03
mean MASE SeasonalNaive: 1.18
mean MASE Global: 0.91
mean MASE NHITS SUBSET8: 1.17
mean MASE SeasonalNaive SUBSET8: 1.16
mean MASE Global SUBSET8: 1.03
quantile loss 0.1 NHITS: 27.66
quantile loss 0.5 NHITS: 68.31
quantile loss 0.9 NHITS: 40.58
coverage 80 NHITS: 0.700
interval score 80 NHITS: 682.37
coverage 95 NHITS: 0.877
interval score 95 NHITS: 1250.94
| Model | Mean MASE, ALL30 | Mean MASE, SUBSET8 | MAE, ALL30 (kWh) |
|---|---|---|---|
| Seasonal naive | 1.18 | 1.16 | 148.26 |
| Global model (Chapter 6) | 0.91 | 1.03 | 122.81 |
| Chronos-2, median | 0.81 | 0.87 | 100.13 |
| NHITS, median | 1.03 | 1.17 | 136.61 |
| Prophet, point forecast | 1.35 | 1.39 | 182.44 |
Chronos-2 has the lowest mean per-client MASE of the five models on both sets of clients: 0.81 over the thirty clients and 0.87 on SUBSET8. It beats seasonal naive on 27 of the thirty clients. The global model from Chapter 6 is second on both sets. Chapter 4's fixed seasonal ARIMA, on the same eight clients, cutoffs and scale, scores 1.09 on SUBSET8, between the global model and seasonal naive.
NHITS is third on the thirty clients and fourth on SUBSET8, where it is narrowly behind seasonal naive, 1.17 against 1.16. Prophet has the highest mean on both sets.
The lead needs its caveat. Chronos-2's pretraining corpus includes this dataset (the recording section), so these readings may have been part of its training, and its score here is an upper bound on what to expect on readings it has not seen. NHITS has no such advantage, because each of its cutoffs was trained only on that cutoff's own context. NHITS trails Chronos-2 on the same hours, but the two are not on equal footing, and this chapter does not rank them on this comparison.
The median is also a check on the quantile arithmetic. The pinball loss at 0.5 is half the absolute error, so the median's loss should be half its MAE. For Chronos-2 the loss at 0.5 is 50.07, against the MAE of 100.13 kWh divided by two. For NHITS the loss at 0.5 is 68.31, against half its MAE of 136.61 kWh.
Chronos-2's 95% range covers 0.944 of the readings, close to its level, and its 80% range covers 0.749, a little under its level of 0.80. The Chronos-2 95% range is partly interpolated, so its score is for the model and the package together.
Prophet's 95% range has an interval score a little lower than Chronos-2's, 932.66 against 958.19, but its coverage of 0.918 sits further from its level.
NHITS's ranges are too narrow for their levels on these hours: its 80% range covers 0.700 of the readings, and its 95% range covers 0.877.
Its 95% interval score is the highest of the three 95% ranges, and its quantiles are trained directly, so no range is interpolated. The recording does not put its quantiles in order, though. The check counts 22 rows with an adjacent pair out of order, by at most 21.11 kWh, and 0 inverted 80% or 95% ranges. The scores use the values as recorded.
Three limits apply. These are three cutoffs in one week of one dataset, so the results are not a general claim about any model. Neither range was calibrated on the calibration windows that Chapter 8 used, so the coverage here is a check, not a calibration. The comparison makes no choice, because decisions in this book are made on decision windows, and the recordings have no forecasts there, so nothing is picked with these scores.
Working rule: Score a recorded forecast on the same windows, the same scale and the same readings as every other model, and name any interpolated quantile wherever its range is reported. Before you read a lead, check what the model was trained on, because a model that saw the scored readings cannot show skill on them. Do not choose a model or a setting on these windows. A recorded forecast with no forecasts on the decision windows cannot be chosen for production until it has them.
为什么记录神经模型
本版次不运行任何神经模型。书中的两个神经预测结果都是预先记录的,原因在于运行环境的成本,而不是对神经模型的评判。
NHITS 是 neuralforecast 库中的一个神经预测器,该库为多条序列组成的面板数据封装了多种神经预测器。该库的许可证是 Apache-2.0(neuralforecast 元数据)。我们记录 NHITS 的方式与记录 Chronos-2 相同:有各自的脚本、固定版本和清单,并在相同的上下文、截断点、尺度和指标下,与 Chronos-2 并列评分。它的记录过程针对每个截断点训练一次,使用的是提供给 Chronos-2 的同样三十段 1,344 小时的上下文。输入窗口为 168 小时,即一周,预测范围为 24 小时。五个分位水平在 MQLoss 损失下直接训练,每个水平对应一个输出。缩放器是 neuralforecast 的标准缩放器,随机种子固定,运行在 CPU 上。每个截断点的模型只看到该截断点的上下文,因此截断点之后的读数不会进入模型。权重不随书发布。
NHITS 同样要接受检查,对照其自身清单中的哈希值进行。
1"""Checks the recorded NHITS outputs before any score reads them: each file's SHA-256 against manifest.json, the shape2of the forecast file, and any quantiles out of order, without running the model. The point that the recording is3intact rests on the hash mismatches line; the interpolated levels line shows that every level is trained directly, and4the rows with quantiles out of order line is the one the chapter reports."""5from lib.compare import check_recording6from lib.settings import show78for label, value in check_recording("nhits").items(): # "nhits" is the NHITS folder in examples/data/recorded/9 show(label, value)
python recorded-nhits-check.pyoutputs checked: 1
hash mismatches: 0
rows: 2160
clients: 30
windows: 3
interpolated levels: 0
rows with quantiles out of order: 22
largest quantile crossing, kWh: 21.11
inverted 80 or 95 ranges: 0
NHITS 的文件与其清单一致,并且它的各个水平都不是插值得到的,因为每个水平都是直接训练的。
有两项成本使 NHITS 不能进入示例。它的库会引入 torch、ray 和 pytorch-lightning,使示例的环境从 687 MB 增大到 1.7 GB;而在发布服务的 2 个 vCPU 和 4 GiB 内存上,一次 CPU 训练的耗时超出了一个示例应有的长度。其中不涉及版本冲突。NHITS 的依赖文件 recordings/requirements-nhits.txt 安装 neuralforecast 时,使用的 pandas 版本与示例固定的版本相同(2.3.3)。
Chronos-2 有自己的一套依赖。它的依赖文件 recordings/requirements.txt 会安装 chronos-forecasting,以及 transformers 和 torch,并为记录过程固定使用 pandas 3。该软件包本身并不需要 pandas 3。它的元数据接受从 2.0 起、但不含 4 的 pandas 版本(chronos-forecasting 源码,pyproject.toml)。示例从不导入这两个软件包,只读取保存下来的输出。
在后续版次中,只要有一个符合发布服务限制的运行环境,或者有它自己的运行环境,NHITS 就可以实时运行。无论哪种情况,本章中的窗口、尺度和评分都保持不变,因此后来的模型会与其他模型在相同的小时上并列评分。
工作准则: 当某个模型的依赖会突破运行环境的预算时,就把它排除在示例的运行环境之外,改为记录它,并把它的固定版本写在单独的依赖文件中。添加依赖之前,先在干净的环境中安装它,并测量它在体积和软件包数量上增加了多少。如果依赖解析会改动某个已固定的软件包,这就是记录该模型的理由,而不是修改固定版本的理由。
这部分尚未翻译,因此以英文显示。
What goes wrong in production
Prophet's holiday table covers 2013 and 2014 only. The book's runs leave it off, because the holiday decision windows gave no reason to turn it on, but the standard cutoffs show that it moves the result. A forecast made in 2015 with the table unchanged gets no holiday effect on later holidays, and the run reports nothing wrong. A team that turns the table on must extend it to the forecast's years. Holidays are known in advance, so the fix is to extend the table. Count the holiday hours in each window as well. A window that contains a holiday should show a count above zero, and a count of zero where a holiday falls is the sign to look for.
Interval width fails differently: a Prophet interval left at its default of 80% gets read as a 95% range by a reader who sees only the word "interval" in a report. The coverage line is the check. Printed beside its level, a coverage near 0.8 is correct for an 80% interval and a failure for a 95% one. Print the level with the coverage every time.
A recorded forecast goes stale in a different way. The recording covers three cutoffs. A day-ahead forecast in production needs one cutoff every day, and the release service does not run the model. Covering another cutoff means running the model, saving its output and hashing it, which is work outside the examples. The sign that the recording is out of date is a window with no rows in the file, which the check's window count shows.
A later model revision changes every number that depends on the recording, even when the code does not change. Treat the revision as an update trigger. Record it, score it on the same windows as the old one, and compare the two before the replacement recording is used. A package that changes its interpolation between versions changes the 95% range without touching the median, so the manifest's list of interpolated levels should be compared as well.
A recorded forecast can also break the order of its own quantiles. The NHITS recording does in 22 of its 2160 rows, by at most 21.11 kWh. A range built from a crossed pair would be read as inverted. The check counts the crossings, and a consumer should sort or reject such rows before it uses them.
Prophet's trend assumption fails when a client's demand moves to a different level, as it does after a change of contract or the loss of a large load. Prophet expects the trend to change as often as it did in the history, and its default interval does not cover the seasonal pattern. The sign is residuals that keep one sign across successive windows, and a coverage line that falls over them. The fix is a refit on a shorter history for that client, not a wider interval.
Working rule: Before any Prophet or recorded forecast is reported, check that the holiday table covers the forecast's years when the table is on, that the interval level is printed beside its coverage, that the recording has a window for every cutoff in use, that its quantiles are in order, and that the model revision matches the one scored. A failed check stops the report, and the report says which check failed.
本章做出的决定
- Prophet 在全部三十个客户端上运行,而不只是在
SUBSET8上。 运行全部三十个客户端,可以让 Prophet 与全局模型在相同的客户端上并列比较。SUBSET8的均值取自同一次运行并已打印。 - Prophet 的区间取 95%,在代码中设定并打印。 这样,它的覆盖率和区间得分就能与录制结果的 95% 范围并列比较。默认的 80% 不予使用。
- Prophet 的季节性开关在代码中固定。 日季节性和周季节性开启,年季节性关闭。这些是 Prophet 对八周的小时级数据会做出的选择,这里只是显式地设定出来。
- 节假日表为关闭状态。 在节假日决策窗口上,该表使平均 MASE 升高,因此保留了较简单的模型。在标准截断点上,该表本会使 Prophet 的平均 MASE 降低,我们报告这一数字,但不用它来做选择。
- Chronos-2 和 NHITS 只作记录,不运行。 这些示例读取已保存的输出并校验其哈希值,从不调用模型,因为示例必须能够离线运行,并且每次给出相同的输出。
- NHITS 的记录是出于运行时成本的考虑。 它的依赖项会使示例的运行时从 687 MB 增至 1.7 GB,超出发布服务的 2 个 vCPU 和 4 GiB,而且在 CPU 上训练一次所需的时间,比一个示例应有的时间更长。
- Chronos-2 的 95% 范围用它的两个插值分位数来评分,本章对此有明确说明。 NHITS 的分位数是直接训练得到的,因此它的范围都不是插值得来的。
- Chronos-2 的领先优势只作为上界来解读。 它的预训练语料包含这一数据集,因此它在这里的得分不是样本外的。
- 选择只来自决策窗口。 录制结果在决策窗口中没有预测,所以本章不用它们做任何选择。
本章,在本版次中
版次 2026.11.11示例于 2026年11月20日 通过本章 6个示例 个示例中有 6 个通过
译自英文版图书,版次 2026.11.11。