Prophet과 파운데이션 모델 비교
읽는 데 약 54분분 걸린다, 7절에 6개 예제이(가) 있음.
AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식
영어판 2026.11.11판을 번역한 것이다. 원문 읽기
이 장을 마치면 책에서 표준으로 쓰는 세 개의 기준 시점(cutoff)에서 Prophet을 적합하고, 기록된 파운데이션 모델(Chronos-2)과 기록된 신경망 모델(NHITS)을 불러와, 모든 모델을 같은 관측값에 대해 하나의 MASE 척도로 평가할 수 있다.
독자는 이 책 밖에서도 Prophet과 신경망 예측 모델을 접하게 되며, 각 모델에는 저마다의 주장이 따른다. 이 장은 표준 기준 시점인 2014년 12월 29일부터 31일까지, 공급업체의 고객들에 대해 각 모델이 어떻게 동작하는지를 묻는다. 공급업체가 어떤 모델에 요구하는 것은 더 좁다. 고객별 하루 앞 예측과, 그것을 근거로 구매할 수 있는 구간이다. 8장에서는 이전 윈도우(window)들에서 이러한 구간을 보정했다. 여기의 구간은 그런 방식으로 보정되지 않았으므로, 이 장은 구간의 포함률(coverage)을 점검할 뿐 바로잡지는 않는다. 모든 모델은 2장의 계절 나이브 예측과 6장의 전역 모델을 기준으로, 같은 시간들과 하나의 MASE 척도에서 평가한다. 모델 중 둘은 실행하지 않고 기록된 결과를 사용한다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Where Prophet fits
Prophet is the first model a reader tries beyond the baselines, so we test it first, and its assumptions decide where it can help.
Prophet writes a series as a trend, plus Fourier seasonal terms, plus holiday effects, plus an error term. The authors describe it as a modular regression model whose parameters an analyst can read and adjust (Taylor and Letham (2018)). By default Prophet's fit is a maximum a posteriori estimate, an optimisation rather than a set of sampled draws. The code of version 1.5.0 sets that default in its optimisation call (models.py, lines 165 and 172), and its mcmc_samples argument defaults to 0 (forecaster.py, line 144). Both are in the Prophet source distribution for the pinned version. Each client gets its own fit, so Prophet does not learn from the other clients.
Prophet does several things well. It needs no interpolation of missing observations: the code drops rows whose y is missing before the fit (forecaster.py, line 1335). The clock-change hours are missing in the history by the rule in Chapter 3, so Prophet leaves them out without any extra code. Prophet takes holidays as a table of dates and effects, and the table this chapter tests holds the Portuguese national holidays for 2013 and 2014. The authors report that, on their own series with default settings, Prophet had lower error than auto.arima, ETS, seasonal naive and TBATS (Taylor and Letham (2018)). That result rests on their data, so we cite it as the authors' result and do not use it as evidence about this case.
Several defaults get in the way here. Yearly seasonality turns on only with at least 730 days of history (forecaster.py, line 1187), so it stays off with eight weeks. The default interval is 80%, not 95% (forecaster.py, line 145). It covers trend uncertainty and observation noise, not the seasonal pattern, unless the model is fitted with full sampling (Prophet documentation). Its coverage therefore makes a narrower claim than the word "interval" suggests. Prophet also assumes that the future trend changes as often, and by as much, as it did in the history (Prophet documentation). With eight weeks of history there is little to learn trend changes from, so in our reading that assumption carries most of the forecast.
Working rule: Use Prophet per client when a reader needs its trend and seasonal settings as parameters they can read, and when its scored forecasts beat seasonal naive on the decision windows you chose before scoring. Set its interval width in code, and read its default 80% interval as an 80% interval, never as a 95% range. Check its coverage against the level it claims.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Run Prophet on the same cutoffs
The last section named Prophet's assumptions. This one fixes its settings and runs it on the same hours as every other model.
The comparison uses the three standard cutoffs, at 00:00 on Dec 29, Dec 30 and Dec 31, 2014. These are the windows Chapters 4 to 8 score. Each window's history is the 1,344 hours, eight weeks, before its cutoff, with the start rule applied, so a late starter's history begins at its first reading. Prophet is fitted once for each client and cutoff, which makes 90 fits.
We chose the holiday table on decision windows, not on the standard cutoffs. The ordinary decision windows hold no holiday in their scored hours, so they cannot test a holiday table at all, while the holiday decision windows can. So the choice uses both kinds. A run with the table and one without it shows what the table does on those windows for SUBSET8: the four ordinary days and seven holidays, Apr 18, Apr 20, Apr 25, May 1, Jun 10, Aug 15 and Dec 8, 2014. All of them come before the held-out week, and none has Christmas Day in its history.
1"""Prophet with and without the Portuguese holiday table, on the eleven decision windows for SUBSET8 (B-9, B-10),2before any standard cutoff is scored. The point that the table stays off rests on the two mean MASE lines and on3holiday table kept, which says whether the table lowers the mean per-client MASE."""4from lib.compare import prophet_windows, seasonal_naive, score5from lib.electricity import load6from lib.features import is_holiday7from lib.panel import DECISION_CUTOFFS, HOLIDAY_DECISION_CUTOFFS8from lib.quirks import mask_clock_changes9from lib.settings import SUBSET8, show1011# the four ordinary and seven holiday windows, in date order12DECISION = sorted([*DECISION_CUTOFFS, *HOLIDAY_DECISION_CUTOFFS])13raw = load(SUBSET8)14# clock-change hours set to missing, so Prophet fits without them15df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]16keys = ["unique_id", "ds"]17# one fit per client and window, with the Portuguese holidays as Prophet's holiday table; two processes, and the same18# fits at any count19with_holidays = prophet_windows(df, DECISION, use_holidays=True, workers=2)[keys + ["Prophet"]].rename(20 columns={"Prophet": "WithHolidays"})21without = prophet_windows(df, DECISION, use_holidays=False, workers=2)[keys + ["Prophet"]].rename(22 columns={"Prophet": "NoHolidays"})23forecasts = with_holidays.merge(without, on=keys, validate="one_to_one").merge(24 seasonal_naive(df, DECISION), on=keys, validate="one_to_one")25models = ["WithHolidays", "NoHolidays", "SeasonalNaive"]26# per_client: one MASE per client and model; scored: forecasts beside readings27per_client, scored = score(raw, forecasts, models, SUBSET8, DECISION)2829show("decision windows", len(DECISION))30show("holiday decision windows", len(HOLIDAY_DECISION_CUTOFFS))31show("clients", len(SUBSET8))32show("scored hours", len(scored))33# 1 for each hour on a Portuguese national holiday34show("holiday hours in the decision windows", int(is_holiday(scored["ds"]).sum()))35for model in models:36 # the mean of the per-client MASEs, not a pooled error37 show(f"mean MASE {model}", per_client[model].mean(), digits=3)38show("hours where the forecasts differ", int((scored["WithHolidays"] != scored["NoHolidays"]).sum()))39show("clients where holidays help", int((per_client["WithHolidays"] < per_client["NoHolidays"]).sum()))40# the table is kept only on a strictly lower mean; a tie keeps the simpler model, with the table off41show("holiday table kept", "yes" if per_client["WithHolidays"].mean() < per_client["NoHolidays"].mean() else "no")
python prophet-holiday-decision.pydecision windows: 11
holiday decision windows: 7
clients: 8
scored hours: 2112
holiday hours in the decision windows: 1344
mean MASE WithHolidays: 1.070
mean MASE NoHolidays: 1.061
mean MASE SeasonalNaive: 0.934
hours where the forecasts differ: 768
clients where holidays help: 2
holiday table kept: no
On the decision windows the table makes Prophet worse. Over the eleven of them, its mean per-client MASE is 1.070 with the table and 1.061 without it, against 0.934 for seasonal naive.
A setting is kept only if it lowers the decision-window MASE, and a tie keeps the simpler model. The example's verdict on keeping the table is no, so the table stays off.
The holiday windows hold 1344 client-hours of holiday, and they are the only place the table can change a forecast. 768 of the 2112 scored hours differ between the two variants.
The standard cutoffs tell a different story, which we report without using it to choose. Their histories include Christmas Day, and the holiday windows' histories do not. That is a likely reason for the gap, but we did not test which holiday causes it. With the table on, Prophet's mean per-client MASE on the standard cutoffs would be 1.27, against 1.35 with it off. A team that wants the table should test it on its own decision windows before it scores anything.
Daily and weekly seasonality are on, and yearly is off. These are the choices Prophet makes for itself on eight weeks of hourly data: yearly needs at least 730 days, weekly at least two weeks, and daily at least two days of sub-daily data (forecaster.py, lines 1187, 1209 and 1223). We set them explicitly and did not tune them. The interval is 95%, set in code and printed as Prophet interval level, so coverage sits beside the recordings' 95% ranges. Prophet draws its interval samples from numpy's global generator, so the code sets the seed (SEED) before each predict. The fit itself is deterministic.
Every model is scored on the book's one MASE scale: the seasonal naive error in sample, over every reading before Dec 29, 2014, with the clock-change hours masked. A MASE below 1 means the model's error on the scored hours is smaller than seasonal naive's in-sample error. Each client has its own scale, so a large client does not outweigh a small one. The next run fits Prophet for all thirty clients and scores it beside seasonal naive and the global model on this scale, which is the comparison every later result in the chapter uses.
1"""Prophet for each of the thirty clients on the three standard cutoffs, with a 95% interval, scored on the book's MASE2scale beside seasonal naive and the Chapter 6 global model. The point that Prophet's mean MASE is higher rests on3the mean MASE lines, and the coverage line says whether its 95% interval holds."""4import pandas as pd56from lib.compare import (PROPHET_HOLIDAYS, PROPHET_LEVEL, global_forecasts, prophet_windows, report_range,7 report_subset, score, seasonal_naive)8from lib.electricity import load9from lib.panel import ALL3010from lib.quirks import mask_clock_changes11from lib.scoring import report_scores12from lib.settings import EVAL_CUTOFFS, show1314raw = load(ALL30)15# clock-change hours set to missing, as chapter 3 sets them16df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]17cutoffs = pd.DatetimeIndex(EVAL_CUTOFFS)18# Prophet at the book's settings from lib.compare: holiday table off, 95% interval, one fit per client and cutoff19forecasts = prophet_windows(df, cutoffs).merge(20 seasonal_naive(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one").merge(21 global_forecasts(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one")22models = ["Prophet", "SeasonalNaive", "Global"]23# per_client: one MASE per client and model; scored: forecasts beside readings24per_client, scored = score(raw, forecasts, models, ALL30)2526show("Prophet interval level", PROPHET_LEVEL)27show("Prophet holiday table", "on" if PROPHET_HOLIDAYS else "off")28show("clients scored", scored["unique_id"].nunique())29show("scored hours", len(scored))30show("clients where Prophet beats seasonal naive", int((per_client["Prophet"] < per_client["SeasonalNaive"]).sum()))31show("clients where Prophet beats the global model", int((per_client["Prophet"] < per_client["Global"]).sum()))32report_scores(scored, per_client, models)33report_subset(per_client, models)34# the 95% range from Prophet's lower and upper interval columns35report_range(scored, "Prophet", "ProphetLo", "ProphetHi", PROPHET_LEVEL)36# Sensitivity only, not used to choose: the holiday table on the same cutoffs, which the decision windows37# did not favour.38# two processes; each fit is the same at any worker count39on = prophet_windows(df, cutoffs, use_holidays=True, workers=2)[["unique_id", "ds", "Prophet"]].rename(40 columns={"Prophet": "ProphetHolidays"})41per_holiday, _ = score(raw, forecasts.merge(on, on=["unique_id", "ds"], validate="one_to_one"),42 ["ProphetHolidays"], ALL30)43show("mean MASE Prophet with holidays, not chosen", per_holiday["ProphetHolidays"].mean())
python prophet-thirty-clients-backtest.pyProphet interval level: 95
Prophet holiday table: off
clients scored: 30
scored hours: 2160
clients where Prophet beats seasonal naive: 9
clients where Prophet beats the global model: 6
MAE Prophet: 182.44
MAE SeasonalNaive: 148.26
MAE Global: 122.81
RMSE Prophet: 764.88
RMSE SeasonalNaive: 780.48
RMSE Global: 678.81
mean MASE Prophet: 1.35
mean MASE SeasonalNaive: 1.18
mean MASE Global: 0.91
mean MASE Prophet SUBSET8: 1.39
mean MASE SeasonalNaive SUBSET8: 1.16
mean MASE Global SUBSET8: 1.03
coverage 95 Prophet: 0.918
interval score 95 Prophet: 932.66
mean MASE Prophet with holidays, not chosen: 1.27
On these cutoffs Prophet has a higher mean per-client MASE than seasonal naive: 1.35 over the thirty clients, against 1.18 for seasonal naive and 0.91 for the global model.
The loss is on most clients too, not only on average: Prophet beats seasonal naive on only 9 of the thirty clients, and the global model on 6.
Pooled over every scored hour, Prophet's RMSE is 764.88 kWh, against 780.48 for seasonal naive, so it makes fewer very large misses. It loses to seasonal naive on MAE and MASE, and to the global model on every measure. Its 95% interval covers 0.918 of the readings, and its interval score is 932.66. Coverage below 0.95 means the interval is too narrow for its stated level on these hours.
Working rule: Set Prophet's seasonality, holidays and interval width in code before you score it, and choose any of them on decision windows, never on the standard cutoffs. Keep the holiday table only if it lowers the MASE on the holiday decision windows. Compare Prophet with seasonal naive and the global model on the same windows and the same scale, and list the clients it loses. Set the seed before each predict, and check that a second run prints the same lines.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Load a recorded foundation model's forecasts
Prophet fits each client from its own eight weeks. A foundation model brings what it learned from other series, and this section shows how the book holds its forecasts.
A foundation model is trained once on many series and then applied to a context, with no fitting on that series. Chronos-2 is one such model. Chronos-2 is an encoder-only transformer that takes a context of past values and outputs quantiles for the next steps (Ansari et al. (2025); the earlier Chronos model is described in Ansari et al. (2024)). Its weights are published as amazon/chronos-2 on Hugging Face under the Apache-2.0 licence, not gated (model card).
Its pretraining covers this dataset, which limits how far its score can be read. The Chronos-2 report lists the electricity dataset in its pretraining corpus. Appendix A, Table 6 gives "Electricity", with 370 series in the energy domain (Ansari et al. (2025)). Those series match the UCI ElectricityLoadDiagrams20112014 dataset the book reads, and the book's thirty clients are drawn from them. The report notes partial overlap with some GIFT-Eval training portions. For strictly zero-shot results it points to a variant trained only on synthetic data, in its Section 5.4. We have not run that variant. The report does not show which readings of these clients the model saw, so the scores below are upper bounds, not out-of-sample results.
We do not run Chronos-2 in an example. Its forecasts are recorded: the model ran once, outside the examples, on a CPU, with the revision pinned to 29ec3766d36d6f73f0696f85560a422f50e8498c, and the examples read the saved output. The recording is dated Nov 10, 2026. Examples must run offline and give the same output every time, and a model call does neither. A downloaded model can change, and a reader's machine may not hold the files at all.
The input for each client and cutoff is the 1,344 hours before the cutoff, with the start rule applied. None of the standard cutoffs needs that step. Their histories begin on Nov 3, 2014 or later, after the autumn clock change, and the three late starters had their first readings before Nov 3, 2014. The manifest counts no missing values in the contexts, so the clock-change handling changes no scored number here. The package builds a mask from missing values and scales around them (chronos-forecasting source, chronos2/model.py, line 398). The 90 contexts went in one batch, with each series kept separate. Cross-learning, which would let one series inform another, is off by default, and the recording does not turn it on (chronos2/pipeline.py, line 833). So the Dec 30 context does not inform the Dec 29 forecast.
The output is, for each client, cutoff and hour, five quantiles for the next 24 hours: 0.025, 0.1, 0.5, 0.9 and 0.975. That is 2160 rows in all. The manifest lists the SHA-256 of the output and of inputs.csv.gz, which keeps the exact context each window was given, along with the package versions and the machine, described only as a 4-vCPU Linux container with no GPU.
Not every requested quantile is trained. The package is trained on 21 levels, from 0.01 to 0.99 (Ansari et al. (2025)). The levels 0.1, 0.5 and 0.9 are among them, so they are read directly. The levels 0.025 and 0.975 are not, so the package interpolates each one linearly between the two nearest trained levels: 0.01 and 0.05 for the lower, and 0.95 and 0.99 for the upper. The code is interpolate_quantiles in chronos/utils.py, called from predict_quantiles in chronos2/pipeline.py (chronos-forecasting source). The manifest names the interpolated levels, and a 95% range built from them is partly the package's interpolation.
Before any score reads the recording, run the check. It hashes each output and compares it with manifest.json, counts the rows, clients and cutoffs in the forecast file, and looks for quantiles that run out of order.
1"""Checks the recorded Chronos-2 outputs before any score reads them: each file's SHA-256 against manifest.json, the2shape of the forecast file, and any quantiles out of order, without running the model. The point that the recording3is intact rests on the hash mismatches line and the rows with quantiles out of order line."""4from lib.compare import check_recording5from lib.settings import show67for label, value in check_recording().items(): # the default name is the chronos-2 folder in examples/data/recorded/8 show(label, value)
python recorded-chronos2-check.pyoutputs checked: 2
hash mismatches: 0
rows: 2160
clients: 30
windows: 3
interpolated levels: 2
rows with quantiles out of order: 0
largest quantile crossing, kWh: 0.00
inverted 80 or 95 ranges: 0
The recording is intact: 0 of its 2 outputs fail their hashes, and 0 rows have quantiles out of order. The check verifies the output files, not the prompt hash, which covers the input and the configuration.
The licence decides which weights the book can run or record: this book uses only Apache-2.0 or MIT weights. Chronos-2's weights and the chronos-forecasting package are Apache-2.0 (model card; package metadata). Two other families were considered and are not used. The TimesFM 3.0 weights carry a non-commercial licence, which allows non-commercial and non-production use only (TimesFM 3.0 licence). TimesFM 2.5 is Apache-2.0 (TimesFM 2.5 card), so the licence depends on the version named. The Moirai R-series checkpoints are tagged CC-BY-NC-4.0, which is also non-commercial (Moirai 2.0 card).
A later edition can run Chronos-2 live once its weights and package fit the release service's limits. Until then the recording stands in for the live run.
Working rule: Record a foundation model only with its script, its pinned revision, its package versions and a manifest that hashes every output. Read the recorded outputs in every example, and check the hashes before any score reads them. Check the licence of the weights, not only the licence of the package, and name the model version in any licence sentence.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Score the recordings on the same cutoffs
The recordings mean something only when they are scored on the same hours as Prophet and the baselines.
A forecast with five quantiles can be scored in two ways. The median, the 0.5 quantile, is a point forecast, scored with MASE on the book's scale. The 0.1 and 0.9 quantiles form an 80% range, and the 0.025 and 0.975 quantiles form a 95% range. Each range is scored by coverage, the share of readings inside it, and by the interval score, which adds the width to a penalty for each reading outside it. The penalty is 2 divided by the nominal miss rate, 0.05 for a 95% range, times the distance outside. Chapter 8 defines the interval score with its source. Each quantile is also scored by its quantile loss, which charges tau for each kWh a reading falls above the quantile and one minus tau for each kWh below it (Chapter 8).
Two examples score the recordings, one for each model, on the same thirty clients, cutoffs and MASE scale as the Prophet run. Each rebuilds the global model and seasonal naive instead of copying them from Chapter 6, so every model goes through one scoring path.
1"""Scores the recorded Chronos-2 forecasts on the three standard cutoffs and the MASE scale of the other models: the2median is the point forecast, the 0.1 and 0.9 quantiles the 80% range, and the 0.025 and 0.975 quantiles the 95%3range. The point about the lead rests on the mean MASE lines, and the code reads the recording only, never the4model."""5import pandas as pd67from lib.compare import (global_forecasts, recorded_forecasts, report_pinball, report_range, report_subset, score,8 seasonal_naive)9from lib.electricity import load10from lib.panel import ALL3011from lib.quirks import mask_clock_changes12from lib.scoring import report_scores13from lib.settings import EVAL_CUTOFFS, show1415raw = load(ALL30)16# clock-change hours set to missing, as chapter 3 sets them17df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]18cutoffs = pd.DatetimeIndex(EVAL_CUTOFFS)19# the recording, read only after each output matches its manifest hash; its median is the Chronos column20forecasts = recorded_forecasts().merge(21 seasonal_naive(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one").merge(22 global_forecasts(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one")23models = ["Chronos", "SeasonalNaive", "Global"]24# per_client: one MASE per client and model; scored: forecasts beside readings25per_client, scored = score(raw, forecasts, models, ALL30)2627show("clients scored", scored["unique_id"].nunique())28show("scored hours", len(scored))29show("clients where Chronos beats seasonal naive", int((per_client["Chronos"] < per_client["SeasonalNaive"]).sum()))30report_scores(scored, per_client, models)31report_subset(per_client, models)32report_pinball(scored, "Chronos", "q10", 0.1) # quantile loss at tau 0.1: the 0.1 quantile column33report_pinball(scored, "Chronos", "q50", 0.5) # at tau 0.5 the loss is half the absolute error34report_pinball(scored, "Chronos", "q90", 0.9)35report_range(scored, "Chronos", "q10", "q90", 80)36report_range(scored, "Chronos", "q025", "q975", 95)
python recorded-chronos2-score.pyclients scored: 30
scored hours: 2160
clients where Chronos beats seasonal naive: 27
MAE Chronos: 100.13
MAE SeasonalNaive: 148.26
MAE Global: 122.81
RMSE Chronos: 655.39
RMSE SeasonalNaive: 780.48
RMSE Global: 678.81
mean MASE Chronos: 0.81
mean MASE SeasonalNaive: 1.18
mean MASE Global: 0.91
mean MASE Chronos SUBSET8: 0.87
mean MASE SeasonalNaive SUBSET8: 1.16
mean MASE Global SUBSET8: 1.03
quantile loss 0.1 Chronos: 39.44
quantile loss 0.5 Chronos: 50.07
quantile loss 0.9 Chronos: 17.85
coverage 80 Chronos: 0.749
interval score 80 Chronos: 572.89
coverage 95 Chronos: 0.944
interval score 95 Chronos: 958.19
1"""Scores the recorded NHITS forecasts on the same three cutoffs and MASE scale as Chronos-2 and the other models: the2median is the point forecast, and each range comes from levels trained directly, so none is interpolated. The point3about its ranges rests on the coverage and interval score lines, and the code reads the recording only, never the4model."""5import pandas as pd67from lib.compare import (global_forecasts, recorded_forecasts, report_pinball, report_range, report_subset, score,8 seasonal_naive)9from lib.electricity import load10from lib.panel import ALL3011from lib.quirks import mask_clock_changes12from lib.scoring import report_scores13from lib.settings import EVAL_CUTOFFS, show1415raw = load(ALL30)16# clock-change hours set to missing, as chapter 3 sets them17df = mask_clock_changes(raw)[["unique_id", "ds", "y"]]18cutoffs = pd.DatetimeIndex(EVAL_CUTOFFS)19# the NHITS recording, read only after each output matches its manifest hash; its median is the NHITS column20forecasts = recorded_forecasts("nhits", "NHITS").merge(21 seasonal_naive(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one").merge(22 global_forecasts(df, cutoffs), on=["unique_id", "ds"], validate="one_to_one")23models = ["NHITS", "SeasonalNaive", "Global"]24# per_client: one MASE per client and model; scored: forecasts beside readings25per_client, scored = score(raw, forecasts, models, ALL30)2627show("clients scored", scored["unique_id"].nunique())28show("scored hours", len(scored))29show("clients where NHITS beats seasonal naive", int((per_client["NHITS"] < per_client["SeasonalNaive"]).sum()))30report_scores(scored, per_client, models)31report_subset(per_client, models)32report_pinball(scored, "NHITS", "q10", 0.1) # quantile loss at tau 0.1: the 0.1 quantile column33report_pinball(scored, "NHITS", "q50", 0.5) # at tau 0.5 the loss is half the absolute error34report_pinball(scored, "NHITS", "q90", 0.9)35report_range(scored, "NHITS", "q10", "q90", 80)36report_range(scored, "NHITS", "q025", "q975", 95)
python recorded-nhits-score.pyclients scored: 30
scored hours: 2160
clients where NHITS beats seasonal naive: 24
MAE NHITS: 136.61
MAE SeasonalNaive: 148.26
MAE Global: 122.81
RMSE NHITS: 649.41
RMSE SeasonalNaive: 780.48
RMSE Global: 678.81
mean MASE NHITS: 1.03
mean MASE SeasonalNaive: 1.18
mean MASE Global: 0.91
mean MASE NHITS SUBSET8: 1.17
mean MASE SeasonalNaive SUBSET8: 1.16
mean MASE Global SUBSET8: 1.03
quantile loss 0.1 NHITS: 27.66
quantile loss 0.5 NHITS: 68.31
quantile loss 0.9 NHITS: 40.58
coverage 80 NHITS: 0.700
interval score 80 NHITS: 682.37
coverage 95 NHITS: 0.877
interval score 95 NHITS: 1250.94
| Model | Mean MASE, ALL30 | Mean MASE, SUBSET8 | MAE, ALL30 (kWh) |
|---|---|---|---|
| Seasonal naive | 1.18 | 1.16 | 148.26 |
| Global model (Chapter 6) | 0.91 | 1.03 | 122.81 |
| Chronos-2, median | 0.81 | 0.87 | 100.13 |
| NHITS, median | 1.03 | 1.17 | 136.61 |
| Prophet, point forecast | 1.35 | 1.39 | 182.44 |
Chronos-2 has the lowest mean per-client MASE of the five models on both sets of clients: 0.81 over the thirty clients and 0.87 on SUBSET8. It beats seasonal naive on 27 of the thirty clients. The global model from Chapter 6 is second on both sets. Chapter 4's fixed seasonal ARIMA, on the same eight clients, cutoffs and scale, scores 1.09 on SUBSET8, between the global model and seasonal naive.
NHITS is third on the thirty clients and fourth on SUBSET8, where it is narrowly behind seasonal naive, 1.17 against 1.16. Prophet has the highest mean on both sets.
The lead needs its caveat. Chronos-2's pretraining corpus includes this dataset (the recording section), so these readings may have been part of its training, and its score here is an upper bound on what to expect on readings it has not seen. NHITS has no such advantage, because each of its cutoffs was trained only on that cutoff's own context. NHITS trails Chronos-2 on the same hours, but the two are not on equal footing, and this chapter does not rank them on this comparison.
The median is also a check on the quantile arithmetic. The pinball loss at 0.5 is half the absolute error, so the median's loss should be half its MAE. For Chronos-2 the loss at 0.5 is 50.07, against the MAE of 100.13 kWh divided by two. For NHITS the loss at 0.5 is 68.31, against half its MAE of 136.61 kWh.
Chronos-2's 95% range covers 0.944 of the readings, close to its level, and its 80% range covers 0.749, a little under its level of 0.80. The Chronos-2 95% range is partly interpolated, so its score is for the model and the package together.
Prophet's 95% range has an interval score a little lower than Chronos-2's, 932.66 against 958.19, but its coverage of 0.918 sits further from its level.
NHITS's ranges are too narrow for their levels on these hours: its 80% range covers 0.700 of the readings, and its 95% range covers 0.877.
Its 95% interval score is the highest of the three 95% ranges, and its quantiles are trained directly, so no range is interpolated. The recording does not put its quantiles in order, though. The check counts 22 rows with an adjacent pair out of order, by at most 21.11 kWh, and 0 inverted 80% or 95% ranges. The scores use the values as recorded.
Three limits apply. These are three cutoffs in one week of one dataset, so the results are not a general claim about any model. Neither range was calibrated on the calibration windows that Chapter 8 used, so the coverage here is a check, not a calibration. The comparison makes no choice, because decisions in this book are made on decision windows, and the recordings have no forecasts there, so nothing is picked with these scores.
Working rule: Score a recorded forecast on the same windows, the same scale and the same readings as every other model, and name any interpolated quantile wherever its range is reported. Before you read a lead, check what the model was trained on, because a model that saw the scored readings cannot show skill on them. Do not choose a model or a setting on these windows. A recorded forecast with no forecasts on the decision windows cannot be chosen for production until it has them.
신경망 모델을 기록으로 남기는 이유
이번 판은 신경망 모델을 직접 실행하지 않는다. 두 개의 신경망 예측은 기록으로 남기며, 그 이유는 런타임 비용 때문이지 신경망 모델에 대한 평가 때문이 아니다.
NHITS는 neuralforecast 라이브러리가 제공하는 신경망 예측기이다. neuralforecast는 여러 시계열의 패널에 쓰는 신경망 예측기를 묶어 놓은 라이브러리이다. 이 라이브러리의 라이선스는 Apache-2.0이다(neuralforecast 메타데이터). NHITS는 Chronos-2를 기록한 방식과 같이 자체 스크립트, 고정된 버전, 매니페스트를 두고 기록하며, 같은 컨텍스트, 같은 컷오프, 같은 스케일, 같은 지표로 Chronos-2와 나란히 채점한다. 이 기록은 Chronos-2에 주어진 것과 같은 1,344시간짜리 컨텍스트 서른 개에서 컷오프마다 한 번씩 학습한다. 입력 윈도우는 168시간, 즉 1주이고 예측 구간은 24시간이다. 다섯 개의 수준은 MQLoss 손실로 직접 학습하며, 수준마다 출력이 하나씩 있다. 스케일러는 neuralforecast의 표준 스케일러이고 시드는 고정했으며, 실행은 CPU에서 한다. 각 컷오프의 모델은 그 컷오프의 컨텍스트만 보므로 컷오프 이후의 관측값은 모델에 들어가지 않는다. 가중치는 배포하지 않는다.
NHITS도 자체 매니페스트에 있는 해시와 대조하여 같은 검사를 받는다.
1"""Checks the recorded NHITS outputs before any score reads them: each file's SHA-256 against manifest.json, the shape2of the forecast file, and any quantiles out of order, without running the model. The point that the recording is3intact rests on the hash mismatches line; the interpolated levels line shows that every level is trained directly, and4the rows with quantiles out of order line is the one the chapter reports."""5from lib.compare import check_recording6from lib.settings import show78for label, value in check_recording("nhits").items(): # "nhits" is the NHITS folder in examples/data/recorded/9 show(label, value)
python recorded-nhits-check.pyoutputs checked: 1
hash mismatches: 0
rows: 2160
clients: 30
windows: 3
interpolated levels: 0
rows with quantiles out of order: 22
largest quantile crossing, kWh: 21.11
inverted 80 or 95 ranges: 0
NHITS 파일은 매니페스트와 일치하며, 각 수준을 직접 학습하므로 보간된 수준은 하나도 없다.
두 가지 비용 때문에 NHITS는 예제에서 빠진다. 이 라이브러리는 torch, ray, pytorch-lightning을 함께 가져오는데, 이 때문에 예제 환경이 687 MB에서 1.7 GB로 커진다. 또한 CPU 학습은 릴리스 서비스의 2 vCPU와 4 GiB 메모리에서 예제가 걸려야 할 시간보다 오래 걸린다. 버전 충돌은 관련이 없다. NHITS 요구 사항 파일 recordings/requirements-nhits.txt은 예제가 고정한 것과 같은 pandas 버전(2.3.3)으로 neuralforecast를 설치한다.
Chronos-2는 별도의 스택을 쓴다. 요구 사항 파일 recordings/requirements.txt은 transformers, torch와 함께 chronos-forecasting을 설치하며, 기록을 위해 pandas 3을 고정한다. 이 패키지가 pandas 3을 필요로 하는 것은 아니다. 패키지 메타데이터는 pandas 2.0 이상 4 미만을 허용한다(chronos-forecasting 소스, pyproject.toml). 예제는 두 패키지 중 어느 것도 임포트하지 않는다. 저장된 출력을 읽을 뿐이다.
릴리스 서비스의 제한에 맞는 런타임이 있거나 NHITS 전용 런타임이 있다면, NHITS는 이후 판에서 직접 실행할 수 있다. 어느 쪽이든 이 장의 윈도우, 스케일, 채점은 그대로 유지되므로, 이후의 모델도 같은 시간대에서 다른 모델과 나란히 채점된다.
작업 규칙: 의존성이 런타임 예산을 깨뜨릴 모델은 예제의 런타임에 넣지 말고 기록으로 남기며, 그 모델의 고정 사항은 별도의 요구 사항 파일에 둔다. 의존성을 추가하기 전에 깨끗한 환경에 설치해 보고, 그것이 크기와 패키지 수에서 얼마나 늘리는지 측정한다. 의존성 해석 결과가 고정된 패키지를 움직인다면, 그것은 모델을 기록으로 남겨야 할 이유이지 고정을 바꿔야 할 이유가 아니다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
What goes wrong in production
Prophet's holiday table covers 2013 and 2014 only. The book's runs leave it off, because the holiday decision windows gave no reason to turn it on, but the standard cutoffs show that it moves the result. A forecast made in 2015 with the table unchanged gets no holiday effect on later holidays, and the run reports nothing wrong. A team that turns the table on must extend it to the forecast's years. Holidays are known in advance, so the fix is to extend the table. Count the holiday hours in each window as well. A window that contains a holiday should show a count above zero, and a count of zero where a holiday falls is the sign to look for.
Interval width fails differently: a Prophet interval left at its default of 80% gets read as a 95% range by a reader who sees only the word "interval" in a report. The coverage line is the check. Printed beside its level, a coverage near 0.8 is correct for an 80% interval and a failure for a 95% one. Print the level with the coverage every time.
A recorded forecast goes stale in a different way. The recording covers three cutoffs. A day-ahead forecast in production needs one cutoff every day, and the release service does not run the model. Covering another cutoff means running the model, saving its output and hashing it, which is work outside the examples. The sign that the recording is out of date is a window with no rows in the file, which the check's window count shows.
A later model revision changes every number that depends on the recording, even when the code does not change. Treat the revision as an update trigger. Record it, score it on the same windows as the old one, and compare the two before the replacement recording is used. A package that changes its interpolation between versions changes the 95% range without touching the median, so the manifest's list of interpolated levels should be compared as well.
A recorded forecast can also break the order of its own quantiles. The NHITS recording does in 22 of its 2160 rows, by at most 21.11 kWh. A range built from a crossed pair would be read as inverted. The check counts the crossings, and a consumer should sort or reject such rows before it uses them.
Prophet's trend assumption fails when a client's demand moves to a different level, as it does after a change of contract or the loss of a large load. Prophet expects the trend to change as often as it did in the history, and its default interval does not cover the seasonal pattern. The sign is residuals that keep one sign across successive windows, and a coverage line that falls over them. The fix is a refit on a shorter history for that client, not a wider interval.
Working rule: Before any Prophet or recorded forecast is reported, check that the holiday table covers the forecast's years when the table is on, that the interval level is printed beside its coverage, that the recording has a window for every cutoff in use, that its quantiles are in order, and that the model revision matches the one scored. A failed check stops the report, and the report says which check failed.
이 장에서 내린 결정
- Prophet은
SUBSET8만이 아니라 서른 개 클라이언트 전체에서 실행한다. 서른 개를 모두 실행하면 Prophet을 같은 클라이언트에서 전역 모델과 나란히 놓을 수 있다.SUBSET8의 평균은 같은 실행에서 출력한 값이다. - Prophet의 구간은 95%이며, 코드에서 설정하고 출력한다. 그러면 Prophet의 커버리지와 구간 점수를 녹화 데이터의 95% 범위 옆에 놓을 수 있다. 기본값인 80%는 사용하지 않는다.
- Prophet의 계절성 플래그는 코드에서 고정한다. 일별 계절성과 주별 계절성은 켜고 연별 계절성은 끈다. 이는 8주 분량의 시간별 데이터에 대해 Prophet이 내리는 선택을 명시적으로 설정한 것이다.
- 휴일 표는 사용하지 않는다. 휴일 결정 윈도에서는 이 표가 평균 MASE를 높였으므로 더 단순한 모델을 유지했다. 표준 컷오프에서는 이 표가 Prophet의 평균 MASE를 낮췄을 것이며, 이 수치는 보고하되 선택에는 사용하지 않는다.
- Chronos-2와 NHITS는 실행하지 않고 기록한다. 예제는 저장된 출력을 읽고 해시를 확인할 뿐 모델을 호출하지 않는다. 예제는 오프라인에서 실행되어야 하고 매번 같은 출력을 내야 하기 때문이다.
- NHITS는 런타임 비용 때문에 기록한다. NHITS의 의존성을 설치하면 예제의 런타임이 687 MB에서 1.7 GB로 늘어나 릴리스 서비스의 2 vCPU와 4 GiB를 넘고, CPU 학습 실행은 예제에 적합한 시간보다 오래 걸린다.
- Chronos-2의 95% 범위는 보간한 두 분위수로 채점하며, 이 장은 그 사실을 밝힌다. NHITS의 분위수는 직접 학습한 것이므로 NHITS의 범위에는 보간한 것이 없다.
- Chronos-2의 우위는 상한으로 읽는다. Chronos-2의 사전 학습 코퍼스에 이 데이터셋이 포함되어 있으므로, 여기서의 점수는 표본 외 점수가 아니다.
- 선택은 결정 윈도에서만 내린다. 녹화 데이터에는 그 구간의 예측이 없으므로, 이 장은 녹화 데이터로는 어떤 선택도 하지 않는다.
이 판의 이 장
2026.11.11판예제 통과 2026년 11월 20이 장에서 6개 예제개 중 6개가 통과한다
영어판 2026.11.11판을 번역한 것이다.