用一个模型预测多个客户
阅读约需 76分钟 分钟,7节中含4个示例。
读完本章,你将得到一个覆盖全部三十个客户的全局模型,一条防止后期启用客户的前导零进入模型的起始规则,以及一个在相同截止点上逐客户将该模型与季节性朴素方法并列比较的评分。供应商的日前购电量是一个总量,由第 7 章根据三十个客户的预测汇总得出。按客户分别建模时,每个窗口都需要拟合三十次,每个客户一次。这些客户有共同的日内节律,因此用全部客户拟合的一个模型,只需学习一次这一节律。
第 5 章用SUBSET8中的八个客户拟合模型。改用三十个客户拟合同一个模型,会带来三项选择,它们决定评分是否可信:是否为模型提供客户标识符,如何处理三个后期启用的客户(这些客户的电表在数据中途才开始记录),以及如何在同一尺度上评分三十个预测。本章用四个示例展开。每个示例先输出计数,使你在查看任何误差之前,能够核对模型是否覆盖了应当覆盖的客户。
这部分尚未翻译,因此以英文显示。
One model for many clients
Montero-Manso and Hyndman (2021), in Principles and algorithms for forecasting groups of time series: Locality and globality, show that a global model is no more restrictive than a local one, because both can produce the same forecasts. They compare global methods, which fit one model across a group of series, with local methods, which fit each series on its own. A global model fits one set of parameters to many series in a single pass, so every client contributes to what it learns. Their abstract starts from the view that for groups of similar series, global methods outperform the more established local methods, and it notes that later results in that literature show good performance of global models even in heterogeneous datasets. It also reports that global linear models can give competitive accuracy with two orders of magnitude fewer parameters than local methods. The claims in this paragraph come from that abstract. Bojer and Meldgaard (2020) found that global ensemble models and gradient-boosted trees did well in six Kaggle forecasting competitions (arXiv:2009.07701). Those are not the supplier's data, so these sources are a reason to try the approach. They are not evidence about this case. The supplier's clients have the same daily shape and different sizes, so the test has to be run on them.
The data is held in long format: one row per client and hour, with three columns. unique_id names the client, ds is the hour, and y is the kilowatt hours used in that hour. Each row carries its own lags, the readings a day and a week earlier. Chapter 5 found that calendar and holiday features did not lower the error on these eight weeks of history, so this chapter uses lags only, at 24 and 168 hours. Both lags are at least the 24-hour horizon, so neither reads the day being forecast.
The lags cost rows. A lag of 168 hours reads a week back, so the first week of each client's history has no row with every lag available. Chapter 5 showed that MLForecast drops such rows from training by default. A client's training rows are therefore fewer than the hours in its history. The forecast itself is unaffected, because each forecast hour reads only readings from before its cutoff.
Clients differ in size, and a squared-error loss over raw kilowatt hours lets the largest set the splits, while the small ones carry little weight. So each client's target is scaled before the fit. The scale is LocalStandardScaler from mlforecast 1.1.0. It gives each client its own mean and standard deviation, and the model learns the scaled values:
z = (y - mean) / standard deviation
The mean and deviation come from that client's history before the cutoff only, and the forecasts are converted back with the same two numbers. The library's source shows where this happens. MLForecast.fit reaches TimeSeries._fit in mlforecast/core.py, which fits the scaler on each client's history before the cutoff. That history includes the first week, whose rows the lags then drop from training. TimeSeries.predict in the same file inverts the scaling. The class is _BaseLocalScaler in mlforecast/target_transforms.py. Chapter 5 compared the scaled and unscaled targets on SUBSET8, and the unscaled model did worse than seasonal naive. This chapter keeps the scaled target and does not repeat that comparison.
make_forecaster in the shared module examples/lib/features.py builds one MLForecast with one LGBMRegressor, set by LGBM_PARAMS. The model is deterministic, uses one thread and a fixed seed, and runs under LightGBM 4.7.0. The module examples/lib/panel.py fits that forecaster once per window, on the panel's history before the window's cutoff. It fills any clock-change hour from that same history, as Chapter 3 does, and returns a forecast for every client.
Working rule: Fit one global model when your series share a shape and you must forecast many of them on each window. Scale each series' target with statistics from that window's history only, use only lags at least as long as the horizon, and fit once per window, not once per series. Then score the model per series against a baseline on the same windows, as the sections below do, before you trust its pooled error.
这部分尚未翻译,因此以英文显示。
Identify each client
Once the rows share one table, the model no longer knows which client a row belongs to, unless you tell it. Scaling removes each client's mean and spread, so what remains is mostly the shape of the day. A client identifier puts the client in the table as a static categorical feature. The trees can then split on the client and learn that client's own habits. The cost is one more split to learn for each client, and a client with no history has nothing to learn from. A new client therefore needs a refit before the model can forecast it.
The rule for keeping an identifier is the one Chapter 5 sets for every feature: keep it only if it lowers the mean per-client MASE on decision windows that the standard cutoffs do not use. The held-out week runs from Dec 25 to Dec 31, 2014, and it has seven daily cutoffs. The three standard cutoffs, Dec 29 to Dec 31, 2014, are the reported scores, so the decision does not use them, and it does not use Dec 25 to Dec 28, 2014 either. The decision windows are four ordinary days before the held-out week: cutoffs on Nov 25, Nov 26, Dec 2 and Dec 3, all in 2014, each forecasting 24 hours. None of their scored hours falls on a Portuguese holiday. The one holiday of 2014 between these windows and the held-out week, Dec 8, 2014, is not among them. The calendar lists no holiday on Dec 1, 2014.
Before the identifier goes into the model, test it on windows the reported scores never touch. The check fits the model twice on each decision cutoff, with and without the identifier, and refuses to start if a scored hour is a holiday or in the held-out week.
1"""Tests whether a client identifier lowers the global model's error on four ordinary decision windows, before the2held-out week, on the book's MASE scale. The point rests on the mean per-client MASE lines and on `identifier kept`."""3from lib.electricity import load4from lib.panel import ALL30, DECISION_CUTOFFS, check_decision_hours, mae_change, panel_windows5from lib.quirks import mask_clock_changes6from lib.scoring import actual_hours, book_scale, report_scores, score_windows7from lib.settings import show89raw = load(ALL30) # all thirty clients, not the eight of SUBSET810df = mask_clock_changes(raw)[["unique_id", "ds", "y"]] # clock-change hours are set to missing, so the scores leave them out11cutoffs = DECISION_CUTOFFS12actual = actual_hours(df, cutoffs)13holiday_hours = check_decision_hours(actual) # raises before any fit if a scored hour is a holiday or in the held-out week1415plain = panel_windows(df, cutoffs, client_id=False, name="plain") # the model without the identifier16identified = panel_windows(df, cutoffs, client_id=True, name="identifier") # the same model, with the client as a feature17models = ["plain", "identifier"]18# the MASE scale reads only the readings before the first decision cutoff, so the scored hours cannot leak into it19scored, per_client = score_windows(actual, plain.merge(identified, on=["unique_id", "ds"], validate="one_to_one"),20 models, book_scale(raw, cutoffs[0]))21lowers = per_client["identifier"] < per_client["plain"]22size = raw.groupby("unique_id")["y"].mean().sort_values(ascending=False) # mean kWh per client, largest first23contrib = mae_change(scored, "plain", "identifier") # each client's share of the pooled MAE change, kWh; the shares sum to the total2425show("first decision cutoff", f"{cutoffs[0]:%Y-%m-%d}")26show("last decision cutoff", f"{cutoffs[-1]:%Y-%m-%d}")27show("clients", scored["unique_id"].nunique())28show("windows", len(cutoffs))29show("scored hours", len(scored))30show("decision hours on a holiday", holiday_hours)31report_scores(scored, per_client, models)32show("clients where the identifier lowers MASE", int(lowers.sum()))33show("of the 15 largest clients, where the identifier lowers MASE", int(lowers[size.index[:15]].sum()))34show("of the 15 smallest clients, where the identifier lowers MASE", int(lowers[size.index[-15:]].sum()))35show("largest client", size.index[0])36show("pooled MAE change from the largest client, kWh", contrib[size.index[0]])37show("pooled MAE change, all clients, kWh", contrib.sum())38# the decision rule keeps the identifier only if the mean per-client MASE is lower, not the pooled MAE39show("identifier kept", "yes" if per_client["identifier"].mean() < per_client["plain"].mean() else "no")
python client-identifier-check.pyfirst decision cutoff: 2014-11-25
last decision cutoff: 2014-12-03
clients: 30
windows: 4
scored hours: 2880
decision hours on a holiday: 0
MAE plain: 70.72
MAE identifier: 66.81
RMSE plain: 316.50
RMSE identifier: 303.68
mean MASE plain: 0.77
mean MASE identifier: 0.75
clients where the identifier lowers MASE: 23
of the 15 largest clients, where the identifier lowers MASE: 10
of the 15 smallest clients, where the identifier lowers MASE: 13
largest client: MT_370
pooled MAE change from the largest client, kWh: -3.80
pooled MAE change, all clients, kWh: -3.92
identifier kept: yes
The identifier lowers the mean per-client MASE, from 0.77 to 0.75, and it lowers the MASE of 23 of the thirty clients.
On these windows the pooled MAE falls too, from 70.72 kWh to 66.81 kWh. The two errors weigh clients differently, because the pooled MAE is weighted toward the largest.
The pooled change of -3.92 kWh is almost all from one client, MT_370, which accounts for -3.80 kWh of it.
The per-client MASE gives every client the same weight, and the identifier helps across sizes. It lowers the MASE of 10 of the fifteen largest clients and of 13 of the fifteen smallest. It does not help only the large clients.
The printed verdict is yes, so the identifier is kept and the final model uses it. The margin is small. Other windows could give the opposite sign by a similar small amount, so the verdict is close and depends on the windows. A reader with other windows should measure it again before trusting it.
The identifier has a cost at the standard cutoffs. There, the final model's mean per-client MASE is 0.91 with the identifier, against 0.98 without it, but its pooled MAE is 122.81 kWh, against 119.03 kWh without it. Chapter 7 has to check its total against that. The rest of the chapter uses the model with the identifier.
Working rule: Treat a client identifier as one more feature under Chapter 5's rule: keep it only if it lowers the mean per-client MASE on the decision windows, with the pooled MAE beside it. If the decision is close, say so, and measure it again on other windows. If the identifier wins, write down its cost: a new client has no learned split until it has history, and the model must be refitted before it can forecast that client.
这部分尚未翻译,因此以英文显示。
Apply the start rule to every client
A series begins at its first reading. Before that, the meter reports zeros, and those zeros are not low use. Chapter 4 met this for one client. With thirty clients, three are late starters. MT_039 first reads on 2013-10-03, MT_179 on 2013-12-20, and MT_115 on 2014-01-14. A model fitted on their leading zeros learns a level near zero for them. The start rule drops every reading before a client's first reading. It finds that reading in the rows before the cutoff only.
The rule refuses a cutoff in two cases only: a client has no non-zero reading before the cutoff, or it has fewer than 48 hours of history since its first reading. It does not skip the client, because a skipped client leaves the total short. The rule does not catch a gap or a run of zeros in the middle of a series, and a meter replaced under the same id passes too, because its zeros are not leading. The scale catches a gap, because it refuses a series that is not one unbroken hourly series. Nothing in the forecast checks for one, so a gap there shows only as fewer training rows.
The leak check asks whether the rule reads only the past and refuses the right cutoffs. It runs nine cutoffs, refused or not, and poisons each from the cutoff on, once with zeros and once with a value no meter could record. The history must not change, and every accepted cutoff must hold all thirty clients.
1"""Checks that the start rule reads only rows before each cutoff, refuses the cutoffs the data's flag predicts, and2removes only leading zeros, on nine cutoffs. The same checks run on a deliberately broken history function, and each3must fail there; the point rests on the refused, changed and missing counts. The checks are start_rule_checks in lib/panel.py."""4import pandas as pd56from lib.electricity import load7from lib.panel import ALL30, FEBRUARY_CUTOFFS, leaky_history, panel_history, start_rule_checks8from lib.settings import EVAL_CUTOFFS, show910raw = load(ALL30)11# nine cutoffs: three around MT_179's and MT_115's first readings (refused), the February cutoffs and the standard cutoffs12CHECKED = pd.DatetimeIndex(["2013-12-15", "2014-01-01", "2014-01-15", *FEBRUARY_CUTOFFS, *EVAL_CUTOFFS])131415def report(prefix, r):16 show(f"{prefix}cutoffs checked", len(CHECKED))17 show(f"{prefix}cutoffs refused by the start rule", r["refused"])18 show(f"{prefix}cutoffs expected to be refused, from the flag", r["expected"])19 show(f"{prefix}cutoffs where the refusal differs from the flag", r["mismatch"])20 show(f"{prefix}cutoffs accepted", r["accepted"])21 show(f"{prefix}accepted cutoffs with a client missing from the history", r["missing"])22 show(f"{prefix}cutoffs whose history changes when the future is poisoned", r["changed"])232425# the book's history function, with the count of rows the rule removes switched on26real = start_rule_checks(raw, CHECKED, panel_history, removed_rows=True)27report("", real)28show("rows the start rule removes, accepted cutoffs", real["removed"])29show("removed rows that are not zero", real["removed_nonzero"])30# the deliberately broken history, which reads each first reading from the whole series, must fail every check31report("broken version, ", start_rule_checks(raw, CHECKED, leaky_history))
python start-rule-leak-check.pycutoffs checked: 9
cutoffs refused by the start rule: 3
cutoffs expected to be refused, from the flag: 3
cutoffs where the refusal differs from the flag: 0
cutoffs accepted: 6
accepted cutoffs with a client missing from the history: 0
cutoffs whose history changes when the future is poisoned: 0
rows the start rule removes, accepted cutoffs: 2064
removed rows that are not zero: 0
broken version, cutoffs checked: 9
broken version, cutoffs refused by the start rule: 1
broken version, cutoffs expected to be refused, from the flag: 3
broken version, cutoffs where the refusal differs from the flag: 2
broken version, cutoffs accepted: 8
broken version, accepted cutoffs with a client missing from the history: 2
broken version, cutoffs whose history changes when the future is poisoned: 2
The real rule passes all three checks. Its refusals match the data's own flag, which uses each first reading and the 48-hour rule: the rule refuses 3 of the nine cutoffs, the flag predicts 3, and 0 cutoffs differ from it.
Poisoning the future changes no history. Its history changes at 0 cutoffs, and 0 accepted cutoffs are missing a client.
The rule removes leading zeros and nothing else. Across the accepted cutoffs it drops 2064 rows, and 0 of them are not zero.
The same checks run on a deliberately broken version, which reads each first reading from the whole series, the leak the checks are written for. It fails each of them. Its refusals differ from the flag at 2 cutoffs, its history changes at 2, and 2 accepted cutoffs drop a client without a message. So a pass on the real rule means something.
The model runs with the start rule and without it, on the standard cutoffs and then the February cutoffs. The counts that differ, and each run's error, show what the rule does.
1"""Runs the global model with the start rule and without it, on the February cutoffs, where a late starter's zeros enter2the history, and on the standard cutoffs, where the rule changes no row. The point rests on the February mean MASE lines3and the count of forecast hours that differ; the scale lines compare the book's scale with a labelled counterfactual."""4import numpy as np5import pandas as pd67from lib.electricity import load8from lib.panel import (ALL30, FEBRUARY_CUTOFFS, IDENTIFIER_KEPT, LATE_STARTERS, late_starter_facts, mae_by_group,9 mase_by_run, scale_facts, with_and_without_start_rule)10from lib.quirks import mask_clock_changes11from lib.settings import EVAL_CUTOFFS, show1213raw = load(ALL30)14df = mask_clock_changes(raw)[["unique_id", "ds", "y"]] # clock-change hours are set to missing, so the scores leave them out15for label, value in late_starter_facts(raw, df).items():16 show(label, value)1718standard = pd.to_datetime(EVAL_CUTOFFS) # the three standard cutoffs19# two runs of the same model on the same cutoffs, differing only in their histories20scored = with_and_without_start_rule(df, standard, client_id=IDENTIFIER_KEPT)21# compared with isclose, so a rounding difference between two runs does not count as a changed forecast22show("forecast hours that differ, standard", int((~np.isclose(scored["start"], scored["raw"])).sum()))2324# the February cutoffs, where the zeros of MT_115 and MT_179 enter the history25feb = with_and_without_start_rule(df, FEBRUARY_CUTOFFS, client_id=IDENTIFIER_KEPT)26show("scored hours, february", len(feb))27show("forecast hours that differ, february", int((~np.isclose(feb["start"], feb["raw"])).sum()))28for model in ["start", "raw"]:29 for group, value in mae_by_group(feb, model).items():30 show(f"MAE {group}, {model}, february", value)31# both runs on one scale, read from the first February cutoff32feb_mase = mase_by_run(raw, feb, ["start", "raw"])33show("clients in MASE, february", int(feb_mase["start"].notna().sum()))34for uid in LATE_STARTERS:35 show(f"MASE {uid}, start, february", feb_mase.loc[uid, "start"])36 show(f"MASE {uid}, raw, february", feb_mase.loc[uid, "raw"])37for run in ["start", "raw"]:38 show(f"mean MASE, {run}, february", feb_mase[run].mean())3940# 'counterfactual raw' keeps the zeros before each first reading, a scale the book does not use; 'start rule' is the book's41for label, value in scale_facts(raw, standard, IDENTIFIER_KEPT).items():42 show(label, value)
python global-model-raw-starts.pyfirst reading MT_039: 2013-10-03
first reading MT_115: 2014-01-14
first reading MT_179: 2013-12-20
zero share of history MT_115, 2014-02-01: 68
zero share of history MT_179, 2014-02-01: 23
zero share of history MT_115, 2014-02-15: 43
zero share of history MT_179, 2014-02-15: 0
zero share of history MT_115, 2014-02-28: 20
zero share of history MT_179, 2014-02-28: 0
forecast hours that differ, standard: 0
scored hours, february: 2160
forecast hours that differ, february: 2160
MAE late starters, start, february: 4.60
MAE other clients, start, february: 77.53
MAE MT_039, start, february: 4.64
MAE MT_115, start, february: 4.62
MAE MT_179, start, february: 4.54
MAE late starters, raw, february: 4.60
MAE other clients, raw, february: 77.95
MAE MT_039, raw, february: 4.57
MAE MT_115, raw, february: 4.68
MAE MT_179, raw, february: 4.53
clients in MASE, february: 30
MASE MT_039, start, february: 0.78
MASE MT_039, raw, february: 0.77
MASE MT_115, start, february: 0.90
MASE MT_115, raw, february: 0.91
MASE MT_179, start, february: 0.54
MASE MT_179, raw, february: 0.54
mean MASE, start, february: 0.70
mean MASE, raw, february: 0.71
MASE scale MT_039, start rule: 5.21
MASE scale MT_039, counterfactual raw: 3.28
MASE scale MT_115, start rule: 4.58
MASE scale MT_115, counterfactual raw: 2.22
MASE scale MT_179, start rule: 5.91
MASE scale MT_179, counterfactual raw: 3.04
mean MASE global, counterfactual raw scale, standard cutoffs: 1.01
mean MASE global, book scale, standard cutoffs: 0.91
At the standard cutoffs the rule matters less than the late starters might suggest. The three standard cutoffs have 8 weeks of history, reaching back to early November, and no client's history in that window holds a zero from before its first reading. So the rule changes no training row there, and 0 of the 2160 forecast hours change. The rule still changes the score through the MASE scale.
The February cutoffs, which Chapter 4 used for MT_115, are where the zeros enter the history. MT_115's history is 68 percent zeros on Feb 1, 2014, 43 percent on Feb 15 and 20 percent on Feb 28. MT_179's history is 23 percent zeros on Feb 1 and none after.
Here the forecasts change, because one model learns from all thirty clients at once: 2160 of the 2160 forecast hours in these three windows differ. Both runs are scored on the one book scale, read from Feb 1, 2014, so only the histories differ. The mean MASE is 0.70 with the rule and 0.71 without it.
Each late starter's MASE moves very little, and the direction is mixed. MT_039 scores 0.78 with the rule and 0.77 without it, and MT_115 scores 0.90 with the rule and 0.91 without.
MT_179 scores 0.54 with the rule and 0.54 without it, the same to two decimals.
The other 27 clients' MAE moves from 77.95 kWh without the rule to 77.53 kWh with it. The effect is small and mixed, and three windows are too few to call it. We keep the rule because the zeros are not demand, and because the scale depends on it.
You could instead mark the zeros as missing, as the clock rule does for the two clock-change hours in Chapter 3. That would be the wrong fix for this data. A zero before a first reading is the meter being silent, not a reading that is missing, and a missing value would still be a row the model learns from. The start rule removes the rows instead, and its check is simple to state: it removes only rows before the first reading, and every one of them is zero.
Working rule: Apply the start rule to every series that has leading zeros, and find each first reading from the rows before the cutoff. When the rule refuses a cutoff, stop the run and name the client and the cutoff. Do not fill the zeros, and do not publish a panel with a client missing. Judge the rule by the scale and by the windows where zeros enter the history, and expect the MAE effect to be small and mixed.
这部分尚未翻译,因此以英文显示。
Score thirty clients on the same cutoffs
The three standard cutoffs are Dec 29 to Dec 31, 2014, each at 00:00 on that day, with 24 hours forecast from each. For thirty clients that is 72 scored hours per client. Seasonal naive is the baseline Chapter 2 set. It takes each cutoff's own history, the 8 weeks before it, and copies the last day's readings forward. At the standard cutoffs no client's history holds a leading zero, so the global model and seasonal naive read the same readings. The comparison is fair because the inputs match, not because the methods are alike.
The headline is the mean per-client MASE, the measure Chapter 2 defines. The book uses one scale per client per backtest, the rule Chapter 2 sets, and the function book_scale in examples/lib/scoring.py builds it. The scale is the mean absolute change between each reading and the reading one day earlier, taken over every reading before the first scored cutoff. A late starter is counted from its first reading, and the clock-change readings are masked out. A client with no non-zero reading before the cutoff has no scale, so it is left out of MASE and reported, not treated as an error. Because the scale reads only readings before the scored windows, it is known at every cutoff and cannot leak. Each client counts once in the mean, so one large client cannot set the headline. The pooled MAE and RMSE follow as second lines. They show the error in kilowatt hours that the supplier pays, and they are weighted toward the large clients.
The scale is part of the score. MT_115's scale is 2.22 kWh if the zeros before its first reading are kept, and 4.58 kWh with the book's rule, which counts it from its first reading.
MT_039 and MT_179 move the same way. MT_039's scale rises from 3.28 to 5.21 kWh, and MT_179's from 3.04 to 5.91 kWh.
On the standard cutoffs the same forecasts score a mean MASE of 1.01 with the scale that keeps the zeros, and 0.91 with the book's scale. That difference is the scale, not the model. A score computed with a different scale rule is not comparable with one computed with the book's rule, so every score should record the rule it used.
The score below sets the global model beside seasonal naive on the three standard cutoffs.
1"""Scores one global LightGBM model, with the identifier that client-identifier-check kept, against seasonal naive on the2three standard cutoffs, per client on the book's MASE scale. The point rests on the mean per-client MASE lines; the model3without the identifier is printed for reference only."""4import pandas as pd5from statsforecast import StatsForecast6from statsforecast.models import SeasonalNaive78from lib.backtest import forecast_windows9from lib.baselines import SEASON10from lib.electricity import load11from lib.panel import ALL30, IDENTIFIER_KEPT, panel_windows12from lib.quirks import mask_clock_changes13from lib.scoring import actual_hours, book_scale, left_out_facts, report_scores, score_windows14from lib.settings import EVAL_CUTOFFS, show1516raw = load(ALL30)17df = mask_clock_changes(raw)[["unique_id", "ds", "y"]] # clock-change hours are set to missing, so the scores leave them out18cutoffs = pd.to_datetime(EVAL_CUTOFFS) # the three standard cutoffs, Dec 29 to Dec 31, 201419actual = actual_hours(df, cutoffs)2021# seasonal naive from each cutoff's own history: the baseline of Chapter 222naive = forecast_windows(StatsForecast(models=[SeasonalNaive(season_length=SEASON)], freq="h", n_jobs=1), df, cutoffs)23global_fc = panel_windows(df, cutoffs, client_id=IDENTIFIER_KEPT) # the identifier setting that client-identifier-check kept24plain_fc = panel_windows(df, cutoffs, client_id=False, name="plain") # the same model without the identifier, for reference25forecasts = naive.merge(global_fc, on=["unique_id", "ds"], validate="one_to_one").merge(26 plain_fc, on=["unique_id", "ds"], validate="one_to_one")27models = ["SeasonalNaive", "global", "plain"]28# the book's scale, read from the readings before the first cutoff only29scored, per_client = score_windows(actual, forecasts, models, book_scale(raw, cutoffs[0]))30beats = per_client["global"] < per_client["SeasonalNaive"] # per client: True where the global model's MASE is lower3132show("clients scored", scored["unique_id"].nunique())33show("windows", len(cutoffs))34show("scored hours", len(scored))35# a client with no non-zero reading before the first cutoff is left out of MASE, and is not an error36for label, value in left_out_facts(raw, df, cutoffs[0]).items():37 show(label, value)38show("clients in MASE", per_client["global"].notna().sum())39show("clients where global beats seasonal naive", int(beats.sum()))40show("clients where seasonal naive wins", ", ".join(sorted(per_client.index[~beats])))41report_scores(scored, per_client, models)
python global-model-thirty-clients.pyclients scored: 30
windows: 3
scored hours: 2160
clients left out of MASE: 0
clients with no reading before 2013-12-15, check: 2
clients with any row before 2013-12-15, broken count: 30
clients in MASE: 30
clients where global beats seasonal naive: 23
clients where seasonal naive wins: MT_001, MT_128, MT_141, MT_179, MT_217, MT_268, MT_294
MAE SeasonalNaive: 148.26
MAE global: 122.81
MAE plain: 119.03
RMSE SeasonalNaive: 780.48
RMSE global: 678.81
RMSE plain: 657.63
mean MASE SeasonalNaive: 1.18
mean MASE global: 0.91
mean MASE plain: 0.98
The example scores 30 clients on 3 windows, 2160 scored hours, and it counts 0 clients left out of MASE, which is zero: every client has a non-zero reading before the first cutoff, so every client is in the MASE table.
The count is only as good as its rule. At a cutoff of Dec 15, 2013, 2 clients have no reading yet, and counting any row before the cutoff would give 30, which reports no gap.
One global model beats seasonal naive on these cutoffs, for most clients and in total: its mean per-client MASE is 0.91 against 1.18, and it wins for 23 of the thirty clients. It loses for MT_001, MT_128, MT_141, MT_179, MT_217, MT_268, MT_294, and those clients are the first place to look when the model is refreshed.
The pooled figures point the same way. The MAE is 122.81 kWh against 148.26 kWh, and the RMSE is 678.81 kWh against 780.48.
The model without the identifier, printed for reference, has a mean MASE of 0.98 and an MAE of 119.03 kWh, which is the trade-off the identifier section describes.
The printed figures are means and counts, and a mean can hide one client that is far worse than the rest. When a figure changes, open the per-client table first.
The chapter does not explain why seasonal naive wins for those clients, and that is a reason not to report only the pooled figure. The result covers three windows in one week of one season, so it describes these cutoffs, not every day of the year. The longer test is the seven-window held-out week, which the rolling example in Chapter 10 uses.
Working rule: Score a global model per client on the same cutoffs as the baseline, with the MASE scale fixed before the first scored cutoff. Report the mean per-client MASE beside seasonal naive, and print the counts of clients and scored hours so a reader can check them. State any client left out of MASE and why. Read the pooled MAE and RMSE as the cost in kilowatt hours, not as the measure of model quality.
这部分尚未翻译,因此以英文显示。
Budget the run time
A backtest costs fits, and the count is what to check before you run one. A per-client model is fitted once for each client, each window and each method, so the count is clients times windows times methods. Thirty clients over three windows is ninety fits for one per-client method. A global model is fitted once for each window and method, however many clients it covers, so the same three windows cost three fits. More clients make each global fit larger, but they do not add fits.
The book sets a subset rule for the expensive per-client methods. SUBSET8 is the eight-client subset that Chapter 4 runs its per-client statistical models on. A global model covers all thirty in one fit, so it runs on all thirty clients, as ALL30. The subsets are fixed for the whole book, so a number in one chapter can be set beside the next. The window count follows the same logic. Expensive methods use three windows, and cheap methods can use more. Each window of the identifier check costs two global fits, one with the identifier and one without, so its four windows cost eight fits.
The two kinds of cost move differently when the number of clients grows. A per-client method gets more fits, one for each new client. A global model gets bigger fits, because each one reads more rows, and it keeps the same number of fits.
The book does not quote timings. The examples run with one thread, so their output repeats on the machine that recorded the book's outputs, and that choice costs run time. A time depends on the machine, and a reader's machine may differ. Count the fits, not the seconds.
Working rule: Before you run a backtest, count its fits: clients times windows times methods for a per-client method, and windows times methods for a global one. If the count is over budget, cut the subset or the window count, never the scored windows, and write the cut down beside the score. Then a reader can compare like with like.
生产中会出什么问题
运行可能会中止。如果某个客户在某个截止时间点之前没有非零读数,或者自其首次读数以来的历史不足 48 小时,起点规则就会拒绝该截止时间点。该规则会抛出错误,而不是跳过这个客户,因为跳过客户会使总量偏小,而第 7 章正是通过对各客户求和得到总量的。在生产中,运行应当停止,指明客户和截止时间点,并等待人工处理。泄漏检查在本书的数据中显示了同样的拒绝情形。
一个模型驱动所有客户。在 2 月的截止时间点,后期起步客户的零值改变了 2160 个预测小时,其他 27 个客户的 MAE 也随之变化,正如起点规则一节所示。对某个客户的历史、某条规则或某个特征所做的任何改动,都会改变整个面板的预测。任何改动之后,都要检查每一个客户,而不只是检查被改动的那个。
尺度是得分的一部分。同样的预测,其平均 MASE 可能是 1.01,也可能是 0.91,这取决于尺度是否保留首次读数之前的零值。评分一节给出了这些尺度。每次记录得分时都要同时记录这条规则,并把规则的变更视为得分的变更。
决策可能会漂移。标识符的决策依据的是位于留出周之外的四个普通窗口,其结论十分接近。当客户加入、离开或规模发生变化时,应当重新做出该决策。
缺失的客户不会在表中表现为错误。由二十九个客户组成的面板仍然是一张有效的表,对它的预测看起来仍然像预测。如果一次运行评分的客户数或小时数少于预期,或者全局模型优于季节性朴素方法的客户数在两次运行之间急剧变化,就应当停止。如果某个客户的单客户 MASE 骤然跳变,而汇总的 MAE 保持不变,也应当停止。
**工作准则:**每次刷新时,在读取任何得分之前,先打印并核对各项计数(客户数、窗口数、已评分小时数、被拒绝的截止时间点数、未纳入 MASE 的客户数)。然后在季节性朴素方法旁报告各客户 MASE 的均值。遇到任何拒绝就停止运行,并且不要发布由不完整面板构成的总量。
本章所做的决定
第 7 章从本章移交的结果出发:每个标准截断点各有三十个客户预测,由同一个模型在应用客户标识符和起始规则后生成,并按本书统一的 MASE 尺度评分。标识符的保留由 client-identifier-check 中的决定确定,examples/lib/panel.py 在 IDENTIFIER_KEPT 中保存了这一结论。在这些截断点上,使用标识符后的汇总误差反而更差,因此第 7 章在依赖客户预测之前,应先核对总量。
- 一个全局 LightGBM 模型覆盖全部三十个客户,滞后阶数为 24 和 168,不使用日历或节假日特征,每个客户的目标值仅依据该窗口的历史数据进行缩放,这正是第 5 章所确定的缩放方式。
- 保留客户标识符。该决定使用了四个普通窗口,时间为 2014 年 11 月 25 日至 12 月 3 日,位于留出的那一周之外,且不含葡萄牙的节假日,这符合本书选择设置的规则。在这些窗口上,平均 MASE 由 0.77 变为 0.75。差距很小,如标识符一节所示。
- 起始规则适用于每个客户。对于无法起始的截断点,它会拒绝该截断点,而不是跳过该客户。
- 每份面板历史中的时钟调整小时,仅依据该历史数据进行填补,因此跨越时钟调整的滚动窗口可以拟合,而不会被拒绝。第 10 章的滚动窗口依赖这一点。
- 主要结果是全部三十个客户的逐客户 MASE 均值,采用本书统一的尺度(在第 2 章中设定,由
book_scale计算),并在相同截断点上与季节性朴素方法并列。截断点之前没有非零读数的客户不计入 MASE,并会单独报告。汇总的 MAE 和 RMSE 作为次要结果列于其后。在标准截断点上没有客户被排除,因为每个客户在第一个截断点之前都有读数。 - 模型以单线程和固定随机种子运行,因此在记录本书输出的那台机器上,其输出可以重现。本书尚未在其他机器上测试。
本章,在本版次中
版次 2026.11.11示例于 2026年11月20日 通过本章 4个示例 个示例中有 4 个通过
译自英文版图书,版次 2026.11.11。