Realistic delays shrink gains from adapting forecast models
Accounting for realistic delays left published adaptation methods improving by at most half a percent over frozen models across four of five benchmarks.
Edited by Boris Djordjevic · Paperfoot
This empirical study audited four recent adaptation methods for time-series forecasting across five benchmark datasets using a newly built, leakage-free testing harness. These methods update a deployed forecasting model using incoming data. However, real future values arrive only after a delay. Earlier evaluations allowed immediate updates without this delay, which inflated apparent gains by up to 110%.
When the audit enforced realistic label delays, published methods showed little benefit. On four of the five datasets, the largest statistically significant improvement any published method achieved over its frozen model was half a percent. On all four, at least one published method was significantly worse than the frozen model at the minimum causal delay. On the remaining benchmark, every audited method adapted, but a simple reference filter beat three of the four at a fraction of their computational cost.
Why it matters
The findings show that reported gains from continually adapting forecasting models can be artifacts of unrealistic testing setups that ignore real-world data delays. By providing an open, delay-aware testing harness tested across five public benchmarks, the work gives engineers a standard way to evaluate whether online adaptation actually helps in practice.
Caveats
The evaluation was limited to four recent adaptation methods tested on five benchmark datasets. Additionally, the paper is a preprint that has not yet undergone peer review.
From the paper
- Limitation
We evaluate five dataset families but one deep architecture (PatchTST), the backbone all four audited methods release checkpoints for.
Quoted word for word from the full text on arXiv.
In this paper
test-time adaptation for time-series forecasting
- Methods
- recursive least squares (RLS) filters combined by a per-coordinate median (this paper)TAFAS (baseline)COSA (baseline)PETSA (baseline)DynaTTA (baseline)ELF-style linear corrector (baseline)ELF-mix (baseline)ELF-linear (baseline)persistence (baseline)seasonal-naive (baseline)bias-EMA (baseline)NLMS (baseline)
- Datasets
- ETTm1 (evaluated)ETTh2 (evaluated)Weather (evaluated)Electricity (evaluated)Traffic (evaluated)
- Links
- Code
Results
- recursive least squares (RLS) filters combined by a per-coordinate median, cost per step, 7-channel streams: 56 microseconds
As references we add two closed-form correctors: a bank of recursive least squares (RLS) filters combined by a per-coordinate median, with no tunable hyperparameters and 56 microseconds per step on the 7-channel streams, and an ELF-style linear corrector.
Abstract
- TAFAS, improvement in standardized MSE, on ETTm1: 0.12; COSA, improvement in standardized MSE, on ETTm1: 0.50; DynaTTA, improvement in standardized MSE, on ETTm1: 0.91; PETSA, improvement in standardized MSE, on ETTm1: 0.02
On ETTm1 all four published methods genuinely adapt: TAFAS improves on the frozen model by 0.12 in standardized MSE, COSA by 0.50, DynaTTA by 0.91 and PETSA by 0.02 (significant in every seed for the first three, in one of three seeds for PETSA).
ETTm1 is the positive control.
- recursive least squares (RLS) filters combined by a per-coordinate median, improvement in standardized MSE, on ETTm1: 0.86
The bank improves by 0.86.
ETTm1 is the positive control.
- seasonal-naive, MSE, on ETTm1: 0.9633
Seasonal-naive, at 0.9633, beats TAFAS, PETSA and the frozen model on this stream.
ETTm1 is the positive control.
- TAFAS, improvement, on Traffic, H: 0.5%; PETSA, relative error, on Traffic: 11%
On Traffic every method sits within 12% of the frozen model and most within 1%: the bank is best at every delay and significantly better than the frozen model at 2H and 4H (by 1.5 to 1.8%), TAFAS is significantly better at H by 0.5% and significantly worse at 2H and 4H, COSA is tied throughout, and PETSA is significantly worse by 11 …
Weather, Electricity, Traffic: the frozen model and the closed-form correctors set the pace.
- recursive least squares (RLS) filters combined by a per-coordinate median, cost per step, C{=}7, H{=}24 and F{=}4: 56 \mu s
The cost per step is O(CHF^{2}), which is 56 \mu s on one CPU core for C{=}7, H{=}24 and F{=}4.
RLS with anti-windup.
- recursive least squares (RLS) filters combined by a per-coordinate median, cost, over all ten cells including the 862-channel Traffic stream: 2.9 ms per step
The bank updates 4,032 closed-form weights (168 coordinates × 4 features × 6 experts) in 0.056 ms per stream step on one CPU core on the 7-channel ETT streams, with no tunable hyperparameters; its cost scales as O(CHF^{2}) and averages 2.9 ms per step over all ten cells including the 862-channel Traffic stream …
5.3 Cost
What it shows
- On ETTm1 every audited method genuinely adapts, but the RLS bank beats three of the four at a fraction of their cost, while DynaTTA beats the bank only at the minimum causal delay and at roughly 2,500 times the cost.
On ETTm1 every audited method genuinely adapts, yet the RLS bank still beats three of the four at a fraction of their cost; only DynaTTA beats the bank, only at the minimum causal delay, and at roughly 2,500 times the per-update cost; the ELF-style corrector beats all four.
Abstract
- Outside ETTm1, the largest statistically significant gain any published method achieves over its own frozen checkpoint is half a percent, and longer delays make adapters significantly harmful on drift-heavy ETTh2.
On the other four datasets, the largest statistically significant improvement any published method achieves over its own frozen checkpoint is half a percent, on all four at least one published method is significantly worse than the frozen model at the minimum causal delay, and on drift-heavy ETTh2 longer label delays make every adapter that separates from the frozen model, ours included, significantly harmful …
Abstract
- Leaky next-step updates inflate simple adapters' apparent gains by up to 110%, while the backbone training recipe changes frozen online error by up to a factor of 25.
Leaky next-step updates inflate the apparent gains of simple adapters by up to 110%, and the backbone training recipe moves frozen online error by up to a factor of 25, more than any adaptation effect we measure.
Abstract
- The RLS bank's median combination follows the majority of adapting experts and turns significantly harmful on ETTh2 under extended label delays.
On ETTh2 with a well-trained frozen backbone the bank is harmful: significantly so at d=2H ( +0.16 to +0.26 across the three checkpoint seeds) and at d=4H ( +0.33 to +0.44 ).
6 When Closed-Form Adaptation Fails
- The bank's accuracy sits within 7% of the oracle forgetting factor in nine of ten cells, tracking the shift in optimal plasticity without being told.
The bank follows this shift without being told; its MSE sits within 7% of the per-cell oracle \lambda in nine of ten cells (from 0.7% better to 6.6% worse), with one outlier, Weather/DLinear at d=H, where the bank lands 24% above the best single \lambda before the gap closes by d=4H (Section 6).
5.4 Diagnostics: label staleness, plasticity and the training recipe
Quoted word for word from the full text on arXiv.
The paper
Training on the Future: A Delay-Aware Audit of Test-Time Adaptation for Time-Series Forecasting
arXiv, Oct 8, 2026, Open access, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.12232
arXiv:2610.12232v1 in cs.LG
Comments: 34 pages, 9 figures. Code and cached results: https://github.com/Xodios/TRAINING-ON-THE-FUTURE-A-DELAY-AWARE-AUDIT-OF-TEST-TIME-ADAPTATION-FOR-TIME-SERIES-FORECASTING
Cite as BibTeX
@misc{fentazi2026training,
title = {Training on the Future: A Delay-Aware Audit of Test-Time Adaptation for Time-Series Forecasting},
author = {Fentazi, Mohamed Readh and Ameur, Mazene and Ksentini, Adlen},
year = {2026},
eprint = {2610.12232},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
doi = {10.48550/arxiv.2610.12232},
url = {https://arxiv.org/abs/2610.12232}
}