IBM Bundles Four Granite Forecasting Models Into One, Easing the Model-Picking Burden
It delivers future values along with an expected range, no additional training required. In benchmark tests the error was slightly lower than any single model's, but runtime and memory costs go up.
On October 22 local time, IBM introduced “Granite Timeseries Ensemble,” a package that bundles four Granite models into one. It is a tool that looks at data collected over time and predicts future values. Without any additional training, it produces both a forecast and an expected range. IBM said the goal is to reduce the burden of agonizing over which model to pick for each dataset.
Why run four models at once
Even pre-trained forecasting models perform differently depending on the data. A model that handles the smoothly changing signals of factory equipment well will not necessarily forecast choppy retail demand accurately. This tool pools the forecasts of four models that read trend and seasonal variation in different ways. The approach spares users from having to choose a single model suited to their data from the outset.
The default setting, “Probabilistic Ensemble,” gives equal weight to the forecasts of all four models. There is no need to separately learn per-model weights for new data. The alternative, “IQR Ensemble,” assigns greater weight to models that produce narrower expected ranges. IBM noted that a narrow range does not mean accuracy, and that this method can favor overconfident models.
How much did accuracy improve
IBM evaluated both methods on 97 dataset configurations from the GIFT-Eval benchmark. No additional training was performed, and the company said it did not use the benchmark’s ground-truth data to select or train the ensembling method. In results as of October 20, both ensembling methods beat all four individual models on aggregate forecast error and average rank. The default method’s error was 0.683 on MASE. That is slightly lower than the 0.685 of PatchTST-FM-r2, the best of the individual models. For this figure, lower means more accurate.
Image: huggingface.co
The ranking shifts once the expected range is factored in as well. PatchTST-FM-r2 had the lowest aggregate error when the expected range was included. By contrast, on the average of per-dataset ranks, IQR Ensemble was the best of the six contenders. IBM said the reduction in forecast error relative to individual models is small. Instead, it framed the value as getting comparable or better accuracy across a variety of datasets without knowing in advance which model will fit well.
So should you use it at work
This tool can be used to examine not just a single forecast value but the range of possible outcomes around it. The default output contains values spanning from the 10th percentile to the 90th percentile of the expected distribution. The span between the 10th and the 90th is the middle 80% interval. IBM recommended checking against historical data how often actual outcomes fall inside that range.
Image: huggingface.co
The default configuration can be downloaded and run from Hugging Face. On the first run it downloads each of the four models, and generating forecasts also takes more time and memory than using a single model. For now it forecasts each data series separately and does not use the relationships between series. The default configuration and the models included in it carry a license with few usage restrictions. A separately prepared extended configuration includes models that may only be used for noncommercial or research purposes.
Adding compatible external models or changing the ensembling method leaves the output format unchanged. If you are weighing adoption, the first step is to compare accuracy and how trustworthy the expected ranges are using your own company’s historical data. Whether the time and memory needed to run four models together are worth that much of an accuracy gain is another point to weigh.