Salesforce open-sources GIFT-Eval-CTX: feeding promotions and weather to agents cuts forecast error by up to 40.7%
Synopsis
Salesforce AI Research has open-sourced GIFT-Eval-CTX, a benchmark for whether forecasting systems can turn forward-looking business context into accurate numerical predictions. Traditional time-series foundation models take numerical history and predefined variables, so they struggle to use known future events such as heatwaves, road closures or software releases. The benchmark splits context into macro trends and micro events across 10,000 tasks from 20 real-world datasets; adding structured future context cut GPT-5.6 Sol's error by 40.7% and Claude Opus 4.8's by 29.6%.
Interpretation
Forecasting systems now have a public testbed for whether they can convert known future business information into numerical predictions rather than only extrapolating historical curves. Traditional time-series foundation models mainly accept numerical history and predefined variables, and lack a common interface for open-ended information in documents, plans or expert judgment. GIFT-Eval-CTX spans 10,000 tasks from 20 real-world datasets in energy, logistics, weather, retail and cloud operations, and selects test windows where traditional models fail using past data alone.
The benchmark decomposes future context into two structured layers, macro and micro, leaving the model to fill in step-by-step numbers. Macro effects record trend direction, seasonal cycles and value bounds across the whole horizon, while micro effects pin down specific time windows and magnitudes. Micro effects come in three types: excursions that spike or dip temporarily, mean shifts that move the baseline permanently, and seasonal changes that alter recurring cycles; the final benchmark holds 23,476 validated micro effects with matching macro horizon annotations.
An agentic system combining an LLM with domain models and code execution delivers the best accuracy across every forecast length. Pure LLMs underperform traditional forecasting models when given history alone and are error-prone on numerical precision and output length. MoiraiAgent's aggregate error is 36.3% lower than the best history-only forecasting model, with the advantage widening on longer forecasts; without tools, GPT-5.6 Sol produced the correct forecast length in only 38.9%–40.5% of medium-to-long tasks, and Claude Opus 4.8 in under 2.3%.
Perspective
The evaluation covers 10,000 tasks from 20 real-world datasets, focused on operational forecasting in energy, logistics, weather, retail and cloud operations, for researchers and enterprise teams who need to bring forward-looking business information into forecasts; the benchmark, dataset and leaderboard are open source.
Pure LLMs remain unreliable on numerical precision and output length, with a low share of correct lengths without tools; the reported error reductions are the releasing organization's aggregated results on its own benchmark, and the material does not break performance down by domain.