Probabilistic & ensemble forecasting★ Highlight
A Practical Probabilistic Benchmark for AI Weather Models
Why it matters. Lagged-ensemble method enables fair probabilistic comparison of AI weather models; shows multi-step losses hurt probabilistic skill.
In one sentence. Lagged ensembles give a parameter-free probabilistic comparison showing GraphCast and Pangu tie on CRPS, and that multi-step losses hurt probabilistic skill.
Abstract
Since the weather is chaotic, forecasts aim to predict the distribution of future states rather than make a single prediction. Recently, multiple data driven weather models have emerged claiming breakthroughs in skill. However, these have mostly been benchmarked using deterministic skill scores, and little is known about their probabilistic skill. Unfortunately, it is hard to fairly compare AI weather models in a probabilistic sense, since variations in choice of ensemble initialization, definition of state, and noise injection methodology become confounding. Moreover, even obtaining ensemble forecast baselines is a substantial engineering challenge given the data volumes involved. We sidestep both problems by applying a decades-old idea -- lagged ensembles -- whereby an ensemble can be constructed from a moderately-sized library of deterministic forecasts. This allows the first parameter-free intercomparison of leading AI weather models' probabilistic skill against an operational baseline. The results reveal that two leading AI weather models, i.e. GraphCast and Pangu, are tied on the probabilistic CRPS metric even though the former outperforms the latter in deterministic scoring. We also reveal how multiple time-step loss functions, which many data-driven weather models have employed, are counter-productive: they improve deterministic metrics at the cost of increased dissipation, deteriorating probabilistic skill. This is confirmed through ablations applied to a spherical Fourier Neural Operator (SFNO) approach to AI weather forecasting. Separate SFNO ablations modulating effective resolution reveal it has a useful effect on ensemble dispersion relevant to achieving good ensemble calibration. We hope these and forthcoming insights from lagged ensembles can help guide the development of AI weather forecasts and have thus shared the diagnostic code.
Labels
- Topic
- Probabilistic & ensemble forecasting, Global medium-range NWP
- Type
- Evaluation or intercomparison of existing models
- Method
- Neural operator (FNO, SFNO, ...), Transformer / ViT
- Time scale
- 3–15 days
- Domain
- Global
- Data
- ERA5 / reanalysis, NWP analysis or forecasts (IFS, GFS, ...)
- Status
- Research
- Identifiers
- arXiv:2401.15305 DOI:10.1029/2024gl113656 OpenAlex:W4409263600
- Added
- 2026-10-22
Labels and summary are generated by Claude and reviewed weekly. Wrong? Edit this entry.