Explainability, uncertainty & physics
Strategic Governance of AI Models in Earth Science
In one sentence. Argues AI weather and climate foundation models need physical evaluation beyond skill scores, proposing five priorities and recommendations such as shared physics-based reporting standards.
Abstract
AI foundation models pretrained on weather and climate data are increasingly fine-tuned to Earth science tasks well beyond weather forecasting. Their development and adoption are outpacing the scientific community's ability to evaluate them. These models are judged almost entirely by benchmark skill metrics, which measure how closely a forecast reproduces a reference product but not whether a model represents the physical processes governing the system it predicts. Forecast skill and physical reliability are therefore distinct properties. The distinction is most consequential under the nonstationary conditions of a changing climate for which these models were never trained. We identify five priorities for the physical evaluation of AI models in Earth science from task-specific emulators to foundation models, spanning training data, fine-tuning, behavioral testing, mechanistic interpretability, and output validation. We recommend three activities for the coming decade: 1) open AI-ready evaluation datasets, 2) a shared reporting standard for physics-based evaluation, and 3) a dedicated research program on the safety of these models.
Labels
- Topic
- Explainability, uncertainty & physics, Foundation models, Benchmarks, datasets & evaluation
- Type
- Position or perspective paper
- Domain
- Global
- Status
- Research
- Identifiers
- arXiv:2610.10560 OpenAlex:W7221137919
- Added
- 2026-10-22
Labels and summary are generated by Claude and reviewed weekly. Wrong? Edit this entry.