ML4WX

Benchmarks, datasets & evaluation

Recommendations for Comprehensive and Independent Evaluation of Machine Learning-Based Earth System Models

Paul A. Ullrich, Elizabeth A. Barnes, William D. Collins, Katherine Dagon, Shiheng Duan, Joshua Elms, Jiwoo Lee, L. Ruby Leung, Dan Lu, Maria J. Molina, Travis A. O'Brien, Finn O. Rebassoo

Journal of Geophysical Research: Machine Learning and Computation · · Colorado State University (incl. CIRA), Lawrence Berkeley National Laboratory / NERSC, NSF NCAR / UCAR, Oak Ridge National Laboratory, University of California, Berkeley

PaperPDF

In one sentence. Five recommendations for standardized, independent evaluation of ML-based Earth system models, stressing physical consistency for climate projections without historical analogues.

Abstract

Machine learning (ML) is a revolutionary technology with demonstrable applications across multiple disciplines. Within the Earth science community, ML has been most visible for weather forecasting, producing forecasts that rival modern physics-based models. Given the importance of deepening our understanding and improving predictions of the Earth system on all time scales, efforts are now underway to develop forecasting models into Earth-system models (ESMs), capable of representing all components of the coupled Earth system (or their aggregated behavior) and their response to external changes. Modeling the Earth system is a much more difficult problem than weather forecasting, not least because the model must represent the alternate (e.g., future) coupled states of the system for which there are no historical observations. Given that the physical principles that enable predictions about the response of the Earth system are often not explicitly coded in these ML-based models, demonstrating the credibility of ML-based ESMs thus requires us to build evidence of their consistency with the physical system. To this end, this paper puts forward five recommendations to enhance comprehensive, standardized, and independent evaluation of ML-based ESMs to strengthen their credibility and promote their wider use.

Labels

Topic
Benchmarks, datasets & evaluation, Climate emulation, Explainability, uncertainty & physics
Type
Position or perspective paper
Method
Other
Time scale
Climate projections / centennial
Domain
Global
Data
Other
Status
Research
Identifiers
arXiv:2410.19882 DOI:10.1029/2024jh000496 OpenAlex:W4408457719
Added
2026-10-22

Labels and summary are generated by Claude and reviewed weekly. Wrong? Edit this entry.