Benchmarks, datasets & evaluation
OmniEarth-Bench: Towards Holistic Evaluation of Earth's Six Spheres and Cross-Spheres Interactions with Multimodal Observational Earth Data
In one sentence. OmniEarth-Bench is a 29,855-annotation multimodal benchmark spanning six Earth spheres including atmosphere; none of 9 MLLMs exceeds 35% accuracy.
Abstract
Existing benchmarks for multimodal learning in Earth science offer limited, siloed coverage of Earth's spheres and their cross-sphere interactions, typically restricting evaluation to the human-activity sphere of atmosphere and to at most 16 tasks. These limitations: narrow-source heterogeneity (single/few data sources), constrained scientific granularity, and limited-sphere extensibility. Therefore, we introduce OmniEarth-Bench, the first multimodal benchmark that systematically spans all six spheres: atmosphere, lithosphere, oceanosphere, cryosphere, biosphere, and human-activity sphere, and cross-spheres. Built with a scalable, modular-topology data inference framework and native multi-observation sources and expert-in-the-loop curation, OmniEarth-Bench produces 29,855 standardized, expert-curated annotations. All annotations are organized into a four-level hierarchy (Sphere, Scenario, Ability, Task), encompassing 109 expert-curated evaluation tasks. Experiments on 9 state-of-the-art MLLMs reveal that even the most advanced models struggle with our benchmarks, where none of them reach 35% accuracy, revealing systematic gaps in Earth-system cognitive ability. The dataset and evaluation code were released at OmniEarth-Bench (https://anonymous.4open.science/r/OmniEarth-Bench-B1BD).
Labels
- Topic
- Benchmarks, datasets & evaluation, LLMs & agents
- Type
- Benchmark or dataset
- Method
- Language model
- Domain
- Global
- Data
- Other
- Status
- Research
- Identifiers
- arXiv:2505.23522 OpenAlex:W4416611775
- Added
- 2026-10-22
Labels and summary are generated by Claude and reviewed weekly. Wrong? Edit this entry.