ClimateBert: A Pretrained Language Model for Climate-Related Text
In one sentence. ClimateBERT is a language model pretrained on climate-related text; it does not target the atmosphere or climate system itself.
Abstract
Over the recent years, large pretrained language models (LM) have revolutionized the field of natural language processing (NLP). However, while pretraining on general language has been shown to work very well for common language, it has been observed that niche language poses problems. In particular, climate-related texts include specific language that common LMs can not represent accurately. We argue that this shortcoming of today's LMs limits the applicability of modern NLP to the broad field of text processing of climate-related texts. As a remedy, we propose CLIMATEBERT, a transformer-based language model that is further pretrained on over 2 million paragraphs of climate-related texts, crawled from various sources such as common news, research articles, and climate reporting of companies. We find that CLIMATEBERT leads to a 48% improvement on a masked language model objective which, in turn, leads to lowering error rates by 3.57% to 35.71% for various climate-related downstream tasks like text classification, sentiment analysis, and fact-checking.
Labels
- Topic
- LLMs & agents
- Type
- New model or method
- Method
- Transformer / ViT
- Data
- Other
- Status
- Research
- Identifiers
- arXiv:2110.12010 OpenAlex:W4286896917
- Added
- 2026-10-22
Labels and summary are generated by Claude and reviewed weekly. Wrong? Edit this entry.