ClimateBert: A Pretrained Language Model for Climate-Related Text

Webersinke N, Kraus M, Bingler JA, Leippold M (2022)

Publication Language: English

Publication Type: Conference contribution, Conference Contribution

Publication year: 2022

Journal

arXiv arXiv.org

Conference Proceedings Title: Proceedings of AAAI 2022 Fall Symposium: The Role of AI in Responding to Climate Challenges

Event location: Arlington, VA US

URI: https://www.climatechange.ai/papers/aaaifss2022/12

DOI: 10.2139/ssrn.4229146

Open Access Link: https://www.climatechange.ai/papers/aaaifss2022/12

Abstract

Over the recent years, large pretrained language models (LM) have revolutionized the field of natural language processing (NLP). However, while pretraining on general language has been shown to work very well for common language, it has been observed that niche language poses problems. In particular, climate-related texts include specific language that common LMs can not represent accurately. We argue that this shortcoming of today's LMs limits the applicability of modern NLP to the broad field of text processing of climate-related texts. As a remedy, we propose ClimateBERT, a transformer-based language model that is further pretrained on over 2 million paragraphs of climate-related texts, crawled from various sources such as common news, research articles, and climate reporting of companies. We find that ClimateBERT leads to a 48% improvement on a masked language model objective which, in turn, leads to lowering error rates by 3.57% to 35.71% for various climate-related downstream tasks like text classification, sentiment analysis, and fact-checking.

Authors with CRIS profile

Nicolas Webersinke Lehrstuhl für Betriebswirtschaftslehre, insbesondere Finanzierung und Banken Mathias Kraus Juniorprofessur für Data Analytics

Involved external institutions

Eidgenössische Technische Hochschule Zürich (ETHZ) / Swiss Federal Institute of Technology in Zurich CH Switzerland (CH) University of Zurich / Universität Zürich (UZH) CH Switzerland (CH)

How to cite

APA:

Webersinke, N., Kraus, M., Bingler, J.A., & Leippold, M. (2022). ClimateBert: A Pretrained Language Model for Climate-Related Text. In Proceedings of AAAI 2022 Fall Symposium: The Role of AI in Responding to Climate Challenges. Arlington, VA, US.

MLA:

Webersinke, Nicolas, et al. "ClimateBert: A Pretrained Language Model for Climate-Related Text." Proceedings of the AAAI 2022 Fall Symposium: The Role of AI in Responding to Climate Challenges, Arlington, VA 2022.

BibTeX: Download