Insights into the accuracy of social scientists’ forecasts of societal change

Grossmann I, Rotella A, Hutcherson CA, Sharpinskyi K, Varnum ME, Achter S, Dhami MK, Guo XE, Kara-Yakoubian M, Mandel DR, Raes L, Tay L, Vie A, Wagner L, Adamkovic M, Arami A, Arriaga P, Bandara K, Baník G, Bartoš F, Baskin E, Bergmeir C, Białek M, Børsting CK, Browne DT, Caruso EM, Chen R, Chie BT, Chopik WJ, Collins RN, Cong CW, Conway LG, Davis M, Day MV, Dhaliwal NA, Durham JD, Dziekan M, Elbaek CT, Shuman E, Fabrykant M, Firat M, Fong GT, Frimer JA, Gallegos JM, Goldberg SB, Gollwitzer A, Goyal J, Graf-Vlachy L, Gronlund SD, Hafenbrädl S, Hartanto A, Hirshberg MJ, Hornsey MJ, Howe PD, Izadi A, Jaeger B, Kačmár P, Kim YJ, Krenzler R, Lannin DG, Lin HW, Lou NM, Lua VY, Lukaszewski AW, Ly AL, Madan CR, Maier M, Majeed NM, March DS, Marsh AA, Misiak M, Myrseth KOR, Napan JM, Nicholas J, Nikolopoulos K, O J, Otterbring T, Paruzel-Czachura M, Pauer S, Protzko J, Raffaelli Q, Ropovik I, Ross RM, Roth Y, Røysamb E, Schnabel L, Schütz A, Seifert M, Sevincer AT, Sherman GT, Simonsson O, Sung MC, Tai CC, Talhelm T, Teachman BA, Tetlock PE, Thomakos D, Tse DC, Twardus OJ, Tybur JM, Ungar L, Vandermeulen D, Vaughan Williams L, Vosgerichian HA, Wang Q, Wang K, Whiting ME, Wollbrant CE, Yang T, Yogeeswaran K, Yoon S, Alves VR, Andrews-Hanna JR, Bloom PA, Boyles A, Charis L, Choi M, Darling-Hammond S, Ferguson ZE, Kaiser CR, Karg ST, Ortega AL, Mahoney L, Marsh MS, Martinie MF, Michaels EK, Millroth P, Naqvi JB, Ng W, Rutledge RB, Slattery P, Smiley AH, Strijbis O, Sznycer D, Tsukayama E, van Loon A, Voelkel JG, Wienk MN, Wilkening T (2023)

Publication Type: Journal article

Publication year: 2023

Journal

Book Volume: 7

Pages Range: 484-501

Journal Issue: 4

DOI: 10.1038/s41562-022-01517-1

Abstract

How well can social scientists predict societal change, and what processes underlie their predictions? To answer these questions, we ran two forecasting tournaments testing the accuracy of predictions of societal change in domains commonly studied in the social sciences: ideological preferences, political polarization, life satisfaction, sentiment on social media, and gender–career and racial bias. After we provided them with historical trend data on the relevant domain, social scientists submitted pre-registered monthly forecasts for a year (Tournament 1; N = 86 teams and 359 forecasts), with an opportunity to update forecasts on the basis of new data six months later (Tournament 2; N = 120 teams and 546 forecasts). Benchmarking forecasting accuracy revealed that social scientists’ forecasts were on average no more accurate than those of simple statistical models (historical means, random walks or linear regressions) or the aggregate forecasts of a sample from the general public (N = 802). However, scientists were more accurate if they had scientific expertise in a prediction domain, were interdisciplinary, used simpler models and based predictions on prior data.

Involved external institutions

University of Waterloo (UW) Canada (CA) Northumbria University United Kingdom (GB) University of Toronto Canada (CA) Arizona State University (ASU) United States (USA) (US) Technische Universität Hamburg-Harburg (TUHH) Germany (DE) Middlesex University United Kingdom (GB) University of California, San Diego (UC San Diego, UCSD) United States (USA) (US) Toronto Metropolitan University / Ryerson University Canada (CA) Defence Research and Development Canada Canada (CA) Tilburg University Netherlands (NL) Purdue University United States (USA) (US) University of Oxford United Kingdom (GB) University of Zurich / Universität Zürich (UZH) Switzerland (CH) University of Prešov / Prešovská univerzita v Prešove Slovakia (SK) Iscte – Instituto Universitário de Lisboa (ISCTE-IUL) / Instituto Superior de Ciências do Trabalho e da Empresa Portugal (PT) The University of Melbourne Australia (AU) Adam Mickiewicz University / Uniwersytet im. Adama Mickiewicza w Poznaniu (UAM) Poland (PL) University of Wroclaw / Uniwersytet Wrocławski Poland (PL) University of York United Kingdom (GB) IE University Spain (ES) Hebrew University of Jerusalem Israel (IL) Nottingham Trent University United Kingdom (GB) Northwestern University United States (USA) (US) University of Washington United States (USA) (US) Stanford University United States (USA) (US) Columbia University United States (USA) (US) University of Amsterdam Netherlands (NL) Saint Joseph's University United States (USA) (US) Monash University Australia (AU) Aarhus University Denmark (DK) University of California Los Angeles (UCLA) United States (USA) (US) Dominican University of California United States (USA) (US) Tamkang University / 淡江大學 Taiwan (TW) Michigan State University United States (USA) (US) Grove City College United States (USA) (US) Siena College United States (USA) (US) Memorial University of Newfoundland (MUN) Canada (CA) University of British Columbia Canada (CA) University of Oklahoma United States (USA) (US) University of Groningen / Rijksuniversiteit Groningen Netherlands (NL) National Research University - Higher School of Economics (HSE) / Высшая школа экономики Russian Federation (RU) Radboud University Nijmegen Netherlands (NL) Ontario Institute for Cancer Research Canada (CA) The University of Winnipeg / Université de winnipeg Canada (CA) University of Wisconsin - Madison United States (USA) (US) BI Norwegian Business School / Handelshøyskolen BI Norway (NO) Technische Universität Dortmund Germany (DE) Singapore Management University (SMU) Singapore (SG) University of Queensland Australia (AU) UMass University of Massachusetts Dartmouth United States (USA) (US) Vrije Universiteit Amsterdam (VU) / University Amsterdam Netherlands (NL) Pavol Jozef Šafárik University in Kosice / Univerzita Pavla Jozefa Šafárika v Košiciach Slovakia (SK) University of Cambridge United Kingdom (GB) Hermes Germany GmbH Germany (DE) Illinois State University (ISU) United States (USA) (US) National Pingtung University Taiwan (TW) University of Victoria (UVic) Canada (CA) California State University (CSUF) United States (USA) (US) Loma Linda University (LLU) United States (USA) (US) University of Nottingham United Kingdom (GB) University College London (UCL) United Kingdom (GB) Florida State University (FSU) United States (USA) (US) Georgetown University United States (USA) (US) Durham University United Kingdom (GB) Aberystwyth University United Kingdom (GB) University of Agder / Universitetet i Agder / Agderin yliopisto Norway (NO) University of Silesia in Katowice / Uniwersytet Śląski w Katowicach Poland (PL) Central Connecticut State University CCSU United States (USA) (US) University of Arizona United States (USA) (US) Univerzita Karlova v Praze / Charles University in Prague Czech Republic (CZ) Macquarie University Australia (AU) University of Haifa / אוניברסיטת חיפה Israel (IL) University of Oslo Norway (NO) Cornell University United States (USA) (US) Otto-Friedrich-Universität Bamberg Germany (DE) Universität Hamburg (UHH) Germany (DE) Karolinska Institute Sweden (SE) University of Southampton United Kingdom (GB) University of Chicago United States (USA) (US) University of Virginia (UVA) United States (USA) (US) University of Pennsylvania (UPenn) United States (USA) (US) National and Kapodistrian University of Athens Greece (GR) University of Strathclyde United Kingdom (GB) University of Guelph (U of G) Canada (CA) Harvard University United States (USA) (US) University of St Andrews United Kingdom (GB) University of North Carolina Wilmington (UNCW) United States (USA) (US) University of Canterbury New Zealand (NZ) University of Dayton United States (USA) (US) Universidade Lusófona Portugal (PT) Boston University United States (USA) (US) Pusan National University (PNU) Korea, Republic of (KR) Wright State University United States (USA) (US) Georgia Southern University United States (USA) (US) University of California, Berkeley (UC Berkeley) United States (USA) (US) Uppsala University Sweden (SE) Carnegie Mellon University United States (USA) (US) Singapore University of Social Sciences SUSS / 新加坡社科大學 Singapore (SG) Yale University United States (USA) (US) Oklahoma State University United States (USA) (US) University of Hawaiʻi-West Oʻahu United States (USA) (US)

How to cite

APA:

Grossmann, I., Rotella, A., Hutcherson, C.A., Sharpinskyi, K., Varnum, M.E., Achter, S.,... Wilkening, T. (2023). Insights into the accuracy of social scientists’ forecasts of societal change. Nature Human Behaviour, 7(4), 484-501. https://doi.org/10.1038/s41562-022-01517-1

MLA:

Grossmann, Igor, et al. "Insights into the accuracy of social scientists’ forecasts of societal change." Nature Human Behaviour 7.4 (2023): 484-501.

BibTeX: Download