Towards fair decentralized benchmarking of healthcare AI algorithms with the Federated Tumor Segmentation (FeTS) challenge

Zenk M, Baid U, Pati S, Linardos A, Edwards B, Sheller M, Foley P, Aristizabal A, Zimmerer D, Gruzdev A, Martin J, Shinohara RT, Reinke A, Isensee F, Parampottupadam S, Parekh K, Floca R, Kassem H, Baheti B, Thakur S, Chung V, Kushibar K, Lekadir K, Jiang M, Yin Y, Yang H, Liu Q, Chen C, Dou Q, Heng PA, Zhang X, Zhang S, Khan MI, Azeem MA, Jafaritadi M, Alhoniemi E, Kontio E, Khan SA, Mächler L, Ezhov I, Kofler F, Shit S, Paetzold JC, Loehr T, Wiestler B, Peiris H, Pawar K, Zhong S, Chen Z, Hayat M, Egan G, Harandi M, Isik Polat E, Polat G, Kocyigit A, Temizel A, Tuladhar A, Tyagi L, Souza R, Forkert ND, Mouches P, Wilms M, Shambhat V, Maurya A, Danannavar SS, Kalla R, Anand VK, Krishnamurthi G, Nalawade S, Ganesh C, Wagner B, Reddy D, Das Y, Yu FF, Fei B, Madhuranthakam AJ, Maldjian J, Singh G, Ren J, Zhang W, An N, Hu Q, Zhang Y, Zhou Y, Siomos V, Tarroni G, Passerrat-Palmbach J, Rawat A, Zizzo G, Kadhe SR, Epperlein JP, Braghin S, Wang Y, Kanagavelu R, Wei Q, Yang Y, Liu Y, Kotowski K, Adamski S, Machura B, Malara W, Zarudzki L, Nalepa J, Shi Y, Gao H, Avestimehr S, Yan Y, Akbar AS, Kondrateva E, Yang H, Li Z, Wu HY, Roth J, Saueressig C, Milesi A, Nguyen QD, Gruenhagen NJ, Huang TM, Ma J, Singh HSH, Pan NY, Zhang D, Zeineldin RA, Futrega M, Yuan Y, Conte GM, Feng X, Pham QD, Xia Y, Jiang Z, Luu HM, Dobko M, Carré A, Tuchinov B, Mohy-ud-Din H, Alam S, Singh A, Shah N, Wang W, Sako C, Bilello M, Ghodasara S, Mohan S, Davatzikos C, Calabrese E, Rudie J, Villanueva-Meyer J, Cha S, Hess C, Mongan J, Ingalhalikar M, Jadhav M, Pandey U, Saini J, Huang RY, Chang K, To MS, Bhardwaj S, Chong C, Agzarian M, Kozubek M, Lux F, Michálek J, Matula P, Ker^kovský M, Kopr^ivová T, Dostál M, Vybíhal V, Pinho MC, Holcomb J, Metz M, Jain R, Lee MD, Lui YW, Tiwari P, Verma R, Bareja R, Yadav I, Chen J, Kumar N, Gusev Y, Bhuvaneshwar K, Sayah A, Bencheqroun C, Belouali A, Madhavan S, Colen RR, Kotrotsou A, Vollmuth P, Brugnara G, Preetha CJ, Sahm F, Bendszus M, Wick W, Mahajan A, Balaña C, Capellades J, Puig J, Choi YS, Lee SK, Chang JH, Ahn SS, Shaykh HF, Herrera-Trujillo A, Trujillo M, Escobar W, Abello A, Bernal J, Gómez J, LaMontagne P, Marcus DS, Milchenko M, Nazeri A, Landman B, Ramadass K, Xu K, Chotai S, Chambless LB, Mistry A, Thompson RC, Srinivasan A, Bapuraj JR, Rao A, Wang N, Yoshiaki O, Moritani T, Turk S, Lee J, Prabhudesai S, Garrett J, Larson M, Jeraj R, Li H, Weiss T, Weller M, Bink A, Pouymayou B, Sharma S, Tseng TC, Adabi S, Xavier Falcão A, Martins SB, Teixeira BC, Sprenger F, Menotti D, Lucio DR, Niclou SP, Keunen O, Hau AC, Pelaez E, Franco-Maldonado H, Loayza F, Quevedo S, McKinley R, Slotboom J, Radojewski P, Meier R, Wiest R, Trenkler J, Pichler J, Necker G, Haunschmidt A, Meckel S, Guevara P, Torche E, Mendoza C, Vera F, Ríos E, López E, Velastin SA, Choi J, Baek S, Kim Y, Ismael H, Allen B, Buatti JM, Zampakis P, Panagiotopoulos V, Tsiganos P, Alexiou S, Haliassos I, Zacharaki EI, Moustakas K, Kalogeropoulou C, Kardamakis DM, Luo B, Poisson LM, Wen N, Vallières M, Loutfi MAL, Fortin D, Lepage M, Morón F, Mandel J, Shukla G, Liem S, Alexandre GS, Lombardo J, Palmer JD, Flanders AE, Dicker AP, Ogbole G, Oyekunle D, Odafe-Oyibotha O, Osobu B, Shu’aibu Hikima M, Soneye M, Dako F, Dorcas A, Murcia D, Fu E, Haas R, Thompson JA, Ormond DR, Currie S, Fatania K, Frood R, Simpson AL, Peoples JJ, Hu R, Cutler D, Moraes FY, Tran A, Hamghalam M, Boss MA, Gimpel J, Kattil Veettil D, Schmidt K, Cimino L, Price C, Bialecki B, Marella S, Apgar C, Jakab A, Weber MA, Colak E, Kleesiek J, Freymann JB, Kirby JS, Maier-Hein L, Albrecht J, Mattson P, Karargyris A, Shah P, Menze B, Maier-Hein K, Bakas S (2025)

Publication Type: Journal article

Publication year: 2025

Journal

Book Volume: 16

Article Number: 6274

Journal Issue: 1

DOI: 10.1038/s41467-025-60466-1

Abstract

Computational competitions are the standard for benchmarking medical image analysis algorithms, but they typically use small curated test datasets acquired at a few centers, leaving a gap to the reality of diverse multicentric patient data. To this end, the Federated Tumor Segmentation (FeTS) Challenge represents the paradigm for real-world algorithmic performance evaluation. The FeTS challenge is a competition to benchmark (i) federated learning aggregation algorithms and (ii) state-of-the-art segmentation algorithms, across multiple international sites. Weight aggregation and client selection techniques were compared using a multicentric brain tumor dataset in realistic federated learning simulations, yielding benefits for adaptive weight aggregation, and efficiency gains through client sampling. Quantitative performance evaluation of state-of-the-art segmentation algorithms on data distributed internationally across 32 institutions yielded good generalization on average, albeit the worst-case performance revealed data-specific modes of failure. Similar multi-site setups can help validate the real-world utility of healthcare AI algorithms in the future.

Involved external institutions

Indiana University United States (USA) (US) Henry Ford Health System (HFHS) United States (USA) (US) Université de Sherbrooke Canada (CA) Thomas Jefferson University Hospitals United States (USA) (US) Ohio State University United States (USA) (US) University College Hospital Nigeria (NG) University of Colorado Anschutz Medical Campus United States (USA) (US) Leeds Teaching Hospitals NHS Trust United Kingdom (GB) Queen's University Canada (CA) Brown University United States (USA) (US) Universitätsspital Zürich (USZ) Switzerland (CH) Shanghai Jiao Tong University / 上海交通大学 China (CN) Icahn School of Medicine at Mount Sinai United States (USA) (US) Nvidia Corporation United States (USA) (US) Shanghai Artificial Intelligence Laboratory / 上海人工智能实验室 China (CN) École pour l'informatique et les techniques avancées (EPITA) France (FR) National Taiwan Normal University (NTNU) / 國立臺灣師範大學 Taiwan (TW) Turku Amk - Turku University of Applied Sciences Finland (FI) Universidade Estadual de Campinas (UNICAMP) / University of Campinas Brazil (BR) Nanjing University of Science and Technology China (CN) Korea Advanced Institute of Science and Technology (KAIST) Korea, Republic of (KR) Luxembourg Institute of Health (CRP-Santé) Luxembourg (LU) École Normale Supérieure - PSL (ENS) France (FR) Indian Institute of Technology (IIT), Delhi India (IN) Technische Universität München (TUM) Germany (DE) Mazumdar Shaw Medical Foundation (MSMF) India (IN) National Taiwan University (NTU) Taiwan (TW) Escuela Superior Politécnica del Litoral (ESPOL) Ecuador (EC) Kepler Universitätsklinikum (KUK) Austria (AT) Middle East Technical University (METU) / Orta Doğu Teknik Üniversitesi Turkey (TR) Universidad de Concepcion Chile (CL) University of Calgary Canada (CA) Flinders Medical Centre Australia (AU) New York University (NYU) United States (USA) (US) Indian Institute of Technology Kanpur (IITK) India (IN) University of Iowa United States (USA) (US) University of Wisconsin - Madison United States (USA) (US) Case Western Reserve University United States (USA) (US) IBM Researc Ireland Ireland (IE) Sociedad de Lucha Contra el Cáncer del Ecuador (SOLCE) Ecuador (EC) Washington University in St. Louis United States (USA) (US) Vanderbilt University United States (USA) (US) Inselspital, Universitätsspital Bern Switzerland (CH) A-Star Institute of High Performance Computing Singapore (SG) University of Michigan United States (USA) (US) Deutsches Krebsforschungszentrum (DKFZ) Germany (DE) Intel Corporation United States (USA) (US) American College of Radiology United States (USA) (US) Universitätsmedizin Rostock Germany (DE) Universitat de Barcelona (UB) / University of Barcelona Spain (ES) Northwestern Polytechnical University / 西北工业大学 (NPU/NWPU) China (CN) Monash University Australia (AU) Universitätsklinikum Heidelberg Germany (DE) The Chinese University of Hong Kong (CUHK) China (CN) Graylight Imaging Poland (PL) Clinix Healthcare Nigeria (NG) University of Alberta Canada (CA) University of Southern California (USC) United States (USA) (US) University of North Carolina at Charlotte (UNCC) United States (USA) (US) Georgetown University United States (USA) (US) Karlsruhe Institute of Technology (KIT) Germany (DE) Helsingin yliopisto / University of Helsinki Finland (FI) Universitäts-Kinderspital Zürich Switzerland (CH) Universitätsklinikum Essen Germany (DE) Leidos Biomedical Research Inc. United States (USA) (US) Indian Institute of Technology Madras India (IN) Penn Medicine United States (USA) (US) Obafemi Awolowo University Nigeria (NG) Sage Bionetworks United States (USA) (US) MLCommons Association United States (USA) (US) University of Patras (UPATRAS) Greece (GR) University of Texas Southwestern Medical Center (UT Southwestern) United States (USA) (US) Changping Laboratory China (CN) University of Virginia (UVA) United States (USA) (US) Lahore University of Management Sciences (LUMS) Pakistan (PK) University of Pennsylvania (UPenn) United States (USA) (US) University of California San Francisco (UCSF) United States (USA) (US) St. Michael's Hospital Canada (CA) Symbiosis International University (SIU) India (IN) Ruprecht-Karls-Universität Heidelberg Germany (DE) Consorci Mar Parc de Salut de Barcelona Spain (ES) Harvard University United States (USA) (US) Clínica Imbanaco Quirónsalud Colombia (CO) Massachusetts General Hospital United States (USA) (US) Yonsei University Health System (YUHS) Korea, Republic of (KR) Masaryk University Czech Republic (CZ) University of Valle / Universidad del Valle (Univalle) Colombia (CO) ScaDS.AI Dresden/Leipzig (Center for Scalable Data Analytics and Artificial Intelligence) Germany (DE) National Institute of Mental Health and Neuroscience (NIMHANS) India (IN) University of Texas MD Anderson Cancer Center United States (USA) (US) Tata Memorial Hospital India (IN) Catalan Institute of Oncology / Institut Català d'Oncologia (ICO) Spain (ES) Baylor College of Medicine United States (USA) (US) Hospital Clínic de Barcelona Spain (ES) National University of Singapore (NUS) Singapore (SG) University Hospital Brno Czech Republic (CZ) University of Texas at Dallas (UTD / UT Dallas) United States (USA) (US) Klinikum rechts der Isar Germany (DE) Instituto Federal de Educação, Ciência e Tecnologia de Sergipe (IFS) Brazil (BR) Cornell University United States (USA) (US) City St George's, University of London United Kingdom (GB) University of Zurich / Universität Zürich (UZH) Switzerland (CH) Universitas Islam Nahdlatul Ulama Jepara (UNISNU) Indonesia (ID) Fuzhou University (FZU) China (CN) Universidade Federal do Paraná (UFPR) Brazil (BR) Queen Mary University of London United Kingdom (GB) Maria Skłodowska-Curie Institute of Oncology / Centrum Onkologii–Instytut im. Marii Skłodowskiej-Curie w Warszawie Poland (PL) National Tsing Hua University (NTHU) Taiwan (TW) Medical College of Wisconsin (MCW) United States (USA) (US) Hong Kong University of Science and Technology (HKUST) / 香港科技大學 China (CN) Mayo Clinic United States (USA) (US) Columbian College of Arts and Sciences United States (USA) (US) Ukrainian Catholic University / Ukraiński Uniwersytet Katolicki Poland (PL) Institut Gustave-Roussy France (FR) Novosibirsk State University (NSU) Russian Federation (RU) University of Bergen / Universitetet i Bergen Norway (NO) Zhejiang University / Chekiang University / 浙江大学 China (CN) Indian Institute of Information Technology Vadodara (IIIT V) India (IN) Skolkovo Institute of Science and Technology Russian Federation (RU) Fujian Normal University / 福建师范大学 China (CN) National Taiwan University of Science and Technology (NTUST) / 國立臺灣科技大學 / Taiwan Tech / 臺科大 Taiwan (TW) MedStar Georgetown University Hospital United States (USA) (US) Columbia University Irving Medical Center (CUIMC) United States (USA) (US) Hospital INC – Neurological Institute of Curitiba / Hospital INC – Instituto de Neurologia de Curitiba Brazil (BR) Indiana University – Purdue University Indianapolis United States (USA) (US) Beihang University, Beijing University of Aeronautics and Astronautics (BUAA) / 北京航空航天大学 China (CN) Universidad Católica de Cuenca Poland (PL) Riphah International University Pakistan (PK) Helmholtz Zentrum München - Deutsches Forschungszentrum für Gesundheit und Umwelt (HMGU) / Helmholtz Munich Germany (DE) University of Pittsburgh United States (USA) (US) University of Alabama at Birmingham (UAB) United States (USA) (US) University of Edinburgh United Kingdom (GB)

How to cite

APA:

Zenk, M., Baid, U., Pati, S., Linardos, A., Edwards, B., Sheller, M.,... Bakas, S. (2025). Towards fair decentralized benchmarking of healthcare AI algorithms with the Federated Tumor Segmentation (FeTS) challenge. Nature Communications, 16(1). https://doi.org/10.1038/s41467-025-60466-1

MLA:

Zenk, Maximilian, et al. "Towards fair decentralized benchmarking of healthcare AI algorithms with the Federated Tumor Segmentation (FeTS) challenge." Nature Communications 16.1 (2025).

BibTeX: Download