The Liver Tumor Segmentation Benchmark (LiTS)

Bilic P, Christ P, Li HB, Vorontsov E, Ben-Cohen A, Kaissis G, Szeskin A, Jacobs C, Mamani GEH, Chartrand G, Lohoefer F, Holch JW, Sommer W, Hofmann F, Hostettler A, Lev-Cohain N, Drozdzal M, Amitai MM, Vivanti R, Sosna J, Ezhov I, Sekuboyina A, Navarro F, Kofler F, Paetzold JC, Shit S, Hu X, Lipkova J, Rempfler M, Piraud M, Kirschke J, Wiestler B, Zhang Z, Huelsemeyer C, Beetz M, Ettlinger F, Antonelli M, Bae W, Bellver M, Bi L, Chen H, Chlebus G, Dam EB, Dou Q, Fu CW, Georgescu B, Giro-I-Nieto X, Gruen F, Han X, Heng PA, Hesser J, Moltz JH, Igel C, Isensee F, Jaeger P, Jia F, Kaluva KC, Khened M, Kim I, Kim JH, Kim S, Kohl S, Konopczynski T, Kori A, Krishnamurthi G, Li F, Li H, Li J, Li X, Lowengrub J, Ma J, Maier-Hein K, Maninis KK, Meine H, Merhof D, Pai A, Perslev M, Petersen J, Pont-Tuset J, Qi J, Qi X, Rippel O, Roth K, Sarasua I, Schenk A, Shen Z, Torres J, Wachinger C, Wang C, Weninger L, Wu J, Xu D, Yang X, Yu SCH, Yuan Y, Yue M, Zhang L, Cardoso J, Bakas S, Braren R, Heinemann V, Pal C, Tang A, Kadoury S, Soler L, Van Ginneken B, Greenspan H, Joskowicz L, Menze B (2023)

Publication Type: Journal article

Publication year: 2023

Journal

Book Volume: 84

Article Number: 102680

DOI: 10.1016/j.media.2022.102680

Abstract

In this work, we report the set-up and results of the Liver Tumor Segmentation Benchmark (LiTS), which was organized in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI) 2017 and the International Conferences on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2017 and 2018. The image dataset is diverse and contains primary and secondary tumors with varied sizes and appearances with various lesion-to-background levels (hyper-/hypo-dense), created in collaboration with seven hospitals and research institutions. Seventy-five submitted liver and liver tumor segmentation algorithms were trained on a set of 131 computed tomography (CT) volumes and were tested on 70 unseen test images acquired from different patients. We found that not a single algorithm performed best for both liver and liver tumors in the three events. The best liver segmentation algorithm achieved a Dice score of 0.963, whereas, for tumor segmentation, the best algorithms achieved Dices scores of 0.674 (ISBI 2017), 0.702 (MICCAI 2017), and 0.739 (MICCAI 2018). Retrospectively, we performed additional analysis on liver tumor detection and revealed that not all top-performing segmentation algorithms worked well for tumor detection. The best liver tumor detection method achieved a lesion-wise recall of 0.458 (ISBI 2017), 0.515 (MICCAI 2017), and 0.554 (MICCAI 2018), indicating the need for further research. LiTS remains an active benchmark and resource for research, e.g., contributing the liver-related segmentation tasks in http://medicaldecathlon.com/. In addition, both data and online evaluation are accessible via https://competitions.codalab.org/competitions/17094.

Involved external institutions

Hebrew University of Jerusalem Israel (IL) Technische Universität München (TUM) Germany (DE) École Polytechnique de Montréal Canada (CA) Tel Aviv University Israel (IL) University of Sydney (USYD) Australia (AU) Ruprecht-Karls-Universität Heidelberg Germany (DE) Radboud University Nijmegen Netherlands (NL) Centre de recherche du C.H.U.M. Canada (CA) Ludwig-Maximilians-Universität (LMU) Germany (DE) Research Institute against Digestive Cancer / Institut de Recherche contre les Cancers de l’Appareil Digestif (IRCAD) France (FR) Rafael Advanced Defense Systems / רפאל - מערכות לחימה מתקדמות בעמ Israel (IL) Imperial College London / The Imperial College of Science, Technology and Medicine United Kingdom (GB) Harvard University United States (USA) (US) Nanjing University China (CN) King’s College London United Kingdom (GB) Barcelona Supercomputing Center / Centro Nacional de Supercomputación Spain (ES) Hong Kong University of Science and Technology (HKUST) / 香港科技大學 China (CN) Fraunhofer-Institut für Bildgestützte Medizin (MEVIS) Germany (DE) University of Copenhagen Denmark (DK) The Chinese University of Hong Kong (CUHK) China (CN) Siemens AG, Healthcare Sector Germany (DE) Polytechnic University of Catalonia / Universitat Politècnica de Catalunya Spain (ES) Technische Universität Braunschweig Germany (DE) University of North Carolina at Chapel Hill United States (USA) (US) Rheinisch-Westfälische Technische Hochschule (RWTH) Aachen Germany (DE) Deutsches Krebsforschungszentrum (DKFZ) Germany (DE) Chinese Academy of Sciences (CAS) / 中国科学院 China (CN) Indian Institute of Technology Madras India (IN) SenseTime / 商汤科技 Hong Kong (HK) Guangdong University of Foreign Studies (GDUFS) / 广东外语外贸大学 China (CN) Philips China (CN) University of Hong Kong (HKU) / 香港大學 Hong Kong (HK) University of California Irvine United States (USA) (US) Nanjing University of Science and Technology China (CN) Universitätsklinikum Heidelberg Germany (DE) Eidgenössische Technische Hochschule Zürich (ETHZ) / Swiss Federal Institute of Technology in Zurich Switzerland (CH) Sungkyunkwan University (SKKU) Korea, Republic of (KR) University of Electronic Science and Technology of China (UESTC) / 电子科技大学 China (CN) Eberhard Karls Universität Tübingen Germany (DE) University of Illinois at Urbana-Champaign United States (USA) (US) Royal Institute of Technology / Kungliga Tekniska Högskolan (KTH) Sweden (SE) Tencent Holdings Ltd / 腾讯控股有限公司 China (CN) Nvidia Corporation United States (USA) (US) Icahn School of Medicine at Mount Sinai United States (USA) (US) CGG France (FR) University of Pennsylvania United States (USA) (US) Universitätsklinikum der Ludwig-Maximilians-Universität München Germany (DE) Université de Montréal Canada (CA)

How to cite

APA:

Bilic, P., Christ, P., Li, H.B., Vorontsov, E., Ben-Cohen, A., Kaissis, G.,... Menze, B. (2023). The Liver Tumor Segmentation Benchmark (LiTS). Medical Image Analysis, 84. https://dx.doi.org/10.1016/j.media.2022.102680

MLA:

Bilic, Patrick, et al. "The Liver Tumor Segmentation Benchmark (LiTS)." Medical Image Analysis 84 (2023).

BibTeX: Download