METhodological RadiomICs Score (METRICS): a quality scoring tool for radiomics research endorsed by EuSoMII

Kocak B, Akinci D’Antonoli T, Mercaldo N, Alberich-Bayarri A, Baessler B, Ambrosini I, Andreychenko AE, Bakas S, Beets-Tan RG, Bressem K, Buvat I, Cannella R, Cappellini LA, Cavallo AU, Chepelev LL, Chu LCH, Demircioglu A, deSouza NM, Dietzel M, Fanni SC, Fedorov A, Fournier LS, Giannini V, Girometti R, Groot Lipman KB, Kalarakis G, Kelly BS, Klontzas ME, Koh DM, Kotter E, Lee HY, Maas M, Marti-Bonmati L, Müller H, Obuchowski N, Orlhac F, Papanikolaou N, Petrash E, Pfaehler E, Pinto dos Santos D, Ponsiglione A, Sabater S, Sardanelli F, Seeböck P, Sijtsema NM, Stanzione A, Traverso A, Ugga L, Vallières M, van Dijk LV, van Griethuysen JJ, van Hamersvelt RW, van Ooijen P, Vernuccio F, Wang A, Williams S, Witowski J, Zhang Z, Zwanenburg A, Cuocolo R (2024)

Publication Type: Journal article

Publication year: 2024

Journal

Book Volume: 15

Article Number: 8

Journal Issue: 1

DOI: 10.1186/s13244-023-01572-w

Abstract

Purpose: To propose a new quality scoring tool, METhodological RadiomICs Score (METRICS), to assess and improve research quality of radiomics studies. Methods: We conducted an online modified Delphi study with a group of international experts. It was performed in three consecutive stages: Stage#1, item preparation; Stage#2, panel discussion among EuSoMII Auditing Group members to identify the items to be voted; and Stage#3, four rounds of the modified Delphi exercise by panelists to determine the items eligible for the METRICS and their weights. The consensus threshold was 75%. Based on the median ranks derived from expert panel opinion and their rank-sum based conversion to importance scores, the category and item weights were calculated. Result: In total, 59 panelists from 19 countries participated in selection and ranking of the items and categories. Final METRICS tool included 30 items within 9 categories. According to their weights, the categories were in descending order of importance: study design, imaging data, image processing and feature extraction, metrics and comparison, testing, feature processing, preparation for modeling, segmentation, and open science. A web application and a repository were developed to streamline the calculation of the METRICS score and to collect feedback from the radiomics community. Conclusion: In this work, we developed a scoring tool for assessing the methodological quality of the radiomics research, with a large international panel and a modified Delphi protocol. With its conditional format to cover methodological variations, it provides a well-constructed framework for the key methodological concepts to assess the quality of radiomic research papers. Critical relevance statement: A quality assessment tool, METhodological RadiomICs Score (METRICS), is made available by a large group of international domain experts, with transparent methodology, aiming at evaluating and improving research quality in radiomics and machine learning. Key points: • A methodological scoring tool, METRICS, was developed for assessing the quality of radiomics research, with a large international expert panel and a modified Delphi protocol. • The proposed scoring tool presents expert opinion-based importance weights of categories and items with a transparent methodology for the first time. • METRICS accounts for varying use cases, from handcrafted radiomics to entirely deep learning-based pipelines. • A web application has been developed to help with the calculation of the METRICS score (https://metricsscore.github.io/metrics/METRICS.html) and a repository created to collect feedback from the radiomics community (https://github.com/metricsscore/metrics). Graphical Abstract: [Figure not available: see fulltext.]

Authors with CRIS profile

Involved external institutions

University of Toronto Canada (CA) University of Pisa / Università di Pisa (UniPi) Italy (IT) Veterans Affairs Healthcare System Boston and Harvard Medical School United States (USA) (US) Universitätsklinikum Essen Germany (DE) HES-SO Valais-Wallis / University of Applied Sciences and Arts Western Switzerland Valais / Fachhochschule Westschweiz - Wallis Switzerland (CH) Cleveland Clinic United States (USA) (US) Nationales Centrum für Tumorerkrankungen Dresden (NCT/UCC) Germany (DE) Institut Curie France (FR) Champalimaud Foundation / Fundação Champalimaud Portugal (PT) Ministry of Health of the Russian Federation / Министерство здравоохранения Российской Федерации Russian Federation (RU) Forschungszentrum Jülich / Research Centre Jülich (FZJ) Germany (DE) Universitätsklinikum Köln Germany (DE) Istituto Dermopatico dell'Immacolata Italy (IT) New York University (NYU) United States (USA) (US) Griffith University Australia (AU) Università degli Studi di Salerno Italy (IT) Netherlands Cancer Institute (NKI) Netherlands (NL) Charité - Universitätsmedizin Berlin Germany (DE) Università degli Studi di Palermo Italy (IT) Humanitas University Italy (IT) Medizinische Universität Wien Austria (AT) University Medical Center Groningen (UMCG) / Universitair Medisch Centrum Groningen Netherlands (NL) University of Auckland New Zealand (NZ) Maastro Clinic Netherlands (NL) Università degli Studi di Napoli Federico II Italy (IT) Université de Sherbrooke Canada (CA) Georges Pompidou European Hospital / Hôpital Européen Georges-Pompidou (HEGP) France (FR) St Vincent's University Hospital Ireland (IE) University General Hospital of Heraklion Greece (GR) University of Turin / Università degli Studi di Torino (UNITO) Italy (IT) University of Udine / Università degli Studi di Udine Italy (IT) Karolinska University Hospital / Karolinska Universitetssjukhuset Sweden (SE) Royal Marsden Hospital / The Royal Marsden NHS Foundation Trust United Kingdom (GB) Universitätsklinikum Freiburg Germany (DE) University of Health Sciences Vientiane / ມະຫາວິທະຍາໄລ ວິທະຍາສາດ ສຸຂະພາບ Lao People's Democratic Republic (LA) Sungkyunkwan University (SKKU) Korea, Republic of (KR) Kantonsspital Baselland Switzerland (CH) Massachusetts General Hospital United States (USA) (US) University of Amsterdam Netherlands (NL) Quantitative Imaging Biomarkers in Medicine Spain (ES) Hospital Universitario y Politécnico de La Fe Spain (ES) Universitätsklinikum Würzburg Germany (DE) Johns Hopkins University (JHU) United States (USA) (US) ITMO University / Национальный исследовательский университет ИТМО Russian Federation (RU) Indiana University – Purdue University Indianapolis United States (USA) (US) The Institute of Cancer Research (ICR) United Kingdom (GB) Complejo Hospitalario Universitario de Albacete Spain (ES) Università degli studi di Milano Italy (IT) University Medical Centre Utrecht (UMC Utrecht) Netherlands (NL) Nottingham University Hospitals United Kingdom (GB)

How to cite

APA:

Kocak, B., Akinci D’Antonoli, T., Mercaldo, N., Alberich-Bayarri, A., Baessler, B., Ambrosini, I.,... Cuocolo, R. (2024). METhodological RadiomICs Score (METRICS): a quality scoring tool for radiomics research endorsed by EuSoMII. Insights into Imaging, 15(1). https://dx.doi.org/10.1186/s13244-023-01572-w

MLA:

Kocak, Burak, et al. "METhodological RadiomICs Score (METRICS): a quality scoring tool for radiomics research endorsed by EuSoMII." Insights into Imaging 15.1 (2024).

BibTeX: Download