CG-SENSE revisited: Results from the first ISMRM reproducibility challenge

Maier O, Baete SH, Fyrdahl A, Hammernik K, Harrevelt S, Kasper L, Karakuzu A, Loecher M, Patzig F, Tian Y, Wang K, Gallichan D, Uecker M, Knoll F (2021)

Publication Type: Journal article, Review article

Publication year: 2021

Journal

Magnetic Resonance in Medicine Wiley

Book Volume: 85

Pages Range: 1821-1839

Journal Issue: 4

DOI: 10.1002/mrm.28569

Abstract

Purpose: The aim of this work is to shed light on the issue of reproducibility in MR image reconstruction in the context of a challenge. Participants had to recreate the results of “Advances in sensitivity encoding with arbitrary k-space trajectories" by Pruessmann et al. Methods: The task of the challenge was to reconstruct radially acquired multicoil k-space data (brain/heart) following the method in the original paper, reproducing its key figures. Results were compared to consolidated reference implementations created after the challenge, accounting for the two most common programming languages used in the submissions (Matlab/Python). Results: Visually, differences between submissions were small. Pixel-wise differences originated from image orientation, assumed field-of-view, or resolution. The reference implementations were in good agreement, both visually and in terms of image similarity metrics. Discussion and Conclusion: While the description level of the published algorithm enabled participants to reproduce CG-SENSE in general, details of the implementation varied, for example, density compensation or Tikhonov regularization. Implicit assumptions about the data lead to further differences, emphasizing the importance of sufficient metadata accompanying open datasets. Defining reproducibility quantitatively turned out to be nontrivial for this image reconstruction challenge, in the absence of ground-truth results. Typical similarity measures like NMSE of SSIM were misled by image intensity scaling and outlier pixels. Thus, to facilitate reproducibility, researchers are encouraged to publish code and data alongside the original paper. Future methodological papers on MR image reconstruction might benefit from the consolidated reference implementations of CG-SENSE presented here, as a benchmark for methods comparison.

Authors with CRIS profile

Florian Knoll

Involved external institutions

Technische Universität Graz AT Austria (AT) New York University (NYU) US United States (USA) (US) Karolinska University Hospital / Karolinska Universitetssjukhuset SE Sweden (SE) Imperial College London / The Imperial College of Science, Technology and Medicine GB United Kingdom (GB) Eindhoven University of Technology / Technische Universiteit Eindhoven (TU/e) NL Netherlands (NL) Eidgenössische Technische Hochschule Zürich (ETHZ) / Swiss Federal Institute of Technology in Zurich CH Switzerland (CH) École Polytechnique de Montréal CA Canada (CA) Stanford University US United States (USA) (US) University of Utah US United States (USA) (US) University of California, Berkeley US United States (USA) (US) Cardiff University GB United Kingdom (GB) Universitätsklinikum Göttingen DE Germany (DE)

How to cite

APA:

Maier, O., Baete, S.H., Fyrdahl, A., Hammernik, K., Harrevelt, S., Kasper, L.,... Knoll, F. (2021). CG-SENSE revisited: Results from the first ISMRM reproducibility challenge. Magnetic Resonance in Medicine, 85(4), 1821-1839. https://dx.doi.org/10.1002/mrm.28569

MLA:

Maier, Oliver, et al. "CG-SENSE revisited: Results from the first ISMRM reproducibility challenge." Magnetic Resonance in Medicine 85.4 (2021): 1821-1839.

BibTeX: Download