pix2xray: converting RGB images into X-rays using generative adversarial networks
Haiderbhai M, Ledesma S, Lee SC, Seibold M, Fuernstahl P, Navab N, Fallavollita P (2020)
Publication Type: Journal article
Publication year: 2020
Journal
Book Volume: 15
Pages Range: 973-980
Journal Issue: 6
DOI: 10.1007/s11548-020-02159-2
Abstract
Purpose: We propose a novel methodology for generating synthetic X-rays from 2D RGB images. This method creates accurate simulations for use in non-diagnostic visualization problems where the only input comes from a generic camera. Traditional methods are restricted to using simulation algorithms on 3D computer models. To solve this problem, we propose a method of synthetic X-ray generation using conditional generative adversarial networks (CGANs). Methods: We create a custom synthetic X-ray dataset generator to generate image triplets for X-ray images, pose images, and RGB images of natural hand poses sampled from the NYU hand pose dataset. This dataset is used to train two general-purpose CGAN networks, pix2pix and CycleGAN, as well as our novel architecture called pix2xray which expands upon the pix2pix architecture to include the hand pose into the network. Results: Our results demonstrate that our pix2xray architecture outperforms both pix2pix and CycleGAN in producing higher-quality X-ray images. We measure higher similarity metrics in our approach, with pix2pix coming in second, and CycleGAN producing the worst results. Our network performs better in the difficult cases which involve high occlusion due to occluded poses or large rotations. Conclusion: Overall our work establishes a baseline that synthetic X-rays can be simulated using 2D RGB input. We establish the need for additional data such as the hand pose to produce clearer results and show that future research must focus on more specialized architectures to improve overall image clarity and structure.
Involved external institutions
University of Ottawa
Canada (CA)
Johns Hopkins University (JHU)
United States (USA) (US)
University of Zurich / Universität Zürich (UZH)
Switzerland (CH)
Canada (CA)
Johns Hopkins University (JHU)
United States (USA) (US)
University of Zurich / Universität Zürich (UZH)
Switzerland (CH)
How to cite
APA:
Haiderbhai, M., Ledesma, S., Lee, S.C., Seibold, M., Fuernstahl, P., Navab, N., & Fallavollita, P. (2020). pix2xray: converting RGB images into X-rays using generative adversarial networks. International Journal of Computer Assisted Radiology and Surgery, 15(6), 973-980. https://dx.doi.org/10.1007/s11548-020-02159-2
MLA:
Haiderbhai, Mustafa, et al. "pix2xray: converting RGB images into X-rays using generative adversarial networks." International Journal of Computer Assisted Radiology and Surgery 15.6 (2020): 973-980.
BibTeX: Download