Known Operator Learning for a Hybrid Magnetic Resonance/X-ray Imaging Acquisition Scheme - Syben, Christopher (Friedrich-Alexander-Universität Erlangen-Nürnberg, 2021)
With today’s technology, various non-invasive imaging methods provide detailed insight into the patient’s anatomy and support the physician during surgery. Depending on the signal type of each modality, the information collected is limited to narrow parts of the overall available information. A combination of several modalities makes it possible to increase the amount of information simultaneously made available to the physician. Especially a possible combination of the two most frequently used modalities, namely magnetic resonance imaging (MRI) and X-ray/computed tomography (CT) is of particular interest. Apart from the physical challenges, such a combination also poses a challenge to the acquisition scheme for a meaningful simultaneous acquisition with both modalities. This is especially true for the interventional environment, which implies additional requirements such as the real-time capability of the acquisition scheme. Such a combination, comes along with limitations for the signal recording schemes, up to the degree that an analytically correct solution can no longer be derived. Recently, there have been different attempts to overcome such limitations with machine learning. However, the solutions found are only partially comprehensible, and signal authenticity cannot be guaranteed. With the integration of prior knowledge about signal acquisition processes, such models can also be used in the medical field. In this thesis, we outline an acquisition scheme for a novel hybrid magnetic resonance (MR)/X-ray imaging system for the interventional environment and investigate the realization of parts of the scheme. Further, we investigate the concept of known operator learning, derive an implementation for operators in CT, and utilize the concept to enable the aforementioned hybrid MR/X-ray acquisition scheme. First, we investigate the possibilities and limitations for incorporating prior knowledge about the signal processing chain into machine learning algorithms. Using the universal approximation theorem (UAT), the benefits of mixing prior knowledge with neural networks are theoretically analyzed. Such a learning pipeline enriched with known operators allows the reduction of trainable parameters, and constrains the pipeline such that signal authenticity can be met, while the approximative power of deep learning (DL) can be utilized. In the course of the thesis, the concept is implemented for the CT reconstruction and evaluated from a methodological and algorithmic point of view. The results show that CT operators can be integrated into neural networks allowing gradient flow through the whole pipeline. The experiments suggest that such mixed pipelines can be trained only with numerical data if designed specifically for the problem. Furthermore, an open-source software framework called PYRO-NN is developed to make these benefits accessible to a broader community. Second, we develop a novel hybrid MR/X-ray acquisition scheme for image-guided interventions. The scheme acknowledges the high contrast diversity of MRI and the benefits of fast, high-resolution X-ray imaging. The information provided by both modalities is captured in different domains, making a subsequent exploitation of the complementary signals difficult, and is identified as a major obstacle to fully benefit from the simultaneous hybrid acquisition. The proposed scheme results in multiple orthographic MR projections and, in one perspectively distorted X-ray projection per frame. In the course of the thesis, we first develop a novel tomographic conversion scheme to overcome the limitations of classical geometrical rebinning. The tomographic rebinning is derived and performed utilizing the known operator learning scheme to learn an efficient convolution-based algorithm for the conversion. The novel concept is first designed and comprehensively evaluated on the 1-D/2-D case against the baseline method. Subsequently, the tomographic rebinning concept is developed for the clinically relevant 2-D/3-D case. In the course of the experiments, we can show that the efficient convolution-based algorithm can be learned with purely numeric training data. The presented method overcomes the limitation of the baseline method and provides sharper and more distinct projection than the baseline method. Hence, the learned tomographic rebinning algorithm is promising for the proposed hybrid MR/X-ray acquisition scheme, and the underlying known operator learning concept encourages further integration of the acquisition and contrast parameters into the trainable pipeline. In general, the benefit of incorporating domain knowledge into neural networks is highly promising and is possible for various other tasks even beyond medical imaging.<br
Keywords: CT; Machine Learning; Known Operator Learaning; Hybrid Imaging