3D fetal skull reconstruction from 2DUS via deep conditional generative networks - Cerrolaza, Juan J.; Li, Yuanwei; Biffi, Carlo; Gomez, Alberto; Sinclair, Matthew; Matthew, Jacqueline; Knight, Caronline; Kainz, Bernhard; Rueckert, Daniel (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2018)
2D ultrasound (US) is the primary imaging modality in antenatal healthcare. Despite the limitations of traditional 2D biometrics to characterize the true 3D anatomy of the fetus, the adoption of 3DUS is still very limited. This is particularly significant in developing countries and remote areas, due to the lack of experienced sonographers and the limited access to 3D technology. In this paper, we present a new deep conditional generative network for the 3D reconstruction of the fetal skull from 2DUS standard planes of the head routinely acquired during the fetal screening process. Based on the generative properties of conditional variational autoencoders (CVAE), our reconstruction architecture (REC-CVAE) directly integrates the three US standard planes as conditional variables to generate a unified latent space of the skull. Additionally, we propose HiREC-CVAE, a hierarchical generative network based on the different clinical relevance of each predictive view. The hierarchical structure of HiREC-CVAE allows the network to learn a sequence of nested latent spaces, providing superior predictive capabilities even in the absence of some of the 2DUS scans. The performance of the proposed architectures was evaluated on a dataset of 72 cases, showing accurate reconstruction capabilities from standard non-registered 2DUS images.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Fetal ultrasound; Generative model; Variational autoencoder

3D Probabilistic Segmentation and Volumetry from 2D Projection Images - Vlontzos, Athanasios; Budd, Samuel; Hou, Benjamin; Rueckert, Daniel; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2020)
X-Ray imaging is quick, cheap and useful for front-line care assessment and intra-operative real-time imaging (e.g., C-Arm Fluoroscopy). However, it suffers from projective information loss and lacks vital volumetric information on which many essential diagnostic biomarkers are based on. In this paper we explore probabilistic methods to reconstruct 3D volumetric images from 2D imaging modalities and measure the models’ performance and confidence. We show our models’ performance on large connected structures and we test for limitations regarding fine structures and image domain sensitivity. We utilize fast end-to-end training of a 2D-3D convolutional networks, evaluate our method on 117 CT scans segmenting 3D structures from digitally reconstructed radiographs (DRRs) with a Dice score of. Source code will be made available by the time of the conference.
CRIS-Team Scopus Importer:2021-08-17

3-D Reconstruction in Canonical Co-Ordinate Space from Arbitrarily Oriented 2-D Images - Hou, Benjamin; Khanal, Bishesh; Alansary, Amir; Mcdonagh, Steven; Davidson, Alice; Rutherford, Mary; Hajnal, Jo; Rueckert, Daniel; Glocker, Ben; Kainz, Bernhard (IEEE Transactions on Medical Imaging, 2018)
Limited capture range, and the requirement to provide high quality initialization for optimization-based 2-D/3-D image registration methods, can significantly degrade the performance of 3-D image reconstruction and motion compensation pipelines. Challenging clinical imaging scenarios, which contain significant subject motion, such as fetal in-utero imaging, complicate the 3-D image and volume reconstruction process. In this paper, we present a learning-based image registration method capable of predicting 3-D rigid transformations of arbitrarily oriented 2-D image slices, with respect to a learned canonical atlas co-ordinate system. Only image slice intensity information is used to perform registration and canonical alignment, no spatial transform initialization is required. To find image transformations, we utilize a convolutional neural network architecture to learn the regression function capable of mapping 2-D image slices to a 3-D canonical atlas space. We extensively evaluate the effectiveness of our approach quantitatively on simulated magnetic resonance imaging (MRI), fetal brain imagery with synthetic motion and further demonstrate qualitative results on real fetal MRI data where our method is integrated into a full reconstruction and motion compensation pipeline. Our learning based registration achieves an average spatial prediction error of 7 mm on simulated data and produces qualitatively improved reconstructions for heavily moving fetuses with gestational ages of approximately 20 weeks. Our model provides a general and computationally efficient solution to the 2-D/3-D registration initialization problem and is suitable for real-time scenarios.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Biomedical imaging; image reconstruction; image registration; machine learning; magnetic resonance imaging; motion compensation

Accessible GLSL shader programming - Toisoul, Antoine; Rueckert, Daniel; Kainz, Bernhard (European Association for Computer Graphics - 38th Annual Conference, EUROGRAPHICS 2017 - Education Papers, 2017)
Teaching fundamental principles of Computer Graphics requires a thoroughly prepared lecture alongside practical training. Modern graphics programming rarely provides a straightforward application programming interface (API) and the available APIs pose high entry barriers to students. Shader-based programming of standard graphics pipelines is often inaccessible through complex setup procedures and convoluted programming environments. In this paper we discuss an undergraduate entry level lecture with its according lab exercises. We present a programming framework that makes interactive graphics programming accessible while allowing to design individual tasks as instructive exercises to solidify the content of individual lecture units. The discussed teaching framework provides a well defined programmable graphics pipeline with geometry shading stages and image-based post processing functionality based on framebuffer objects. It is open-source and available online.
CRIS-Team Scopus Importer:2021-08-17

Adaptive scan strategies for fetal MRI imaging using slice to volume techniques - Kainz, Bernhard; Malamateniou, Christina; Ferrazzi, Giulio; Murgasova, Maria; Egger, Jan; Keraudren, Kevin; Rutherford, Mary; Hajnal, Joseph V.; Rueckert, Daniel (Proceedings - International Symposium on Biomedical Imaging, 2015)
In this paper several novel methods to account for fetal movements during fetal Magnetic Resonance Imaging (fetal MRI) are explored. We show how slice-to-volume reconstruction methods can be used to account for motion adaptively during the scan. Three candidate methods are tested for their feasibility and integrated into a computer simulation of fetal MRI. The first alters the main orientation of the stacks used for reconstruction, the second stops if too much motion occurs during slice acquisition and the third steers the orientation of each slice individually. Reconstruction informed adaptive scanning can provide a peak signal-to-noise ratio (PSNR) improvement of up to 2 dB after only two stacks of scanned slices and is more efficient with respect to the uncertainty of the final reconstruction.
CRIS-Team Scopus Importer:2021-08-17

Adnexal Mass Segmentation with Ultrasound Data Synthesis - Lebbos, Clara; Barcroft, Jen; Tan, Jeremy; Müller, Johanna; Baugh, Matthew; Vlontzos, Athanasios; Saso, Srdjan; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2022)
Ovarian cancer is the most lethal gynaecological malignancy. The disease is most commonly asymptomatic at its early stages and its diagnosis relies on expert evaluation of transvaginal ultrasound images. Ultrasound is the first-line imaging modality for characterising adnexal masses, it requires significant expertise and its analysis is subjective and labour-intensive, therefore open to error. Hence, automating processes to facilitate and standardise the evaluation of scans is desired in clinical practice. Using supervised learning, we have demonstrated that segmentation of adnexal masses is possible, however, prevalence and label imbalance restricts the performance on under-represented classes. To mitigate this we apply a novel pathology-specific data synthesiser. We create synthetic medical images with their corresponding ground truth segmentations by using Poisson image editing to integrate less common masses into other samples. Our approach achieves the best performance across all classes, including an improvement of up to 8% when compared with nnU-Net baseline approaches.
CRIS-Team Scopus Importer:2022-10-14

Altered biomechanical stimulation of the developing hip joint in presence of hip dysplasia risk factors - Verbruggen, Stefaan W.; Kainz, Bernhard; Shelmerdine, Susan C.; Arthurs, Owen J.; Hajnal, Joseph V.; Rutherford, Mary A.; Phillips, Andrew T. M.; Nowlan, Niamh C. (Journal of Biomechanics, 2018)
Fetal kicking and movements generate biomechanical stimulation in the fetal skeleton, which is important for prenatal musculoskeletal development, particularly joint shape. Developmental dysplasia of the hip (DDH) is the most common joint shape abnormality at birth, with many risk factors for the condition being associated with restricted fetal movement. In this study, we investigate the biomechanics of fetal movements in such situations, namely fetal breech position, oligohydramnios and primiparity (firstborn pregnancy). We also investigate twin pregnancies, which are not at greater risk of DDH incidence, despite the more restricted intra-uterine environment. We track fetal movements for each of these situations using cine-MRI technology, quantify the kick and muscle forces, and characterise the resulting stress and strain in the hip joint, testing the hypothesis that altered biomechanical stimuli may explain the link between certain intra-uterine conditions and risk of DDH. Kick force, stress and strain were found to be significantly lower in cases of breech position and oligohydramnios. Similarly, firstborn fetuses were found to generate significantly lower kick forces than non-firstborns. Interestingly, no significant difference was observed in twins compared to singletons. This research represents the first evidence of a link between the biomechanics of fetal movements and the risk of DDH, potentially informing the development of future preventative measures and enhanced diagnosis. Our results emphasise the importance of ultrasound screening for breech position and oligohydramnios, particularly later in pregnancy, and suggest that earlier intervention to correct breech position through external cephalic version could reduce the risk of hip dysplasia.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Cine MRI; Computational model; Developmental dysplasia of the hip; ECV; External cephalic version; Joint biomechanics; Musculoskeletal development

Anatomically Constrained Neural Networks (ACNNs): Application to Cardiac Image Enhancement and Segmentation - Oktay, Ozan; Ferrante, Enzo; Kamnitsas, Konstantinos; Heinrich, Mattias; Bai, Wenjia; Caballero, Jose; Cook, Stuart A.; De Marvao, Antonio; Dawes, Timothy; O'Regan, Declan P.; Kainz, Bernhard; Glocker, Ben; et al. (IEEE Transactions on Medical Imaging, 2018)
Incorporation of prior knowledge about organ shape and location is key to improve performance of image analysis approaches. In particular, priors can be useful in cases where images are corrupted and contain artefacts due to limitations in image acquisition. The highly constrained nature of anatomical objects can be well captured with learning-based techniques. However, in most recent and promising techniques such as CNN-based segmentation it is not obvious how to incorporate such prior knowledge. State-of-the-art methods operate as pixel-wise classifiers where the training objectives do not incorporate the structure and inter-dependencies of the output. To overcome this limitation, we propose a generic training strategy that incorporates anatomical prior knowledge into CNNs through a new regularisation model, which is trained end-to-end. The new framework encourages models to follow the global anatomical properties of the underlying anatomy (e.g. shape, label structure) via learnt non-linear representations of the shape. We show that the proposed approach can be easily adapted to different analysis tasks (e.g. image enhancement, segmentation) and improve the prediction accuracy of the state-of-the-art models. The applicability of our approach is shown on multi-modal cardiac data sets and public benchmarks. In addition, we demonstrate how the learnt deep models of 3-D shapes can be interpreted and used as biomarkers for classification of cardiac pathologies.
CRIS-Team Scopus Importer:2021-08-17
Keywords: convolutional neural network; image super-resolution; medical image segmentation; Shape prior

Artificial intelligence, fetal echocardiography, and congenital heart disease - Day, Thomas G.; Kainz, Bernhard; Hajnal, Jo; Razavi, Reza; Simpson, John M. (Prenatal Diagnosis, 2021)
There has been a recent explosion in the use of artificial intelligence (AI), which is now part of our everyday lives. Uptake in medicine has been more limited, although in several fields there have been encouraging results showing excellent performance when AI is used to assist in a well-defined medical task. Most of this work has been performed using retrospective data, and there have been few clinical trials published using prospective data. This review focuses on the potential uses of AI in the field of fetal cardiology. Ultrasound of the fetal heart is highly specific and sensitive in experienced hands, but despite this there is significant room for improvement in the rates of prenatal diagnosis of congenital heart disease in most countries. AI may be one way of improving this. Other potential applications in fetal cardiology include the provision of more accurate prognoses for individuals, and automatic quantification of various metrics including cardiac function. However, there are also ethical and governance concerns. These will need to be overcome before AI can be widely accepted in mainstream use. It is likely that a familiarity of the uses, and pitfalls, of AI will soon be mandatory for many healthcare professionals working in fetal cardiology.
CRIS-Team Scopus Importer:2021-08-17

Artificial intelligence to assist in the screening fetal anomaly ultrasound scan (PROMETHEUS): a randomised controlled trial - Day, T. G.; Matthew, J.; Budd, S.; Venturini, L.; Wright, R.; Farruggia, A.; Jamshidi, B.; Syngelaki, A.; Ling, H.; Tan, M.; Lai, J.; Brown, M.; et al. (Abstracts of the 34th World Congress on Ultrasound in Obstetrics and Gynecology, 15–18 September 2024, Budapest, Hungary, 2024)
CRIS-Team Scopus Importer:2024-09-20

A survey on active learning and human-in-the-loop deep learning for medical image analysis - Budd, Samuel; Robinson, Emma C.; Kainz, Bernhard (Medical Image Analysis, 2021)
Fully automatic deep learning has become the state-of-the-art technique for many tasks including image acquisition, analysis and interpretation, and for the extraction of clinically useful information for computer-aided detection, diagnosis, treatment planning, intervention and therapy. However, the unique challenges posed by medical image analysis suggest that retaining a human end-user in any deep learning enabled system will be beneficial. In this review we investigate the role that humans might play in the development and deployment of deep learning enabled diagnostic applications and focus on techniques that will retain a significant input from a human end user. Human-in-the-Loop computing is an area that we see as increasingly important in future research due to the safety-critical nature of working in the medical domain. We evaluate four key areas that we consider vital for deep learning in the clinical practice: (1) Active Learning to choose the best data to annotate for optimal model performance; (2) Interaction with model outputs - using iterative feedback to steer models to optima for a given prediction and offering meaningful ways to interpret and respond to predictions; (3) Practical considerations - developing full scale applications and the key considerations that need to be made before deployment; (4) Future Prospective and Unanswered Questions - knowledge gaps and related research fields that will benefit human-in-the-loop computing as they evolve. We offer our opinions on the most promising directions of research and how various aspects of each area might be unified towards common goals.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Active learning; Deep Learning; Human-in-the-Loop; Medical image analysis

Attention gated networks: Learning to leverage salient regions in medical images - Schlemper, Jo; Oktay, Ozan; Schaap, Michiel; Heinrich, Mattias; Kainz, Bernhard; Glocker, Ben; Rueckert, Daniel (Medical Image Analysis, 2019)
We propose a novel attention gate (AG) model for medical image analysis that automatically learns to focus on target structures of varying shapes and sizes. Models trained with AGs implicitly learn to suppress irrelevant regions in an input image while highlighting salient features useful for a specific task. This enables us to eliminate the necessity of using explicit external tissue/organ localisation modules when using convolutional neural networks (CNNs). AGs can be easily integrated into standard CNN models such as VGG or U-Net architectures with minimal computational overhead while increasing the model sensitivity and prediction accuracy. The proposed AG models are evaluated on a variety of tasks, including medical image classification and segmentation. For classification, we demonstrate the use case of AGs in scan plane detection for fetal ultrasound screening. We show that the proposed attention mechanism can provide efficient object localisation while improving the overall prediction performance by reducing false positives. For segmentation, the proposed architecture is evaluated on two large 3D CT abdominal datasets with manual annotations for multiple organs. Experimental results show that AG models consistently improve the prediction performance of the base architectures across different datasets and training sizes while preserving computational efficiency. Moreover, AGs guide the model activations to be focused around salient regions, which provides better insights into how model predictions are made. The source code for the proposed AG models is publicly available.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Attention gates; Fully convolutional networks; Image classification; Localisation; Segmentation; Soft attention

AutoDVT: Joint real-time classification for vein compressibility analysis in deep vein thrombosis ultrasound diagnostics - Tanno, Ryutaro; Makropoulos, Antonios; Arslan, Salim; Oktay, Ozan; Mischkewitz, Sven; Al-Noor, Fouad; Oppenheimer, Jonas; Mandegaran, Ramin; Kainz, Bernhard; Heinrich, Mattias P. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2018)
We propose a dual-task convolutional neural network (CNN) to fully automate the real-time diagnosis of deep vein thrombosis (DVT). DVT can be reliably diagnosed through evaluation of vascular compressibility at anatomically defined landmarks in streams of ultrasound (US) images. The combined real-time evaluation of these tasks has never been achieved before. As proof-of-concept, we evaluate our approach on two selected landmarks of the femoral vein, which can be identified with high accuracy by our approach. Our CNN is able to identify if a vein fully compresses with a F1 score of more than 90% while applying manual pressure with the ultrasound probe. Fully compressible veins robustly rule out DVT and such patients do not need to be referred to further specialist examination. We have evaluated our method on 1150 5–10 s compression image sequences from 115 healthy volunteers, which results in a data set size of approximately 200k labelled images. Our method yields a theoretical inference frame rate of more than 500 fps and we thoroughly evaluate the performance of 15 possible configurations.
CRIS-Team Scopus Importer:2021-08-17

Automated cardiovascular magnetic resonance image analysis with fully convolutional networks 08 Information and Computing Sciences 0801 Artificial Intelligence and Image Processing - Bai, Wenjia; Sinclair, Matthew; Tarroni, Giacomo; Oktay, Ozan; Rajchl, Martin; Vaillant, Ghislain; Lee, Aaron M.; Aung, Nay; Lukaschuk, Elena; Sanghvi, Mihir M.; Zemrak, Filip; Fung, Kenneth; et al. (Journal of Cardiovascular Magnetic Resonance, 2018)
Background: Cardiovascular resonance (CMR) imaging is a standard imaging modality for assessing cardiovascular diseases (CVDs), the leading cause of death globally. CMR enables accurate quantification of the cardiac chamber volume, ejection fraction and myocardial mass, providing information for diagnosis and monitoring of CVDs. However, for years, clinicians have been relying on manual approaches for CMR image analysis, which is time consuming and prone to subjective errors. It is a major clinical challenge to automatically derive quantitative and clinically relevant information from CMR images. Methods: Deep neural networks have shown a great potential in image pattern recognition and segmentation for a variety of tasks. Here we demonstrate an automated analysis method for CMR images, which is based on a fully convolutional network (FCN). The network is trained and evaluated on a large-scale dataset from the UK Biobank, consisting of 4,875 subjects with 93,500 pixelwise annotated images. The performance of the method has been evaluated using a number of technical metrics, including the Dice metric, mean contour distance and Hausdorff distance, as well as clinically relevant measures, including left ventricle (LV) end-diastolic volume (LVEDV) and end-systolic volume (LVESV), LV mass (LVM); right ventricle (RV) end-diastolic volume (RVEDV) and end-systolic volume (RVESV). Results: By combining FCN with a large-scale annotated dataset, the proposed automated method achieves a high performance in segmenting the LV and RV on short-axis CMR images and the left atrium (LA) and right atrium (RA) on long-axis CMR images. On a short-axis image test set of 600 subjects, it achieves an average Dice metric of 0.94 for the LV cavity, 0.88 for the LV myocardium and 0.90 for the RV cavity. The mean absolute difference between automated measurement and manual measurement is 6.1 mL for LVEDV, 5.3 mL for LVESV, 6.9 gram for LVM, 8.5 mL for RVEDV and 7.2 mL for RVESV. On long-axis image test sets, the average Dice metric is 0.93 for the LA cavity (2-chamber view), 0.95 for the LA cavity (4-chamber view) and 0.96 for the RA cavity (4-chamber view). The performance is comparable to human inter-observer variability. Conclusions: We show that an automated method achieves a performance on par with human experts in analysing CMR images and deriving clinically relevant measures.
CRIS-Team Scopus Importer:2021-08-17
Keywords: CMR image analysis; Fully convolutional networks; Machine learning

Automated Detection of Congenital Heart Disease in Fetal Ultrasound Screening - Tan, Jeremy; Au, Anselm; Meng, Qingjie; FinesilverSmith, Sandy; Simpson, John; Rueckert, Daniel; Razavi, Reza; Day, Thomas; Lloyd, David; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2020)
Prenatal screening with ultrasound can lower neonatal mortality significantly for selected cardiac abnormalities. However, the need for human expertise, coupled with the high volume of screening cases, limits the practically achievable detection rates. In this paper we discuss the potential for deep learning techniques to aid in the detection of congenital heart disease (CHD) in fetal ultrasound. We propose a pipeline for automated data curation and classification. During both training and inference, we exploit an auxiliary view classification task to bias features toward relevant cardiac structures. This bias helps to improve in F1-scores from 0.72 and 0.77 to 0.87 and 0.85 for healthy and CHD classes respectively.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Congenital heart disease; Fetal ultrasound

Automated fetal brain segmentation from 2D MRI slices for motion correction - Keraudren, K.; Kuklisova-Murgasova, M.; Kyriakopoulou, Vanessa; Malamateniou, C.; Rutherford, M. A.; Kainz, Bernhard; Hajnal, J. V.; Rueckert, Daniel (NeuroImage, 2014)
Motion correction is a key element for imaging the fetal brain in-utero using Magnetic Resonance Imaging (MRI). Maternal breathing can introduce motion, but a larger effect is frequently due to fetal movement within the womb. Consequently, imaging is frequently performed slice-by-slice using single shot techniques, which are then combined into volumetric images using slice-to-volume reconstruction methods (SVR). For successful SVR, a key preprocessing step is to isolate fetal brain tissues from maternal anatomy before correcting for the motion of the fetal head. This has hitherto been a manual or semi-automatic procedure. We propose an automatic method to localize and segment the brain of the fetus when the image data is acquired as stacks of 2D slices with anatomy misaligned due to fetal motion. We combine this segmentation process with a robust motion correction method, enabling the segmentation to be refined as the reconstruction proceeds. The fetal brain localization process uses Maximally Stable Extremal Regions (MSER), which are classified using a Bag-of-Words model with Scale-Invariant Feature Transform (SIFT) features. The segmentation process is a patch-based propagation of the MSER regions selected during detection, combined with a Conditional Random Field (CRF). The gestational age (GA) is used to incorporate prior knowledge about the size and volume of the fetal brain into the detection and segmentation process. The method was tested in a ten-fold cross-validation experiment on 66 datasets of healthy fetuses whose GA ranged from 22 to 39. weeks. In 85% of the tested cases, our proposed method produced a motion corrected volume of a relevant quality for clinical diagnosis, thus removing the need for manually delineating the contours of the brain before motion correction. Our method automatically generated as a side-product a segmentation of the reconstructed fetal brain with a mean Dice score of 93%, which can be used for further processing.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Bag-of-words; Brain extraction; Fetal MRI; Motion correction; MSER; SIFT

Automated localization of fetal organs in MRI using random forests with steerable features - Keraudren, Kevin; Kainz, Bernhard; Oktay, Ozan; Kyriakopoulou, Vanessa; Rutherford, Mary; Hajnal, Joseph V.; Rueckert, Daniel (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2015)
Fetal MRI is an invaluable diagnostic tool complementary to ultrasound thanks to its high contrast and resolution. Motion artifacts and the arbitrary orientation of the fetus are two main challenges of fetal MRI. In this paper, we propose a method based on Random Forests with steerable features to automatically localize the heart, lungs and liver in fetal MRI. During training, all MR images are mapped into a standard coordinate system that is defined by landmarks on the fetal anatomy and normalized for fetal age. Image features are then extracted in this coordinate system. During testing, features are computed for different orientations with a search space constrained by previously detected landmarks. The method was tested on healthy fetuses as well as fetuses with intrauterine growth restriction (IUGR) from 20 to 38 weeks of gestation. The detection rate was above 90% for all organs of healthy fetuses in the absence of motion artifacts. In the presence of motion, the detection rate was 83% for the heart, 78% for the lungs and 67% for the liver. Growth restriction did not decrease the performance of the heart detection but had an impact on the detection of the lungs and liver. The proposed method can be used to initialize subsequent processing steps such as segmentation or motion correction, as well as automatically orient the 3D volume based on the fetal anatomy to facilitate clinical examination.
CRIS-Team Scopus Importer:2021-08-17

Automated quality control in image segmentation: Application to the UK Biobank cardiovascular magnetic resonance imaging study - Robinson, Robert; Valindria, Vanya V.; Bai, Wenjia; Oktay, Ozan; Kainz, Bernhard; Suzuki, Hideaki; Sanghvi, Mihir M.; Aung, Nay; Paiva, Jose Miguel; Zemrak, Filip; Fung, Kenneth; Lukaschuk, Elena; et al. (Journal of Cardiovascular Magnetic Resonance, 2019)
Background: The trend towards large-scale studies including population imaging poses new challenges in terms of quality control (QC). This is a particular issue when automatic processing tools such as image segmentation methods are employed to derive quantitative measures or biomarkers for further analyses. Manual inspection and visual QC of each segmentation result is not feasible at large scale. However, it is important to be able to automatically detect when a segmentation method fails in order to avoid inclusion of wrong measurements into subsequent analyses which could otherwise lead to incorrect conclusions. Methods: To overcome this challenge, we explore an approach for predicting segmentation quality based on Reverse Classification Accuracy, which enables us to discriminate between successful and failed segmentations on a per-cases basis. We validate this approach on a new, large-scale manually-annotated set of 4800 cardiovascular magnetic resonance (CMR) scans. We then apply our method to a large cohort of 7250 CMR on which we have performed manual QC. Results: We report results used for predicting segmentation quality metrics including Dice Similarity Coefficient (DSC) and surface-distance measures. As initial validation, we present data for 400 scans demonstrating 99% accuracy for classifying low and high quality segmentations using the predicted DSC scores. As further validation we show high correlation between real and predicted scores and 95% classification accuracy on 4800 scans for which manual segmentations were available. We mimic real-world application of the method on 7250 CMR where we show good agreement between predicted quality metrics and manual visual QC scores. Conclusions: We show that Reverse classification accuracy has the potential for accurate and fully automatic segmentation QC on a per-case basis in the context of large-scale population imaging as in the UK Biobank Imaging Study.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Automatic quality control; Population imaging; Segmentation

Automatic brain localization in fetal MRI using superpixel graphs - Alansary, Amir; Lee, Matthew; Keraudren, Kevin; Kainz, Bernhard; Malamateniou, Christina; Rutherford, Mary; Hajnal, Joseph V.; Glocker, Ben; Rueckert, Daniel (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2015)
Fetal MRI is emerging as an effective, non-invasive tool in prenatal diagnosis and pregnancy follow-up. However, there is a significant variability of the position and orientation of the fetus in the MR images. This makes these images more difficult to analyze and interpret compared to standard adult MR imaging, which standardized anatomical imaging aligned planes. We address this issue by automatic localization of the fetal anatomy, in particular, the brain which is a structure of interest for many fetal MRI studies. We first extract superpixels followed by the computation of a histogram of features for each superpixel using bag of words based on dense scale invariant feature transform (DSIFT) descriptors. We construct a graph of superpixels and train a random forest classifier to distinguish between brain and non-brain superpixels. The localization framework has been tested on 55 MR datasets at gestational ages between 20–38 weeks. The proposed method was evaluated using 5-fold cross validation achieving a 94.55% brain detection accuracy rate.
CRIS-Team Scopus Importer:2021-08-17

Automatic detection of bowel disease with residual networks - Holland, Robert; Patel, Uday; Lung, Phillip; Chotzoglou, Elisa; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2019)
Crohn’s disease, one of two inflammatory bowel diseases (IBD), affects 200,000 people in the UK alone, or roughly one in every 500. We explore the feasibility of deep learning algorithms for identification of terminal ileal Crohn’s disease in Magnetic Resonance Enterography images on a small dataset. We show that they provide comparable performance to the current clinical standard, the MaRIA score, while requiring only a fraction of the preparation and inference time. Moreover, bowels are subject to high variation between individuals due to the complex and free-moving anatomy. Thus we also explore the effect of difficulty of the classification at hand on performance. Finally, we employ soft attention mechanisms to amplify salient local features and add interpretability.
CRIS-Team Scopus Importer:2021-08-17

Automatic differentiation for GPU-accelerated 2D/3D registration - Kainz, Bernhard (2008)

Automatic Segmentation of Lymphatic Perfusion in Patients with Congenital Single Ventricle Defects - Stegmaier, Marietta; Schröder, Christian; Müller, Johanna; Day, Thomas; Cuomo, Michela; Dewald, Oliver; Dittrich, Sven; Kainz, Bernhard (Bildverarbeitung für die Medizin 2024. BVM 2024, 2024)
The Fontan circulation is the surgical end-point for a variety of singleventricle congenital heart lesions. While recent decades have witnessed substantial improvements in survival rates, the associated physiology remains susceptible to severe complications such as protein-losing enteropathy and plastic bronchitis. These complications are often indicative of abnormal congestion in the lymphatic system, underscoring their significance as harbingers of impending comorbidities. An accurate assessment of congestion severity requires the detailed annotation of lymphatic perfusion patterns in each volumetric scan slice. The manual labelling of such intricate data is time-consuming and demands a high level of expertise, rendering it unfeasible within the confines of standard clinical protocols. We use a curated database consisting of manually annotated T2-weighted magnetic resonance imaging (MRI) scans from 71 Fontan patients post-surgery. Following the current state-of-the-art method for biomedical image segmentation, we evaluate its performance on multiple independent test sets regarding the degree of severity and imaging quality. Incorporating the best-performing model,we have developed a user-friendly interface for the automatic segmentation of lymphatic malformations, which will be published before the conference starts.
CRIS-Team Scopus Importer:2024-03-29

Automatic shadow detection in 2D ultrasound images - Meng, Qingjie; Baumgartner, Christian; Sinclair, Matthew; Housden, James; Rajchl, Martin; Gomez, Alberto; Hou, Benjamin; Toussaint, Nicolas; Zimmer, Veronika; Tan, Jeremy; Matthew, Jacqueline; Rueckert, Daniel; et al. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2018)
Automatically detecting acoustic shadows is of great importance for automatic 2D ultrasound analysis ranging from anatomy segmentation to landmark detection. However, variation in shape and similarity in intensity to other structures make shadow detection a very challenging task. In this paper, we propose an automatic shadow detection method to generate a pixel-wise, shadow-focused confidence map from weakly labelled, anatomically-focused images. Our method: (1) initializes potential shadow areas based on a classification task. (2) extends potential shadow areas using a GAN model. (3) adds intensity information to generate the final confidence map using a distance matrix. The proposed method accurately highlights the shadow areas in 2D ultrasound datasets comprising standard view planes as acquired during fetal screening. Moreover, the proposed method outperforms the state-of-the-art quantitatively and improves failure cases for automatic biometric measurement.
CRIS-Team Scopus Importer:2021-08-17

Automatic view planning with multi-scale deep reinforcement learning agents - Alansary, Amir; Le Folgoc, Loic; Vaillant, Ghislain; Oktay, Ozan; Li, Yuanwei; Bai, Wenjia; Passerat-Palmbach, Jonathan; Guerrero, Ricardo; Kamnitsas, Konstantinos; Hou, Benjamin; Mcdonagh, Steven; Glocker, Ben; et al. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2018)
We propose a fully automatic method to find standardized view planes in 3D image acquisitions. Standard view images are important in clinical practice as they provide a means to perform biometric measurements from similar anatomical regions. These views are often constrained to the native orientation of a 3D image acquisition. Navigating through target anatomy to find the required view plane is tedious and operator-dependent. For this task, we employ a multi-scale reinforcement learning (RL) agent framework and extensively evaluate several Deep Q-Network (DQN) based strategies. RL enables a natural learning paradigm by interaction with the environment, which can be used to mimic experienced operators. We evaluate our results using the distance between the anatomical landmarks and detected planes, and the angles between their normal vector and target. The proposed algorithm is assessed on the mid-sagittal and anterior-posterior commissure planes of brain MRI, and the 4-chamber long-axis plane commonly used in cardiac MRI, achieving accuracy of 1.53 mm, 1.98 mm and 4.84 mm, respectively.
CRIS-Team Scopus Importer:2021-08-17

A variational Bayesian method for similarity learning in non-rigid image registration - Grzech, Daniel; Azampour, Mohammad Farid; Glocker, Ben; Schnabel, Julia; Navab, Nassir; Kainz, Bernhard; Folgoc, Loic Le (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2022)
We propose a novel variational Bayesian formulation for diffeomorphic non-rigid registration of medical images, which learns in an unsupervised way a data-specific similarity metric. The proposed framework is general and may be used together with many existing image registration models. We evaluate it on brain MRI scans from the UK Biobank and show that use of the learnt similarity metric, which is parametrised as a neural network, leads to more accurate results than use of traditional functions, e.g. SSD and LCC, to which we initialise the model, without a negative impact on image registration speed or transformation smoothness. In addition, the method estimates the uncertainty associated with the transformation. The code and the trained models are available in a public repository: https://github.com/dgrzech/learnsim.
CRIS-Team Scopus Importer:2022-11-25
Keywords: Machine learning; Self-& semi-& meta- & unsupervised learning

Can Non-specialists Provide High Quality Gold Standard Labels in Challenging Modalities? - Budd, Samuel; Day, Thomas G.; Simpson, John M.; Lloyd, Karen; Matthew, Jacqueline; Skelton, Emily; Razavi, Reza; Kainz, Bernhard (Springer Science and Business Media Deutschland GmbH, 2021)
Probably yes.—Supervised Deep Learning dominates performance scores for many computer vision tasks and defines the state-of-the-art. However, medical image analysis lags behind natural image applications. One of the many reasons is the lack of well annotated medical image data available to researchers. One of the first things researchers are told is that we require significant expertise to reliably and accurately interpret and label such data. We see significant inter- and intra-observer variability between expert annotations of medical images. Still, it is a widely held assumption that novice annotators are unable to provide useful annotations for use by clinical Deep Learning models. In this work we challenge this assumption and examine the implications of using a minimally trained novice labelling workforce to acquire annotations for a complex medical image dataset. We study the time and cost implications of using novice annotators, the raw performance of novice annotators compared to gold-standard expert annotators, and the downstream effects on a trained Deep Learning segmentation model’s performance for detecting a specific congenital heart disease (hypoplastic left heart syndrome) in fetal ultrasound imaging.
Keywords: Labels; Novice; Annotations; Expert

CAS-Net: Conditional Atlas Generation and Brain Segmentation for Fetal MRI - Kainz, Bernhard; et al. (3rd International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, UNSURE 2021, and the 6th International Workshop on Perinatal, Preterm and Paediatric Image Analysis, PIPPI 2021, held in conjunction with the 24th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2021, 2021)

Causal reasoning in medical imaging - Vlontzos, Athanasios; Müller, Christine; Kainz, Bernhard (Trustworthy AI in Medical Imaging, 2025)
Medical image analysis is a vibrant research area that offers doctors and medical practitioners valuable insight and the ability to accurately diagnose and monitor disease. Machine learning provides an additional boost for this area. However, machine learning for medical image analysis is particularly vulnerable to natural biases like domain shifts that affect algorithmic performance and robustness. This chapter analyzes machine learning for medical image analysis within the framework of Technology Readiness Levels and reviews how causal analysis methods can fill a gap when creating robust and adaptable medical image analysis algorithms. This work provides a review of the methods using causality in medical imaging AI/ML, and highlights the potential of causal analysis to mitigate critical problems for clinical translatio

Complete Fetal Head Compounding from Multi-view 3D Ultrasound - Wright, Robert; Toussaint, Nicolas; Gomez, Alberto; Zimmer, Veronika; Khanal, Bishesh; Matthew, Jacqueline; Skelton, Emily; Kainz, Bernhard; Rueckert, Daniel; Hajnal, Joseph V.; Schnabel, Julia A. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2019)
Ultrasound (US) images suffer from artefacts which limit its diagnostic value, notably acoustic shadow. Shadows are dependent on probe orientation, with each view giving a distinct, partial view of the anatomy. In this work, we fuse the partially imaged fetal head anatomy, acquired from numerous views, into a single coherent compounding of the full anatomy. Firstly, a stream of freehand 3D US images is acquired, capturing as many different views as possible. The imaged anatomy at each time-point is then independently aligned to a canonical pose using an iterative spatial transformer network (iSTN), making our approach robust to fast fetal and probe motion. Secondly, images are fused by averaging only the best (most salient) features from all images, producing a more detailed compounding. Finally, the compounding is iteratively refined using a groupwise registration approach. We evaluate our compounding approach quantitatively and qualitatively, comparing it with average compounding and individual US frames. We also evaluate our alignment accuracy using two physically attached probes, that capture separate views simultaneously, providing ground-truth. Lastly, we demonstrate the potential clinical impact of our method for assessing cranial, facial and external ear abnormalities, with automated atlas-based masking and 3D volume rendering.
CRIS-Team Scopus Importer:2021-08-17

Computer oriented image acquisition of the liver: Toward a better numerical model for radiofrequency ablation - Muehl, Judith K.; Kainz, Bernhard; Portugaller, Horst R.; Stiegler, Philipp B.; Bauer, Christian H. (Proceedings of the 31st Annual International Conference of the IEEE Engineering in Medicine and Biology Society: Engineering the Future of Biomedicine, EMBC 2009, 2009)
Simulating physiological interventions for planning purposes requires an accurate virtual liver model as computation input. To best meet the demands the data acquisition has to be oriented on image processing purposes. We provide a CT imaging protocol which makes it possible to extract much more vessels with the same segmentation algorithms than when using them on data sets from the clinical routine. Medical evaluation of physiological models demand a statistical evaluation in a pre-clinical study, that means in a first step reproducible results for a large number of subjects. So data acquisition should be as automatic as possible without neglecting modeling demands. Image quality should be reproducible to guarantee an ongoing high quality of image processing results. We evaluate the protocol by comparison of segmentation results with results achieved on standard data sets from clinical routine using the same segmentation methods. Results show that typically up to ten times more vessels can be extracted and the surface accuracy is improved. ©2009 IEEE.
CRIS-Team Scopus Importer:2021-08-17

Computing CNN loss and gradients for pose estimation with Riemannian geometry - Hou, Benjamin; Miolane, Nina; Khanal, Bishesh; Lee, Matthew C. H.; Alansary, Amir; Mcdonagh, Steven; Hajnal, Jo V.; Rueckert, Daniel; Glocker, Ben; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2018)
Pose estimation, i.e. predicting a 3D rigid transformation with respect to a fixed co-ordinate frame in, SE(3), is an omnipresent problem in medical image analysis. Deep learning methods often parameterise poses with a representation that separates rotation and translation. As commonly available frameworks do not provide means to calculate loss on a manifold, regression is usually performed using the L2-norm independently on the rotation’s and the translation’s parameterisations. This is a metric for linear spaces that does not take into account the Lie group structure of SE(3). In this paper, we propose a general Riemannian formulation of the pose estimation problem, and train CNNs directly on SE(3) equipped with a left-invariant Riemannian metric. The loss between the ground truth and predicted pose (elements of the manifold) is calculated as the Riemannian geodesic distance, which couples together the translation and rotation components. Network weights are updated by back-propagating the gradient with respect to the predicted pose on the tangent space of the manifold SE(3). We thoroughly evaluate the effectiveness of our loss function by comparing its performance with popular and most commonly used existing methods, on tasks such as image-based localisation and intensity-based 2D/3D registration. We also show that hyper-parameters, used in our loss function to weight the contribution between rotations and translations, can be intrinsically calculated from the dataset to achieve greater performance margins.
CRIS-Team Scopus Importer:2021-08-17

Conditional Temporal Attention Networks for Neonatal Cortical Surface Reconstruction - Ma, Qiang; Li, Liu; Kyriakopoulou, Vanessa; Hajnal, Joseph V.; Robinson, Emma C.; Kainz, Bernhard; Rueckert, Daniel (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2023)
Cortical surface reconstruction plays a fundamental role in modeling the rapid brain development during the perinatal period. In this work, we propose Conditional Temporal Attention Network (CoTAN), a fast end-to-end framework for diffeomorphic neonatal cortical surface reconstruction. CoTAN predicts multi-resolution stationary velocity fields (SVF) from neonatal brain magnetic resonance images (MRI). Instead of integrating multiple SVFs, CoTAN introduces attention mechanisms to learn a conditional time-varying velocity field (CTVF) by computing the weighted sum of all SVFs at each integration step. The importance of each SVF, which is estimated by learned attention maps, is conditioned on the age of the neonates and varies with the time step of integration. The proposed CTVF defines a diffeomorphic surface deformation, which reduces mesh self-intersection errors effectively. It only requires 0.21 s to deform an initial template mesh to cortical white matter and pial surfaces for each brain hemisphere. CoTAN is validated on the Developing Human Connectome Project (dHCP) dataset with 877 3D brain MR images acquired from preterm and term born neonates. Compared to state-of-the-art baselines, CoTAN achieves superior performance with only 0.12 ± 0.03 mm geometric error and 0.07 ± 0.03% self-intersecting faces. The visualization of our attention maps illustrates that CoTAN indeed learns coarse-to-fine surface deformations automatically without intermediate supervision.
CRIS-Team Scopus Importer:2023-11-03

Confidence-Aware and Self-supervised Image Anomaly Localisation - Müller, Johanna; Baugh, Matthew; Tan, Jeremy; Dombrowski, Mischa; Kainz, Bernhard (Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, 2023)
Universal anomaly detection still remains a challenging problem in machine learning and medical image analysis. It is possible to learn an expected distribution from a single class of normative samples, e.g., through epistemic uncertainty estimates, auto-encoding models, or from synthetic anomalies in a self-supervised way. The performance of self-supervised anomaly detection approaches is still inferior compared to methods that use examples from known unknown classes to shape the decision boundary. However, outlier exposure methods often do not identify unknown unknowns. Here we discuss an improved self-supervised single-class training strategy that supports the approximation of probabilistic inference with loosen feature locality constraints. We show that up-scaling of gradients with histogram-equalised images is beneficial for recently proposed self-supervision tasks. Our method is integrated into several out-of-distribution (OOD) detection models and we show evidence that our method outperforms the state-of-the-art on various benchmark datasets.<br
Keywords: Anomaly detection; Out-of-distribution detection; Poisson image interpolation; Self-supervision

Confident head circumference measurement from ultrasound with real-time feedback for sonographers - Budd, Samuel; Sinclair, Matthew; Khanal, Bishesh; Matthew, Jacqueline; Lloyd, David; Gomez, Alberto; Toussaint, Nicolas; Robinson, Emma C.; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2019)
Manual estimation of fetal Head Circumference (HC) from Ultrasound (US) is a key biometric for monitoring the healthy development of fetuses. Unfortunately, such measurements are subject to large inter-observer variability, resulting in low early-detection rates of fetal abnormalities. To address this issue, we propose a novel probabilistic Deep Learning approach for real-time automated estimation of fetal HC. This system feeds back statistics on measurement robustness to inform users how confident a deep neural network is in evaluating suitable views acquired during free-hand ultrasound examination. In real-time scenarios, this approach may be exploited to guide operators to scan planes that are as close as possible to the underlying distribution of training images, for the purpose of improving inter-operator consistency. We train on freehand ultrasound data from over 2000 subjects (2848 training/540 test) and show that our method is able to predict HC measurements within 1.81±1.65 mm deviation from the ground truth, with 50% of the test images fully contained within the predicted confidence margins, and an average of 1.82±1.78 mm deviation from the margin for the remaining cases that are not fully contained.
CRIS-Team Scopus Importer:2021-08-17

Context-sensitive super-resolution for fast fetal magnetic resonance imaging - Mcdonagh, Steven; Hou, Benjamin; Alansary, Amir; Oktay, Ozan; Kamnitsas, Konstantinos; Rutherford, Mary; Hajnal, Jo V.; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2017)
3D Magnetic Resonance Imaging (MRI) is often a trade-off between fast but low-resolution image acquisition and highly detailed but slow image acquisition. Fast imaging is required for targets that move to avoid motion artefacts. This is in particular difficult for fetal MRI. Spatially independent upsampling techniques, which are the state-of-the-art to address this problem, are error prone and disregard contextual information. In this paper we propose a context-sensitive upsampling method based on a residual convolutional neural network model that learns organ specific appearance and adopts semantically to input data allowing for the generation of high resolution images with sharp edges and fine scale detail. By making contextual decisions about appearance and shape, present in different parts of an image, we gain a maximum of structural detail at a similar contrast as provided by high-resolution data. We experiment on 145 fetal scans and show that our approach yields an increased PSNR of 1.25 dB when applied to under-sampled fetal data cf. baseline upsampling. Furthermore, our method yields an increased PSNR of 1.73 dB when utilizing under-sampled fetal data to perform brain volume reconstruction on motion corrupted captured data.
CRIS-Team Scopus Importer:2021-08-17

Contrastive Learning for View Classification of Echocardiograms - Kainz, Bernhard; et al. (Springer Science and Business Media Deutschland GmbH, 2021)
Keywords: Echocardiography; Contrastive learning; Classification

Crepuscular rays for tumor accessibility planning - Khlebnikov, Rostislav; Kainz, Bernhard; Muehl, Judith; Schmalstieg, Dieter (IEEE Transactions on Visualization and Computer Graphics, 2011)
In modern clinical practice, planning access paths to volumetric target structures remains one of the most important and most complex tasks, and a physician's insufficient experience in this can lead to severe complications or even the death of the patient. In this paper, we present a method for safety evaluation and the visualization of access paths to assist physicians during preoperative planning. As a metaphor for our method, we employ a well-known, and thus intuitively perceivable, natural phenomenon that is usually called crepuscular rays. Using this metaphor, we propose several ways to compute the safety of paths from the region of interest to all tumor voxels and show how this information can be visualized in real-time using a multi-volume rendering system. Furthermore, we show how to estimate the extent of connected safe areas to improve common medical 2D multi-planar reconstruction (MPR) views. We evaluate our method by means of expert interviews, an online survey, and a retrospective evaluation of 19 real abdominal radio-frequency ablation (RFA) interventions, with expert decisions serving as a gold standard. The evaluation results show clear evidence that our method can be successfully applied in clinical practice without introducing substantial overhead work for the acting personnel. Finally, we show that our method is not limited to medical applications and that it can also be useful in other fields. © 2011 IEEE.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Accessibility; medical visualization.; ray casting

D’ARTAGNAN: Counterfactual Video Generation - Reynaud, Hadrien; Vlontzos, Athanasios; Dombrowski, Mischa; Gilligan Lee, Ciarán; Beqiri, Arian; Leeson, Paul; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2022)
Causally-enabled machine learning frameworks could help clinicians to identify the best course of treatments by answering counterfactual questions. We explore this path for the case of echocardiograms by looking into the variation of the Left Ventricle Ejection Fraction, the most essential clinical metric gained from these examinations. We combine deep neural networks, twin causal networks and generative adversarial methods for the first time to build D’ARTAGNAN (Deep ARtificial Twin-Architecture GeNerAtive Networks), a novel causal generative model. We demonstrate the soundness of our approach on a synthetic dataset before applying it to cardiac ultrasound videos to answer the question: “What would this echocardiogram look like if the patient had a different ejection fraction?”. To do so, we generate new ultrasound videos, retaining the video style and anatomy of the original patient, while modifying the Ejection Fraction conditioned on a given input. We achieve an SSIM score of 0.79 and an R2 score of 0.51 on the counterfactual videos. Code and models are available at: https://github.com/HReynaud/dartagnan.
CRIS-Team Scopus Importer:2022-10-14
Keywords: Causal inference; Counterfactual video generation; Twin networks

DeepCut: Object Segmentation from Bounding Box Annotations Using Convolutional Neural Networks - Rajchl, Martin; Lee, Matthew C. H.; Oktay, Ozan; Kamnitsas, Konstantinos; Passerat-Palmbach, Jonathan; Bai, Wenjia; Damodaram, Mellisa; Rutherford, Mary A.; Hajnal, Joseph V.; Kainz, Bernhard; Rueckert, Daniel (IEEE Transactions on Medical Imaging, 2017)
In this paper, we propose DeepCut, a method to obtain pixelwise object segmentations given an image dataset labelled weak annotations, in our case bounding boxes. It extends the approach of the well-known GrabCut [1] method to include machine learning by training a neural network classifier from bounding box annotations. We formulate the problem as an energy minimisation problem over a densely-connected conditional random field and iteratively update the training targets to obtain pixelwise object segmentations. Additionally, we propose variants of the DeepCut method and compare those to a naïve approach to CNN training under weak supervision. We test its applicability to solve brain and lung segmentation problems on a challenging fetal magnetic resonance dataset and obtain encouraging results in terms of accuracy.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Bounding box; convolutional neural networks; DeepCut; image segmentation; machine learning; weak annotations

Deep radiance caching: Convolutional autoencoders deeper in ray tracing - Jiang, Giulio; Kainz, Bernhard (Computers & Graphics, 2021)
Rendering realistic images with global illumination is a computationally demanding task and often requires dedicated hardware for feasible runtime. Recent research uses Deep Neural Networks to predict indirect lighting on image level, but such methods are commonly limited to diffuse materials and require training on each scene. We present Deep Radiance Caching (DRC), an efficient variant of Radiance Caching utilizing Convolutional Autoencoders for rendering global illumination. DRC employs a denoising neural network with Radiance Caching to support a wide range of material types, without the requirement of offline pre-computation or training for each scene. This offers high performance CPU rendering for maximum accessibility. Our method has been evaluated on interior scenes, and is able to produce high-quality images within 180 s on a single CPU.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Deep learning; Radiance caching; Ray tracing

Detecting Hypo-plastic Left Heart Syndrome in Fetal Ultrasound via Disease-Specific Atlas Maps - Budd, Samuel; Sinclair, Matthew; Day, Thomas G.; Vlontzos, Athanasios; Tan, Jeremy; Liu, Tianrui; Matthew, Jaqueline; Skelton, Emily; Simpson, John M.; Razavi, Reza; Glocker, Ben; Rueckert, Daniel; et al. (Springer Science and Business Media Deutschland GmbH, 2021)
Fetal ultrasound screening during pregnancy plays a vital role in the early detection of fetal malformations which have potential long-term health impacts. The level of skill required to diagnose such malformations from live ultrasound during examination is high and resources for screening are often limited. We present an interpretable, atlas-learning segmentation method for automatic diagnosis of Hypo-plastic Left Heart Syndrome (HLHS) from a single ‘4 Chamber Heart’ view image. We propose to extend the recently introduced Image-and-Spatial Transformer Networks (Atlas-ISTN) into a framework that enables sensitising atlas generation to disease. In this framework we can jointly learn image segmentation, registration, atlas construction and disease prediction while providing a maximum level of clinical interpretability compared to direct image classification methods. As a result our segmentation allows diagnoses competitive with expert-derived manual diagnosis and yields an AUC-ROC of 0.978 (1043 cases for training, 260 for validation and 325 for testing).
Keywords: Atlas; Segmentation; Ultrasound

Detecting Outliers with Poisson Image Interpolation - Tan, Jeremy; Hou, Benjamin; Day, Thomas G.; Simpson, John M.; Rueckert, Daniel; Kainz, Bernhard (Springer Science and Business Media Deutschland GmbH, 2021)
Supervised learning of every possible pathology is unrealistic for many primary care applications like health screening. Image anomaly detection methods that learn normal appearance from only healthy data have shown promising results recently. We propose an alternative to image reconstruction-based and image embedding-based methods and propose a new self-supervised method to tackle pathological anomaly detection. Our approach originates in the foreign patch interpolation (FPI) strategy that has shown superior performance on brain MRI and abdominal CT data. We propose to use a better patch interpolation strategy, Poisson image interpolation (PII), which makes our method suitable for applications in challenging data regimes. PII outperforms state-of-the-art methods by a good margin when tested on surrogate tasks like identifying common lung anomalies in chest X-rays or hypo-plastic left heart syndrome in prenatal, fetal cardiac ultrasound images.
Keywords: Self-supervised learning; Outlier detection

DISYRE: Diffusion-Inspired Synthetic Restoration for Unsupervised Anomaly Detection - Marimont, Sergio Naval; Baugh, Matthew; Siomos, Vasilis; Tzelepis, Christos; Kainz, Bernhard; Tarroni, Giacomo (Proceedings - International Symposium on Biomedical Imaging, 2024)
Unsupervised Anomaly Detection (UAD) techniques aim to identify and localize anomalies without relying on annotations, only leveraging a model trained on a dataset known to be free of anomalies. Diffusion models learn to modify inputs x to increase the probability of it belonging to a desired distribution, i.e., they model the score function ∇x log p(x). Such a score function is potentially relevant for UAD, since ∇x log p(x) is itself a pixel-wise anomaly score. However, diffusion models are trained to invert a corruption process based on Gaussian noise and the learned score function is unlikely to generalize to medical anomalies. This work addresses the problem of how to learn a score function relevant for UAD and proposes DISYRE: Diffusion-Inspired SYnthetic REstoration. We retain the diffusion-like pipeline but replace the Gaussian noise corruption with a gradual, synthetic anomaly corruption so the learned score function generalizes to medical, naturally occurring anomalies. We evaluate DISYRE on three common Brain MRI UAD benchmarks and substantially outperform other methods in two out of the three tasks.
CRIS-Team Scopus Importer:2024-09-20
Keywords: diffusion models; out-of-distribution detection; synthetic anomalies; Unsupervised anomaly detection

Divergent search for image classification behaviors - Tan, Jeremy; Kainz, Bernhard (GECCO 2020 Companion - Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, 2020)
When data is unlabelled and the target task is not known a priori, divergent search offers a strategy for learning a wide range of skills. Having such a repertoire allows a system to adapt to new, unforeseen tasks. Unlabelled image data is plentiful, but it is not always known which features will be required for downstream tasks. We propose a method for divergent search in the few-shot image classification setting and evaluate with Omniglot and Mini-ImageNet. This high-dimensional behavior space includes all possible ways of partitioning the data. To manage divergent search in this space, we rely on a meta-learning framework to integrate useful features from diverse tasks into a single model. The final layer of this model is used as an index into the 'archive' of all past behaviors. We search for regions in the behavior space that the current archive cannot reach. As expected, divergent search is outperformed by models with a strong bias toward the evaluation tasks. But it is able to match and sometimes exceed the performance of models that have a weak bias toward the target task or none at all. This demonstrates that divergent search is a viable approach, even in high-dimensional behavior spaces.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Divergent search; Meta-learning; Unsupervised

EchoFusion: Tracking and reconstruction of objects in 4D freehand ultrasound imaging without external trackers - Khanal, Bishesh; Gomez, Alberto; Toussaint, Nicolas; Mcdonagh, Steven; Zimmer, Veronika; Skelton, Emily; Matthew, Jacqueline; Grzech, Daniel; Wright, Robert; Gupta, Chandni; Hou, Benjamin; Rueckert, Daniel; et al. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2018)
Ultrasound (US) is the most widely used fetal imaging technique. However, US images have limited capture range, and suffer from view dependent artefacts such as acoustic shadows. Compounding of overlapping 3D US acquisitions into a high-resolution volume can extend the field of view and remove image artefacts, which is useful for retrospective analysis including population based studies. However, such volume reconstructions require information about relative transformations between probe positions from which the individual volumes were acquired. In prenatal US scans, the fetus can move independently from the mother, making external trackers such as electromagnetic or optical tracking unable to track the motion between probe position and the moving fetus. We provide a novel methodology for image-based tracking and volume reconstruction by combining recent advances in deep learning and simultaneous localisation and mapping (SLAM). Tracking semantics are established through the use of a Residual 3D U-Net and the output is fed to the SLAM algorithm. As a proof of concept, experiments are conducted on US volumes taken from a whole body fetal phantom, and from the heads of real fetuses. For the fetal head segmentation, we also introduce a novel weak annotation approach to minimise the required manual effort for ground truth annotation. We evaluate our method qualitatively, and quantitatively with respect to tissue discrimination accuracy and tracking robustness.
CRIS-Team Scopus Importer:2021-08-17

Echonet-synthetic: Privacy-preserving video generation for safe medical data sharing - Reynaud, Hadrien; Meng, Qingjie; Dombrowski, Mischa; Ghosh, Arijit; Day, Thomas; Gomez, Alberto; Leeson, Paul; Kainz, Bernhard (International Conference on Medical Image Computing and Computer-Assisted Intervention, 2024)

Ensembles of multiple models and architectures for robust brain tumour segmentation - Kamnitsas, Konstantinos; Bai, Wenjia; Ferrante, Enzo; Mcdonagh, S.; Sinclair, M.; Pawlowski, N.; Rajchl, M.; Lee, M.; Kainz, Bernhard; Rueckert, D.; Glocker, B. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2018)
Deep learning approaches such as convolutional neural nets have consistently outperformed previous methods on challenging tasks such as dense, semantic segmentation. However, the various proposed networks perform differently, with behaviour largely influenced by architectural choices and training settings. This paper explores Ensembles of Multiple Models and Architectures (EMMA) for robust performance through aggregation of predictions from a wide range of methods. The approach reduces the influence of the meta-parameters of individual models and the risk of overfitting the configuration to a particular database. EMMA can be seen as an unbiased, generic deep learning model which is shown to yield excellent performance, winning the first position in the BRATS 2017 competition among 50+ participating teams.
CRIS-Team Scopus Importer:2021-08-17

Estimating categorical counterfactuals via deep twin networks - Vlontzos, Athanasios; Kainz, Bernhard; Gilligan-Lee, Ciarán M. (Nature Machine Intelligence, 2023)
Counterfactual inference is a powerful tool, capable of solving challenging problems in high-profile sectors. To perform counterfactual inference, we require knowledge of the underlying causal mechanisms. However, causal mechanisms cannot be uniquely determined from observations and interventions alone. This raises the question of how to choose the causal mechanisms so that the resulting counterfactual inference is trustworthy in a given domain. This question has been addressed in causal models with binary variables, but for the case of categorical variables, it remains unanswered. We address this challenge by introducing for causal models with categorical variables the notion of counterfactual ordering, a principle positing desirable properties that causal mechanisms should possess and prove that it is equivalent to specific functional constraints on the causal mechanisms. To learn causal mechanisms satisfying these constraints, and perform counterfactual inference with them, we introduce deep twin networks. These are deep neural networks that, when trained, are capable of twin network counterfactual inference—an alternative to the abduction–action–prediction method. We empirically test our approach on diverse real-world and semisynthetic data from medicine, epidemiology and finance, reporting accurate estimation of counterfactual probabilities while demonstrating the issues that arise with counterfactual reasoning when counterfactual ordering is not enforced
CRIS-Team Scopus Importer:2023-03-03

Evaluating reinforcement learning agents for anatomical landmark detection - Alansary, Amir; Oktay, Ozan; Li, Yuanwei; Le Folgoc, Loic; Hou, Benjamin; Vaillant, Ghislain; Kamnitsas, Konstantinos; Vlontzos, Athanasios; Glocker, Ben; Kainz, Bernhard; Rueckert, Daniel (Medical Image Analysis, 2019)
Automatic detection of anatomical landmarks is an important step for a wide range of applications in medical image analysis. Manual annotation of landmarks is a tedious task and prone to observer errors. In this paper, we evaluate novel deep reinforcement learning (RL) strategies to train agents that can precisely and robustly localize target landmarks in medical scans. An artificial RL agent learns to identify the optimal path to the landmark by interacting with an environment, in our case 3D images. Furthermore, we investigate the use of fixed- and multi-scale search strategies with novel hierarchical action steps in a coarse-to-fine manner. Several deep Q-network (DQN) architectures are evaluated for detecting multiple landmarks using three different medical imaging datasets: fetal head ultrasound (US), adult brain and cardiac magnetic resonance imaging (MRI). The performance of our agents surpasses state-of-the-art supervised and RL methods. Our experiments also show that multi-scale search strategies perform significantly better than fixed-scale agents in images with large field of view and noisy background such as in cardiac MRI. Moreover, the novel hierarchical steps can significantly speed up the searching process by a factor of 4–5 times.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Automatic landmark detection; Deep learning; DQN; Reinforcement learning

Exploring a new paradigm for the fetal anomaly ultrasound scan: Artificial intelligence in real time - Matthew, Jacqueline; Skelton, Emily; Day, Thomas G.; Zimmer, Veronika A.; Gomez, Alberto; Wheeler, Gavin; Toussaint, Nicolas; Liu, Tianrui; Budd, Samuel; Lloyd, Karen; Wright, Robert; Deng, Shujie; et al. (Prenatal Diagnosis, 2022)
Objective: Advances in artificial intelligence (AI) have demonstrated potential to improve medical diagnosis. We piloted the end-to-end automation of the mid-trimester screening ultrasound scan using AI-enabled tools. Methods: A prospective method comparison study was conducted. Participants had both standard and AI-assisted US scans performed. The AI tools automated image acquisition, biometric measurement, and report production. A feedback survey captured the sonographers' perceptions of scanning. Results: Twenty-three subjects were studied. The average time saving per scan was 7.62 min (34.7%) with the AI-assisted method (p < 0.0001). There was no difference in reporting time. There were no clinically significant differences in biometric measurements between the two methods. The AI tools saved a satisfactory view in 93% of the cases (four core views only), and 73% for the full 13 views, compared to 98% for both using the manual scan. Survey responses suggest that the AI tools helped sonographers to concentrate on image interpretation by removing disruptive tasks. Conclusion: Separating freehand scanning from image capture and measurement resulted in a faster scan and altered workflow. Removing repetitive tasks may allow more attention to be directed identifying fetal malformation. Further work is required to improve the image plane detection algorithm for use in real time.
CRIS-Team Scopus Importer:2023-03-08

Exploring the relationship between segmentation uncertainty, segmentation performance and inter-observer variability with probabilistic networks - Chotzoglou, Elisa; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2019)
Medical image segmentation is an essential tool for clinical decision making and treatment planning. Automation of this process led to significant improvements in diagnostics and patient care, especially after recent breakthroughs that have been triggered by deep learning. However, when integrating automatic tools into patient care, it is crucial to understand their limitations and to have means to assess their confidence for individual cases. Aleatoric and epistemic uncertainties have been subject of recent research. Methods have been developed to calculate these quantities automatically during segmentation inference. However, it is still unclear how much human factors affect these metrics. Varying image quality and different levels of human annotator expertise are an integral part of aleatoric uncertainty. It is unknown how much this variability affects uncertainty in the final segmentation. Thus, in this work we explore potential links between deep network segmentation uncertainties with inter-observer variance and segmentation performance. We show how the area of disagreement between different ground-truth annotators can be developed into model confidence metrics and evaluate them on the LIDC-IDRI dataset, which contains multiple expert annotations for each subject. Our results indicate that a probabilistic 3D U-Net and a 3D U-Net using Monte-Carlo dropout during inference both show a similar correlation between our segmentation uncertainty metrics, segmentation performance and human expert variability.
CRIS-Team Scopus Importer:2021-08-17

Exploring the role of different cell types on cortical folding in the developing human brain through computational modeling - Zarzor, Mohammad Saeed; Ma, Qiang; Almurey, Median; Kainz, Bernhard; Budday, Silvia (Scientific Reports, 2024)
The human brain’s distinctive folding pattern has attracted the attention of researchers from different fields. Neuroscientists have provided insights into the role of four fundamental cell types crucial during embryonic development: radial glial cells, intermediate progenitor cells, outer radial glial cells, and neurons. Understanding the mechanisms by which these cell types influence the number of cortical neurons and the emerging cortical folding pattern necessitates accounting for the mechanical forces that drive the cortical folding process. Our research aims to explore the correlation between biological processes and mechanical forces through computational modeling. We introduce cell-density fields, characterized by a system of advection-diffusion equations, designed to replicate the characteristic behaviors of various cell types in the developing brain. Concurrently, we adopt the theory of finite growth to describe cortex expansion driven by increasing cell density. Our model serves as an adjustable tool for understanding how the behavior of individual cell types reflects normal and abnormal folding patterns. Through comparison with magnetic resonance images of the fetal brain, we explore the correlation between morphological changes and underlying cellular mechanisms. Moreover, our model sheds light on the spatiotemporal relationships among different cell types in the human brain and enables cellular deconvolution of histological sections.
CRIS-Team Scopus Importer:2024-11-15
Keywords: Cortical folding; Coupled problems; Human brain development; Multi-field modeling; Outer radial glial cells

Fast fully automatic brain detection in fetal MRI using dense rotation invariant image descriptors - Kainz, Bernhard; Keraudren, Kevin; Kyriakopoulou, Vanessa; Rutherford, Mary; Hajnal, Joseph V.; Rueckert, Daniel (2014 IEEE 11th International Symposium on Biomedical Imaging, ISBI 2014, 2014)
Automatic detection of the fetal brain in Magnetic Resonance (MR) Images is especially difficult due to arbitrary orientation of the fetus and possible movements during the scan. In this paper, we propose a method to facilitate fully automatic brain voxel classification by means of rotation invariant volume descriptors. We calculate features for a set of 50 prenatal fast spin echo T2 volumes of the uterus and learn the appearance of the fetal brain in the feature space. We evaluate our novel classification method and show that we can localize the fetal brain with an accuracy of 100% and classify fetal brain voxels with an accuracy above 97%. Furthermore, we show how the classification process can be used for a direct segmentation of the brain by simple refinement methods within the raw MR scan data leading to a final segmentation with a Dice score above 0.90.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Fetal brain localization; Fetal brain segmentation; Fetal MRI reconstruction

Fast fully automatic segmentation of the human placenta from motion corrupted MRI - Alansary, Amir; Kamnitsas, Konstantinos; Davidson, Alice; Khlebnikov, Rostislav; Rajchl, Martin; Malamateniou, Christina; Rutherford, Mary; Hajnal, Joseph V.; Glocker, Ben; Rueckert, Daniel; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2016)
Recently,magnetic resonance imaging has revealed to be important for the evaluation of placenta’s health during pregnancy. Quantitative assessment of the placenta requires a segmentation,which proves to be challenging because of the high variability of its position,orientation,shape and appearance. Moreover,image acquisition is corrupted by motion artifacts from both fetal and maternal movements. In this paper we propose a fully automatic segmentation framework of the placenta from structural T2-weighted scans of the whole uterus,as well as an extension in order to provide an intuitive pre-natal view into this vital organ. We adopt a 3D multi-scale convolutional neural network to automatically identify placental candidate pixels. The resulting classification is subsequently refined by a 3D dense conditional random field,so that a high resolution placental volume can be reconstructed from multiple overlapping stacks of slices. Our segmentation framework has been tested on 66 subjects at gestational ages 20–38 weeks achieving a Dice score of 71.95 ± 19.79% for healthy fetuses with a fixed scan sequence and 66.89 ± 15.35% for a cohort mixed with cases of intrauterine fetal growth restriction using varying scan parameters.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Fetal; MRI; Placenta; Segmentation

Fast marker based C-Arm pose estimation - Kainz, Bernhard; et al. (11th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2008, 2008)
To estimate the pose of a C-Arm during interventions therapy we have developed a small sized X-Ray Target including a special set of beads with known locations in 3D space. Since the patient needs of remain the the X-Ray path for all feasible poses of the C-Arm during the interventions, we cannot construct a single marker which is entirely visible in all images. Therefore findings 2D-3D point correspondences is a non-trivial task. The marker pattern has to be chosen in a way such that its projection onto the image plane is unique in a minimal-sized window for all relevant poses of C-Arm. We use a two dimensional adaption of a linear feedback shift register (LFSR) to generate a two-dimensional pattern with unique sub-patterns in a certain window range. Thereby uniqueness is not achieved by placing unique 2D sub patterns side by side but by the code property itself. The code is designed in a way that any sub window of a minimal size guarantees uniqueness and that even occlusions from medical instruments can be handled. Experiments showed that we were able to estimate the C-Arm's pose from a single image within one second with a precision below one millimeter and one degree.

Fast multiple landmark localisation using a patch-based iterative network - Li, Yuanwei; Alansary, Amir; Cerrolaza, Juan J.; Khanal, Bishesh; Sinclair, Matthew; Matthew, Jacqueline; Gupta, Chandni; Knight, Caroline; Kainz, Bernhard; Rueckert, Daniel (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2018)
We propose a new Patch-based Iterative Network (PIN) for fast and accurate landmark localisation in 3D medical volumes. PIN utilises a Convolutional Neural Network (CNN) to learn the spatial relationship between an image patch and anatomical landmark positions. During inference, patches are repeatedly passed to the CNN until the estimated landmark position converges to the true landmark location. PIN is computationally efficient since the inference stage only selectively samples a small number of patches in an iterative fashion rather than a dense sampling at every location in the volume. Our approach adopts a multi-task learning framework that combines regression and classification to improve localisation accuracy. We extend PIN to localise multiple landmarks by using principal component analysis, which models the global anatomical relationships between landmarks. We have evaluated PIN using 72 3D ultrasound images from fetal screening examinations. PIN achieves quantitatively an average landmark localisation error of 5.59 mm and a runtime of 0.44 s to predict 10 landmarks per volume. Qualitatively, anatomical 2D standard scan planes derived from the predicted landmark locations are visually similar to the clinical ground truth.
CRIS-Team Scopus Importer:2021-08-17

Fast Volume Reconstruction from Motion Corrupted Stacks of 2D Slices - Kainz, Bernhard; Steinberger, Markus; Wein, Wolfgang; Kuklisova-Murgasova, Maria; Malamateniou, Christina; Keraudren, Kevin; Torsney-Weir, Thomas; Rutherford, Mary; Aljabar, Paul; Hajnal, Joseph V.; Rueckert, Daniel (IEEE Transactions on Medical Imaging, 2015)
Capturing an enclosing volume of moving subjects and organs using fast individual image slice acquisition has shown promise in dealing with motion artefacts. Motion between slice acquisitions results in spatial inconsistencies that can be resolved by slice-to-volume reconstruction (SVR) methods to provide high quality 3D image data. Existing algorithms are, however, typically very slow, specialised to specific applications and rely on approximations, which impedes their potential clinical use. In this paper, we present a fast multi-GPU accelerated framework for slice-to-volume reconstruction. It is based on optimised 2D/3D registration, super-resolution with automatic outlier rejection and an additional (optional) intensity bias correction. We introduce a novel and fully automatic procedure for selecting the image stack with least motion to serve as an initial registration target. We evaluate the proposed method using artificial motion corrupted phantom data as well as clinical data, including tracked freehand ultrasound of the liver and fetal Magnetic Resonance Imaging. We achieve speed-up factors greater than 30 compared to a single CPU system and greater than 10 compared to currently available state-of-the-art multi-core CPU methods. We ensure high reconstruction accuracy by exact computation of the point-spread function for every input data point, which has not previously been possible due to computational limitations. Our framework and its implementation is scalable for available computational infrastructures and tests show a speed-up factor of 1.70 for each additional GPU. This paves the way for the online application of image based reconstruction methods during clinical examinations. The source code for the proposed approach is publicly available.
CRIS-Team Scopus Importer:2021-08-17
Keywords: fetal imaging; freehand compound ultrasound; GPU acceleration; magnetic resonance imaging; Motion correction

Feature-Conditioned Cascaded Video Diffusion Models for Precise Echocardiogram Synthesis - Reynaud, Hadrien; Qiao, Mengyun; Dombrowski, Mischa; Day, Thomas; Razavi, Reza; Gomez, Alberto; Leeson, Paul; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2023)
Image synthesis is expected to provide value for the translation of machine learning methods into clinical practice. Fundamental problems like model robustness, domain transfer, causal modelling, and operator training become approachable through synthetic data. Especially, heavily operator-dependant modalities like Ultrasound imaging require robust frameworks for image and video generation. So far, video generation has only been possible by providing input data that is as rich as the output data, e.g., image sequence plus conditioning in → video out. However, clinical documentation is usually scarce and only single images are reported and stored, thus retrospective patient-specific analysis or the generation of rich training data becomes impossible with current approaches. In this paper, we extend elucidated diffusion models for video modelling to generate plausible video sequences from single images and arbitrary conditioning with clinical parameters. We explore this idea within the context of echocardiograms by looking into the variation of the Left Ventricle Ejection Fraction, the most essential clinical metric gained from these examinations. We use the publicly available EchoNet-Dynamic dataset for all our experiments. Our image to sequence approach achieves an R2 score of 93%, which is 38 points higher than recently proposed sequence to sequence generation methods. Code and weights are available at https://github.com/HReynaud/EchoDiffusion.
CRIS-Team Scopus Importer:2023-11-03
Keywords: Cardiac; Diffusion; Generative; Ultrasound; Video

Federated deep learning for detecting COVID-19 lung abnormalities in CT: a privacy-preserving multinational validation study - Dou, Qi; So, Tiffany Y.; Jiang, Meirui; Liu, Quande; Vardhanabhuti, Varut; Kaissis, Georgios; Li, Zeju; Si, Weixin; Lee, Heather H. C.; Yu, Kevin; Feng, Zuxin; Dong, Li; et al. (npj Digital Medicine, 2021)
Data privacy mechanisms are essential for rapidly scaling medical training databases to capture the heterogeneity of patient data distributions toward robust and generalizable machine learning systems. In the current COVID-19 pandemic, a major focus of artificial intelligence (AI) is interpreting chest CT, which can be readily used in the assessment and management of the disease. This paper demonstrates the feasibility of a federated learning method for detecting COVID-19 related CT abnormalities with external validation on patients from a multinational study. We recruited 132 patients from seven multinational different centers, with three internal hospitals from Hong Kong for training and testing, and four external, independent datasets from Mainland China and Germany, for validating model generalizability. We also conducted case studies on longitudinal scans for automated estimation of lesion burden for hospitalized COVID-19 patients. We explore the federated learning algorithms to develop a privacy-preserving AI model for COVID-19 medical image diagnosis with good generalization capability on unseen multinational datasets. Federated learning could provide an effective mechanism during pandemics to rapidly develop clinically useful AI across institutions and countries overcoming the burden of central aggregation of large amounts of sensitive data.
CRIS-Team Scopus Importer:2021-08-17

Fetal Cortex Segmentation with Topology and Thickness Loss Constraints - Li, Liu; Ma, Qiang; Li, Zeju; Ouyang, Cheng; Zhang, Weitong; Price, Anthony; Kyriakopoulou, Vanessa; Grande, Lucilio C.; Makropoulos, Antonis; Hajnal, Joseph; Rueckert, Daniel; Kainz, Bernhard; et al. (ETHICAL AND PHILOSOPHICAL ISSUES IN MEDICAL IMAGING, MULTIMODAL LEARNING AND FUSION ACROSS SCALES FOR CLINICAL DECISION SUPPORT, AND TOPOLOGICAL DATA ANALYSIS FOR BIOMEDICAL IMAGING, EPIMI 2022, ML-CDS 2022, TDA4BIOMEDICALIMAGING, 2022)
CRIS-Team WoS Importer:2023-02-17

Fetal skull segmentation in 3D ultrasound via structured geodesic random forest - Cerrolaza, Juan J.; Oktay, Ozan; Gomez, Alberto; Matthew, Jacqueline; Knight, Caroline; Kainz, Bernhard; Rueckert, Daniel (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2017)
Ultrasound is the primary imaging method for prenatal screening and diagnosis of fetal anomalies. Thanks to its non-invasive and non-ionizing properties, ultrasound allows quick, safe and detailed evaluation of the unborn baby, including the estimation of the gestational age, brain and cranium development. However, the accuracy of traditional 2D fetal biometrics is dependent on operator expertise and subjectivity in 2D plane finding and manual marking. 3D ultrasound has the potential to reduce the operator dependence. In this paper, we propose a new random forest-based segmentation framework for fetal 3D ultrasound volumes, able to efficiently integrate semantic and structural information in the classification process. We introduce a new semantic features space able to encode spatial context via generalized geodesic distance transform. Unlike alternative auto-context approaches, this new set of features is efficiently integrated into the same forest using contextual trees. Finally, we use a new structured labels space as alternative to the traditional atomic class labels, able to capture morphological variability of the target organ. Here, we show the potential of this new general framework segmenting the skull in 3D fetal ultrasound volumes, significantly outperforming alternative random forest-based approaches.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Generalized geodesic distance; Random forest; Structured class

Flexible Conditional Image Generation of Missing Data with Learned Mental Maps - Hou, Benjamin; Vlontzos, Athanasios; Alansary, Amir; Rueckert, Daniel; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2019)
Real-world settings often do not allow acquisition of high-resolution volumetric images for accurate morphological assessment and diagnostic. In clinical practice it is frequently common to acquire only sparse data (e.g. individual slices) for initial diagnostic decision making. Thereby, physicians rely on their prior knowledge (or mental maps) of the human anatomy to extrapolate the underlying 3D information. Accurate mental maps require years of anatomy training, which in the first instance relies on normative learning, i.e. excluding pathology. In this paper, we leverage Bayesian Deep Learning and environment mapping to generate full volumetric anatomy representations from none to a small, sparse set of slices. We evaluate proof of concept implementations based on Generative Query Networks (GQN) and Conditional BRUNO using abdominal CT and brain MRI as well as in a clinical application involving sparse, motion-corrupted MR acquisition for fetal imaging. Our approach allows to reconstruct 3D volumes from 1 to 4 tomographic slices, with a SSIM of 0.7+ and cross-correlation of 0.8+ compared to the 3D ground truth.
CRIS-Team Scopus Importer:2021-08-17

Flexible reconstruction and correction of unpredictable motion from stacks of 2D images - Kainz, Bernhard; Alansary, Amir; Malamateniou, Christina; Keraudren, Kevin; Rutherford, Mary; Hajnal, Joseph V.; Rueckert, Daniel (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2015)
We present a method to correct motion in fetal in-utero scan sequences. The proposed approach avoids previously necessary manual segmentation of a region of interest. We solve the problem of non-rigid motion by splitting motion corrupted slices into overlapping patches of finite size. In these patches the assumption of rigid motion approximately holds and they can thus be used to perform a slice-to-volumebased (SVR) reconstruction during which their consistency with the other patches is learned. The learned information is used to reject patches that are not conform with the motion corrected reconstruction in their local areas. We evaluate rectangular and evenly distributed patches for the reconstruction as well as patches that have been derived from superpixels. Both approaches achieve on 29 subjects aged between 22–37 weeks a sufficient reconstruction quality and facilitate following 3D segmentation of fetal organs and the placenta.
CRIS-Team Scopus Importer:2021-08-17

Foreground-Background Separation through Concept Distillation from Generative Image Foundation Models - Dombrowski, Mischa; Reynaud, Hadrien; Baugh, Matthew; Kainz, Bernhard (2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023)
Curating datasets for object segmentation is a difficult task. With the advent of large-scale pre-trained generative models, conditional image generation has been given a significant boost in result quality and ease of use. In this paper, we present a novel method that enables the generation of general foreground-background segmentation models from simple textual descriptions, without requiring segmentation labels. We leverage and explore pre-trained latent diffusion models, to automatically generate weak segmentation masks for concepts and objects. The masks are then used to fine-tune the diffusion model on an inpainting task, which enables fine-grained removal of the object, while at the same time providing a synthetic foreground and background dataset. We demonstrate that using this method beats previous methods in both discriminative and generative performance and closes the gap with fully supervised training while requiring no pixel-wise object labels. We show results on the task of segmenting four different objects (humans, dogs, cars, birds) and a use case scenario in medical image analysis. The code is available at https://github.com/MischaD/fobadiffusion.
CRIS-Team Scopus Importer:2024-12-16

Foreword to the special section on the Eurographics Workshop on Visual Computing for Biology and Medicine (VCBM) at Medical Image Computing and Computer Assisted Intervention (MICCAI) 2018 - Schultz, Thomas; Puig, Anna; Kainz, Bernhard (Computers & Graphics, 2019)
CRIS-Team Scopus Importer:2021-08-17

Geomstats: A python package for riemannian geometry in machine learning - Miolane, Nina; Guigui, Nicolas; Le Brigant, Alice; Mathe, Johan; Hou, Benjamin; Thanwerdas, Yann; Heyder, Stefan; Peltre, Olivier; Koep, Niklas; Zaatiti, Hadi; Hajri, Hatem; Cabanes, Yann; et al. (Journal of Machine Learning Research, 2020)
We introduce Geomstats, an open-source Python package for computations and statistics on nonlinear manifolds such as hyperbolic spaces, spaces of symmetric positive definite matrices, Lie groups of transformations, and many more. We provide object-oriented and extensively unit-tested implementations. Manifolds come equipped with families of Riemannian metrics with associated exponential and logarithmic maps, geodesics, and parallel transport. Statistics and learning algorithms provide methods for estimation, clustering, and dimension reduction on manifolds. All associated operations are vectorized for batch computation and provide support for different execution backends-namely NumPy, PyTorch, and TensorFlow. This paper presents the package, compares it with related libraries, and provides relevant code examples. We show that Geomstats provides reliable building blocks to both foster research in differential geometry and statistics and democratize the use of Riemannian geometry in machine learning applications. The source code is freely available under the MIT license at geomstats.ai.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Differential geometry; Machine learning; Manifold; Riemannian geometry; Statistics

High-resolution contrast enhanced multi-phase hepatic computed tomography data fromaporcine Radio-Frequency Ablation study - Kainz, Bernhard; Voglreiter, Philip; Sereinigg, Michael; Wiederstein-Grasser, Iris; Mayrhauser, Ursula; Köstenbauer, Sonja; Pollari, Mika; Khlebnikov, Rostislav; Seise, Matthias; Alhonnoro, Tuomas; Häme, Yrjö; Seider, Daniel; et al. (2014 IEEE 11th International Symposium on Biomedical Imaging, ISBI 2014, 2014)
Data below 1 mm voxel size is getting more and more common in the clinical practice but it is still hard to obtain a consistent collection of such datasets for medical image processing research. With this paper we provide a large collection of Contrast Enhanced (CE) Computed Tomography (CT) data from porcine animal experiments and describe their acquisition procedure and peculiarities. We have acquired three CE-CT phases at the highest available scanner resolution of 57 porcine livers during induced respiratory arrest. These phases capture contrast enhanced hepatic arteries, portal venous veins and hepatic veins. Therefore, we provide scan data that allows for a highly accurate reconstruction of hepatic vessel trees. Several datasets have been acquired during Radio-Frequency Ablation (RFA) experiments. Hence, many datasets show also artificially induced hepatic lesions, which can be used for the evaluation of structure detection methods.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Contrast enhanced computed tomography; Liver tumor ablation; Porcine study data

Human-level Performance on Automatic Head Biometrics in Fetal Ultrasound Using Fully Convolutional Neural Networks - Sinclair, Matthew; Baumgartner, Christian F.; Matthew, Jacqueline; Bai, Wenjia; Martinez, Juan Cerrolaza; Li, Yuanwei; Smith, Sandra; Knight, Caroline L.; Kainz, Bernhard; Hajnal, Jo; King, Andrew P.; Rueckert, Daniel (Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, 2018)
Measurement of head biometrics from fetal ultrasonography images is of key importance in monitoring the healthy development of fetuses. However, the accurate measurement of relevant anatomical structures is subject to large inter-observer variability in the clinic. To address this issue, an automated method utilizing Fully Convolutional Networks (FCN) is proposed to determine measurements of fetal head circumference (HC) and biparietal diameter (BPD). An FCN was trained on approximately 2000 2D ultrasound images of the head with annotations provided by 45 different sonographers during routine screening examinations to perform semantic segmentation of the head. An ellipse is fitted to the resulting segmentation contours to mimic the annotation typically produced by a sonographer. The model's performance was compared with inter-observer variability, where two experts manually annotated 100 test images. Mean absolute model-expert error was slightly better than inter-observer error for HC (1.99mm vs 2.16mm), and comparable for BPD (0.61mm vs 0.59mm), as well as Dice coefficient (0.980 vs 0.980). Our results demonstrate that the model performs at a level similar to a human expert, and learns to produce accurate predictions from a large dataset annotated by many sonographers. Additionally, measurements are generated in near real-time at 15fps on a GPU, which could speed up clinical workflow for both skilled and trainee sonographers.
CRIS-Team Scopus Importer:2021-08-17

Image distillation for safe data sharing in histopathology - Li, Zhe; Kainz, Bernhard (International Conference on Medical Image Computing and Computer-Assisted Intervention, 2024)

Image Registration via Stochastic Gradient Markov Chain Monte Carlo - Grzech, Daniel; Kainz, Bernhard; Glocker, Ben; le Folgoc, Loïc (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2020)
We develop a fully Bayesian framework for non-rigid registration of three-dimensional medical images, with a focus on uncertainty quantification. Probabilistic registration of large images along with calibrated uncertainty estimates is difficult for both computational and modelling reasons. To address the computational issues, we explore connections between the Markov chain Monte Carlo by backprop and the variational inference by backprop frameworks in order to efficiently draw thousands of samples from the posterior distribution. Regarding the modelling issues, we carefully design a Bayesian model for registration to overcome the existing barriers when using a dense, high-dimensional, and diffeomorphic parameterisation of the transformation. This results in improved calibration of uncertainty estimates.
CRIS-Team Scopus Importer:2021-08-17

Improved Post-hoc Probability Calibration for Out-of-Domain MRI Segmentation - Ouyang, Cheng; Wang, Shuo; Chen, Chen; Li, Zeju; Bai, Wenjia; Kainz, Bernhard; Rueckert, Daniel (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2022)
Probability calibration for deep models is highly desirable in safety-critical applications such as medical imaging. It makes output probabilities of deep networks interpretable, by aligning prediction probability with the actual accuracy in test data. In image segmentation, well-calibrated probabilities allow radiologists to identify regions where model-predicted segmentations are unreliable. These unreliable predictions often occur to out-of-domain (OOD) images that are caused by imaging artifacts or unseen imaging protocols. Unfortunately, most previous calibration methods for image segmentation perform sub-optimally on OOD images. To reduce the calibration error when confronted with OOD images, we propose a novel post-hoc calibration model. Our model leverages the pixel susceptibility against perturbations at the local level, and the shape prior information at the global level. The model is tested on cardiac MRI segmentation datasets that contain unseen imaging artifacts and images from an unseen imaging protocol. We demonstrate reduced calibration errors compared with the state-of-the-art calibration algorithm.
CRIS-Team Scopus Importer:2022-10-14

Improving image labelling quality - Day, Thomas G.; Simpson, John M.; Razavi, Reza; Kainz, Bernhard (Nature Machine Intelligence, 2023)
There is a continuing demand for high-quality, large-scale annotated datasets in medical imaging supported by machine learning. A new study investigates the importance of what type of instructions crowdsourced annotators receive.
CRIS-Team WoS Importer:2023-05-26

Interactive Volumetry of Liver Ablation Zones - Egger, Jan; Busse, Harald; Brandmaier, Philipp; Seider, Daniel; Gawlitza, Matthias; Strocka, Steffen; Voglreiter, Philip; Dokter, Mark; Hofmann, Michael; Kainz, Bernhard; Hann, Alexander; Chen, Xiaojun; et al. (Scientific Reports, 2015)
Percutaneous radiofrequency ablation (RFA) is a minimally invasive technique that destroys cancer cells by heat. The heat results from focusing energy in the radiofrequency spectrum through a needle. Amongst others, this can enable the treatment of patients who are not eligible for an open surgery. However, the possibility of recurrent liver cancer due to incomplete ablation of the tumor makes post-interventional monitoring via regular follow-up scans mandatory. These scans have to be carefully inspected for any conspicuousness. Within this study, the RF ablation zones from twelve post-interventional CT acquisitions have been segmented semi-automatically to support the visual inspection. An interactive, graph-based contouring approach, which prefers spherically shaped regions, has been applied. For the quantitative and qualitative analysis of the algorithm's results, manual slice-by-slice segmentations produced by clinical experts have been used as the gold standard (which have also been compared among each other). As evaluation metric for the statistical validation, the Dice Similarity Coefficient (DSC) has been calculated. The results show that the proposed tool provides lesion segmentation with sufficient accuracy much faster than manual segmentation. The visual feedback and interactivity make the proposed tool well suitable for the clinical workflow.
CRIS-Team Scopus Importer:2021-08-17

Intervention planning of hepatocellular carcinoma radio-frequency ablations - Kerbl, Bernhard; Voglreiter, Philip; Khlebnikov, Rostislav; Schmalstieg, Dieter; Seider, Daniel; Moche, Michael; Stiegler, Philipp; Portugaller, Rupert H.; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2013)
We present a software solution for planning and simulating radio-frequency ablation (RFA) treatment for patients suffering from hepatocellular carcinoma. Our software provides the graphical front-end for the results of the EU FP7 project IMPPACT. The main planning application was designed to assist with the identification of minimum-risk setups for RFA probes and generation of evaluable 3D representations of the predicted necrosis zones. Patient-specific mesh data describing the involved anatomic structures are used to individually parameterize the simulation procedure for personalized results. Our software supplies tools for extracting the required anatomic meshes from computed tomography (CT) images and for specifying custom intervention protocols. Data collected during simulations allow for detailed illustration of expected effectiveness and progress of heat-induced necrosis over time. Our software was evaluated positively by medical personnel and has been successfully put into practice at two independent European clinical sites. © Springer-Verlag 2013.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Intervention planning; Simulation; Tumor ablation; User-interfaces; Visualization

In vivo interactive visualization of four-dimensional blood flow patterns : RRRealtime assessment of volumetric phase contrast MRI - Kainz, Bernhard; Reiter, Ursula; Reiter, Gert; Schmalstieg, Dieter (Visual Computer, 2009)
In this paper we give an overview over a series of experiments to visualize and measure flow fields in the human vascular system with respect to their diagnostic capabilities. The experiments utilize a selection of GPU-based sparse and dense flow visualization algorithms to show the diagnostic opportunities for volumetric cardiovascular phase contrast magnetic resonance imaging data sets. Besides classical hardware accelerated particle and line-based approaches, an extensible tublet-based visualization, a four-dimensional volumetric line integral convolution and a new two-dimensional cutting plane tool for three-dimensional velocity data sets have been implemented. To evaluate the results, several hearts of human subjects have been investigated and a flow phantom was built to artificially simulate distinctive flow features. Our results demonstrate that we are able to provide an interactive tool for cardiovascular diagnostics with complementary hardware accelerated visualizations. © 2009 Springer-Verlag.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Cardiovascular diagnostics; Flow visualization; Phase contrast magnetic resonance imaging (PC-MRI)

Keynote: Beyond Supervised Learning Exploring Novel Machine Learning Approaches for Robust Medical Image Analysis - Kainz, Bernhard (Informatik aktuell, 2023)
Machine learning has been widely regarded as a solution for diagnostic automation in medical image analysis, but there are still unsolved problems in robust modelling of normal appearance and identification of features pointing into the long tail of population data. In this talk, I will explore the fitness of machine learning for applications at the front line of care and high throughput population health screening, specifically in prenatal health screening with ultrasound and MRI, cardiac imaging, and bedside diagnosis of deep vein thrombosis. I will discuss the requirements for such applications and how quality control can be achieved through robust estimation of algorithmic uncertainties and automatic robust modelling of expected anatomical structures. I will also explore the potential for improving models through active learning and the accuracy of nonexpert labelling workforces. However, I will argue that supervised machine learning might not be fit for purpose, as it cannot handle the unknown and requires a lot of annotated examples from well-defined pathological appearance. This categorization paradigm cannot be deployed earlier in the diagnostic pathway or for health screening, where a growing number of potentially hundred-thousands of medically catalogued illnesses may be relevant for diagnosis. Therefore, I introduce the idea of normative representation learning as a new machine learning paradigm for medical imaging. This paradigm can provide patient-specific computational tools for robust confirmation of normality, image quality control, health screening, and prevention of disease before onset. I will present novel deep learning approaches that can learn without manual labels from healthy patient data only. Our initial success with single class learning and self-supervised learning will be discussed, along with an outlook into the future with causal machine learning methods and the potential of advanced generative models [1].
Created from Fastlane, Scopus look-up

Learnable Slice-to-volume Reconstruction for Motion Compensation in Fetal Magnetic Resonance Imaging - Jehn, Constantin; Müller, Johanna; Kainz, Bernhard (Informatik aktuell, 2023)
Reconstructing motion-free 3D magnetic resonance imaging (MRI) volumes of fetal organs comes with the challenge of motion artefacts due to fetal motion and maternal respiration. Current methods rely on iterative procedures of outlier removal, super-resolution (SR) and slice-to-volume registration (SVR). Long runtimes and missing volume preservation over multiple iterations are still challenges for widespread clinical implementation. We envision an end-to-end learnable reconstruction framework that enables faster inference times and that can be steered by downstream tasks like segmentation. Therefore, we propose a new hybrid architecture for fetal brain reconstruction, consisting of a fully differentiable pre-registration module and a CycleGAN model for 3D image-toimage translation that is pretrained on our custom-generated dataset of 209 pairs of low-resolution (LoRes) and high-resolution (HiRes) fetal brain volumes. Our results are evaluated quantitatively with respect to five different similarity metrics. We incorporate the learned perceptual image patch similarity (LPIPS) metric and apply it to quantify volumetric image similarity for the first time in literature. Furthermore, we evaluate the model outputs qualitatively and conduct an expert survey to compare our method’s reconstruction quality to an established approach.
Created from Fastlane, Scopus look-up

Learning clinically useful information from images: Past, present and future - Rueckert, Daniel; Glocker, Ben; Kainz, Bernhard (Medical Image Analysis, 2016)
Over the last decade, research in medical imaging has made significant progress in addressing challenging tasks such as image registration and image segmentation. In particular, the use of model-based approaches has been key in numerous, successful advances in methodology. The advantage of model-based approaches is that they allow the incorporation of prior knowledge acting as a regularisation that favours plausible solutions over implausible ones. More recently, medical imaging has moved away from hand-crafted, and often explicitly designed models towards data-driven, implicit models that are constructed using machine learning techniques. This has led to major improvements in all stages of the medical imaging pipeline, from acquisition and reconstruction to analysis and interpretation. As more and more imaging data is becoming available, e.g., from large population studies, this trend is likely to continue and accelerate. At the same time new developments in machine learning, e.g., deep learning, as well as significant improvements in computing power, e.g., parallelisation on graphics hardware, offer new potential for data-driven, semantic and intelligent medical imaging. This article outlines the work of the BioMedIA group in this area and highlights some of the challenges and opportunities for future work.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Intelligent imaging; Machine learning; Semantic imaging

LesionMix: A Lesion-Level Data Augmentation Method for Medical Image Segmentation - Basaran, Berke Doga; Zhang, Weitong; Qiao, Mengyun; Kainz, Bernhard; Matthews, Paul M.; Bai, Wenjia (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2024)
Data augmentation has become a de facto component of deep learning-based medical image segmentation methods. Most data augmentation techniques used in medical imaging focus on spatial and intensity transformations to improve the diversity of training images. They are often designed at the image level, augmenting the full image, and do not pay attention to specific abnormalities within the image. Here, we present LesionMix, a novel and simple lesion-aware data augmentation method. It performs augmentation at the lesion level, increasing the diversity of lesion shape, location, intensity and load distribution, and allowing both lesion populating and inpainting. Experiments on different modalities and different lesion datasets, including four brain MR lesion datasets and one liver CT lesion dataset, demonstrate that LesionMix achieves promising performance in lesion image segmentation, outperforming several recent Mix-based data augmentation methods. The code will be released at https://github.com/dogabasaran/lesionmix.
CRIS-Team Scopus Importer:2024-05-24
Keywords: Data augmentation; Image synthesis; Lesion image segmentation; Lesion inpainting; Lesion populating

Machine learning for the automatic localisation of foetal body parts in cine-MRI scans - Bowles, Christopher; Nowlan, Niamh C.; Hayat, Tayyib T. A.; Malamateniou, Christina; Rutherford, Mary; Hajnal, Joseph V.; Rueckert, Daniel; Kainz, Bernhard (Progress in Biomedical Optics and Imaging - Proceedings of SPIE, 2015)
Being able to automate the location of individual foetal body parts has the potential to dramatically reduce the work required to analyse time resolved foetal Magnetic Resonance Imaging (cine-MRI) scans, for example, for use in the automatic evaluation of the foetal development. Currently, manual preprocessing of every scan is required to locate body parts before analysis can be performed, leading to a significant time overhead. With the volume of scans becoming available set to increase as cine-MRI scans become more prevalent in clinical practice, this stage of manual preprocessing is a bottleneck, limiting the data available for further analysis. Any tools which can automate this process will therefore save many hours of research time and increase the rate of new discoveries in what is a key area in understanding early human development. Here we present a series of techniques which can be applied to foetal cine-MRI scans in order to first locate and then differentiate between individual body parts. A novel approach to maternal movement suppression and segmentation using Fourier transforms is put forward as a preprocessing step, allowing for easy extraction of short movements of individual foetal body parts via the clustering of optical flow vector fields. These body part movements are compared to a labelled database and probabilistically classified before being spatially and temporally combined to give a final estimate for the location of each body part.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Cine-MRI; Foetal body part identification; Machine Learning; Motion Suppression; Segmentation

Magnetic resonance-derived 3-dimensional blood flow patterns in the main pulmonary artery as a marker of pulmonary hypertension and a measure of elevated mean pulmonary arterial pressure. - Kainz, Bernhard; et al. (Circulation. Cardiovascular genetics, 2008)
Background-Pulmonary hypertension is a disease characterized by an elevation in pulmonary arterial pressure that is diagnosed invasively via right heart catheterization. Such pathological altered pressures in the pulmonary vascular system should lead to changes in blood flow patterns in the main pulmonary artery.Methods and Results-Forty-eight subjects (22 with manifest pulmonary hypertension, 13 with latent pulmonary hypertension, and 13 normal control subjects) underwent time-resolved 3D magnetic resonance phase-contrast imaging of the main pulmonary artery. Velocity fields that resulted from measurements were calculated, visualized, and analyzed with dedicated software. Main findings were as follows: (1) Manifest pulmonary hypertension coincides with the appearance of a vortex of blood flow in the main pulmonary artery (sensitivity and specificity of 1.00, 95% confidence intervals of 0.84 to 1.00 and 0.87 to 1.00, respectively), and (2) the relative period of existence of the vortex correlates significantly with mean pulmonary arterial pressure at rest (correlation coefficient of 0.94). To test the diagnostic performance of the vortex criterion, we furthermore investigated 55 patients in a blinded prospective study (22 with manifest pulmonary hypertension, 32 with latent pulmonary hypertension, and 1healthy subject), which resulted in a sensitivity of 1.00 and specificity of 0.91 (95% confidence intervals of 0.84 to 1.00 and 0.76 to 0.98, respectively). Comparison of catheter-derived mean pulmonary artery pressure measurements and calculated mean pulmonary artery pressure values resulted in a standard deviation of differences of 3.6 mm Hg.Conclusions-Vortices of blood flow in the main pulmonary artery enable the identification of manifest pulmonary hypertension. Elevated mean pulmonary arterial pressures can be measured from the period of vortex existence. (Circ Cardiovasc Imaging. 2008;1:23-30.)
Keywords: hypertension; pulmonary; magnetic resonance imaging; blood flow; blood pressure; catheterization

Many Tasks Make Light Work: Learning to Localise Medical Anomalies from Multiple Synthetic Tasks - Baugh, Matthew; Tan, Jeremy; Müller, Johanna P.; Dombrowski, Mischa; Batten, James; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2023)
There is a growing interest in single-class modelling and out-of-distribution detection as fully supervised machine learning models cannot reliably identify classes not included in their training. The long tail of infinitely many out-of-distribution classes in real-world scenarios, e.g., for screening, triage, and quality control, means that it is often necessary to train single-class models that represent an expected feature distribution, e.g., from only strictly healthy volunteer data. Conventional supervised machine learning would require the collection of datasets that contain enough samples of all possible diseases in every imaging modality, which is not realistic. Self-supervised learning methods with synthetic anomalies are currently amongst the most promising approaches, alongside generative auto-encoders that analyse the residual reconstruction error. However, all methods suffer from a lack of structured validation, which makes calibration for deployment difficult and dataset-dependant. Our method alleviates this by making use of multiple visually-distinct synthetic anomaly learning tasks for both training and validation. This enables more robust training and generalisation. With our approach we can readily outperform state-of-the-art methods, which we demonstrate on exemplars in brain MRI and chest X-rays. Code is available at https://github.com/matt-baugh/many-tasks-make-light-work.
CRIS-Team Scopus Importer:2023-11-10

MetaDetector: Detecting Outliers by Learning to Learn from Self-supervision - Tan, Jeremy; Kart, Turkay; Hou, Benjamin; Batten, James; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2022)
Using self-supervision in anomaly detection can increase sensitivity to subtle irregularities. However, increasing sensitivity to certain classes of outliers could result in decreased sensitivity to other types. While a single model may have limited coverage, an adaptive method could help detect a broader range of outliers. Our proposed method explores whether meta learning can increase the adaptability of self-supervised methods. Meta learning is often employed in few-shot settings with labelled examples. To use it for anomaly detection, where labelled support data is usually not available, we instead construct a self-supervised task using the test input itself and reference samples from the normal training data. Specifically, patches from the test image are introduced into normal reference images. This forms the basis of the few-shot task. During training, the same few-shot process is used, but the test/query image is substituted with a normal training image that contains a synthetic irregularity. Meta learning is then used to learn how to learn from the few-shot task by computing second order gradients. Given the importance of screening applications, e.g. in healthcare or security, any adaptability in the method must be counterbalanced with robustness. As such, we add strong regularization by i) restricting meta learning to only layers near the bottleneck of our encoder-decoder architecture and ii) computing the loss at multiple points during the few-shot process.
CRIS-Team Scopus Importer:2022-03-25
Keywords: Meta-learning; Outlier detection; Self-supervised learning

Metrics reloaded: recommendations for image analysis validation - Maier-Hein, Lena; Reinke, Annika; Godau, Patrick; Tizabi, Minu D.; Buettner, Florian; Christodoulou, Evangelia; Glocker, Ben; Isensee, Fabian; Kleesiek, Jens; Kozubek, Michal; Reyes, Mauricio; Riegler, Michael A.; et al. (Nature Methods, 2024)
Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint—a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.
CRIS-Team Scopus Importer:2024-03-01

MoCoSR: Respiratory Motion Correction and Super-Resolution for 3D Abdominal MRI - Zhang, Weitong; Basaran, Berke; Meng, Qingjie; Baugh, Matthew; Stelter, Jonathan; Lung, Phillip; Patel, Uday; Bai, Wenjia; Karampinos, Dimitrios; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2023)
Abdominal MRI is critical for diagnosing a wide variety of diseases. However, due to respiratory motion and other organ motions, it is challenging to obtain motion-free and isotropic MRI for clinical diagnosis. Imaging patients with inflammatory bowel disease (IBD) can be especially problematic, owing to involuntary bowel movements and difficulties with long breath-holds during acquisition. Therefore, this paper proposes a deep adversarial super-resolution (SR) reconstruction approach to address the problem of multi-task degradation by utilizing cycle consistency in a staged reconstruction model. We leverage a low-resolution (LR) latent space for motion correction, followed by super-resolution reconstruction, compensating for imaging artefacts caused by respiratory motion and spontaneous bowel movements. This alleviates the need for semantic knowledge about the intestines and paired data. Both are examined through variations of our proposed approach and we compare them to conventional, model-based, and learning-based MC and SR methods. Learned image reconstruction approaches are believed to occasionally hide disease signs. We investigate this hypothesis by evaluating a downstream task, automatically scoring IBD in the area of the terminal ileum on the reconstructed images and show evidence that our method does not suffer a synthetic domain bias.
CRIS-Team Scopus Importer:2023-11-03
Keywords: Abdominal MR; Deep Learning; Motion Correction; Super-resolution

MOOD 2020: A Public Benchmark for Out-of-Distribution Detection and Localization on Medical Images - Zimmerer, David; Full, Peter M.; Isensee, Fabian; Jaeger, Paul; Adler, Tim; Petersen, Jens; Koehler, Gregor; Ross, Tobias; Reinke, Annika; Kascenas, Antanas; Jensen, Bjorn Sand; O'Neil, Alison Q.; et al. (IEEE Transactions on Medical Imaging, 2022)
Detecting Out-of-Distribution (OoD) data is one of the greatest challenges in safe and robust deployment of machine learning algorithms in medicine. When the algorithms encounter cases that deviate from the distribution of the training data, they often produce incorrect and over-confident predictions. OoD detection algorithms aim to catch erroneous predictions in advance by analysing the data distribution and detecting potential instances of failure. Moreover, flagging OoD cases may support human readers in identifying incidental findings. Due to the increased interest in OoD algorithms, benchmarks for different domains have recently been established. In the medical imaging domain, for which reliable predictions are often essential, an open benchmark has been missing. We introduce the Medical-Out-Of-Distribution-Analysis-Challenge (MOOD) as an open, fair, and unbiased benchmark for OoD methods in the medical imaging domain. The analysis of the submitted algorithms shows that performance has a strong positive correlation with the perceived difficulty, and that all algorithms show a high variance for different anomalies, making it yet hard to recommend them for clinical practice. We also see a strong correlation between challenge ranking and performance on a simple toy test set, indicating that this might be a valuable addition as a proxy dataset during anomaly detection algorithm development.
CRIS-Team Scopus Importer:2022-10-21
Keywords: Anomaly detection; anomaly localization; biomedical challenge; out-of-distribution analysis

Morpho-Mnist: Quantitative assessment and diagnostics for representation learning - Castro, Daniel C.; Tan, Jeremy; Kainz, Bernhard; Konukoglu, Ender; Glocker, Ben (Journal of Machine Learning Research, 2019)
Revealing latent structure in data is an active field of research, having introduced exciting technologies such as variational autoencoders and adversarial networks, and is essential to push machine learning towards unsupervised knowledge discovery. However, a major challenge is the lack of suitable benchmarks for an objective and quantitative evaluation of learned representations. To address this issue we introduce Morpho-MNIST, a framework that aims to answer: “to what extent has my model learned to represent specific factors of variation in the data?” We extend the popular MNIST dataset by adding a morphometric analysis enabling quantitative comparison of trained models, identification of the roles of latent variables, and characterisation of sample diversity. We further propose a set of quantifiable perturbations to assess the performance of unsupervised and supervised methods on challenging tasks such as outlier detection and domain adaptation. Data and code are available at https://github.com/dccastro/Morpho-MNIST.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Disentanglement; Empirical evaluation; Generative models; Morphometrics; Representation learning

Motion corrected 3D reconstruction of the fetal thorax from prenatal MRI - Kainz, Bernhard; Malamateniou, Christina; Murgasova, Maria; Keraudren, Kevin; Rutherford, Mary; Hajnal, Joseph V.; Rueckert, Daniel (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2014)
In this paper we present a semi-automatic method for analysis of the fetal thorax in genuine three-dimensional volumes. After one initial click we localize the spine and accurately determine the volume of the fetal lung from high resolution volumetric images reconstructed from motion corrupted prenatal Magnetic Resonance Imaging (MRI). We compare the current state-of-the-art method of segmenting the lung in a slice-by-slice manner with the most recent multi-scan reconstruction methods. We use fast rotation invariant spherical harmonics image descriptors with Classification Forest ensemble learning methods to extract the spinal cord and show an efficient way to generate a segmentation prior for the fetal lung from this information for two different MRI field strengths. The spinal cord can be segmented with a DICE coefficient of 0.89 and the automatic lung segmentation has been evaluated with a DICE coefficient of 0.87. We evaluate our method on 29 fetuses with a gestational age (GA) between 20 and 38 weeks and show that our computed segmentations and the manual ground truth correlate well with the recorded values in literature.
CRIS-Team Scopus Importer:2021-08-17

MOTION CORRECTED FETAL CARDIAC MRI INCREASES DIAGNOSTIC CONFIDENCE IN CLINICALLY CHALLENGING CASES - Kainz, Bernhard; Lloyd, David FA; van Poppel, M; et al. (Heart, 2018)

Multiple landmark detection using multi-agent reinforcement learning - Vlontzos, Athanasios; Alansary, Amir; Kamnitsas, Konstantinos; Rueckert, Daniel; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2019)
The detection of anatomical landmarks is a vital step for medical image analysis and applications for diagnosis, interpretation and guidance. Manual annotation of landmarks is a tedious process that requires domain-specific expertise and introduces inter-observer variability. This paper proposes a new detection approach for multiple landmarks based on multi-agent reinforcement learning. Our hypothesis is that the position of all anatomical landmarks is interdependent and non-random within the human anatomy, thus finding one landmark can help to deduce the location of others. Using a Deep Q-Network (DQN) architecture we construct an environment and agent with implicit inter-communication such that we can accommodate K agents acting and learning simultaneously, while they attempt to detect K different landmarks. During training the agents collaborate by sharing their accumulated knowledge for a collective gain. We compare our approach with state-of-the-art architectures and achieve significantly better accuracy by reducing the detection error by 50%, while requiring fewer computational resources and time to train compared to the naïve approach of training K agents separately. Code and visualizations available: https://github.com/thanosvlo/MARL-for-Anatomical-Landmark-Detection.
CRIS-Team Scopus Importer:2021-08-17

Multitask Weakly Supervised Generative Network for MR-US Registration - Azampour, Mohammad Farid; Mach, Kristina; Fatemizadeh, Emad; Demiray, Beatrice; Westenfelder, Kay; Steiger, Katja; Eiber, Matthias; Wendler, Thomas; Kainz, Bernhard; Navab, Nassir (IEEE Transactions on Medical Imaging, 2024)
Registering pre-operative modalities, such as magnetic resonance imaging or computed tomography, to ultrasound images is crucial for guiding clinicians during surgeries and biopsies. Recently, deep-learning approaches have been proposed to increase the speed and accuracy of this registration problem. However, all of these approaches need expensive supervision from the ultrasound domain. In this work, we propose a multitask generative framework that needs weak supervision only from the pre-operative imaging domain during training. To perform a deformable registration, the proposed framework translates a magnetic resonance image to the ultrasound domain while preserving the structural content. To demonstrate the efficacy of the proposed method, we tackle the registration problem of pre-operative 3D MR to transrectal ultrasonography images as necessary for targeted prostate biopsies. We use an in-house dataset of 600 patients, divided into 540 for training, 30 for validation, and the remaining for testing. An expert manually segmented the prostate in both modalities for validation and test sets to assess the performance of our framework. The proposed framework achieves a 3.58 mm target registration error on the expert-selected landmarks, 89.2% in the Dice score, and 1.81 mm 95th percentile Hausdorff distance on the prostate masks in the test set. Our experiments demonstrate that the proposed generative model successfully translates magnetic resonance images into the ultrasound domain. The translated image contains the structural content and fine details due to an ultrasound-specific two-path design of the generative model. The proposed framework enables training learning-based registration methods while only weak supervision from the pre-operative domain is available.
CRIS-Team Scopus Importer:2024-06-14
Keywords: Biopsy; Convolutional neural network; Deep learning; Image segmentation; Measurement; Medical image registration; Multi-task learning; Multitasking; Training; Ultrasonic imaging; Unpaired image-to-image translation; US-MR registration

Mutual Information-Based Disentangled Neural Networks for Classifying Unseen Categories in Different Domains: Application to Fetal Ultrasound Imaging - Meng, Qingjie; Matthew, Jacqueline; Zimmer, Veronika A.; Gomez, Alberto; Lloyd, David F. A.; Rueckert, Daniel; Kainz, Bernhard (IEEE Transactions on Medical Imaging, 2021)
Deep neural networks exhibit limited generalizability across images with different entangled domain features and categorical features. Learning generalizable features that can form universal categorical decision boundaries across domains is an interesting and difficult challenge. This problem occurs frequently in medical imaging applications when attempts are made to deploy and improve deep learning models across different image acquisition devices, across acquisition parameters or if some classes are unavailable in new training databases. To address this problem, we propose Mutual Information-based Disentangled Neural Networks (MIDNet), which extract generalizable categorical features to transfer knowledge to unseen categories in a target domain. The proposed MIDNet adopts a semi-supervised learning paradigm to alleviate the dependency on labeled data. This is important for real-world applications where data annotation is time-consuming, costly and requires training and expertise. We extensively evaluate the proposed method on fetal ultrasound datasets for two different image classification tasks where domain features are respectively defined by shadow artifacts and image acquisition devices. Experimental results show that the proposed method outperforms the state-of-The-Art on the classification of unseen categories in a target domain with sparsely labeled training data.
CRIS-Team Scopus Importer:2021-08-17
Keywords: domain adaptation; image classification; Representation disentanglement; semi-supervised learning

Natural Synthetic Anomalies for Self-supervised Anomaly Detection and Localization - Schlüter, Hannah M.; Tan, Jeremy; Hou, Benjamin; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2022)
We introduce a simple and intuitive self-supervision task, Natural Synthetic Anomalies (NSA), for training an end-to-end model for anomaly detection and localization using only normal training data. NSA integrates Poisson image editing to seamlessly blend scaled patches of various sizes from separate images. This creates a wide range of synthetic anomalies which are more similar to natural sub-image irregularities than previous data-augmentation strategies for self-supervised anomaly detection. We evaluate the proposed method using natural and medical images. Our experiments with the MVTec AD dataset show that a model trained to localize NSA anomalies generalizes well to detecting real-world a priori unknown types of manufacturing defects. Our method achieves an overall detection AUROC of 97.2 outperforming all previous methods that learn without the use of additional datasets. Code available at https://github.com/hmsch/natural-synthetic-anomalies.
CRIS-Team Scopus Importer:2022-12-09
Keywords: Image anomaly localization; self-supervised learning

nnOOD: A Framework for Benchmarking Self-supervised Anomaly Localisation Methods - Baugh, Matthew; Tan, Jeremy; Vlontzos, Athanasios; Müller, Johanna; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2022)
The wide variety of in-distribution and out-of-distribution data in medical imaging makes universal anomaly detection a challenging task. Recently a number of self-supervised methods have been developed that train end-to-end models on healthy data augmented with synthetic anomalies. However, it is difficult to compare these methods as it is not clear whether gains in performance are from the task itself or the training pipeline around it. It is also difficult to assess whether a task generalises well for universal anomaly detection, as they are often only tested on a limited range of anomalies. To assist with this we have developed nnOOD, a framework that adapts nnU-Net to allow for comparison of self-supervised anomaly localisation methods. By isolating the synthetic, self-supervised task from the rest of the training process we perform a more faithful comparison of the tasks, whilst also making the workflow for evaluating over a given dataset quick and easy. Using this we have implemented the current state-of-the-art tasks and evaluated them on a challenging X-ray dataset.
CRIS-Team Scopus Importer:2022-10-14
Keywords: Anomaly localisation; Self-supervised learning

Noise-based volume rendering for the visualization of multivariate volumetric data - Khlebnikov, Rostislav; Kainz, Bernhard; Steinberger, Markus; Schmalstieg, Dieter (IEEE Transactions on Visualization and Computer Graphics, 2013)
Analysis of multivariate data is of great importance in many scientific disciplines. However, visualization of 3D spatially-fixed multivariate volumetric data is a very challenging task. In this paper we present a method that allows simultaneous real-time visualization of multivariate data. We redistribute the opacity within a voxel to improve the readability of the color defined by a regular transfer function, and to maintain the see-through capabilities of volume rendering. We use predictable procedural noise-random-phase Gabor noise-to generate a high-frequency redistribution pattern and construct an opacity mapping function, which allows to partition the available space among the displayed data attributes. This mapping function is appropriately filtered to avoid aliasing, while maintaining transparent regions. We show the usefulness of our approach on various data sets and with different example applications. Furthermore, we evaluate our method by comparing it to other visualization techniques in a controlled user study. Overall, the results of our study indicate that users are much more accurate in determining exact data values with our novel 3D volume visualization method. Significantly lower error rates for reading data values and high subjective ranking of our method imply that it has a high chance of being adopted for the purpose of visualization of multivariate 3D data. © 2013 IEEE.
CRIS-Team Scopus Importer:2021-08-17
Keywords: multi-variate data visualization; multi-volume rendering; scientific visualization; Volume rendering

Non-invasive diagnosis of deep vein thrombosis from ultrasound imaging with machine learning - Kainz, Bernhard; Heinrich, Mattias P.; Makropoulos, Antonios; Oppenheimer, Jonas; Mandegaran, Ramin; Sankar, Shrinivasan; Deane, Christopher; Mischkewitz, Sven; Al-Noor, Fouad; Rawdin, Andrew C.; Ruttloff, Andreas; Klein-Weigel, Peter F.; et al. (npj Digital Medicine, 2021)
Deep vein thrombosis (DVT) is a blood clot most commonly found in the leg, which can lead to fatal pulmonary embolism (PE). Compression ultrasound of the legs is the diagnostic gold standard, leading to a definitive diagnosis. However, many patients with possible symptoms are not found to have a DVT, resulting in long referral waiting times for patients and a large clinical burden for specialists. Thus, diagnosis at the point of care by non-specialists is desired. We collect images in a pre-clinical study and investigate a deep learning approach for the automatic interpretation of compression ultrasound images. Our method provides guidance for free-hand ultrasound and aids non-specialists in detecting DVT. We train a deep learning algorithm on ultrasound videos from 255 volunteers and evaluate on a sample size of 53 prospectively enrolled patients from an NHS DVT diagnostic clinic and 30 prospectively enrolled patients from a German DVT clinic. Algorithmic DVT diagnosis performance results in a sensitivity within a 95% CI range of (0.82, 0.94), specificity of (0.70, 0.82), a positive predictive value of (0.65, 0.89), and a negative predictive value of (0.99, 1.00) when compared to the clinical gold standard. To assess the potential benefits of this technology in healthcare we evaluate the entire clinical DVT decision algorithm and provide cost analysis when integrating our approach into diagnostic pathways for DVT. Our approach is estimated to generate a positive net monetary benefit at costs up to 72 pound to 175 pound per software-supported examination, assuming a willingness to pay of 20,000 pound/QALY.
CRIS-Team WoS Importer:2021-09-24

Omni kinect: Real-time dense volumetric data acquisition and applications - Kainz, Bernhard; Hauswiesner, Stefan; Reitmayr, Gerhard; Steinberger, Markus; Grasset, Raphael; Gruber, Lukas; Veas, Eduardo; Kalkofen, Denis; Seichter, Hartmut; Schmalstieg, Dieter (Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST, 2012)
Real-time three-dimensional acquisition of real-world scenes has many important applications in computer graphics, computer vision and human-computer interaction. Inexpensive depth sensors such as the Microsoft Kinect allow to leverage the development of such applications. However, this technology is still relatively recent, and no detailed studies on its scalability to dense and view-independent acquisition have been reported. This paper addresses the question of what can be done with a larger number of Kinects used simultaneously. We describe an interference-reducing physical setup, a calibration procedure and an extension to the KinectFusion algorithm, which allows to produce high quality volumetric reconstructions from multiple Kinects whilst overcoming systematic errors in the depth measurements. We also report on enhancing image based visual hull rendering by depth measurements, and compare the results to KinectFusion. Our system provides practical insight into achievable spatial and radial range and into bandwidth requirements for depth data acquisition. Finally, we present a number of practical applications of our system. Copyright 2012 ACM.
CRIS-Team Scopus Importer:2021-08-17
Keywords: 4D reconstruction; Depth sensors; Microsoft Kinect

On-the-fly generation and rendering of infinite cities on the GPU - Steinberger, Markus; Kenzel, Michael; Kainz, Bernhard; Wonka, Peter; Schmalstieg, Dieter (Computer Graphics Forum, 2014)
In this paper, we present a new approach for shape-grammar-based generation and rendering of huge cities in real-time on the graphics processing unit (GPU). Traditional approaches rely on evaluating a shape grammar and storing the geometry produced as a preprocessing step. During rendering, the pregenerated data is then streamed to the GPU. By interweaving generation and rendering, we overcome the problems and limitations of streaming pregenerated data. Using our methods of visibility pruning and adaptive level of detail, we are able to dynamically generate only the geometry needed to render the current view in real-time directly on the GPU. We also present a robust and efficient way to dynamically update a scene's derivation tree and geometry, enabling us to exploit frame-to-frame coherence. Our combined generation and rendering is significantly faster than all previous work. For detailed scenes, we are capable of generating geometry more rapidly than even just copying pregenerated data from main memory, enabling us to render cities with thousands of buildings at up to 100 frames per second, even with the camera moving at supersonic speed. © 2014 The Author(s) Computer Graphics Forum © 2014 The Eurographics Association and John Wiley & Sons Ltd. Published by John Wiley & Sons Ltd.
CRIS-Team Scopus Importer:2021-08-17

ParaDime: A Framework for Parametric Dimensionality Reduction - Hinterreiter, Andreas; Humer, Christina; Kainz, Bernhard; Streit, Marc (Computer Graphics Forum, 2023)
ParaDime is a framework for parametric dimensionality reduction (DR). In parametric DR, neural networks are trained to embed high-dimensional data items in a low-dimensional space while minimizing an objective function. ParaDime builds on the idea that the objective functions of several modern DR techniques result from transformed inter-item relationships. It provides a common interface for specifying these relations and transformations and for defining how they are used within the losses that govern the training process. Through this interface, ParaDime unifies parametric versions of DR techniques such as metric MDS, t-SNE, and UMAP. It allows users to fully customize all aspects of the DR process. We show how this ease of customization makes ParaDime suitable for experimenting with interesting techniques such as hybrid classification/embedding models and supervised DR. This way, ParaDime opens up new possibilities for visualizing high-dimensional data.
CRIS-Team Scopus Importer:2023-07-21
Keywords: CCS Concepts; Information visualization; Learning latent representations; • Computing methodologies → Neural networks; • Human-centered computing → Visualization systems and tools

Parallel generation of architecture on the GPU - Steinberger, Markus; Kenzel, Michael; Kainz, Bernhard; Mueller, Joerg; Wonka, Peter; Schmalstieg, Dieter (Computer Graphics Forum, 2014)
In this paper, we present a novel approach for the parallel evaluation of procedural shape grammars on the graphics processing unit (GPU). Unlike previous approaches that are either limited in the kind of shapes they allow, the amount of parallelism they can take advantage of, or both, our method supports state of the art procedural modeling including stochasticity and context-sensitivity. To increase parallelism, we explicitly express independence in the grammar, reduce inter-rule dependencies required for context-sensitive evaluation, and introduce intra-rule parallelism. Our rule scheduling scheme avoids unnecessary back and forth between CPU and GPU and reduces round trips to slow global memory by dynamically grouping rules in on-chip shared memory. Our GPU shape grammar implementation is multiple orders of magnitude faster than the standard in CPU-based rule evaluation, while offering equal expressive power. In comparison to the state of the art in GPU shape grammar derivation, our approach is nearly 50 times faster, while adding support for geometric context-sensitivity. © 2014 The Author(s) Computer Graphics Forum © 2014 The Eurographics Association and John Wiley & Sons Ltd. Published by John Wiley & Sons Ltd.
CRIS-Team Scopus Importer:2021-08-17

Parallel irradiance caching for interactive monte-carlo direct volume rendering - Khlebnikov, Rostislav; Voglreiter, Philip; Steinberger, Markus; Kainz, Bernhard; Schmalstieg, Dieter (Computer Graphics Forum, 2014)
We propose a technique to build the irradiance cache for isotropic scattering simultaneously with Monte Carlo progressive direct volume rendering on a single GPU, which allows us to achieve up to four times increased convergence rate for complex scenes with arbitrary sources of light. We use three procedures that run concurrently on a single GPU. The first is the main rendering procedure. The second procedure computes new cache entries, and the third one corrects the errors that may arise after creation of new cache entries. We propose two distinct approaches to allow massive parallelism of cache entry creation. In addition, we show a novel extrapolation approach which outputs high quality irradiance approximations and a suitable prioritization scheme to increase the convergence rate by dedicating more computational power to more complex rendering areas. © 2014 The Eurographics Association and John Wiley & Sons Ltd. Published by John Wiley & Sons Ltd.
CRIS-Team Scopus Importer:2021-08-17

PialNN: A Fast Deep Learning Framework for Cortical Pial Surface Reconstruction - Kainz, Bernhard; et al. (Springer Science and Business Media Deutschland GmbH, 2021)

Placenta maps: In utero placental health assessment of the human fetus - Miao, Haichao; Mistelbauer, Gabriel; Karimov, Alexey; Alansary, Amir; Davidson, Alice; Lloyd, David F. A.; Damodaram, Mellisa; Story, Lisa; Hutter, Jana; Hajnal, Joseph V.; Rutherford, Mary; Preim, Bernhard; et al. (IEEE Transactions on Visualization and Computer Graphics, 2017)
The human placenta is essential for the supply of the fetus. To monitor the fetal development, imaging data is acquired using (US). Although it is currently the gold-standard in fetal imaging, it might not capture certain abnormalities of the placenta. (MRI) is a safe alternative for the in utero examination while acquiring the fetus data in higher detail. Nevertheless, there is currently no established procedure for assessing the condition of the placenta and consequently the fetal health. Due to maternal respiration and inherent movements of the fetus during examination, a quantitative assessment of the placenta requires fetal motion compensation, precise placenta segmentation and a standardized visualization, which are challenging tasks. Utilizing advanced motion compensation and automatic segmentation methods to extract the highly versatile shape of the placenta, we introduce a novel visualization technique that presents the fetal and maternal side of the placenta in a standardized way. Our approach enables physicians to explore the placenta even in utero. This establishes the basis for a comparative assessment of multiple placentas to analyze possible pathologic arrangements and to support the research and understanding of this vital organ. Additionally, we propose a three-dimensional structure-aware surface slicing technique in order to explore relevant regions inside the placenta. Finally, to survey the applicability of our approach, we consulted clinical experts in prenatal diagnostics and imaging. We received mainly positive feedback, especially the applicability of our technique for research purposes was appreciated.
CRIS-Team Scopus Importer:2021-08-17
Keywords: fetal; flattening; peeling; Placenta; structure-aware slicing

Placenta segmentation in ultrasound imaging: Addressing sources of uncertainty and limited field-of-view - Zimmer, Veronika A.; Gomez, Alberto; Skelton, Emily; Wright, Robert; Wheeler, Gavin; Deng, Shujie; Ghavami, Nooshin; Lloyd, Karen; Matthew, Jacqueline; Kainz, Bernhard; Rueckert, Daniel; Hajnal, Joseph V.; et al. (Medical Image Analysis, 2023)
Automatic segmentation of the placenta in fetal ultrasound (US) is challenging due to the (i) high diversity of placenta appearance, (ii) the restricted quality in US resulting in highly variable reference annotations, and (iii) the limited field-of-view of US prohibiting whole placenta assessment at late gestation. In this work, we address these three challenges with a multi-task learning approach that combines the classification of placental location (e.g., anterior, posterior) and semantic placenta segmentation in a single convolutional neural network. Through the classification task the model can learn from larger and more diverse datasets while improving the accuracy of the segmentation task in particular in limited training set conditions. With this approach we investigate the variability in annotations from multiple raters and show that our automatic segmentations (Dice of 0.86 for anterior and 0.83 for posterior placentas) achieve human-level performance as compared to intra- and inter-observer variability. Lastly, our approach can deliver whole placenta segmentation using a multi-view US acquisition pipeline consisting of three stages: multi-probe image acquisition, image fusion and image segmentation. This results in high quality segmentation of larger structures such as the placenta in US with reduced image artifacts which are beyond the field-of-view of single probes.
CRIS-Team Scopus Importer:2022-10-28
Keywords: Multi-task learning; Multi-view imaging; Ultrasound placenta segmentation; Uncertainty/variability

Predicting slice-to-volume transformation in presence of arbitrary subject motion - Hou, Benjamin; Alansary, Amir; McDonagh, Steven; Davidson, Alice; Rutherford, Mary; Hajnal, Jo V.; Rueckert, Daniel; Glocker, Ben; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2017)
This paper aims to solve a fundamental problem in intensity-based 2D/3D registration, which concerns the limited capture range and need for very good initialization of state-of-the-art image registration methods. We propose a regression approach that learns to predict rotations and translations of arbitrary 2D image slices from 3D volumes, with respect to a learned canonical atlas co-ordinate system. To this end, we utilize Convolutional Neural Networks (CNNs) to learn the highly complex regression function that maps 2D image slices into their correct position and orientation in 3D space. Our approach is attractive in challenging imaging scenarios, where significant subject motion complicates reconstruction performance of 3D volumes from 2D slice data. We extensively evaluate the effectiveness of our approach quantitatively on simulated MRI brain data with extreme random motion. We further demonstrate qualitative results on fetal MRI where our method is integrated into a full reconstruction and motion compensation pipeline. With our CNN regression approach we obtain an average prediction error of 7 mm on simulated data, and convincing reconstruction quality of images of very young fetuses where previous methods fail. We further discuss applications to Computed Tomography (CT) and X-Ray projections. Our approach is a general solution to the 2D/3D initialization problem. It is computationally efficient, with prediction times per slice of a few milliseconds, making it suitable for real-time scenarios.
CRIS-Team Scopus Importer:2021-08-17

Preface RAMBO 2017 - Kainz, Bernhard; et al. (Springer Verlag, 2017)

Prenatal diagnosis of hypoplastic left heart syndrome on ultrasound using artificial intelligence: How does performance compare to a current screening programme? - Day, Thomas G.; Budd, Samuel; Tan, Jeremy; Matthew, Jacqueline; Skelton, Emily; Jowett, Victoria; Lloyd, David; Gomez, Alberto; Hajnal, Jo V.; Razavi, Reza; Kainz, Bernhard; Simpson, John M. (Prenatal Diagnosis, 2023)
Background: Artificial intelligence (AI) has the potential to improve prenatal detection of congenital heart disease. We analysed the performance of the current national screening programme in detecting hypoplastic left heart syndrome (HLHS) to compare with our own AI model. Methods: Current screening programme performance was calculated from local and national sources. AI models were trained using four-chamber ultrasound views of the fetal heart, using a ResNet classifier. Results: Estimated current fetal screening programme sensitivity and specificity for HLHS were 94.3% and 99.985%, respectively. Depending on calibration, AI models to detect HLHS were either highly sensitive (sensitivity 100%, specificity 94.0%) or highly specific (sensitivity 93.3%, specificity 100%). Our analysis suggests that our highly sensitive model would generate 45,134 screen positive results for a gain of 14 additional HLHS cases. Our highly specific model would be associated with two fewer detected HLHS cases, and 118 fewer false positives. Conclusion: If used independently, our AI model performance is slightly worse than the performance level of the current screening programme in detecting HLHS, and this performance is likely to deteriorate further when used prospectively. This demonstrates that collaboration between humans and AI will be key for effective future clinical use.
CRIS-Team Scopus Importer:2023-10-20

PRETUS: A plug-in based platform for real-time ultrasound imaging research - Gomez, Alberto; Zimmer, Veronika A.; Wheeler, Gavin; Toussaint, Nicolas; Deng, Shujie; Wright, Robert; Skelton, Emily; Matthew, Jackie; Kainz, Bernhard; Hajnal, Jo; Schnabel, Julia (SoftwareX, 2022)
We present PRETUS — a Plugin-based Real Time UltraSound software platform for live ultrasound image analysis and operator support. The software is lightweight; functionality is brought in via independent plug-ins that can be arranged in sequence. The software allows to capture the real-time stream of ultrasound images from virtually any ultrasound machine, applies computational methods and visualizes the results on-the-fly. Plug-ins can run concurrently without blocking each other. They can be implemented in C++ and Python. A graphical user interface can be implemented for each plug-in, and presented to the user in a compact way. The software is free and open source, and allows for rapid prototyping and testing of real-time ultrasound imaging methods in a manufacturer-agnostic fashion. The software is provided with input, output and processing plug-ins, as well as with tutorials to illustrate how to develop new plug-ins for PRETUS.
CRIS-Team Scopus Importer:2022-03-25
Keywords: Plug-in based; Real time; Ultrasound imaging

Procedural texture synthesis for zoom-independent visualization of multivariate data - Khlebnikov, R.; Kainz, Bernhard; Steinberger, M.; Streit, Marc; Schmalstieg, D. (Computer Graphics Forum, 2012)
Simultaneous visualization of multiple continuous data attributes in a single visualization is a task that is important for many application areas. Unsurprisingly, many methods have been proposed to solve this task. However, the behavior of such methods during the exploration stage, when the user tries to understand the data with panning and zooming, has not been given much attention. In this paper, we propose a method that uses procedural texture synthesis to create zoom-independent visualizations of three scalar data attributes. The method is based on random-phase Gabor noise, whose frequency is adapted for the visualization of the first data attribute. We ensure that the resulting texture frequency lies in the range that is perceived well by the human visual system at any zoom level. To enhance the perception of this attribute, we also apply a specially constructed transfer function that is based on statistical properties of the noise. Additionally, the transfer function is constructed in a way that it does not introduce any aliasing to the texture. We map the second attribute to the texture orientation. The third attribute is color coded and combined with the texture by modifying the value component of the HSV color model. The necessary contrast needed for texture and color perception was determined in a user study. In addition, we conducted a second user study that shows significant advantages of our method over current methods with similar goals. We believe that our method is an important step towards creating methods that not only succeed in visualizing multiple data attributes, but also adapt to the behavior of the user during the data exploration stage. © 2012 The Author(s).
CRIS-Team Scopus Importer:2021-08-17

Projective Latent Interventions for Understanding and Fine-Tuning Classifiers - Hinterreiter, Andreas; Streit, Marc; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2020)
High-dimensional latent representations learned by neural network classifiers are notoriously hard to interpret. Especially in medical applications, model developers and domain experts desire a better understanding of how these latent representations relate to the resulting classification performance. We present Projective Latent Interventions (PLIs), a technique for retraining classifiers by back-propagating manual changes made to low-dimensional embeddings of the latent space. The back-propagation is based on parametric approximations of distributed stochastic neighbourhood embeddings. PLIs allow domain experts to control the latent decision space in an intuitive way in order to better match their expectations. For instance, the performance for specific pairs of classes can be enhanced by manually separating the class clusters in the embedding. We evaluate our technique on a real-world scenario in fetal ultrasound imaging.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Image classification; Latent space; Non-linear embedding

PVR: Patch-to-Volume Reconstruction for Large Area Motion Correction of Fetal MRI - Alansary, Amir; Rajchl, Martin; Mcdonagh, Steven G.; Murgasova, Maria; Damodaram, Mellisa; Lloyd, David F. A.; Davidson, Alice; Rutherford, Mary; Hajnal, Joseph V.; Rueckert, Daniel; Kainz, Bernhard (IEEE Transactions on Medical Imaging, 2017)
In this paper, we present a novel method for the correction of motion artifacts that are present in fetal magnetic resonance imaging (MRI) scans of the whole uterus. Contrary to current slice-to-volume registration (SVR) methods, requiring an inflexible anatomical enclosure of a single investigated organ, the proposed patch-to-volume reconstruction (PVR) approach is able to reconstruct a large field of view of non-rigidly deforming structures. It relaxes rigid motion assumptions by introducing a specific amount of redundant information that is exploited with parallelized patchwise optimization, super-resolution, and automatic outlier rejection. We further describe and provide an efficient parallel implementation of PVR allowing its execution within reasonable time on commercially available graphics processing units, enabling its use in the clinical practice. We evaluate PVR's computational overhead compared with standard methods and observe improved reconstruction accuracy in the presence of affine motion artifacts compared with conventional SVR in synthetic experiments. Furthermore, we have evaluated our method qualitatively and quantitatively on real fetal MRI data subject to maternal breathing and sudden fetal movements. We evaluate peak-signal-to-noise ratio, structural similarity index, and cross correlation with respect to the originally acquired data and provide a method for visual inspection of reconstruction uncertainty. We further evaluate the distance error for selected anatomical landmarks in the fetal head, as well as calculating the mean and maximum displacements resulting from automatic non-rigid registration to a motion-free ground truth image. These experiments demonstrate a successful application of PVR motion compensation to the whole fetal body, uterus, and placenta.
CRIS-Team Scopus Importer:2021-08-17
Keywords: fetal magnetic resonance imaging; GPU acceleration; image reconstruction; Motion correction; super-resolution

RAMBO 2018 Preface - Kainz, Bernhard; Bhatia, Kanwal; Vercauteren, Tom; Oktay, Ozan (Image Analysis for Moving Organ, Breast, and Thoracic Images - Third International Workshop, RAMBO 2018, Fourth International Workshop, BIA 2018, and First International Workshop, TIA 2018, Held in Conjunction with MICCAI 2018, Proceedings, 2018)
CRIS-Team Scopus Importer:2021-08-17

RATCHET: Medical Transformer for Chest X-ray Diagnosis and Reporting - Hou, Benjamin; Kaissis, Georgios; Summers, Ronald; Kainz, Bernhard (Springer Science and Business Media Deutschland GmbH, 2021)
Chest radiographs are one of the most common diagnostic modalities in clinical routine. It can be done cheaply, requires minimal equipment, and the image can be diagnosed by every radiologists. However, the number of chest radiographs obtained on a daily basis can easily overwhelm the available clinical capacities. We propose RATCHET: RAdiological Text Captioning for Human Examined Thoraces. RATCHET is a CNN-RNN-based medical transformer that is trained end-to-end. It is capable of extracting image features from chest radiographs, and generates medically accurate text reports that fit seamlessly into clinical work flows. The model is evaluated for its natural language generation ability using common metrics from NLP literature, as well as its medically accuracy through a surrogate report classification task.

Ray Casting of Multiple Volumetric Datasets with Polyhedral Boundaries on Manycore GPUs - Kainz, Bernhard; Grabner, Markus; Bornik, Alexander; Hauswiesner, Stefan; Muehl, Judith; Schmalstieg, Dieter (Acm Transactions on Graphics, 2009)
We present a new GPU-based rendering system for ray casting of multiple volumes. Our approach supports a large number of volumes, complex translucent and concave polyhedral objects as well as CSG intersections of volumes and geometry in any combination. The system (including the rasterization stage) is implemented entirely in CUDA, which allows full control of the memory hierarchy, in particular access to high bandwidth and low latency shared memory. High depth complexity, which is problematic for conventional approaches based on depth peeling, can be handled successfully. As far as we know, our approach is the first framework for multivolume rendering which provides interactive frame rates when concurrently rendering more than 50 arbitrarily overlapping volumes on current graphics hardware. © 2009, ACM. All rights reserved.
CRIS-Team Scopus Importer:2021-08-17

Ray prioritization using stylization and visual saliency - Steinberger, Markus; Kainz, Bernhard; Hauswiesner, Stefan; Khlebnikov, Rostislav; Kalkofen, Denis; Schmalstieg, Dieter (Computers & Graphics, 2012)
This paper presents a new method to control scene sampling in complex ray-based rendering environments. It proposes to constrain image sampling density with a combination of object features, which are known to be well perceived by the human visual system, and image space saliency, which captures effects that are not based on the objects geometry. The presented method uses Non-Photorealistic Rendering techniques for the object space feature evaluation and combines the image space saliency calculations with image warping to infer quality hints from previously generated frames. In order to map different feature types to sampling densities, we also present an evaluation of the object space and image space features impact on the resulting image quality. In addition, we present an efficient, adaptively aligned fractal pattern that is used to reconstruct the image from sparse sampling data. Furthermore, this paper presents an algorithm which uses our method in order to guarantee a desired minimal frame rate. Our scheduling algorithm maximizes the utilization of each given time slice by rendering features in the order of visual importance values until a time constraint is reached. We demonstrate how our method can be used to boost or stabilize the rendering time in complex ray-based image generation consisting of geometric as well as volumetric data. © 2012 Elsevier Ltd.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Photorealistic rendering; Ray-casting; Ray-tracing; Visual saliency; Volume rendering

Realistic Data Enrichment for Robust Image Segmentation in Histopathology - Cechnicka, Sarah; Ball, James; Reynaud, Hadrien; Arthurs, Callum; Roufosse, Candice; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2024)
Poor performance of quantitative analysis in histopathological Whole Slide Images (WSI) has been a significant obstacle in clinical practice. Annotating large-scale WSIs manually is a demanding and time-consuming task, unlikely to yield the expected results when used for fully supervised learning systems. Rarely observed disease patterns and large differences in object scales are difficult to model through conventional patient intake. Prior methods either fall back to direct disease classification, which only requires learning a few factors per image, or report on average image segmentation performance, which is highly biased towards majority observations. Geometric image augmentation is commonly used to improve robustness for average case predictions and to enrich limited datasets. So far no method provided sampling of a realistic posterior distribution to improve stability, e.g. for the segmentation of imbalanced objects within images. Therefore, we propose a new approach, based on diffusion models, which can enrich an imbalanced dataset with plausible examples from underrepresented groups by conditioning on segmentation maps. Our method can simply expand limited clinical datasets making them suitable to train machine learning pipelines, and provides an interpretable and human-controllable way of generating histopathology images that are indistinguishable from real ones to human experts. We validate our findings on two datasets, one from the public domain and one from a Kidney Transplant study. 1 (The source code and trained models will be publicly available at the time of the conference, on huggingface and github. )
CRIS-Team Scopus Importer:2023-11-17

Realtime 3D graphics programming using the Quake3 engine - Kainz, Bernhard (Proceeding CGEMS: Computer graphics educational materials source, 2008)

Real-Time Prediction of Segmentation Quality - Robinson, Robert; Oktay, Ozan; Bai, Wenjia; Valindria, Vanya V.; Sanghvi, Mihir M.; Aung, Nay; Paiva, Jose M.; Zemrak, Filip; Fung, Kenneth; Lukaschuk, Elena; Lee, Aaron M.; Carapella, Valentina; et al. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2018)
Recent advances in deep learning based image segmentation methods have enabled real-time performance with human-level accuracy. However, occasionally even the best method fails due to low image quality, artifacts or unexpected behaviour of black box algorithms. Being able to predict segmentation quality in the absence of ground truth is of paramount importance in clinical practice, but also in large-scale studies to avoid the inclusion of invalid data in subsequent analysis. In this work, we propose two approaches of real-time automated quality control for cardiovascular MR segmentations using deep learning. First, we train a neural network on 12,880 samples to predict Dice Similarity Coefficients (DSC) on a per-case basis. We report a mean average error (MAE) of 0.03 on 1,610 test samples and 97% binary classification accuracy for separating low and high quality segmentations. Secondly, in the scenario where no manually annotated data is available, we train a network to predict DSC scores from estimated quality obtained via a reverse testing strategy. We report an MAE = 0.14 and 91% binary classification accuracy for this case. Predictions are obtained in real-time which, when combined with real-time segmentation methods, enables instant feedback on whether an acquired scan is analysable while the patient is still in the scanner. This further enables new applications of optimising image acquisition towards best possible analysis results.
CRIS-Team Scopus Importer:2021-08-17

Real-time standard scan plane detection and localisation in fetal ultrasound using fully convolutional neural networks - Baumgartner, Christian F.; Kamnitsas, Konstantinos; Matthew, Jacqueline; Smith, Sandra; Kainz, Bernhard; Rueckert, Daniel (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2016)
Fetal mid-pregnancy scans are typically carried out according to fixed protocols. Accurate detection of abnormalities and correct biometric measurements hinge on the correct acquisition of clearly defined standard scan planes. Locating these standard planes requires a high level of expertise. However,there is a worldwide shortage of expert sonographers. In this paper,we consider a fully automated system based on convolutional neural networks which can detect twelve standard scan planes as defined by the UK fetal abnormality screening programme. The network design allows real-time inference and can be naturally extended to provide an approximate localisation of the fetal anatomy in the image. Such a framework can be used to automate or assist with scan plane selection,or for the retrospective retrieval of scan planes from recorded videos. The method is evaluated on a large database of 1003 volunteer mid-pregnancy scans. We show that standard planes acquired in a clinical scenario are robustly detected with a precision and recall of 69% and 80%,which is superior to the current state-of-the-art. Furthermore,we show that it can retrospectively retrieve correct scan planes with an accuracy of 71% for cardiac views and 81% for non-cardiac views.
CRIS-Team Scopus Importer:2021-08-17

Remote Expert Deep Venous Thrombosis Triaging of Novice-User Compression Sonography with Artificial Intelligence Guidance - Avgerinos, Efthymios; Oppenheimer, Jonas; Al-Noor, Fouad; Karimaghaei, Raham; Adler, Andrea; Singöhl, Stephan; Kainz, Bernhard; Mandegaran, Ramin; Heinrich, Mattias P.; Spiliopoulos, Stavros; Geroulakos, George; Mischkewitz, Sven (Journal of Vascular Surgery: Venous and Lymphatic Disorders, 2023)
Objective Compression ultrasonography of the leg is established for triaging proximal lower extremity deep venous thrombosis (DVT) when performed by a certified operator. It can be associated with long waiting times and high resource utilization. AutoDVT, a machine-learning software, provides a tool for nonspecialists in acquiring appropriate two- or three-point compression sequences. These can be reviewed by an expert to triage patients. The purpose of this study was to test image acquisition and remote triaging in a clinical setting. Methods Patients with a suspected DVT were recruited at two tertiary centers. Enrolled patients underwent an AutoDVT scan by a non-ultrasound-qualified professional before the standard duplex scan. A handheld device with proprietary image-acquisition software to perform a two-point compression examination (groin and popliteal fossa) was used. Images collected by the software were uploaded to a cloud-based platform for blind review by four external, qualified physicians. All reviewers, based on these images, rated the overall quality of all sequences on the ACEP scale (score 1-5, ≥3 defined as sufficient diagnostic-quality) (Figure), and if sufficient chose DVT versus No DVT. Sensitivity and specificity were calculated based on the sufficient quality scans (score ≥3). To simulate a triaging clinical algorithm, an analysis was performed merging the low quality, the incomplete and the incompressible vein scans as high risk for proximal DVT. These were then compared with the standard duplex diagnosis. Results Seventy-three patients (aged 64.2 ± 17.9 years; BMI, 27.7 ± 6.2; 44% female) were eligible for inclusion and scanned by three nurses. Three patients were excluded from further analysis due to incomplete scans. Of the completed scans, 54 (77%) were of diagnostic quality with an average American College for Emergency Physicians score of 3.40 ± 0.86, 12 of which were interpreted as positive for proximal DVT; 6 of them were confirmed with standard duplex and 6 were false positives (sensitivity 100%, specificity 87.50%). The remaining 42 scans were all negative for proximal DVT; 4 isolated calf DVTs were identified in standard duplex. When simulating a triaging algorithm, 31 of 73 patients (42%) would be triaged as high risk and 8 would be confirmed as positive for proximal DVT. Of 42 patients triaged as low risk, all were negative for proximal DVT on standard duplex, thus including AutoDVT in a proximal-DVT triaging algorithm could save 58% of standard duplex scans. Conclusions Machine learning software was able to aid nonexperts in acquiring valid ultrasound images of venous compressions and allowed safe and efficient remote triaging. Such a triaging strategy allows faster diagnosis and treatment of high-risk patients and can spare the need and cost of multiple unnecessary duplex scan

Remote Expert DVT Triaging of Novice-User Compression Sonography with AI-Guidance - Oppenheimer, Jonas; Mandegaran, Ramin; Staabs, Finja; Adler, Andrea; Singöhl, Stephan; Kainz, Bernhard; Heinrich, Matthias; Geroulakos, George; Spiliopoulos, Stavros; Avgerinos, Efthymios (Annals of Vascular Surgery, 2023)
Background: Compression ultrasonography of the leg is established for triaging proximal lower extremity deep vein thrombosis (DVT). AutoDVT, a machine-learning software, provides a tool for nonspecialists in acquiring compression sequences to be reviewed by an expert for patient triage. The purpose of this study was to test image acquisition and remote triaging in a clinical setting. Methods: Patients with a suspected DVT were recruited at 2 centers in Germany and Greece. Enrolled patients underwent an artificial intelligence–guided two-point compression examination by a nonspecialist using a handheld ultrasound device prior to a standard scan. Images collected by the software were uploaded for blind review by 5 qualified physicians. All reviewers rated the quality of all sequences on the American College of Emergency Physicians (ACEP) image quality scale (score 1–5, ≥ 3 defined as adequate imaging quality) and for an ACEP score ≥3, chose “Compressible”, “Incompressible”, or “Other”. Sensitivity and specificity were calculated for adequate quality scans with an assessment as “Compressible” or “Incompressible”. We define this group as diagnostic quality. To simulate a triaging clinical algorithm, a post hoc analysis was performed merging the “incomplete”, the “low quality”, and the “Incompressible” into a high-risk group for proximal DVT. Results: Seventy-three patients (average age 64.2 years, 44% females) were eligible for inclusion and scanned by 3 nonultrasound-qualified healthcare professionals. Three patients were excluded from further analysis due to incomplete scans. Sixty two of 70 (88.57%) of the completed scans were judged to be of adequate image quality with an average ACEP score of 3.35. Forty seven of 62 adequate AutoDVT scans were assessed as diagnostic quality, of which 8 were interpreted as positive for proximal DVT by the reviewers resulting in a sensitivity of 100% and specificity of 95.12%. When simulating a triaging algorithm, 34/73 (46.58%) of patients would be triaged as high risk and 8 would be confirmed as positive for proximal DVT (6 in the diagnostic and 2 in the low-quality cohort). Of 39/73 patients triaged as low risk, all were negative for proximal DVT in standard duplex; thus, this triaging algorithm could potentially save 53.42% of standard duplex scans. Conclusions: Machine learning software was able to aid nonexperts in acquiring valid ultrasound images of venous compressions and allowed remote triaging. This strategy allows faster diagnosis and treatment of high-risk patients and can spare the need for multiple unnecessary duplex scans, the vast majority being negative.
CRIS-Team Scopus Importer:2023-11-10

Representation disentanglement for multi-task learning with application to fetal ultrasound - Meng, Qingjie; Pawlowski, Nick; Rueckert, Daniel; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2019)
One of the biggest challenges for deep learning algorithms in medical image analysis is the indiscriminate mixing of image properties, e.g. artifacts and anatomy. These entangled image properties lead to a semantically redundant feature encoding for the relevant task and thus lead to poor generalization of deep learning algorithms. In this paper we propose a novel representation disentanglement method to extract semantically meaningful and generalizable features for different tasks within a multi-task learning framework. Deep neural networks are utilized to ensure that the encoded features are maximally informative with respect to relevant tasks, while an adversarial regularization encourages these features to be disentangled and minimally informative about irrelevant tasks. We aim to use the disentangled representations to generalize the applicability of deep neural networks. We demonstrate the advantages of the proposed method on synthetic data as well as fetal ultrasound images. Our experiments illustrate that our method is capable of learning disentangled internal representations. It outperforms baseline methods in multiple tasks, especially on images with new properties, e.g. previously unseen artifacts in fetal ultrasound.
CRIS-Team Scopus Importer:2021-08-17

Resource-Efficient Medical Image Analysis with Self-adapting Forward-Forward Networks - Müller, Johanna; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2025)
We introduce a fast Self-adapting Forward-Forward Network (SaFF-Net) for medical imaging analysis, mitigating power consumption and resource limitations, which currently primarily stem from the prevalent reliance on back-propagation for model training and fine-tuning. Building upon the recently proposed Forward-Forward Algorithm (FFA), we introduce the Convolutional Forward-Forward Algorithm (CFFA), a parameter-efficient reformulation that is suitable for advanced image analysis and overcomes the speed and generalisation constraints of the original FFA. To address hyper-parameter sensitivity of FFAs we are also introducing a self-adapting framework SaFF-Net fine-tuning parameters during warmup and training in parallel. Our approach enables more effective model training and eliminates the previously essential requirement for an arbitrarily chosen Goodness function in FFA. We evaluate our approach on several benchmarking datasets in comparison with standard Back-Propagation (BP) neural networks showing that FFA-based networks with notably fewer parameters and function evaluations can compete with standard models, especially, in one-shot scenarios and large batch sizes.
CRIS-Team Scopus Importer:2024-11-15
Keywords: Efficiency; Forward-Forward Algorithm; Goodness Function

RE: Wang et al. Diagnosis of fetal total anomalous pulmonary venous connection based on the post-left atrium space ratio using artificial intelligence - Day, Thomas George; Kainz, Bernhard; Razavi, Reza; Simpson, John (Prenatal Diagnosis, 2023)
CRIS-Team WoS Importer:2023-02-24

RFA-cut: Semi-automatic segmentation of radiofrequency ablation zones with and without needles via optimal s-t-cuts - Egger, Jan; Busse, Harald; Brandmaier, Philipp; Seider, Daniel; Gawlitza, Matthias; Strocka, Steffen; Voglreiter, Philip; Dokter, Mark; Hofmann, Michael; Kainz, Bernhard; Chen, Xiaojun; Hann, Alexander; et al. (Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, 2015)
In this contribution, we present a semi-automatic segmentation algorithm for radiofrequency ablation (RFA) zones via optimal s-t-cuts. Our interactive graph-based approach builds upon a polyhedron to construct the graph and was specifically designed for computed tomography (CT) acquisitions from patients that had RFA treatments of Hepatocellular Carcinomas (HCC). For evaluation, we used twelve post-interventional CT datasets from the clinical routine and as evaluation metric we utilized the Dice Similarity Coefficient (DSC), which is commonly accepted for judging computer aided medical segmentation tasks. Compared with pure manual slice-by-slice expert segmentations from interventional radiologists, we were able to achieve a DSC of about eighty percent, which is sufficient for our clinical needs. Moreover, our approach was able to handle images containing (DSC=75.9%) and not containing (78.1%) the RFA needles still in place. Additionally, we found no statistically significant difference (p<0.423) between the segmentation results of the subgroups for a Mann-Whitney test. Finally, to the best of our knowledge, this is the first time a segmentation approach for CT scans including the RFA needles is reported and we show why another state-of-the-art segmentation method fails for these cases. Intraoperative scans including an RFA probe are very critical in the clinical practice and need a very careful segmentation and inspection to avoid under-treatment, which may result in tumor recurrence (up to 40%). If the decision can be made during the intervention, an additional ablation can be performed without removing the entire needle. This decreases the patient stress and associated risks and costs of a separate intervention at a later date. Ultimately, the segmented ablation zone containing the RFA needle can be used for a precise ablation simulation as the real needle position is known.
CRIS-Team Scopus Importer:2021-08-17

Robust Segmentation via Topology Violation Detection and Feature Synthesis - Li, Liu; Ma, Qiang; Ouyang, Cheng; Li, Zeju; Meng, Qingjie; Zhang, Weitong; Qiao, Mengyun; Kyriakopoulou, Vanessa; Hajnal, Joseph V.; Rueckert, Daniel; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2023)
Despite recent progress of deep learning-based medical image segmentation techniques, fully automatic results often fail to meet clinically acceptable accuracy, especially when topological constraints should be observed, e.g., closed surfaces. Although modern image segmentation methods show promising results when evaluated based on conventional metrics such as the Dice score or Intersection-over-Union, these metrics do not reflect the correctness of a segmentation in terms of a required topological genus. Existing approaches estimate and constrain the topological structure via persistent homology (PH). However, these methods are not computationally efficient as calculating PH is not differentiable. To overcome this problem, we propose a novel approach for topological constraints based on the multi-scale Euler Characteristic (EC). To mitigate computational complexity, we propose a fast formulation for the EC that can inform the learning process of arbitrary segmentation networks via topological violation maps. Topological performance is further facilitated through a corrective convolutional network block. Our experiments on two datasets show that our method can significantly improve topological correctness.
CRIS-Team Scopus Importer:2023-11-03

ScatterAlloc: Massively parallel dynamic memory allocation for the GPU - Steinberger, Markus; Kenzel, Michael; Kainz, Bernhard; Schmalstieg, Dieter (2012 Innovative Parallel Computing, InPar 2012, 2012)
In this paper, we analyze the special requirements of a dynamic memory allocator that is designed for massively parallel architectures such as Graphics Processing Units (GPUs). We show that traditional strategies, which work well on CPUs, are not well suited for the use on GPUs and present the thorough design of ScatterAlloc, which can efficiently deal with hundreds of requests in parallel. Our allocator greatly reduces collisions and congestion by scattering memory requests based on hashing. We analyze ScatterAlloc in terms of allocation speed, data access time and fragmentation, and compare it to current state-of-the-art allocators, including the one provided with the NVIDIA CUDA toolkit. Our results show, that ScatterAlloc clearly outperforms these other approaches, yielding speed-ups between 10 to 100. © 2012 IEEE.
CRIS-Team Scopus Importer:2021-08-17
Keywords: dynamic memory allocation; GPU; hashing; massively parallel

Self-supervised 3D Human Pose Estimation in Static Video via Neural Rendering - Schmidtke, Luca; Hou, Benjamin; Vlontzos, Athanasios; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2023)
Inferring 3D human pose from 2D images is a challenging and long-standing problem in the field of computer vision with many applications including motion capture, virtual reality, surveillance or gait analysis for sports and medicine. We present preliminary results for a method to estimate 3D pose from 2D video containing a single person and a static background without the need for any manual landmark annotations. We achieve this by formulating a simple yet effective self-supervision task: our model is required to reconstruct a random frame of a video given a frame from another timepoint and a rendered image of a transformed human shape template. Crucially for optimisation, our ray casting based rendering pipeline is fully differentiable, enabling end to end training solely based on the reconstruction task.
CRIS-Team Scopus Importer:2023-04-07
Keywords: 3D human pose estimation; 3D pose tracking; Motion capture; Self-supervised learning

Semi-supervised learning of fetal anatomy from ultrasound - Tan, Jeremy; Au, Anselm; Meng, Qingjie; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2019)
Semi-supervised learning methods have achieved excellent performance on standard benchmark datasets using very few labelled images. Anatomy classification in fetal 2D ultrasound is an ideal problem setting to test whether these results translate to non-ideal data. Our results indicate that inclusion of a challenging background class can be detrimental and that semi-supervised learning mostly benefits classes that are already distinct, sometimes at the expense of more similar classes.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Fetal ultrasound; Semi-supervised learning

Simplifying Medical Ultrasound - Kainz, Bernhard; Noble, J. Alison; Schnabel, Julia; Khanal, Bishesh; Müller, Johanna; Day, Thomas (Springer, 2023)
This book constitutes the proceedings of the 4th International Workshop on Advances in Simplifying Medical UltraSound, ASMUS 2023, held in conjunction with MICCAI 2023, the 26th International Conference on Medical Image Computing and Computer-Assisted Intervention. The conference took place in Vancouver, BC, Canada, on October 8, 2023. The 19 papers presented in this book were carefully reviewed and selected from 30 submissions. They were organized in topical sections as follows:​ advanced imaging, segmentation, and ultrasound techniques; predictive analysis, learning, and classification; multimodal imaging, reconstruction, and real-time applications; diagnostic enhancements and novel ultrasound innovation
Keywords: artificial intelligence; deep learning; engineering; image analysis; machine learning; Computer-assisted Diagnosis; Point-of-care ultrasound systems; Ultrasound quality and skills assessment; Deep Neural Networks; Assisting surgery and interventions

Simultaneous whole-liver water T1 and T2 mapping with isotropic resolution during free-breathing - Stelter, Jonathan; Weiss, Kilian; Steinhelfer, Lisa; Spieker, Veronika; Huaroc Moquillaza, Elizabeth; Zhang, Weitong; Makowski, Marcus R.; Schnabel, Julia A.; Kainz, Bernhard; Braren, Rickmer F.; Karampinos, Dimitrios C. (NMR in Biomedicine, 2024)
Purpose: To develop and validate a data acquisition scheme combined with a motion-resolved reconstruction and dictionary-matching-based parameter estimation to enable free-breathing isotropic resolution self-navigated whole-liver simultaneous water-specific (Formula presented.) ((Formula presented.)) and (Formula presented.) ((Formula presented.)) mapping for the characterization of diffuse and oncological liver diseases. Methods: The proposed data acquisition consists of a magnetization preparation pulse and a two-echo gradient echo readout with a radial stack-of-stars trajectory, repeated with different preparations to achieve different (Formula presented.) and (Formula presented.) contrasts in a fixed acquisition time of 6 min. Regularized reconstruction was performed using self-navigation to account for motion during the free-breathing acquisition, followed by water–fat separation. Bloch simulations of the sequence were applied to optimize the sequence timing for (Formula presented.) insensitivity at 3 T, to correct for relaxation-induced blurring, and to map (Formula presented.) and (Formula presented.) using a dictionary. The proposed method was validated on a water–fat phantom with varying relaxation properties and in 10 volunteers against imaging and spectroscopy reference values. The performance and robustness of the proposed method were evaluated in five patients with abdominal pathologies. Results: Simulations demonstrate good (Formula presented.) insensitivity of the proposed method in measuring (Formula presented.) and (Formula presented.) values. The proposed method produces co-registered (Formula presented.) and (Formula presented.) maps with a good agreement with reference methods (phantom: (Formula presented.); (Formula presented.)). The proposed (Formula presented.) and (Formula presented.) mapping exhibits good repeatability and can be robustly performed in patients with pathologies. Conclusions: The proposed method allows whole-liver (Formula presented.) and (Formula presented.) quantification with high accuracy at isotropic resolution in a fixed acquisition time during free-breathing.
CRIS-Team Scopus Importer:2024-09-13
Keywords: BIR-4 preparation; dictionary matching; radial stack-of-stars; relaxometry; water–fat separation

Softshell: Dynamic scheduling on GPUs - Steinberger, Markus; Kainz, Bernhard; Kerbl, Bernhard; Hauswiesner, Stefan; Kenzel, Michael; Schmalstieg, Dieter (Acm Transactions on Graphics, 2012)
In this paper we present Softshell, a novel execution model for devices composed of multiple processing cores operating in a single instruction, multiple data fashion, such as graphics processing units (GPUs). The Softshell model is intuitive and more flexible than the kernel-based adaption of the stream processing model, which is currently the dominant model for general purpose GPU computation. Using the Softshell model, algorithms with a relatively low local degree of parallelism can execute efficiently on massively parallel architectures. Softshell has the following distinct advantages: (1) work can be dynamically issued directly on the device, eliminating the need for synchronization with an external source, i.e., the CPU; (2) its three-tier dynamic scheduler supports arbitrary scheduling strategies, including dynamic priorities and real-time scheduling; and (3) the user can influence, pause, and cancel work already submitted for parallel execution. The Softshell processing model thus brings capabilities to GPU architectures that were previously only known from operating-system designs and reserved for CPU programming. As a proof of our claims, we present a publicly available implementation of the Softshell processing model realized on top of CUDA. The benchmarks of this implementation demonstrate that our processing model is easy to use and also performs substantially better than the state-of-the-art kernel-based processing model for problems that have been difficult to parallelize in the past. © 2012 ACM.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Dynamic parallelism; GPU; Persistent threads; Priority scheduling; Priority work queue; Realtime scheduling

SonoNet: Real-Time Detection and Localisation of Fetal Standard Scan Planes in Freehand Ultrasound - Baumgartner, Christian F.; Kamnitsas, Konstantinos; Matthew, Jacqueline; Fletcher, Tara P.; Smith, Sandra; Koch, Lisa M.; Kainz, Bernhard; Rueckert, Daniel (IEEE Transactions on Medical Imaging, 2017)
Identifying and interpreting fetal standard scan planes during 2-D ultrasound mid-pregnancy examinations are highly complex tasks, which require years of training. Apart from guiding the probe to the correct location, it can be equally difficult for a non-expert to identify relevant structures within the image. Automatic image processing can provide tools to help experienced as well as inexperienced operators with these tasks. In this paper, we propose a novel method based on convolutional neural networks, which can automatically detect 13 fetal standard views in freehand 2-D ultrasound data as well as provide a localization of the fetal structures via a bounding box. An important contribution is that the network learns to localize the target anatomy using weak supervision based on image-level labels only. The network architecture is designed to operate in real-time while providing optimal output for the localization task. We present results for real-time annotation, retrospective frame retrieval from saved videos, and localization on a very large and challenging dataset consisting of images and video recordings of full clinical anomaly screenings. We found that the proposed method achieved an average F1-score of 0.798 in a realistic classification experiment modeling real-time detection, and obtained a 90.09% accuracy for retrospective frame retrieval. Moreover, an accuracy of 77.8% was achieved on the localization task.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Convolutional neural networks; fetal ultrasound; standard plane detection; weakly supervised localisation

Standard plane detection in 3D fetal ultrasound using an iterative transformation network - Li, Yuanwei; Khanal, Bishesh; Hou, Benjamin; Alansary, Amir; Cerrolaza, Juan J.; Sinclair, Matthew; Matthew, Jacqueline; Gupta, Chandni; Knight, Caroline; Kainz, Bernhard; Rueckert, Daniel (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2018)
Standard scan plane detection in fetal brain ultrasound (US) forms a crucial step in the assessment of fetal development. In clinical settings, this is done by manually manoeuvring a 2D probe to the desired scan plane. With the advent of 3D US, the entire fetal brain volume containing these standard planes can be easily acquired. However, manual standard plane identification in 3D volume is labour-intensive and requires expert knowledge of fetal anatomy. We propose a new Iterative Transformation Network (ITN) for the automatic detection of standard planes in 3D volumes. ITN uses a convolutional neural network to learn the relationship between a 2D plane image and the transformation parameters required to move that plane towards the location/orientation of the standard plane in the 3D volume. During inference, the current plane image is passed iteratively to the network until it converges to the standard plane location. We explore the effect of using different transformation representations as regression outputs of ITN. Under a multi-task learning framework, we introduce additional classification probability outputs to the network to act as confidence measures for the regressed transformation parameters in order to further improve the localisation accuracy. When evaluated on 72 US volumes of fetal brain, our method achieves an error of 3.83 mm/12.7° and 3.80 mm/12.6° for the transventricular and transcerebellar planes respectively and takes 0.46 s per plane.
CRIS-Team Scopus Importer:2021-08-17

Stresses and strains on the human fetal skeleton during development - Verbruggen, Stefaan W.; Kainz, Bernhard; Shelmerdine, Susan C.; Hajnal, Joseph V.; Rutherford, Mary A.; Arthurs, Owen J.; Phillips, Andrew T. M.; Nowlan, Niamh C. (Journal of the Royal Society Interface, 2018)
Mechanical forces generated by fetal kicks and movements result in stimulation of the fetal skeleton in the form of stress and strain. This stimulation is known to be critical for prenatal musculoskeletal development; indeed, abnormal or absent movements have been implicated in multiple congenital disorders. However, the mechanical stress and strain experienced by the developing human skeleton in utero have never before been characterized. Here, we quantify the biomechanics of fetal movements during the second half of gestation by modelling fetal movements captured using novel cine-magnetic resonance imaging technology. By tracking these movements, quantifying fetal kick and muscle forces, and applying them to three-dimensional geometries of the fetal skeleton, we test the hypothesis that stress and strain change over ontogeny. We find that fetal kick force increases significantly from 20 to 30 weeks’ gestation, before decreasing towards term. However, stress and strain in the fetal skeleton rises significantly over the latter half of gestation. This increasing trend with gestational age is important because changes in fetal movement patterns in late pregnancy have been linked to poor fetal outcomes and musculoskeletal malformations. This research represents the first quantification of kick force and mechanical stress and strain due to fetal movements in the human skeleton in utero, thus advancing our understanding of the biomechanical environment of the uterus. Further, by revealing a potential link between fetal biomechanics and skeletal malformations, our work will stimulate future research in tissue engineering and mechanobiology.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Biomechanical stimuli; Cine-MRI; Finite element analysis; Joint biomechanics; Musculo-skeletal development

Style-Extracting Diffusion Models for Semi-supervised Histopathology Segmentation - Öttl, Mathias; Wilm, Frauke; Steenpaß, Jana; Qiu, Jingna; Rübner, Matthias; Hartmann, Arndt; Beckmann, Matthias; Fasching, Peter; Maier, Andreas; Erber, Ramona; Kainz, Bernhard; Breininger, Katharina (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2025)
Deep learning-based image generation has seen significant advancements with diffusion models, notably improving the quality of generated images. Despite these developments, generating images with unseen characteristics beneficial for downstream tasks has received limited attention. To bridge this gap, we propose Style-Extracting Diffusion Models, featuring two conditioning mechanisms. Specifically, we utilize 1) a style conditioning mechanism which allows to inject style information of previously unseen images during image generation and 2) a content conditioning which can be targeted to a downstream task, e.g., layout for segmentation. We introduce a trainable style encoder to extract style information from images, and an aggregation block that merges style information from multiple style inputs. This architecture enables the generation of images with unseen styles in a zero-shot manner, by leveraging styles from unseen images, resulting in more diverse generations. In this work, we use the image layout as target condition and first show the capability of our method on a natural image dataset as a proof-of-concept. We further demonstrate its versatility in histopathology, where we combine prior knowledge about tissue composition and unannotated data to create diverse synthetic images with known layouts. This allows us to generate additional synthetic data to train a segmentation network in a semi-supervised fashion. We verify the added value of the generated images by showing improved segmentation results and lower performance variability between patients when synthetic images are included during segmentation training. The code of the method is publicly available at https://github.com/OettlM/STEDM.
Created from Fastlane, Scopus look-up

Stylization-based ray prioritization for guaranteed frame rates - Kainz, Bernhard; Steinberger, Markus; Hauswiesner, Stefan; Khlebnikov, Rostislav; Schmalstieg, Dieter (NPAR Symposium on Non-Photorealistic Animation and Rendering, 2011)
This paper presents a new method to control graceful scene degradation in complex ray-based rendering environments. It proposes to constrain the image sampling density with object features, which are known to support the comprehension of the three-dimensional shape. The presented method uses Non-Photorealistic Rendering (NPR) techniques to extract features such as silhouettes, suggestive contours, suggestive highlights, ridges and valleys. To map different feature types to sampling densities, we also present an evaluation of the features impact on the resulting image quality. To reconstruct the image from sparse sampling data, we use linear interpolation on an adaptively aligned fractal pattern. With this technique, we are able to present an algorithm that guarantees a desired minimal frame rate without much loss of image quality. Our scheduling algorithm maximizes the use of each given time slice by rendering features in order of their corresponding importance values until a time constraint is reached. We demonstrate how our method can be used to boost and guarantee the rendering time in complex raybased environments consisting of geometric as well as volumetric data. © 2011 ACM.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Guaranteed frame rates; Line features; Ray based image generation

Surface Agnostic Metrics for Cortical Volume Segmentation and Regression - Budd, Samuel; Patkee, Prachi; Baburamani, Ana; Rutherford, Mary; Robinson, Emma C.; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2020)
The cerebral cortex performs higher-order brain functions and is thus implicated in a range of cognitive disorders. Current analysis of cortical variation is typically performed by fitting surface mesh models to inner and outer cortical boundaries and investigating metrics such as surface area and cortical curvature or thickness. These, however, take a long time to run, and are sensitive to motion and image and surface resolution, which can prohibit their use in clinical settings. In this paper, we instead propose a machine learning solution, training a novel architecture to predict cortical thickness and curvature metrics from T2 MRI images, while additionally returning metrics of prediction uncertainty. Our proposed model is tested on a clinical cohort (Down Syndrome) for which surface-based modelling often fails. Results suggest that deep convolutional neural networks are a viable option to predict cortical metrics across a range of brain development stages and pathologies.
CRIS-Team Scopus Importer:2021-08-17

The future of volume graphics in medical virtual reality - Muehl, Judith; Kainz, Bernhard; Bornik, Alexander; Grabner, Markus; Hauswiesner, Stefan; Schmalstieg, Dieter (IFMBE Proceedings, 2009)
A recent trend in medical virtual reality is to include information from multiple sources, especially about physiology, into one model and one single visualization. Computer graphics must therefore deal with a huge amount of information in real time. The latest developments in computer graphics hardware allow not only implementing direct volume rendering on the graphics processing unit (GPU). The emerging compute languages enable us to address volume rendering problems of arbitrary complexity without being limited to formulating visualization techniques in an awkward fashion to match the GPU execution model. Utilizing the arising new possibilities we meet next generation's demands in medical visualization.
CRIS-Team Scopus Importer:2021-08-17
Keywords: CUDA; Medical virtual reality; Raycasting; Visualization; Volume rendering

Three-dimensional visualisation of the fetal heart using prenatal MRI with motion-corrected slice-volume registration: a prospective, single-centre cohort study - Lloyd, David F.A.; Pushparajah, Kuberan; Simpson, John M.; van Amerom, Joshua F.P.; van Poppel, Milou P.M.; Schulz, Alexander; Kainz, Bernhard; Deprez, Maria; Lohezic, Maelene; Allsop, Joanna; Mathur, Sujeev; Bellsham-Revell, Hannah; et al. (Lancet, 2019)
Background: Two-dimensional (2D) ultrasound echocardiography is the primary technique used to diagnose congenital heart disease before birth. There is, however, a longstanding need for a reliable form of secondary imaging, particularly in cases when more detailed three-dimensional (3D) vascular imaging is required, or when ultrasound windows are of poor diagnostic quality. Fetal MRI, which is well established for other organ systems, is highly susceptible to fetal movement, particularly for 3D imaging. The objective of this study was to investigate the combination of prenatal MRI with novel, motion-corrected 3D image registration software, as an adjunct to fetal echocardiography in the diagnosis of congenital heart disease. Methods: Pregnant women carrying a fetus with known or suspected congenital heart disease were recruited via a tertiary fetal cardiology unit. After initial validation experiments to assess the general reliability of the approach, MRI data were acquired in 85 consecutive fetuses, as overlapping stacks of 2D images. These images were then processed with a bespoke open-source reconstruction algorithm to produce a super-resolution 3D volume of the fetal thorax. These datasets were assessed with measurement comparison with paired 2D ultrasound, structured anatomical assessment of the 2D and 3D data, and contemporaneous, archived clinical fetal MRI reports, which were compared with postnatal findings after delivery. Findings: Between Oct 8, 2015, and June 30, 2017, 101 patients were referred for MRI, of whom 85 were eligible and had fetal MRI. The mean gestational age at the time of MRI was 32 weeks (range 24–36). High-resolution (0·50–0·75 mm isotropic) 3D datasets of the fetal thorax were generated in all 85 cases. Vascular measurements showed good overall agreement with 2D echocardiography in 51 cases with paired data (intra-class correlation coefficient 0·78, 95% CI 0·68–0·84), with fetal vascular structures more effectively visualised with 3D MRI than with uncorrected 2D MRI (657 [97%] of 680 anatomical areas identified vs 358 [53%] of 680 areas; p<0·0001). When a structure of interest was visualised in both 2D and 3D data (n=358), observers gave a higher diagnostic quality score for 3D data in 321 (90%) of cases, with 37 (10%) scores tied with 2D data, and no lower scores than for 2D data (Wilcoxon signed rank test p<0·0001). Additional anatomical features were described in ten cases, of which all were confirmed postnatally. Interpretation: Standard fetal MRI with open-source image processing software is a reliable method of generating high-resolution 3D imaging of the fetal vasculature. The 3D volumes produced show good spatial agreement with ultrasound, and significantly improved visualisation and diagnostic quality compared with source 2D MRI data. This freely available combination requires minimal infrastructure, and provides safe, powerful, and highly complementary imaging of the fetal cardiovascular system. Funding: Wellcome Trust/EPSRC Centre for Medical Engineering, National Institute for Health Research.
CRIS-Team Scopus Importer:2021-08-17

Towards automated extraction of 2D standard fetal head planes from 3D ultrasound acquisitions: A clinical evaluation and quality assessment comparison - Skelton, Emily; Matthew, J.; Li, Y.; Khanal, B.; Martinez, J. J. Cerrolaza; Toussaint, N.; Gupta, C.; Knight, C.; Kainz, Bernhard; Hajnal, J. V.; Rutherford, M. (Radiography, 2021)
Introduction: Clinical evaluation of deep learning (DL) tools is essential to compliment technical accuracy metrics. This study assessed the image quality of standard fetal head planes automatically-extracted from three-dimensional (3D) ultrasound fetal head volumes using a customised DL-algorithm. Methods: Two observers retrospectively reviewed standard fetal head planes against pre-defined image quality criteria. Forty-eight images (29 transventricular, 19 transcerebellar) were selected from 91 transabdominal fetal scans (mean gestational age = 26 completed weeks, range = 20+5–32+3 weeks). Each had two-dimensional (2D) manually-acquired (2D-MA), 3D operator-selected (3D-OS) and 3D-DL automatically-acquired (3D-DL) images. The proportion of adequate images from each plane and modality, and the number of inadequate images per plane was compared for each method. Inter and intra-observer agreement of overall image quality was calculated. Results: Sixty-seven percent of 3D-OS and 3D-DL transventricular planes were adequate quality. Forty-five percent of 3D-OS and 55% of 3D-DL transcerebellar planes were adequate. Seventy-one percent of 3D-OS and 86% of 3D-DL transventricular planes failed with poor visualisation of intra-cranial structures. Eighty-six percent of 3D-OS and 80% of 3D-DL transcerebellar planes failed due to inadequate visualisation of cerebellar hemispheres. Image quality was significantly different between 2D and 3D, however, no significant difference between 3D-modalities was demonstrated (p < 0.005). Inter-observer agreement of transventricular plane adequacy was moderate for both 3D-modalities, and weak for transcerebellar planes. Conclusion: The 3D-DL algorithm can automatically extract standard fetal head planes from 3D-head volumes of comparable quality to operator-selected planes. Image quality in 3D is inferior to corresponding 2D planes, likely due to limitations with 3D-technology and acquisition technique. Implications for practice: Automated image extraction of standard planes from US-volumes could facilitate use of 3DUS in clinical practice, however image quality is dependent on the volume acquisition technique.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Clinical evaluation; Fetal imaging; Quality assessment; Ultrasound

Trade-Offs in Fine-Tuned Diffusion Models between Accuracy and Interpretability - Dombrowski, Mischa; Reynaud, Hadrien; Müller, Johanna; Baugh, Matthew; Kainz, Bernhard (AAAI-24 Special Track Safe, Robust and Responsible AI Track, 2024)
Recent advancements in diffusion models have significantly impacted the trajectory of generative machine learning re-search, with many adopting the strategy of fine-tuning pre-trained models using domain-specific text-to-image datasets. Notably, this method has been readily employed for medical applications, such as X-ray image synthesis, leveraging the plethora of associated radiology reports. Yet, a prevailing concern is the lack of assurance on whether these models genuinely comprehend their generated content. With the evolution of text conditional image generation, these models have grown potent enough to facilitate object localization scrutiny. Our research underscores this advancement in the critical realm of medical imaging, emphasizing the crucial role of interpretability. We further unravel a consequential trade-off between image fidelity – as gauged by conventional metrics – and model interpretability in generative diffusion models. Specifically, the adoption of learnable text encoders when fine-tuning results in diminished interpretability. Our in-depth exploration uncovers the underlying factors responsible for this divergence. Consequently, we present a set of design principles for the development of truly interpretable generative models. Code is available at https://github.com/MischaD/chest-distillation.<br

Ultrasound Video Summarization Using Deep Reinforcement Learning - Liu, Tianrui; Meng, Qingjie; Vlontzos, Athanasios; Tan, Jeremy; Rueckert, Daniel; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2020)
Video is an essential imaging modality for diagnostics, e.g. in ultrasound imaging, for endoscopy, or movement assessment. However, video hasn’t received a lot of attention in the medical image analysis community. In the clinical practice, it is challenging to utilise raw diagnostic video data efficiently as video data takes a long time to process, annotate or audit. In this paper we introduce a novel, fully automatic video summarization method that is tailored to the needs of medical video data. Our approach is framed as reinforcement learning problem and produces agents focusing on the preservation of important diagnostic information. We evaluate our method on videos from fetal ultrasound screening, where commonly only a small amount of the recorded data is used diagnostically. We show that our method is superior to alternative video summarization methods and that it preserves essential information required by clinical diagnostic standards.
CRIS-Team Scopus Importer:2021-08-17
Keywords: Reinforcement learning; Ultrasound diagnostic; Video summarization

Ultrasound Video Transformers for Cardiac Ejection Fraction Estimation - Reynaud, Hadrien; Vlontzos, Athanasios; Hou, Benjamin; Beqiri, Arian; Leeson, Paul; Kainz, Bernhard (Springer Science and Business Media Deutschland GmbH, 2021)
Cardiac ultrasound imaging is used to diagnose various heart diseases. Common analysis pipelines involve manual processing of the video frames by expert clinicians. This suffers from intra- and inter-observer variability. We propose a novel approach to ultrasound video analysis using a transformer architecture based on a Residual Auto-Encoder Network and a BERT model adapted for token classification. This enables videos of any length to be processed. We apply our model to the task of End-Systolic (ES) and End-Diastolic (ED) frame detection and the automated computation of the left ventricular ejection fraction. We achieve an average frame distance of 3.36 frames for the ES and 7.17 frames for the ED on videos of arbitrary length. Our end-to-end learnable approach can estimate the ejection fraction with a MAE of 5.95 and R2 of 0.52 in 0.15 s per video, showing that segmentation is not the only way to predict ejection fraction.
Keywords: Cardiac; Ultrasound; Transformers

Uncertainty-Aware Vision Transformers for Medical Image Analysis - Erick, Franciskus; Mina, Rezaei; Müller, Johanna; Kainz, Bernhard (Uncertainty for Safe Utilization of Machine Learning in Medical Imaging: 6th International Workshop, UNSURE 2024, Held in Conjunction with MICCAI 2024, Marrakesh, Morocco, October 10, 2024, Proceedings, 2025)

Understanding metric-related pitfalls in image analysis validation - Reinke, Annika; Tizabi, Minu D.; Baumgartner, Michael; Eisenmann, Matthias; Heckmann-Nötzel, Doreen; Kavur, A. Emre; Rädsch, Tim; Sudre, Carole H.; Acion, Laura; Antonelli, Michela; Arbel, Tal; Bakas, Spyridon; et al. (Nature Methods, 2024)
Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.
CRIS-Team Scopus Importer:2024-03-01

Universal Topology Refinement for Medical Image Segmentation with Polynomial Feature Synthesis - Li, Liu; Wang, Hanchun; Baugh, Matthew; Ma, Qiang; Zhang, Weitong; Ouyang, Cheng; Rueckert, Daniel; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2024)
Although existing medical image segmentation methods provide impressive pixel-wise accuracy, they often neglect topological correctness, making their segmentations unusable for many downstream tasks. One option is to retrain such models whilst including a topologydriven loss component. However, this is computationally expensive and often impractical. A better solution would be to have a versatile plug-and-play topology refinement method that is compatible with any domain-specific segmentation pipeline. Directly training a postprocessing model to mitigate topological errors often fails as such models tend to be biased towards the topological errors of a target segmentation network. The diversity of these errors is confined to the information provided by a labelled training set, which is especially problematic for small datasets. Our method solves this problem by training a model-agnostic topology refinement network with synthetic segmentations that cover a wide variety of topological errors. Inspired by the Stone-Weierstrass theorem, we synthesize topology-perturbation masks with randomly sampled coefficients of orthogonal polynomial bases, which ensures a complete and unbiased representation. Practically, we verified the efficiency and effectiveness of our methods as being compatible with multiple families of polynomial bases, and show evidence that our universal plug-and-play topology refinement network outperforms both existing topology-driven learning-based and post-processing methods. We also show that combining our method with learning-based models provides an effortless add-on, which can further improve the performance of existing approaches.
CRIS-Team Scopus Importer:2024-12-13
Keywords: Segmentation; Synthesis; Topology

Unsupervised Cross-domain Image Classification by Distance Metric Guided Feature Alignment - Meng, Qingjie; Rueckert, Daniel; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2020)
Learning deep neural networks that are generalizable across different domains remains a challenge due to the problem of domain shift. Unsupervised domain adaptation is a promising avenue which transfers knowledge from a source domain to a target domain without using any labels in the target domain. Contemporary techniques focus on extracting domain-invariant features using domain adversarial training. However, these techniques neglect to learn discriminative class boundaries in the latent representation space on a target domain and yield limited adaptation performance. To address this problem, we propose distance metric guided feature alignment (MetFA) to extract discriminative as well as domain-invariant features on both source and target domains. The proposed MetFA method explicitly and directly learns the latent representation without using domain adversarial training. Our model integrates class distribution alignment to transfer semantic knowledge from a source domain to a target domain. We evaluate the proposed method on fetal ultrasound datasets for cross-device image classification. Experimental results demonstrate that the proposed method outperforms the state-of-the-art and enables model generalization.
CRIS-Team Scopus Importer:2021-08-17

Unsupervised Human Pose Estimation through Transforming Shape Templates - Schmidtke, Luca; Vlontzos, Athanasios; Ellershaw, Simon; Lukens, Anna; Arichi, Tomoki; Kainz, Bernhard (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2021)
Human pose estimation is a major computer vision problem with applications ranging from augmented reality and video capture to surveillance and movement tracking. In the medical context, the latter may be an important biomarker for neurological impairments in infants. Whilst many methods exist, their application has been limited by the need for well annotated large datasets and the inability to generalize to humans of different shapes and body compositions, e.g. children and infants. In this paper we present a novel method for learning pose estimators for human adults and infants in an unsupervised fashion. We approach this as a learnable template matching problem facilitated by deep feature extractors. Human-interpretable landmarks are estimated by transforming a template consisting of predefined body parts that are characterized by 2D Gaussian distributions. Enforcing a connectivity prior guides our model to meaningful human shape representations. We demonstrate the effectiveness of our approach on two different datasets including adults and infants. Project page: infantmotion.github.io.
CRIS-Team Scopus Importer:2023-03-08

Unsupervised Similarity Learning for Image Registration with Energy-Based Models - Grzech, Daniel; Folgoc, Loïc Le; Azampour, Mohammad Farid; Vlontzos, Athanasios; Glocker, Ben; Navab, Nassir; Schnabel, Julia; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2024)
We present a new model for deformable image registration, which learns in an unsupervised way a data-specific similarity metric. The proposed method consists of two neural networks, one that maps pairs of input images to transformations which align them, and one that provides the similarity metric whose maximisation guides the image alignment. We parametrise the similarity metric as an energy-based model, which is simple to train and allows us to improve the accuracy of image registration compared to other models with learnt similarity metrics by taking advantage of a more general mathematical formulation, as well as larger datasets. We also achieve substantial improvement in the accuracy of inter-patient image registration on MRI scans from the OASIS dataset compared to models that rely on traditional functions.
CRIS-Team Scopus Importer:2024-11-01
Keywords: energy-based models; image registration

URCDM: Ultra-Resolution Image Synthesis in Histopathology - Cechnicka, Sarah; Ball, James; Baugh, Matthew; Reynaud, Hadrien; Simmonds, Naomi; Smith, Andrew P.T.; Horsfield, Catherine; Roufosse, Candice; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2024)
Diagnosing medical conditions from histopathology data requires a thorough analysis across the various resolutions of Whole Slide Images (WSI). However, existing generative methods fail to consistently represent the hierarchical structure of WSIs due to a focus on high-fidelity patches. To tackle this, we propose Ultra-Resolution Cascaded Diffusion Models (URCDMs) which are capable of synthesising entire histopathology images at high resolutions whilst authentically capturing the details of both the underlying anatomy and pathology at all magnification levels. We evaluate our method on three separate datasets, consisting of brain, breast and kidney tissue, and surpass existing state-of-the-art multi-resolution models. Furthermore, an expert evaluation study was conducted, demonstrating that URCDMs consistently generate outputs across various resolutions that trained evaluators cannot distinguish from real images. All code and additional examples can be found on GitHub.
CRIS-Team Scopus Importer:2024-11-08
Keywords: Diffusion; Generative; Gigapixel Images; Histopathology

Using perceptual features to prioritize ray-based image generation - Kainz, Bernhard; Steinberger, M.; Hauswiesner, S.; Khlebnikov, R.; Kalkofen, D.; Schmalstieg, D. (Proceedings of the Symposium on Interactive 3D Graphics, 2011)
A common challenge in interactive image generation is maintaining high interactivity of the applications that use computationally demanding rendering algorithms. This is usually achieved by sacrificing some of the image quality in order to decrease the rendering time. Most of such algorithms achieve interactive frame rates while trying to preserve as much image quality as possible by applying the reduction steps non-uniformly. However, high-end rendering systems, such as those presented by [Parker et al. 2010], aim to generate highly realistic images of very complex scenes. In such systems ordinary sampling approaches often give visually unacceptable results. In order to allow to optimize the ratio between the sampling rate of the scene and its resulting perceptual quality, we present a new sampling strategy which uses the information about object features that are known to support the comprehension of 3D shape [Cole et al. 2009]. We control the sampling density by exploiting line extraction techniques commonly used in Non-Photorealistic Renderings. Copyright © 2011 by the Association for Computing Machinery, Inc.
CRIS-Team Scopus Importer:2021-08-17

Vessel segmentation for ablation treatment planning and simulation - Alhonnoro, Tuomas; Pollari, Mika; Lilja, Mikko; Flanagan, Ronan; Kainz, Bernhard; Muehl, Judith; Mayrhauser, Ursula; Portugaller, Horst; Stiegler, Philipp; Tscheliessnigg, Karlheinz (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2010)
In this paper, a novel segmentation method for liver vasculature is presented, intended for numerical simulation of radio frequency ablation (RFA). The developed method is a semiautomatic hybrid based on multi-scale vessel enhancement combined with ridge-oriented region growing and skeleton-based postprocessing. In addition, an interactive tool for segmentation refinement was developed. Four instances of three-phase contrast enhanced computed tomography (CT) images of porcine liver were used in the evaluation. The results showed improved accuracy over common approaches and illustrated the method's suitability for simulation purposes. © 2010 Springer-Verlag.
CRIS-Team Scopus Importer:2021-08-17

Video-Based Activity Recognition for Automated Motor Assessment of Parkinson's Disease - Sarapata, Grzegorz; Dushin, Yuriy; Morinan, Gareth; Ong, Joshua; Budhdeo, Sanjay; Kainz, Bernhard; O'Keeffe, Jonathan (IEEE Journal of Biomedical and Health Informatics, 2023)
Over the last decade, video-enabled mobile devices have become ubiquitous, while advances in markerless pose estimation allow an individual's body position to be tracked accurately and efficiently across the frames of a video. Previous work by this and other groups has shown that pose-extracted kinematic features can be used to reliably measure motor impairment in Parkinson's disease (PD). This presents the prospect of developing an asynchronous and scalable, video-based assessment of motor dysfunction. Crucial to this endeavour is the ability to automatically recognise the class of an action being performed, without which manual labelling is required. Representing the evolution of body joint locations as a spatio-temporal graph, we implement a deep-learning model for video and frame-level classification of activities performed according to part 3 of the Movement Disorder Society Unified PD Rating Scale (MDS-UPDRS). We train and validate this system using a dataset of n = 7310 video clips, recorded at 5 independent sites. This approach reaches human-level performance in detecting and classifying periods of activity within monocular video clips. Our framework could support clinical workflows and patient care at scale through applications such as quality monitoring of clinical data collection, automated labelling of video streams, or a module within a remote self-assessment system.
CRIS-Team Scopus Importer:2023-11-03
Keywords: Activity recognition; computer vision; graph neural networks; Parkinson's disease; telemedicine

Video Summarization Through Reinforcement Learning with a 3D Spatiooral U-Net - Kainz, Bernhard; et al. (IEEE Transactions on Image Processing, 2022)
Keywords: 3D convolutions; 3D U-Net; ultrasound; Video summarization; medical video processing; reinforcement learning

Video Summarization Through Reinforcement Learning with a 3D Spatiooral U-Net - Liu, Tianrui; Meng, Qingjie; Huang, Jun-Jie; Vlontzos, Athanasios; Rueckert, Daniel; Kainz, Bernhard (IEEE Transactions on Image Processing, 2022)
Intelligent video summarization algorithms allow to quickly convey the most relevant information in videos through the identification of the most essential and explanatory content while removing redundant video frames. In this paper, we introduce the 3DST-UNet-RL framework for video summarization. A 3D spatiooral U-Net is used to efficiently encode spatiooral information of the input videos for downstream reinforcement learning (RL). An RL agent learns from spatiooral latent scores and predicts actions for keeping or rejecting a video frame in a video summary. We investigate if real/inflated 3D spatiooral CNN features are better suited to learn representations from videos than commonly used 2D image features. Our framework can operate in both, a fully unsupervised mode and a supervised training mode. We analyse the impact of prescribed summary lengths and show experimental evidence for the effectiveness of 3DST-UNet-RL on two commonly used general video summarization benchmarks. We also applied our method on a medical video summarization task. The proposed video summarization method has the potential to save storage costs of ultrasound screening videos as well as to increase efficiency when browsing patient video data during retrospective analysis or audit without loosing essential information.
CRIS-Team Scopus Importer:2022-02-25
Keywords: 3D convolutions; 3D U-Net; medical video processing; reinforcement learning; ultrasound; Video summarization

Volumetric real-time particle-based representation of large unstructured tetrahedral polygon meshes - Voglreiter, Philip; Steinberger, Markus; Schmalstieg, Dieter; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2012)
In this paper we propose a particle-based volume rendering approach for unstructured, three-dimensional, tetrahedral polygon meshes. We stochastically generate millions of particles per second and project them on the screen in real-time. In contrast to previous rendering techniques of tetrahedral volume meshes, our method does not need a prior depth sorting of geometry. Instead, the rendered image is generated by choosing particles closest to the camera. Furthermore, we use spatial superimposing. Each pixel is constructed from multiple subpixels. This approach not only increases projection accuracy, but allows also a combination of subpixels into one superpixel that creates the well-known translucency effect of volume rendering. We show that our method is fast enough for the visualization of unstructured three-dimensional grids with hard real-time constraints and that it scales well for a high number of particles. © 2012 Springer-Verlag.
CRIS-Team Scopus Importer:2021-08-17
Keywords: GPU accelerated; mesh representations; particle-based; volume rendering

Volume visualization in the clinical practice - Kainz, Bernhard; Portugaller, Rupert H.; Seider, Daniel; Moche, Michael; Stiegler, Philipp; Schmalstieg, Dieter (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2012)
Volumetric data is common in medicine, geology and engineering, but the O(n 3) complexity in data and algorithms has prevented the widespread use of volume graphics. Recently, 3D image processing and visualization algorithms have been parallelized and ported to graphics processing units. Today, medical diagnostics highly depends on volumetric imaging methods that must be visualized in real-time. However, daily clinical practice shows that physicians still prefer simple 2D multi-planar reconstructions over 3D visualizations for intervention planning. Therefore, a very basic question in this context is, if real-time 3D image synthesis is necessary at all. This paper makes four main observations in a clinical context, which are evaluated with 24 independent physicians from three different European hospitals. © 2012 Springer-Verlag.
CRIS-Team Scopus Importer:2021-08-17
Keywords: evaluation in clinical practice; medical visualization

Weakly Supervised Estimation of Shadow Confidence Maps in Fetal Ultrasound Imaging - Meng, Qingjie; Sinclair, Matthew; Zimmer, Veronika; Hou, Benjamin; Rajchl, Martin; Toussaint, Nicolas; Oktay, Ozan; Schlemper, Jo; Gomez, Alberto; Housden, James; Matthew, Jacqueline; Rueckert, Daniel; et al. (IEEE Transactions on Medical Imaging, 2019)
Detecting acoustic shadows in ultrasound images is important in many clinical and engineering applications. Real-time feedback of acoustic shadows can guide sonographers to a standardized diagnostic viewing plane with minimal artifacts and can provide additional information for other automatic image analysis algorithms. However, automatically detecting shadow regions using learning-based algorithms is challenging because pixel-wise ground truth annotation of acoustic shadows is subjective and time consuming. In this paper, we propose a weakly supervised method for automatic confidence estimation of acoustic shadow regions. Our method is able to generate a dense shadow-focused confidence map. In our method, a shadow-seg module is built to learn general shadow features for shadow segmentation, based on global image-level annotations as well as a small number of coarse pixel-wise shadow annotations. A transfer function is introduced to extend the obtained binary shadow segmentation to a reference confidence map. In addition, a confidence estimation network is proposed to learn the mapping between input images and the reference confidence maps. This network is able to predict shadow confidence maps directly from input images during inference. We use evaluation metrics such as DICE, inter-class correlation, and so on, to verify the effectiveness of our method. Our method is more consistent than human annotation and outperforms the state-of-the-art quantitatively in shadow segmentation and qualitatively in confidence estimation of shadow regions. Furthermore, we demonstrate the applicability of our method by integrating shadow confidence maps into tasks such as ultrasound image classification, multi-view image fusion, and automated biometric measurements.
CRIS-Team Scopus Importer:2021-08-17
Keywords: confidence estimation; deep learning; shadow detection; Ultrasound imaging; weakly supervised

Whole examination AI estimation of fetal biometrics from 20-week ultrasound scans - Venturini, Lorenzo; Budd, Samuel; Farruggia, Alfonso; Wright, Robert; Matthew, Jacqueline; Day, Thomas G.; Kainz, Bernhard; Razavi, Reza; Hajnal, Jo V. (npj Digital Medicine, 2025)
The current approach to fetal anomaly screening is based on biometric measurements derived from individually selected ultrasound images. In this paper, we introduce a paradigm shift that attains human-level performance in biometric measurement by aggregating automatically extracted biometrics from every frame across an entire scan, with no need for operator intervention. We use a neural network to classify each frame of an ultrasound video recording. We then measure fetal biometrics in every frame where appropriate anatomy is visible. We use a Bayesian method to estimate the true value of each biometric from a large number of measurements and probabilistically reject outliers. We performed a retrospective experiment on 1457 recordings (comprising 48 million frames) of 20-week ultrasound scans, estimated fetal biometrics in those scans and compared our estimates to real-time manual measurements. Our method achieves human-level performance in estimating fetal biometrics and estimates well-calibrated credible intervals for the true biometric valu
Keywords: Intrauterine growth; Ultrasonography

Whole Slide Multiple Instance Learning for Predicting Axillary Lymph Node Metastasis - Shkëmbi, Glejdis; Müller, Johanna; Li, Zhe; Breininger, Katharina; Schüffler, Peter; Kainz, Bernhard (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2023)
Breast cancer is a major concern for women’s health globally, with axillary lymph node (ALN) metastasis identification being critical for prognosis evaluation and treatment guidance. This paper presents a deep learning (DL) classification pipeline for quantifying clinical information from digital core-needle biopsy (CNB) images, with one step less than existing methods. A publicly available dataset of 1058 patients was used to evaluate the performance of different baseline state-of-the-art (SOTA) DL models in classifying ALN metastatic status based on CNB images. An extensive ablation study of various data augmentation techniques was also conducted. Finally, the manual tumor segmentation and annotation step performed by the pathologists was assessed. Our proposed training scheme outperformed SOTA by 3.73%. Source code is available here.
CRIS-Team Scopus Importer:2023-11-03