3-D Operation Situs Reconstruction with Time-of-Flight Satellite Cameras Using Photogeometric Data Fusion - Haase, Sven; Bauer, Sebastian; Wasza, Jakob; Kilgus, Thomas; Maier-Hein, Lena; Schneider, Armin; Kranzfelder, Michael; Feußner, Hubertus; Hornegger, Joachim (Medical Image Computing and Computer-Assisted Intervention - MICCAI 2013, Lecture Notes in Computer Science, 2013)

4D photogeometric face recognition with time-of-flight sensors - Bauer, Sebastian; Wasza, Jakob; Müller, Kerstin; Hornegger, Joachim (2011 IEEE Workshop on Applications of Computer Vision, WACV 2011, 2011)
Methods for 2D/3D face recognition typically combine results obtained independently from the 2D and 3D data, respectively. There has not been much emphasis on data fusion at an early stage, even though it is at least potentially more powerful to exploit possible synergies between the two modalities. In this paper, we propose photogeometric features that interpret both the photometric texture and geometric shape information of 2D manifolds in a consistent manner. The 4D features encode the spatial distribution of gradients that are derived generically for any scalar field on arbitrary organized surface meshes. We apply the descriptor for biometric face recognition with a time-of-flight sensor. The method consists of three stages: (i) facial landmark localization with a HOG/SVM sliding window framework, (ii) extraction of photogeometric feature descriptors from time-of-flight data, using the inherent grayscale intensity information of the sensor as the 2D manifold's scalar field, (iii) probe matching against the gallery. Recognition based on the photogeometric features achieved 97.5% rank-1 identification rate on a comprehensive time-of-flight dataset (26 subjects, 364 facial images). © 2010 IEEE.

Automatic Multi-modal ToF/CT Organ Surface Registration - Müller, Kerstin; Bauer, Sebastian; Wasza, Jakob; Hornegger, Joachim (Bildverarbeitung für die Medizin 2011, 2011)

Fast Interpolation of Dense Motion Fields from Synthetic Phantoms - Maier, Andreas; Taubmann, Oliver; Wetzl, Jens; Wasza, Jakob; Forman, Christoph; Fischer, Peter; Hornegger, Joachim; Fahrig, Rebecca (Bildverarbeitung für die Medizin 2014, 2014)
Numerical phantoms are a common tool for the evaluation of registration and reconstruction algorithms. For applications concerning motion, dense deformation fields are of particular interest. Phantoms, however, are often described as surfaces and thus motion vectors can only be generated at these surfaces. In order to create dense motion fields, interpolation is required. A frequently used method for this purpose is the Parzen interpolator. However, with a high number of surface motion vectors and a high voxel count, its run time increases dramatically. In this paper, we investigate different methods to accelerate the creation of these motion fields using hierarchical sampling and the random ball cover. In the results, we show that a 643 volume can be sampled in less than one second with an error below 0.1 mm. Furthermore, we accelerate the interpolation of a 2563 dense deformation field to only 6.5 minutes using the proposed methods from days with previous methods.
UnivIS-Import:2015-04-16:Pub.2014.tech.IMMD.IMMD5.fastin_7
Keywords: interpolation; motion field; phantom

Free-Breathing Whole-Heart Coronary MRI: An Image-Based Motion Compensation Integrated Into Compressed-Sensing Reconstruction - Forman, Christoph; Grimm, Robert; Hutter, Jana; Wasza, Jakob; Kraus, Martin; Hornegger, Joachim; Zenge, Michael O. (Proceedings of the 21st Annual Meeting of ISMRM, 2013)
UnivIS-Import:2015-04-16:Pub.2013.tech.IMMD.IMMD5.freebr_2

High Performance GPU-based Preprocessing for Time-of-Flight Imaging in Medical Applications - Wasza, Jakob; Bauer, Sebastian; Hornegger, Joachim (Bildverarbeitung für die Medizin 2011, 2011)

Kinect-Based Correction of Overexposure Artifacts in Knee Imaging with C-Arm CT Systems - Rausch, Johannes; Maier, Andreas; Fahrig, Rebecca; Choi, Jang-Hwan; Hinshaw, Waldo; Schebesch, Frank; Haase, Sven; Wasza, Jakob; Hornegger, Joachim; Riess, Christian (International Journal of Biomedical Imaging, 2016)

Laparoscopic instrument localization using a 3-D Time-of-Flight/RGB endoscope - Haase, Sven; Wasza, Jakob; Kilgus, Thomas; Hornegger, Joachim (2013 IEEE Workshop on Applications of Computer Vision, WACV 2013, 2013)
Minimally invasive procedures are of importance in modern surgery due to reduced operative trauma and recovery time. To enable robot assisted interventions, automatic tracking of endoscopie tools is an essential task. State-of-the-art techniques rely on 2-D color information only which is error prone for varying illumination and unpredictable color distribution within the human body. In this paper, we use a novel 3-D Time-of-Flight/RGB endoscope that allows to use both color and range information to locate laparoscopic instruments in 3-D. Regarding color and range information the proposed technique calculates a score to indicate which information is more reliable and adopts the next steps of the localization procedure based on this reliability. In experiments on real data the tool tip is located with an average 3-D distance error of less than 4 mm compared to manually labeled ground truth data with a frame-rate of 10 fps. © 2013 IEEE.

Multi-modal surface registration for markerless initial patient setup in radiation therapy using microsoft's Kinect sensor - Bauer, Sebastian; Wasza, Jakob; Haase, Sven; Marosi, Natalia; Hornegger, Joachim (2011 IEEE International Conference on Computer Vision Workshops, ICCV Workshops 2011, 2011)
In radiation therapy, prior to each treatment fraction, the patient must be aligned to computed tomography (CT) data. Patient setup verification systems based on range imaging (RI) can accurately verify the patient position and adjust the treatment table at a fine scale, but require an initial manual setup using lasers and skin markers. We propose a novel markerless solution that enables a fully-automatic initial coarse patient setup. The table transformation that brings template and reference data in congruence is estimated from point correspondences based on matching local surface descriptors. Inherently, this point-based registration approach is capable of coping with gross initial misalignments and partial matching. Facing the challenge of multi-modal surface registration (RI/CT), we have adapted state-of-the-art descriptors to achieve invariance to mesh resolution and robustness to variations in topology. In a case study on real data from a low-cost RI device (Microsoft Kinect), the performance of different descriptors is evaluated on anthropomorphic phantoms. Furthermore, we have investigated the system's resilience to deformations for mono-modal RI/RI registration of data from healthy volunteers. Under gross initial misalignments, our method resulted in an average angular error of 1.5 and an average translational error of 13.4 mm in RI/CT registration. This coarse patient setup provides a feasible initialization for subsequent refinement with verification systems. © 2011 IEEE.

Multi-Sensor Super-Resolution for Hybrid Range Imaging with Application to 3-D Endoscopy and Open Surgery - Köhler, Thomas; Haase, Sven; Bauer, Sebastian; Wasza, Jakob; Kilgus, Thomas; Maier-Hein, Lena; Stock, Christian; Hornegger, Joachim; Feussner, Hubertus (Medical Image Analysis, 2015)
In this paper, we propose a multi-sensor super-resolution framework for hybrid imaging to super-resolve data from one modality by taking advantage of additional guidance images of a complementary modality. This concept is applied to hybrid 3-D range imaging in image-guided surgery, where high-quality photometric data is exploited to enhance range images of low spatial resolution. We formulate super-resolution based on the maximum a-posteriori (MAP) principle and reconstruct high-resolution range data from multiple low-resolution frames and complementary photometric information. Robust motion estimation as required for super-resolution is performed on photometric data to derive displacement fields of subpixel accuracy for the associated range images. For improved reconstruction of depth discontinuities, a novel adaptive regularizer exploiting correlations between both modalities is embedded to MAP estimation. We evaluated our method on synthetic data as well as ex-vivo images in open surgery and endoscopy. The proposed multi-sensor framework improves the peak signal-to-noise ratio by 2 dB and structural similarity by 0.03 on average compared to conventional single-sensor approaches. In ex-vivo experiments on porcine organs, our method achieves substantial improvements in terms of depth discontinuity reconstruction.
UnivIS-Import:2015-10-26:Pub.2015.tech.IMMD.IMMD5.multis
Keywords: 3-D endoscopy; Hybrid range imaging; Open surgery; Sensor fusion; Super-resolution

Outlier Detection for Multi-Sensor Super-Resolution in Hybrid 3-D Endoscopy - Köhler, Thomas; Haase, Sven; Bauer, Sebastian; Wasza, Jakob; Kilgus, Thomas; Maier-Hein, Lena; Feußner, Hubertus; Hornegger, Joachim (Bildverarbeitung für die Medizin 2014, 2014)
UnivIS-Import:2015-04-16:Pub.2014.tech.IMMD.IMMD5.outlie_5

Patch based Specular Reflection Removal for Range Images in Hybrid 3-D Endoscopy - Haase, Sven; Wasza, Jakob; Safak, Mustafa; Kilgus, Thomas; Maier-Hein, Lena; Feußner, Hubertus; Hornegger, Joachim (2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI), 2014)
UnivIS-Import:2015-04-16:Pub.2014.tech.IMMD.IMMD5.patchb

Photometric Estimation of 3-D Surface Motion Fields for Respiration Management - Bauer, Sebastian; Wasza, Jakob; Hornegger, Joachim (Bildverarbeitung für die Medizin 2012, 2012)

Prediction of Respiration-Induced Internal 3-D Deformation Fields From Dense External 3-D Surface Motion - Taubmann, Oliver; Wasza, Jakob; Forman, Christoph; Fischer, Peter; Wetzl, Jens; Maier, Andreas; Hornegger, Joachim (Proceedings of the 28th International Congress and Exhibition, 2014)
UnivIS-Import:2015-04-16:Pub.2014.tech.IMMD.IMMD5.predic_9
Keywords: Respiratory Motion; Motion Modeling; Motion Prediction; External Surrogate

Real-time motion compensated patient positioning and non-rigid deformation estimation using 4-D shape priors - Wasza, Jakob; Bauer, Sebastian; Hornegger, Joachim (Lecture Notes in Computer Science, 2012)
Over the last years, range imaging (RI) techniques have been proposed for patient positioning and respiration analysis in motion compensation. Yet, current RI based approaches for patient positioning employ rigid-body transformations, thus neglecting free-form deformations induced by respiratory motion. Furthermore, RI based respiration analysis relies on non-rigid registration techniques with run-times of several seconds. In this paper we propose a real-time framework based on RI to perform respiratory motion compensated positioning and non-rigid surface deformation estimation in a joint manner. The core of our method are pre-procedurally obtained 4-D shape priors that drive the intra-procedural alignment of the patient to the reference state, simultaneously yielding a rigid-body table transformation and a free-form deformation accounting for respiratory motion. We show that our method outperforms conventional alignment strategies by a factor of 3.0 and 2.3 in the rotation and translation accuracy, respectively. Using a GPU based implementation, we achieve run-times of 40 ms.

Real-time Motion Compensated Patient Positioning and Non-rigid Deformation Estimation using 4-D Shape Priors - Wasza, Jakob; Bauer, Sebastian; Hornegger, Joachim (MICCAI 2012, Part II, LNCS 7511, 2012)
Over the last years, range imaging (RI) techniques have been proposed for patient positioning and respiration analysis in motion compensation. Yet, current RI based approaches for patient positioning employ rigid-body transformations, thus neglecting free-form deformations induced by respiratory motion. Furthermore, RI based respiration analysis relies on non-rigid registration techniques with run-times of several seconds. In this paper we propose a real-time framework based on RI to perform respiratory motion compensated positioning and non-rigid surface deformation estimation in a joint manner. The core of our method are pre-procedurally obtained 4-D shape priors that drive the intra-procedural alignment of the patient to the reference state, simultaneously yielding a rigid-body table transformation and a free-form deformation accounting for respiratory motion. We show that our method outperforms conventional alignment strategies by a factor of 3.0 and 2.3 in the rotation and translation accuracy, respectively. Using a GPU based implementation, we achieve run-times of 40 ms.
UnivIS-Import:2015-04-16:Pub.2012.tech.IMMD.IMMD5.realti_8

Real-time preprocessing for dense 3-D range imaging on the GPU: Defect interpolation, bilateral temporal averaging and guided filtering - Wasza, Jakob; Bauer, Sebastian; Hornegger, Joachim (2011 IEEE International Conference on Computer Vision Workshops, ICCV Workshops 2011, 2011)
Recent advances in range imaging (RI) have enabled dense 3-D scene acquisition in real-time. However, due to physical limitations and the underlying range sampling principles, range data are subject to noise and may contain invalid measurements. Hence, data preprocessing is a prerequisite for practical applications but poses a challenge with respect to real-time constraints. In this paper, we propose a generic and modality-independent pipeline for efficient RI data preprocessing on the graphics processing unit (GPU). The contributions of this work are efficient GPU implementations of normalized convolution for the restoration of invalid measurements, bilateral temporal averaging for dynamic scenes, and guided filtering for edge-preserving denoising. Furthermore, we show that the transformation from range measurements to 3-D world coordinates can be computed efficiently on the GPU. The pipeline has been evaluated on real data from a Time-of-Flight sensor and Microsoft's Kinect. In a run-time performance study, we show that for VGA-resolution data, our preprocessing pipeline runs at 100 fps on an off-the-shelf consumer GPU. © 2011 IEEE.

Real-Time Range Imaging in Health Care: A Survey - Bauer, Sebastian; Seitel, Alexander; Hofmann, Hannes; Blum, Tobias; Wasza, Jakob; Balda, Michael; Meinzer, Hans-Peter; Navab, Nassir; Hornegger, Joachim; Maier-Hein, Lena (Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications, 2013)
The recent availability of dynamic, dense, and low-cost range imaging has gained widespread interest in health care. It opens up new opportunities and has an increasing impact on both research and commercial activities. This chapter presents a state-of-the-art survey on the integration of modern range imaging sensors into medical applications. The scope is to identify promising applications and methods, and to provide an overview of recent developments in this rapidly evolving domain. The survey covers a broad range of topics, including guidance in computer-assisted interventions, operation room monitoring and workflow analysis, touch-less interaction and on-patient visualization, as well as prevention and support in elderly care and rehabilitation. We put emphasis on dynamic and interactive tasks where real-time and dense 3-D imaging forms the key aspect. While considering different range imaging modalities that fulfill these requirements, we particularly investigate the impact of Time-of-Flight imaging in this domain. Eventually, we discuss practical demands and limitations, and open research issues and challenges that are of fundamental importance for the progression of the field. © Springer-Verlag 2013.
UnivIS-Import:2015-04-20:Pub.2013.tech.IMMD.IMMD5.realti_6

Real-Time Respiratory Motion Analysis Using 4-D Shape Priors - Wasza, Jakob; Fischer, Peter; Leutheuser, Heike; Oefner, Tobias; Bert, Christoph; Maier, Andreas; Hornegger, Joachim (IEEE Transactions on Biomedical Engineering, 2016)
Respiratory motion analysis based on range imaging (RI) has emerged as a popular means of generating respiration surrogates to guide motion management strategies in computer-assisted interventions. However, existing approaches employ heuristics, require substantial manual interaction, or yield highly redundant information. In this paper, we propose a framework that uses preprocedurally obtained 4-D shape priors from patient-specific breathing patterns to drive intraprocedural RI-based real-time respiratory motion analysis. As the first contribution, we present a shape motion model enabling an unsupervised decomposition of respiration induced high-dimensional body surface displacement fields into a low-dimensional representation encoding thoracic and abdominal breathing. Second, we propose a method designed for GPU architectures to quickly and robustly align our models to high-coverage multiview RI body surface data. With our fully automatic method, we obtain respiration surrogates yielding a Pearson correlation coefficient (PCC) of 0.98 with conventional surrogates based on manually selected regions on RI body surface data. Compared to impedance pneumography as a respiration signal that measures the change of lung volume, we obtain a PCC of 0.96. Using off-the-shelf hardware, our framework enables high temporal resolution respiration analysis at 50 Hz.
Keywords: Graphics processing unit (GPU); radiation therapy; range imaging (RI); respiratory motion

Real-time Respiratory Motion Analysis Using GPU-accelerated Range Imaging - Wasza, Jakob (Friedrich-Alexander-Universität Erlangen-Nürnberg, 2019)
Respiratory motion analysis and management are crucial issues for a plurality of medical applications. Of particular scientific concern are methods that allow to analyze the patient’s breathing in a non-invasive and real-time manner that does not involve ionizing radiation exposure. For this purpose, range imaging technologies, that allow to dynamically acquire three-dimensional body surface data, have been proposed over the last years. A particular challenge with such methods is a fully automatic investigation and assessment of the body surface data, as well as computation times that comply with real-time constraints. This dissertation is concerned with the application of range imaging principles for real-time automatic respiratory motion analysis. The focus is on the development of efficient methods for data preprocessing and fusion as well as machine learning and surface registration techniques. A particular emphasis of this thesis is the design of the proposed algorithms for GPU architectures to enable real-time computation. The first part of this thesis covers the general challenges and requirements for respiratory motion analysis in diagnostic and therapeutic applications. Furthermore, the range imaging technologies that are relevant for this thesis are introduced and the suitability of GPU architectures for accelerating the computation of several tasks inherent to range imaging based respiratory motion analysis are investigated. The second part of this work is concerned with pre-processing and fusion techniques for range data. To account for the low signal-to-noise ratio that is common with range data, this work proposes a processing pipeline that allows to reconstruct the ideal data with an error trueness less than 1.0mm at run-times of 2 ms. For fusing range image data in a multi-camera setup, as it is required for the simultaneous acquisition of frontal and lateral body surface, this thesis proposes a novel framework that enables the computation of a 180° coverage body surface model consisting of more than 3.0 x 105 points with a computation time of less than 5 ms. The third part of this work is concerned with patient-specific respiratory motion models. The thesis proposes machine learning techniques to generate a continuous motion model that features the ability to automatically differentiate between thoracic and abdominal breathing as well as to quantitatively analyze the patient’s respiration magnitude. By using purposely developed surface registration schemes, these models are then brought in congruence with body surface data acquired by range imaging sensors. This allows for respiratory motion compensated patient positioning that reduces the alignment error observed with conventional approaches by a factor of 3 to less than 4.0mm. Further, this approach allows to automatically derive a multi-dimensional respiration surrogate that yields a correlation coefficient greater than 0.97 compared to commonly employed invasive or semi-automatic approaches and that can be computed in 20 ms. The fourth part concludes this thesis with a summary of the presented methods and results, as well as an outlook regarding future research directions and challenges towards clinical translation.<br

Real-Time Respiratory Motion Analysis Using Manifold Ray Casting of Volumetrically Fused Multi-view Range Imaging - Wasza, Jakob; Bauer, Sebastian; Hornegger, Joachim (MICCAI 2013, Part II, LNCS 8150 (16th International Conference on Medical Image Computing and Computer Assisted Intervention), 2013)

Real-Time RGB-D Mapping and 3-D Modeling on the GPU Using the Random Ball Cover - Bauer, Sebastian; Wasza, Jakob; Lugauer, Felix; Neumann, Dominik; Hornegger, Joachim (Consumer Depth Cameras for Computer Vision, 2013)

Real-time RGB-D mapping and 3-D modeling on the GPU using the random ball cover data structure - Neumann, Dominik; Lugauer, Felix; Bauer, Sebastian; Wasza, Jakob; Hornegger, Joachim (2011 IEEE International Conference on Computer Vision Workshops, ICCV Workshops 2011, 2011)
The modeling of three-dimensional scene geometry from temporal point cloud streams is of particular interest for a variety of computer vision applications. With the advent of RGB-D imaging devices that deliver dense, metric and textured 6-D data in real-time, on-the-fly reconstruction of static environments has come into reach. In this paper, we propose a system for real-time point cloud mapping based on an efficient implementation of the iterative closest point (ICP) algorithm on the graphics processing unit (GPU). In order to achieve robust mappings at real-time performance, our nearest neighbor search evaluates both geometric and photometric information in a direct manner. For acceleration of the search space traversal, we exploit the inherent computing parallelism of GPUs. In this work, we have investigated the fitness of the random ball cover (RBC) data structure and search algorithm, originally proposed for high-dimensional problems, for 6-D data. In particular, we introduce a scheme that enables both fast RBC construction and queries. The proposed system is validated on an indoor scene modeling scenario. For dense data from the Microsoft Kinect sensor (640×480 px), our implementation achieved ICP runtimes of< 20 ms on an off-the-shelf consumer GPU. © 2011 IEEE.

RITK: The Range Imaging Toolkit - A Framework for 3-D Range Image Stream Processing - Wasza, Jakob; Bauer, Sebastian; Haase, Sven; Schmid, M.; Reichert, S.; Hornegger, Joachim (VMV 2011: Vision, Modeling & Visualization, 2011)

Segmentation of kidneys using a new active shape model generation technique based on non-rigid image registration - Spiegel, Martin; Hahn, Dieter; Daum, Volker; Wasza, Jakob; Hornegger, Joachim (Computerized Medical Imaging and Graphics, 2009)
Active shape models (ASMs) are widely used for applications in the field of image segmentation. Building an ASM requires to determine point correspondences for input training data, which usually results in a set of landmarks distributed according to the statistical variations. State-of-the-art methods solve this problem by minimizing the description length of all landmarks using a parametric mapping of the target shape (e.g. a sphere). In case of models composed of multiple sub-parts or highly non-convex shapes, these techniques feature substantial drawbacks. This article proposes a novel technique for solving the crucial correspondence problem using non-rigid image registration. Unlike existing approaches the new method yields more detailed ASMs and does not require explicit or parametric formulations of the problem. Compared to other methods, the already built ASM can be updated with additional prior knowledge in a very efficient manner. For this work, a training set of 3-D kidney pairs has been manually segmented from 41 CT images of different patients and forms the basis for a clinical evaluation. The novel registration based approach is compared to an already established algorithm that uses a minimum description length (MDL) formulation. The presented results indicate that the use of non-rigid image registration to solve the point correspondence problem leads to improved ASMs and more accurate segmentation results. The sensitivity could be increased by approximately 10%. Experiments to analyze the dependency on the user initialization also show a higher sensitivity of 5-15%. The mean squared error of the segmentation results and the ground truth manually classified data could also be reduced by 20-34% with respect to varying numbers of training samples. © 2008 Elsevier Ltd. All rights reserved.

Sparse Active Triangulation Grids for Respiratory Motion Management - Bauer, Sebastian; Ettl, Svenja; Wasza, Jakob; Willomitzer, Florian; Huber, Franz; Hornegger, Joachim; Häusler, Gerd (DGaO Proceedings 2012, 2012)

Sparse Principal Axes Statistical Surface Deformation Models for Respiration Analysis and Classication - Wasza, Jakob; Bauer, Sebastian; Haase, Sven; Hornegger, Joachim (Bildverarbeitung für die Medizin 2012, 2012)

ToF Meets RGB: Novel Multi-Sensor Super-Resolution for Hybrid 3-D Endoscopy - Köhler, Thomas; Haase, Sven; Bauer, Sebastian; Wasza, Jakob; Kilgus, Thomas; Maier-Hein, Lena; Feußner, Hubertus; Hornegger, Joachim (Medical Image Computing and Computer-Assisted Intervention - MICCAI 2013, Lecture Notes in Computer Science, 2013)