A Bregman Learning Framework for Sparse Neural Networks - Bungert, Leon; Roith, Tim; Tenbrinck, Daniel; Burger, Martin (Journal of Machine Learning Research, 2022)
We propose a learning framework based on stochastic Bregman iterations, also known as mirror descent, to train sparse neural networks with an inverse scale space approach. We derive a baseline algorithm called LinBreg, an accelerated version using momentum, and AdaBreg, which is a Bregmanized generalization of the Adam algorithm. In contrast to established methods for sparse training the proposed family of algorithms constitutes a regrowth strategy for neural networks that is solely optimization-based without additional heuristics. Our Bregman learning framework starts the training with very few initial parameters, successively adding only significant ones to obtain a sparse and expressive network. The proposed approach is extremely easy and efficient, yet supported by the rich mathematical theory of inverse scale space methods. We derive a statistically profound sparse parameter initialization strategy and provide a rigorous stochastic convergence analysis of the loss decay and additional convergence proofs in the convex regime. Using only 3.4%" role="presentation" style="display: inline; line-height: normal; font-size: medium; overflow-wrap: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px; border-width: 0px; border-style: initial; color: rgb(0, 0, 0); font-family: "Times New Roman"; position: relative;">3.4%3.4% of the parameters of ResNet-18 we achieve 90.2%" role="presentation" style="display: inline; line-height: normal; font-size: medium; overflow-wrap: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px; border-width: 0px; border-style: initial; color: rgb(0, 0, 0); font-family: "Times New Roman"; position: relative;">90.2.2% test accuracy on CIFAR-10, compared to 93.6%" role="presentation" style="display: inline; line-height: normal; font-size: medium; overflow-wrap: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px; border-width: 0px; border-style: initial; color: rgb(0, 0, 0); font-family: "Times New Roman"; position: relative;">93.6.6% using the dense network. Our algorithm also unveils an autoencoder architecture for a denoising task. The proposed framework also has a huge potential for integrating sparse backpropagation and resource-friendly training. Code is available at https://github.com/TimRoith/BregmanLearning.<br

A Discontinuous Galerkin Method for the Subjective Surfaces Problem - Bungert, Leon; Aizinger, Vadym; Fried, Michael (Journal of Mathematical Imaging and Vision, 2016)
The work formulates and evaluates the local discontinuous Galerkin method for the subjective surfaces problem based on the curvature driven level set equation. A new mixed formulation simplifying the treatment of nonlinearities is proposed. The numerical algorithm is evaluated using several artificial and realistic test cases.

Asymptotic profiles of nonlinear homogeneous evolution equations of gradient flow type - Bungert, Leon; Burger, Martin (Journal of Evolution Equations, 2019)
This work is concerned with the gradient flow of absolutely p-homogeneous convex functionals on a Hilbert space, which we show to exhibit finite (p< 2) or infinite extinction time (p≥ 2). We give upper bounds for the finite extinction time and establish sharp convergence rates of the flow. Moreover, we study next order asymptotics and prove that asymptotic profiles of the solution are eigenfunctions of the subdifferential operator of the functional. To this end, we compare with solutions of an ordinary differential equation which describes the evolution of eigenfunction under the flow. Our work applies, for instance, to local and non-local versions of PDEs like p-Laplacian evolution equations, the porous medium equation, and fast diffusion equations, herewith generalizing many results from the literature to an abstract setting. We also demonstrate how our theory extends to general homogeneous evolution equations which are not necessarily a gradient flow. Here, we discover an interesting integrability condition which characterizes whether or not asymptotic profiles are eigenfunctions.
CRIS-Team Scopus Importer:2019-11-12
Keywords: Asymptotic behavior; Asymptotic profile; Convergence rates; Extinction profile; Extinction time; Gradient flow; Homogeneous functionals; Nonlinear eigenfunctions; Nonlinear evolution equations

Blind image fusion for hyperspectral imaging with the directional total variation - Bungert, Leon; Ehrhardt, Matthias Joachim; Coomes, David; Rasch, Jennifer; Reisenhofer, Rafael; Schönlieb, Carola-Bibiane (Inverse Problems, 2018)
Hyperspectral imaging is a cutting-edge type of remote sensing used for mapping vegetation properties, rock minerals and other materials. A major drawback of hyperspectral imaging devices is their intrinsic low spatial resolution. In this paper, we propose a method for increasing the spatial resolution of a hyperspectral image by fusing it with an image of higher spatial resolution that was obtained with a different imaging modality. This is accomplished by solving a variational problem in which the regularization functional is the directional total variation. To accommodate for possible mis-registrations between the two images, we consider a non-convex blind super-resolution problem where both a fused image and the corresponding convolution kernel are estimated. Using this approach, our model can realign the given images if needed. Our experimental results indicate that the non-convexity is negligible in practice and that reliable solutions can be computed using a variety of different optimization algorithms. Numerical results on real remote sensing data from plant sciences and urban monitoring show the potential of the proposed method and suggests that it is robust with respect to the regularization parameters, mis-registration and the shape of the kernel.
Keywords: remote sensing; super-resolution; pansharpening; blind deconvolution; hyperspectral imaging

CLIP: Cheap Lipschitz Training of Neural Networks - Bungert, Leon; Raab, René; Roith, Tim; Schwinn, Leo; Tenbrinck, Daniel (SSVM 2021: Scale Space and Variational Methods in Computer Vision, 2021)
Despite the large success of deep neural networks (DNN) in recent years, most neural networks still lack mathematical guarantees in terms of stability. For instance, DNNs are vulnerable to small or even imperceptible input perturbations, so called adversarial examples, that can cause false predictions. This instability can have severe consequences in applications which influence the health and safety of humans, e.g., biomedical imaging or autonomous driving. While bounding the Lipschitz constant of a neural network improves stability, most methods rely on restricting the Lipschitz constants of each layer which gives a poor bound for the actual Lipschitz constant. In this paper we investigate a variational regularization method named CLIP for controlling the Lipschitz constant of a neural network, which can easily be integrated into the training procedure. We mathematically analyze the proposed model, in particular discussing the impact of the chosen regularization parameter on the output of the network. Finally, we numerically evaluate our method on both a nonlinear regression problem and the MNIST and Fashion-MNIST classification databases, and compare our results with a weight regularization approac

Comparison of two local discontinuous Galerkin formulations for the subjective surfaces problem - Fried, Michael; Aizinger, Vadym; Bungert, Leon (Computing and Visualization in Science, 2018)
Based on the local discontinuous Galerkin method, two substantially different mixed formulations for the subjective surfaces problem are compared using a number of numerical tests of various types. The work also performs the energy stability analysis for both schemes.
Keywords: Local discontinuous Galerkin method; Image segmentation; Subjective surfaces; Stability analysis; Mixed formulation; Divergence form; Edge detection

Computing Nonlinear Eigenfunctions via Gradient Flow Extinction - Bungert, Leon; Burger, Martin; Tenbrinck, Daniel (Scale Space and Variational Methods in Computer Vision - 7th International Conference, SSVM 2019, Proceedings, 2019)
In this work we investigate the computation of nonlinear eigenfunctions via the extinction profiles of gradient flows. We analyze a scheme that recursively subtracts such eigenfunctions from given data and show that this procedure yields a decomposition of the data into eigenfunctions in some cases as the 1-dimensional total variation, for instance. We discuss results of numerical experiments in which we use extinction profiles and the gradient flow for the task of spectral graph clustering as used, eg, in machine learning applications.<br

Identifying untrustworthy predictions in neural networks by geometric gradient analysis - Schwinn, Leo; Nguyen, An; Raab, René; Bungert, Leon; Tenbrinck, Daniel; Zanca, Dario; Burger, Martin; Eskofier, Björn (Conference on Uncertainty in Artificial Intelligence (UAI), 2021)
The susceptibility of deep neural networks to untrustworthy predictions, including out-of-distribution (OOD) data and adversarial examples, still prevent their widespread use in safety-critical applications. Most existing methods either require a retraining of a given model to achieve robust identification of adversarial attacks or are limited to out-of-distribution sample detection only. In this work, we propose a geometric gradient analysis (GGA) to improve the identification of untrustworthy predictions without retraining of a given model. GGA analyzes the geometry of the loss landscape of neural networks based on the saliency maps of their respective input. We observe considerable differences between the input gradient geometry of trustworthy and untrustworthy predictions. Using these differences, GGA outperforms prior approaches in detecting OOD data and adversarial attacks, including state-of-the-art and adaptive attacks. Code is available at: https://github.com/mad-lab-fau/GGA-Identifying-Untrustworthy-Predictions

Localization of Passive 3-D Coils as an Inverse Problem: Theoretical Analysis and a Numerical Method - Doss, Martin; Bungert, Leon; Cichon, Daniel; Brauer, Hartmut; Psiuk, Rafael (IEEE Transactions on Magnetics, 2020)
This article provides both a theoretical analysis and a numerical method for the inverse source problem of locating multiple passive 3-D coils based on measurements of their superposed magnetic fields. In our context, a 3-D coil consists of three concentric circular coils being mutually perpendicular, and the term "passive" means that these coils are not connected to an active power source. Instead, their current is induced externally by a low-frequency alternating magnetic field which is generated by a closed exciter wire. The underlying inductively coupled system is modeled with regard to the 3-D coils as magnetic dipoles and their localization is formulated as an inverse problem. Since its ill posedness mainly arises from strong sensitivity to observational noise, an approximate upper bound for the localization error is derived mathematically by linearization. The Levenberg-Marquardt algorithm is applied as a method for localization and modified for better performance as well as the ability to estimate the number of 3-D coils in the localization area. Our method is tested with simulated and real data in order to confirm its capability of locating up to three passive 3-D coils within the front of a wooden shelf surrounded by the exciter wire and eight rectangular loop antennas.
Keywords: 3-D coil; inverse problem; Levenberg-Marquardt algorithm; magnetic dipole; magnetic source localization

Localization of Passive 3D-Coils as an Inverse Problem: Theoretical Analysis and a Numerical Method - Doß, Martin; Bungert, Leon; Cichon, Daniel; Brauer, Hartmut; Psiuk, Rafael (IEEE Transactions on Magnetics, 2020)

Nonlinear Spectral Decompositions by Gradient Flows of One-Homogeneous Functionals - Bungert, Leon; Burger, Martin; Chambolle, Antonin; Novaga, Matteo (Analysis & Pde, 2019)
The aim of this paper is to establish a theory of nonlinear spectral decompositions in an infinite dimensional setting by considering the eigenvalue problem related to an absolutely one-homogeneous functional in a Hilbert space. This approach is motivated by works for the total variation and related functionals in , where some interesting results on the eigenvalue problem and the relation to the total variation flow have been proven previously, and by recent results on finite-dimensional polyhedral semi-norms, where gradient flows can yield exact decompositions into eigenvectors. We provide a geometric characterization of eigenvectors via a dual unit ball, which applies to the infinite-dimensional setting and allows applications to several relevant examples. In particular our analysis highlights the role of subgradients of minimal norm and thus connects directly to gradient flows, whose time evolution can be interpreted as a decomposition of the initial condition into subgradients of minimal norm. We can show that if the gradient flow yields a spectral decomposition it is equivalent in an appropriate sense to other schemes such as variational methods with regularization parameter equal to time, and that the decomposition has an interesting orthogonality relation. Indeed we verify that all approaches where these equivalences were known before by other arguments–such as one-dimensional TV flow and multidimensional generalizations to vector fields or the gradient flow of certain polyhedral semi-norms–yield exact spectral decompositions, and we provide further examples. We also investigate extinction times and extinction profiles, which we characterize as eigenvectors in a very general setting, generalizing several results from literature.

Robust Blind Image Fusion for Misaligned Hyperspectral Imaging Data - Bungert, Leon; Ehrhardt, Matthias Joachim; Reisenhofer, Rafael (Proceedings in Applied Mathematics and Mechanics, 2018)
The low spatial resolution of hyperspectral imaging can be significantly improved by fusing the hyperspectral image with a high resolution photograph. In most practical cases, however, the exact alignment between the fused images is not known a priori. In this work, we study how including a blind deconvolution approach in the mathematical model can help resolve translational misalignments. In particular, we investigate the influence of different initialization strategies. The efficiency of the proposed model is validated by numerical experiments using both simulated and real remote sensing data.

Robust Image Reconstruction with Misaligned Structural Information - Bungert, Leon; Ehrhardt, Matthias J. (IEEE Access, 2020)
Multi-modality (or multi-channel) imaging is becoming increasingly important and more widely available, e.g. hyperspectral imaging in remote sensing, spectral CT in material sciences as well as multi-contrast MRI and PET-MR in medicine. Research in the last decades resulted in a plethora of mathematical methods to combine data from several modalities. State-of-the-art methods, often formulated as variational regularization, have shown to significantly improve image reconstruction both quantitatively and qualitatively. Almost all of these models rely on the assumption that the modalities are perfectly registered, which is not the case in most real world applications. We propose a variational framework which jointly performs reconstruction and registration, thereby overcoming this hurdle. Our approach is the first to achieve this for different modalities and outranks established approaches in terms of accuracy of both reconstruction and registration. Numerical results on simulated and real data show the potential of the proposed strategy for various applications in multi-contrast MRI, PET-MR, and hyperspectral imaging: typical misalignments between modalities such as rotations, translations, zooms can be effectively corrected during the reconstruction process. Therefore the proposed framework allows the robust exploitation of shared information across multiple modalities under real conditions.<br

Solution paths of variational regularization methods for inverse problems - Bungert, Leon; Burger, Martin (Inverse Problems, 2019)
We consider a family of variational regularization functionals for a generic inverse problem, where the data fidelity and regularization term are given by powers of a Hilbert norm and an absolutely one-homogeneous functional, respectively. We investigate the small and large time behavior of the associated solution paths and, in particular, prove finite extinction time for a large class of functionals. Depending on the powers, we also show that the solution paths are of bounded variation or even Lipschitz continuous. In addition, it will turn out that the models are" almost" mutually equivalent in terms of the minimizers they admit. Finally, we apply our results to define and compare two different non-linear spectral representations of data and show that only one of it is able to decompose a linear combination of non-linear eigenfunctions into the individual eigenfunctions. For that purpose, we will also briefly address piecewise affine solution paths.<br

Structural analysis of an L-infinity variational problem and relations to distance functions - Bungert, Leon; Korolev, Yury; Burger, Martin (Pure and Applied Analysis, 2020)
We analyse the functional J(u) = ∥∇u∥∞ defined on Lipschitz functions with homogeneous Dirichlet boundary conditions. Our analysis is performed directly on the functional without the need to approximate with smooth p-norms. We prove that its ground states coincide with multiples of the distance function to the boundary of the domain. Furthermore, we compute the L2-subdifferential of J and characterize the distance function as the unique nonnegative eigenfunction of the subdifferential operator. We also study properties of general eigenfunctions, in particular their nodal sets. Furthermore, we prove that the distance function can be computed as the asymptotic profile of the gradient flow of J and construct analytic solutions of fast marching type. In addition, we give a geometric characterization of the extreme points of the unit ball of J. Finally, we transfer many of these results to a discrete version of the functional defined on a finite weighted graph. Here, we analyze properties of distance functions on graphs and their gradients. The main difference between the continuum and discrete setting is that the distance function is not the unique nonnegative eigenfunction on a grap

The Infinity Laplacian Eigenvalue Problem: Reformulation and a Numerical Scheme - Bozorgnia, Farid; Bungert, Leon; Tenbrinck, Daniel (Journal of Scientific Computing, 2024)
In this work, we present an alternative formulation of the higher eigenvalue problem associated to the infinity Laplacian, which opens the door for numerical approximation of eigenfunctions. A rigorous analysis is performed to show the equivalence of the new formulation to the traditional one. Subsequently, we present consistent monotone schemes to approximate infinity ground states and higher eigenfunctions on grids. We prove that our method converges (up to a subsequence) to a viscosity solution of the eigenvalue problem, and perform numerical experiments which investigate theoretical conjectures and compute eigenfunctions on a variety of different domain
Keywords: Infinity Laplace operator; Eigenvalue problems; PDE; Numerical scheme; Monotone scheme

Variational regularisation for inverse problems with imperfect forward operators and general noise models - Bungert, Leon; Burger, Martin; Korolev, Yury; Schönlieb, Carola Bibiane (Inverse Problems, 2020)
We study variational regularisation methods for inverse problems with imperfect forward operators whose errors can be modelled by order intervals in a partial order of a Banach lattice. We carry out analysis with respect to existence and convex duality for general data fidelity terms and regularisation functionals. Both for a priori and a posteriori parameter choice rules, we obtain convergence rates of the regularised solutions in terms of Bregman distances. Our results apply to fidelity terms such as Wasserstein distances, -divergences, norms, as well as sums and infimal convolutions of those.<br
CRIS-Team WoS Importer:2020-12-18