Artificial intelligence trend analysis in German business and politics: a web mining approach - Dumbach, Philipp; Schwinn, Leo; Löhr, Tim; Elsberger, Tassilo; Eskofier, Björn (International Journal of Data Science and Analytics, 2023)
Current research on trend detection in artificial intelligence (AI) mainly concerns academic data sources and industrial applications of AI. However, we argue that industrial trends are influenced by public perception and political decisions (e.g., through industry subsidies and grants) and should be reflected in political data sources. To investigate this hypothesis, we examine the AI trend development in German business and politics from 1998 to 2020. Therefore, we propose a web mining approach to collect a novel data set consisting of business and political data sources combining 1.07 million articles and documents. We identify 246 AI-related buzzwords extracted from various glossaries. We use them to conduct an extensive trend detection and analysis study on the collected data using machine learning-based approaches. This study successfully detects an AI trend and follows its evolution in business and political data sources over the past two decades. Moreover, we find a faster adoption of AI in business than in politics, with a considerable increase in policy discourse in recent years. Finally, we show that the collected data can be used for trend detection besides AI-related topics using topic clustering and the COVID-19 pandemic as examples.<br
Keywords: Artificial intelligence; Web mining; Trend detection; Topic clustering; Business; Politics

Artificial intelligence trend analysis on healthcare podcasts using topic modeling and sentiment analysis: a data-driven approach - Dumbach, Philipp; Schwinn, Leo; Löhr, Tim; Do, Phi Long; Eskofier, Björn (Evolutionary Intelligence, 2023)
Over the past few decades, the topic of artificial intelligence (AI) has gained considerable attention in both research and industry. In particular, the healthcare sector has witnessed a surge in the use of AI applications, as the maturity of these methods increased. However, as the use of machine learning (ML) in healthcare continues to grow, we believe it will become increasingly important to examine public perceptions of this trend to identify potential impediments and future directions. Current work focuses mainly on academic data sources and industrial applications of AI. However, to gain a comprehensive understanding of the increased societal interest in AI, digital media such as podcasts should be consulted, as they are accessible to a broader audience. In order to examine this hypothesis, we investigate the AI trend development in healthcare from 2015 until 2021. In this study, we propose a web mining approach to collect a novel data set consisting of 29 healthcare podcasts with 3449 episodes. We identify 102 AI-related buzzwords that were extracted from various glossaries and hype cycles. These buzzwords were used to conduct an extensive trend detection and analysis study on the collected data using machine learning-based approaches. We successfully detect an AI trend and follow its evolution in healthcare podcasts over several years. Besides the focus area of AI, we are able to detect 14 topic clusters and visualize the trending or decreasing dominant topics over the whole period under consideration. In addition, we analyze the sentiments in podcasts towards the identified topics and deliver further insights for trend detection in healthcare. Finally, the collected data set can be used for trend detection besides AI-related topics using topic clustering.<br
Keywords: Artificial intelligence; Trend analysis; Healthcare podcast; Topic clustering; Sentiment analysis; Web mining

CLIP: Cheap Lipschitz Training of Neural Networks - Bungert, Leon; Raab, René; Roith, Tim; Schwinn, Leo; Tenbrinck, Daniel (SSVM 2021: Scale Space and Variational Methods in Computer Vision, 2021)
Despite the large success of deep neural networks (DNN) in recent years, most neural networks still lack mathematical guarantees in terms of stability. For instance, DNNs are vulnerable to small or even imperceptible input perturbations, so called adversarial examples, that can cause false predictions. This instability can have severe consequences in applications which influence the health and safety of humans, e.g., biomedical imaging or autonomous driving. While bounding the Lipschitz constant of a neural network improves stability, most methods rely on restricting the Lipschitz constants of each layer which gives a poor bound for the actual Lipschitz constant. In this paper we investigate a variational regularization method named CLIP for controlling the Lipschitz constant of a neural network, which can easily be integrated into the training procedure. We mathematically analyze the proposed model, in particular discussing the impact of the chosen regularization parameter on the output of the network. Finally, we numerically evaluate our method on both a nonlinear regression problem and the MNIST and Fashion-MNIST classification databases, and compare our results with a weight regularization approac

Dynamically Sampled Nonlocal Gradients for Stronger Adversarial Attacks - Schwinn, Leo; Nguyen, An; Raab, René; Zanca, Dario; Eskofier, Björn; Tenbrinck, Daniel; Burger, Martin (International Joint Conference on Neural Networks (IJCNN), 2021)
The vulnerability of deep neural networks to small and even imperceptible perturbations has become a central topic in deep learning research. Although several sophisticated defense mechanisms have been introduced, most were later shown to be ineffective. However, a reliable evaluation of model robustness is mandatory for deployment in safety-critical scenarios. To overcome this problem we propose a simple yet effective modification to the gradient calculation of state-of-the-art first-order adversarial attacks. Normally, the gradient update of an attack is directly calculated for the given data point. This approach is sensitive to noise and small local optima of the loss function. Inspired by gradient sampling techniques from non-convex optimization, we propose Dynamically Sampled Nonlocal Gradient Descent (DSNGD). DSNGD calculates the gradient direction of the adversarial attack as the weighted average over past gradients of the optimization history. Moreover, distribution hyperparameters that define the sampling operation are automatically learned during the optimization scheme. We empirically show that by incorporating this nonlocal gradient information, we are able to give a more accurate estimation of the global descent direction on noisy and non-convex loss surfaces. In addition, we show that DSNGD-based attacks are on average 35% faster while achieving 0.9% to 27.1% higher success rates compared to their gradient descent-based counterpart

Exploring Misclassifications of Robust Neural Networks to Enhance Adversarial Attacks - Schwinn, Leo; Raab, René; Nguyen, An; Zanca, Dario; Eskofier, Björn (2021)

Exploring misclassifications of robust neural networks to enhance adversarial attacks - Schwinn, Leo; Raab, René; Nguyen, An; Zanca, Dario; Eskofier, Björn (Applied Intelligence, 2023)
Progress in making neural networks more robust against adversarial attacks is mostly marginal, despite the great efforts of the research community. Moreover, the robustness evaluation is often imprecise, making it challenging to identify promising approaches. We do an observational study on the classification decisions of 19 different state-of-the-art neural networks trained to be robust against adversarial attacks. This analysis gives a new indication of the limits of the robustness of current models on a common benchmark. In addition, our findings suggest that current untargeted adversarial attacks induce misclassification toward only a limited amount of different classes. Similarly, we find that previous attacks under-explore the perturbation space during optimization. This leads to unsuccessful attacks for samples where the initial gradient direction is not a good approximation of the final adversarial perturbation direction. Additionally, we observe that both over- and under-confidence in model predictions result in an inaccurate assessment of model robustness. Based on these observations, we propose a novel loss function for adversarial attacks that consistently improves their efficiency and success rate compared to prior attacks for all 30 analyzed models.
CRIS-Team Scopus Importer:2023-03-31
Keywords: Adversarial attacks; Computer vision; Deep learning; Robustness

Exploring the interrelationship between the skin microbiome and skin volatiles: A pilot study - Härtl, Tobias; Owsienko, Diana; Schwinn, Leo; Hirsch, Cathrin; Eskofier, Björn; Lang, Roland; Wirtz, Stefan; Loos, Helene (Frontiers in Ecology and Evolution, 2023)
Unravelling the interplay between a human’s microbiome and physiology is a relevant task for understanding the principles underlying human health and disease. With regard to human chemical communication, it is of interest to elucidate the role of the microbiome in shaping or generating volatiles emitted from the human body. In this study, we characterized the microbiome and volatile organic compounds (VOCs) sampled from the neck and axilla of ten participants (five male, five female) on two sampling days, by applying different methodological approaches. Volatiles emitted from the respective skin site were collected for 20 min using textile sampling material and analyzed on two analytical columns with varying polarity of the stationary phase. Microbiome samples were analyzed by a culture approach coupled with MALDI-TOF-MS analysis and a 16S ribosomal RNA gene (16S RNA) sequencing approach. Statistical and advanced data analysis methods revealed that classification of body sites was possible by using VOC and microbiome data sets. Higher classification accuracy was achieved by combination of both data pools. Cutibacterium, Staphylococcus, Micrococcus, Streptococcus, Lawsonella, Anaerococcus, and Corynebacterium species were found to contribute to classification of the body sites by the microbiome. Alkanes, esters, ethers, ketones, aldehydes and cyclic structures were used by the classifier when VOC data were considered. The interdisciplinary methodological platform developed here will enable further investigations of skin microbiome and skin VOCs alterations in physiological and pathological conditions.
CRIS-Team Scopus Importer:2023-02-17
Keywords: axilla; body odor; chemical communication; gas chromatography–mass spectrometry; multi-omic analyses; neck

FastAMI - A Monte Carlo Approach to the Adjustment for Chance in Clustering Comparison Metrics - Klede, Kai; Schwinn, Leo; Zanca, Dario; Eskofier, Björn (Proceedings of the AAAI Conference on Artificial Intelligence, 37(7), 2023)
Clustering is at the very core of machine learning, and its applications proliferate with the increasing availability of data. However, as datasets grow, comparing clusterings with an adjustment for chance becomes computationally difficult, preventing unbiased ground-truth comparisons and solution selection. We propose FastAMI, a Monte Carlo-based method to efficiently approximate the Adjusted Mutual Information (AMI) and extend it to the Standardized Mutual Information (SMI). The approach is compared with the exact calculation and a recently developed variant of the AMI based on pairwise permutations, using both synthetic and real data. In contrast to the exact calculation our method is fast enough to enable these adjusted information-theoretic comparisons for large datasets while maintaining considerably more accurate results than the pairwise approach.
Keywords: Artificial Intelligence; Machine Learning; Clustering Comparison

Identifying untrustworthy predictions in neural networks by geometric gradient analysis - Schwinn, Leo; Nguyen, An; Raab, René; Bungert, Leon; Tenbrinck, Daniel; Zanca, Dario; Burger, Martin; Eskofier, Björn (Conference on Uncertainty in Artificial Intelligence (UAI), 2021)
The susceptibility of deep neural networks to untrustworthy predictions, including out-of-distribution (OOD) data and adversarial examples, still prevent their widespread use in safety-critical applications. Most existing methods either require a retraining of a given model to achieve robust identification of adversarial attacks or are limited to out-of-distribution sample detection only. In this work, we propose a geometric gradient analysis (GGA) to improve the identification of untrustworthy predictions without retraining of a given model. GGA analyzes the geometry of the loss landscape of neural networks based on the saliency maps of their respective input. We observe considerable differences between the input gradient geometry of trustworthy and untrustworthy predictions. Using these differences, GGA outperforms prior approaches in detecting OOD data and adversarial attacks, including state-of-the-art and adaptive attacks. Code is available at: https://github.com/mad-lab-fau/GGA-Identifying-Untrustworthy-Predictions

Impact of Pathologies on Automatic Age Estimation - Schwinn, Leo; Haderlein, Tino; Nöth, Elmar; Maier, Andreas (Fortschritte der Akustik - DAGA 2019, 2019)
Automatic Age estimation using speech is a challenging problem. Among other things, the many influences that lead to a change in the voice make it difficult to estimate the exact age. Examples are the microphone used, the distance to this microphone or the sex of the speaker. Stable results can be achieved by extracting Mel-Frequency Cepstral Coefficients (MFCCs) from the speech signal and processing them into i‑vectors, classification and regression tools such as Support Vector Regression (SVR) can then be used for the age estimation from these features. An additional factor influencing the age estimation are speech or voice disorders of a speaker. Rarely is this impact assessed, resulting in systems that are not tailored to the needs of pathological speakers like Siri or Amazon echo. Another example are companies which use automatic age estimation to forward calls to persons of the same age as the caller. These systems are also adapted to the characteristics of healthy speakers. This paper examines the impacts of such pathologies on age estimation and assess the possibility of reducing their influence by using the Word Accuracy (WA) of the speakers. This measure gives information about the intelligibility of a speaker and should help to reduce the variance of the extracted features. The features with low variance should provide more stable results in the age estimation. To achieve this, we compare the results of an age estimation for four different groups of speakers. All speakers at once, only pathological speakers, only healthy speakers and training the SVR with healthy speakers while testing with pathological speakers. Each of these groups are further separated by the WA of the speaker
UnivIS-Import:2019-08-15:Pub.2019.tech.IMMD.IMMD5.impact
Keywords: age estimation; speech recognition; speaker characteristics

Improving Robustness against Real-World and Worst-Case Distribution Shifts through Decision Region Quantification - Schwinn, Leo; Bungert, Leon; Nguyen, An; Raab, René; Pulsmeyer, Falk; Precup, Doina; Eskofier, Björn; Zanca, Dario (162, 2022)
The reliability of neural networks is essential for their use in safety-critical applications. Existing approaches generally aim at improving the robustness of neural networks to either real-world distribution shifts (e.g., common corruptions and perturbations, spatial transformations, and natural adversarial examples) or worst-case distribution shifts (e.g., optimized adversarial examples). In this work, we propose the Decision Region Quantification (DRQ) algorithm to improve the robustness of any differentiable pre-trained model against both real-world and worst-case distribution shifts in the data. DRQ analyzes the robustness of local decision regions in the vicinity of a given data point to make more reliable predictions. We theoretically motivate the DRQ algorithm by showing that it effectively smooths spurious local extrema in the decision surface. Furthermore, we propose an implementation using targeted and untargeted adversarial attacks. An extensive empirical evaluation shows that DRQ increases the robustness of adversarially and non-adversarially trained models against real-world and worst-case distribution shifts on several computer vision benchmark datasets.<br
Keywords: Deep Learning; Robustness; Adversarial examples

Just a Matter of Scale? Reevaluating Scale Equivariance in Convolutional Neural Networks - Altstidl, Thomas Robert; Nguyen, An; Schwinn, Leo; Köferl, Franz; Mutschler, Christopher; Eskofier, Björn; Zanca, Dario (Proc. Intl. Joint Conf. Neural Netw. (IJCNN), 2023)
The widespread success of convolutional neural networks may largely be attributed to their intrinsic property of translation equivariance. However, convolutions are not equivariant to variations in scale and fail to generalize to objects of different sizes. Despite recent advances in this field, it remains unclear how well current methods generalize to unobserved scales on real-world data and to what extent scale equivariance plays a role. To address this, we propose the novel Scaled and Translated Image Recognition (STIR) benchmark based on four different domains. Additionally, we introduce a new family of models that applies many re-scaled kernels with shared weights in parallel and then selects the most appropriate one. Our experimental results on STIR show that both the existing and proposed approaches can improve generalization across scales compared to standard convolutions. We also demonstrate that our family of models is able to generalize well towards larger scales and improve scale equivariance. Moreover, due to their unique design we can validate that kernel selection is consistent with input scale. Even so, none of the evaluated models maintain their performance for large differences in scale, demonstrating that a general understanding of how scale equivariance can improve generalization and robustness is still lacking.<br
Keywords: Image recognition; Neural networks; Benchmark testing; Robustness; Convolutional neural networks; Kernel; Standards

Large-Scale Dataset Pruning in Adversarial Training through Data Importance Extrapolation - Nieth, Björn; Altstidl, Thomas Robert; Schwinn, Leo; Eskofier, Björn (International Conference on Machine Learning 2024 DMLR Workshop: Data-centric Machine Learning Research, 2024)
Their vulnerability to small, imperceptible attacks limits the adoption of deep learning models to real-world systems. Adversarial training has proven to be one of the most promising strategies against these attacks, at the expense of a substantial increase in training time. With the ongoing trend of integrating large-scale synthetic data this is only expected to increase even further. Thus, the need for data-centric approaches that reduce the number of training samples while maintaining accuracy and robustness arises. While data pruning and active learning are prominent research topics in deep learning, they are as of now largely unexplored in the adversarial training literature. We address this gap and propose a new data pruning strategy based on extrapolating data importance scores from a small set of data to a larger set. In an empirical evaluation, we demonstrate that extrapolation-based pruning can efficiently reduce dataset size while maintaining robustnes
Keywords: Machine Learning; Adversarial Attacks; Data Selection

Sensory evaluation of axillary odour samples of younger and older adults by a trained panel - Owsienko, Diana; Schwinn, Leo; Eskofier, Björn; Kiesswetter, Eva; Loos, Helene (Flavour and Fragrance Journal, 2024)
It has been reported that a distinct ‘old person smell’ can develop with advancing age, however, this odour has not yet been sufficiently described in previous research. Sensory evaluation by a trained panel might be useful to describe alterations with age in body odour (BO). To evaluate the alterations and achieve first insights into the ‘old person smell’, this pilot study determined the odour profiles of BO samples from both a younger and an older age group with a trained panel. In addition, we aimed to assess whether the panellists can recognize the age group based on the smell of the BO samples. Eight younger (20–28 years) and eight older (80–83 years) participants sampled their BO by wearing a cotton T-shirt for one night. The samples were sensorially evaluated by a trained panel, including ratings of total intensity and pleasantness. Additionally, an age labelling task was performed as a forced-choice decision. Results revealed that the odour profiles of the BO samples were very similar for both age groups. Nevertheless, trained panellists were able to predict the age group with significantly higher accuracy (p =.042) than expected by chance (61% mean accuracy over all panellists). Furthermore, a linear support vector machine (SVM) classifier achieved an average accuracy of 69%. This finding indicates that the age of a person affects the BO, though it is not reflected in significantly distinct odour profiles.
CRIS-Team Scopus Importer:2023-12-22
Keywords: ageing; body odour; odour profile; old person smell

System Design for a Data-Driven and Explainable Customer Sentiment Monitor Using IoT and Enterprise Data - Nguyen, An; Foerstel, Stefan; Kittler, Thomas; Kurzyukov, Andrey; Schwinn, Leo; Zanca, Dario; Hipp, Tobias; Jun, Sun Da; Schrapp, Michael; Rothgang, Eva; Eskofier, Björn (IEEE Access, 2021)
The most important goal of customer service is to keep the customer satisfied. However, service resources are always limited and must prioritize specific customers. Therefore, it is essential to identify customers who potentially become unsatisfied and might lead to escalations. Data science on IoT data (especially log data) for machine health monitoring and analytics on enterprise data for customer relationship management (CRM) have mainly been researched and applied independently. This paper presents a data-driven decision support system framework that combines IoT and enterprise data to model customer sentiment and predicts escalations. The proposed framework includes a fully automated and interpretable machine learning pipeline using state-of-the-art methods. The framework is applied in a real-world case study with a major medical device manufacturer providing data from a fleet of thousands of high-end medical devices. An anonymized version of this industrial benchmark is released for the research community based on the presented case study, which has interesting and challenging properties. In our extensive experiments, we achieve a Recall@50 of 50.0% for the task of predicting customer escalations. In addition, we show that combining IoT and enterprise data can improve prediction results and ease troubleshooting. Additionally, we propose a practical workflow for end-users when applying the proposed framework.
CRIS-Team Scopus Importer:2021-09-10
Keywords: big data; customer service; Data models; decision support system; Decision support systems; explainable AI; Feature extraction; industrial AI; IoT data; Machine learning; machine learning; Medical devices; Monitoring; Predictive models

Time matters: Time-aware LSTMs for predictive business process monitoring - Nguyen, An; Chatterjee, Srijeet; Weinzierl, Sven; Schwinn, Leo; Matzner, Martin; Eskofier, Björn (Proceedings of the ICPM 2020 International Workshops, 2020)
Predictive business process monitoring (PBPM) aims to predict future process behavior during ongoing process executions based on event log data. Especially, techniques for the next activity and timestamp prediction can help to improve the performance of operational business processes. Recently, many PBPM solutions based on deep learning were proposed by researchers. Due to the sequential nature of event log data, a common choice is to apply recurrent neural networks with long short-term memory (LSTM) cells. We argue, that the elapsed time between events is informative. However, current PBPM techniques mainly use “vanilla” LSTM cells and hand-crafted time-related control flow features. To better model the time dependencies between events, we propose a new PBPM technique based on time-aware LSTM (T-LSTM) cells. T-LSTM cells incorporate the elapsed time between consecutive events inherently to adjust the cell memory. Furthermore, we introduce costsensitive learning to account for the common class imbalance in event logs. Our experiments on publicly available benchmark event logs indicate the effectiveness of the introduced techniques.<br
Keywords: Predictive Business Process Monitoring; Deep Learning; Recurrent Neural Network; LSTM; Time-Awareness

xLength: Predicting Expected Ski Jump Length Shortly after Take-Off Using Deep Learning - Link, Johannes; Schwinn, Leo; Pulsmeyer, Falk; Kautz, Thomas; Eskofier, Björn (Sensors, 2022)
With tracking systems becoming more widespread in sports research and regular training and competitions, more data are available for sports analytics and performance prediction. We analyzed 2523 ski jumps from 205 athletes on five venues. For every jump, the dataset includes the 3D trajectory, 3D velocity, skis’ orientation, and metadata such as wind, starting gate, and ski jumping hill data. Using this dataset, we aimed to predict the expected jump length (xLength) inspired by the expected goals metric in soccer (xG). We evaluate the performance of a fully connected neural network, a convolutional neural network (CNN), a long short-term memory (LSTM), and a ResNet architecture to estimate the xLength. For the prediction of the jump length one second after take-off, we achieve a mean absolute error (MAE) of 5.3 (Formula presented.) for the generalization to new athletes and an MAE of 5.9 (Formula presented.) for the generalization to new ski jumping hills using ResNet architectures. Additionally, we investigated the influence of the input time after the take-off on the predictions’ accuracy. As expected, the MAE becomes smaller with longer inputs. Due to the real-time transmission of the sensor’s data, xLength can be updated during the flight phase and used in live TV broadcasting. xLength could also be used as an analysis tool for experts to quantify the quality of the take-off and flight phases.
CRIS-Team Scopus Importer:2022-11-25
Keywords: inertial measurement unit; performance analysis; performance prediction; sports analytics; ultra-wideband; wearable sensors