Active Learning of Ordinal Embeddings: A User Study on Football Data - Löffler, Christoffer; Fallah, Kion; Fenu, Stefano; Zanca, Dario; Eskofier, Björn; Rozell, Christopher John; Mutschler, Christopher (Transactions on Machine Learning Research, 2023)
Humans innately measure distance between instances in an unlabeled dataset using an unknown similarity function. Distance metrics can only serve as proxy for similarity in information retrieval of similar instances. Learning a good similarity function from human annotations improves the quality of retrievals. This work uses deep metric learning to learn these user-defined similarity functions from few annotations for a large football trajectory dataset. We adapt an entropy-based active learning method with recent work from triplet mining to collect easy-to-answer but still informative annotations from human participants and use them to train a deep convolutional network that generalizes to unseen samples. Our user study shows that our approach improves the quality of the information retrieval compared to a previous deep metric learning approach that relies on a Siamese network. Specifically, we shed light on the strengths and weaknesses of passive sampling heuristics and active learners alike by analyzing the participants' response efficacy. To this end, we collect accuracy, algorithmic time complexity, the participants' fatigue and time-to-response, qualitative self-assessment and statements, as well as the effects of mixed-expertise annotators and their consistency on model performance and transfer-learning.
Keywords: Active Learning; Deep Metric Learning; Ordinal Embedding

Advanced Analysis of Electroretinograms Based on Wavelet Scalogram Processing - Zhdanov, Aleksei; Dolganov, Anton; Zanca, Dario; Borisov, Vasilii; Ronkin, Mikhail (Applied Sciences, 2022)
Featured Application: The results of this work will be used to develop a system to assist ophthalmologist doctor decisions. The electroretinography (ERG) is a diagnostic test that measures the electrical activity of the retina in response to a light stimulus. The current ERG signal analysis uses four components, namely amplitude, and the latency of a-wave and b-wave. Nowadays, the international electrophysiology community established the standard for electroretinography in 2008. However, in terms of signal analysis, there were no major changes. ERG analysis is still based on a four-component evaluation. The article describes the ERG database, including the classification of signals via the advanced analysis of electroretinograms based on wavelet scalogram processing. To implement an extended analysis of the ERG, the parameters extracted from the wavelet scalogram of the signal were obtained using digital image processing and machine learning methods. Specifically, the study focused on the preprocessing of wavelet scalogram as images, and the extraction of connected components and thier evaluation. As a machine learning method, a decision tree was selected as one that incorporated feature selection. The study results show that the proposed algorithm more accurately implements the classification of adult electroretinogram signals by 19%, and pediatric signals by 20%, in comparison with the classical features of ERG. The promising use of ERG is presented using differential diagnostics, which may also be used in preclinical toxicology and experimental modeling. The problem of developing methods for electrophysiological signals analysis in ophthalmology is associated with the complex morphological structures of electrophysiological signal components.
CRIS-Team Scopus Importer:2022-12-23
Keywords: biomedical research; classification; electroretinogram; electroretinography; ERG; machine learning

An Integrated Framework for Data Quality Fusion in Embedded Sensor Systems - Scholl, Christoph; Spiegler, Maximilian; Ludwig, Klaus; Eskofier, Björn; Tobola, Andreas; Zanca, Dario (Sensors, 2023)
The advancement of embedded sensor systems allowed the monitoring of complex processes based on connected devices. As more and more data are produced by these sensor systems, and as the data are used in increasingly vital areas of applications, it is of growing importance to also track the data quality of these systems. We propose a framework to fuse sensor data streams and associated data quality attributes into a single meaningful and interpretable value that represents the current underlying data quality. Based on the definition of data quality attributes and metrics to determine real-valued figures representing the quality of the attributes, the fusion algorithms are engineered. Methods based on maximum likelihood estimation (MLE) and fuzzy logic are used to perform data quality fusion by utilizing domain knowledge and sensor measurements. Two data sets are used to verify the proposed fusion framework. First, the methods are applied to a proprietary data set targeting sample rate inaccuracies of a micro-electro-mechanical system (MEMS) accelerometer and second, to the publicly available Intel Lab Data set. The algorithms are verified against their expected behavior based on data exploration and correlation analysis. We prove that both fusion approaches are capable of detecting data quality issues and providing an interpretable data quality indicator.
CRIS-Team WoS Importer:2023-05-26

A smart capacitive sensor skin with embedded data quality indication for enhanced safety in human–robot interaction - Scholl, Christoph; Tobola, Andreas; Ludwig, Klaus; Zanca, Dario; Eskofier, Björn (Sensors, 2021)
Smart sensors are an integral part of the Fourth Industrial Revolution and are widely used to add safety measures to human–robot interaction applications. With the advancement of machine learning methods in resource-constrained environments, smart sensor systems have become increasingly powerful. As more data-driven approaches are deployed on the sensors, it is of growing importance to monitor data quality at all times of system operation. We introduce a smart capacitive sensor system with an embedded data quality monitoring algorithm to enhance the safety of human– robot interaction scenarios. The smart capacitive skin sensor is capable of detecting the distance and angle of objects nearby by utilizing consumer-grade sensor electronics. To further acknowledge the safety aspect of the sensor, a dedicated layer to monitor data quality in real-time is added to the embedded software of the sensor. Two learning algorithms are used to implement the sensor functionality: (1) a fully connected neural network to infer the position and angle of objects nearby and (2) a one-class SVM to account for the data quality assessment based on out-of-distribution detection. We show that the sensor performs well under normal operating conditions within a range of 200 mm and also detects abnormal operating conditions in terms of poor data quality successfully. A mean absolute distance error of 11.6 mm was achieved without data quality indication. The overall performance of the sensor system could be further improved to 7.5 mm by monitoring the data quality, adding an additional layer of safety for human–robot interaction.
CRIS-Team Scopus Importer:2021-11-05
Keywords: Capacitive sensor system; Data quality; Embedded AI; Human–robot interaction; Signal processing; Smart sensors

Assessing the Performance of Remaining Time Prediction Methods for Business Processes - Roider, Johannes; Nguyen, An; Zanca, Dario; Eskofier, Björn (IEEE Access, 2024)
The prediction of the remaining time for business processes is a major task in predictive process monitoring (PPM). In the last years, various machine learning methods were introduced which reduced error levels steadily. However, the commonly applied metric for optimization and evaluation, the Mean Absolute Error (MAE), has limitations regarding its interpretability. In this work we introduce and evaluate the normalized Mean Absolute Error (nMAE) as an interpretable metric for model evaluation. It accounts for different kinds of label shifts, which are a special type of concept drift that can distort remaining time results. We investigate these concepts in a thorough benchmark study and use them to assess the current state of remaining time prediction for business processes. This includes the evaluation of four different baseline models, identifying the most accurate one. Furthermore, our study compares three different state-of-the-art methods, namely XGBoost, DA-LSTM, and PGT-Net. In contrary to prior studies we find that there is no significant difference in the performance between these models. Additionally, using the nMAE as evaluation metric we find that these models do not perform reasonably well on a range of event logs. Initial ideas for this behaviour are discussed and consolidated along with other findings from the case study into a comprehensive list motivating future research direction
Keywords: Business Process; Graph Neural Network; LSTM; Machine Learning; Predictive Process Monitoring; Process Mining; Remaining Time; XGBoost

Ataxia with oculomotor apraxia type 2 (AOA2): an eye movement study of two siblings - Bargagli, Alessia; Rosini, Francesca; Zanca, Dario; Serchi, Valeria; Rufa, Alessandra (Neurological Sciences, 2021)

A unified computational framework for visual attention dynamics - Zanca, Dario; Gori, Marco; Rufa, Alessandra (Elsevier B.V., 2019)
Keywords: Visual features; Scanpath; Visual attention; Convolutional neural networks; Saliency; Principle of least action

Automated long-term monitoring of stereotypical movement in polar bears under human care using machine learning - Zürl, Matthias; Stoll, Philip; Brehm, Ingrid; Sueskind, Jonas; Raab, René; Petermann, Jan; Zanca, Dario; Simon, Ralph; von Fersen, Lorenzo; Eskofier, Björn (Ecological Informatics, 2024)
The welfare of animals under human care is often assessed by observing behaviours indicative of stress or discomfort, such as stereotypical behaviour (SB), which often shows as repetitive, invariant pacing. Traditional behaviour monitoring methods, however, are labour-intensive and subject to observer bias. Our study presents an innovative automated approach utilising computer vision and machine learning to non-invasively detect and analyse SB in managed populations, exemplified by a longitudinal study of two polar bears. We designed an animal tracking framework to localise and identify individual animals in the enclosure. After determining their position on the enclosure map via homographic transformation, we refined the resulting trajectories using a particle filter. Finally, we classified the trajectory patterns as SB or normal behaviour using a lightweight random forest approach with an accuracy of 94.9 %. The system not only allows for continuous, objective monitoring of animal behaviours but also provides insights into seasonal variations in SB, illustrating its potential for improving animal welfare in zoological settings. Ultimately, we analysed 607 days for the occurrence of SB, allowing us to discuss seasonal patterns of SB in both the male and female polar bear monitored. This work advances the field of animal welfare research by introducing a scalable, efficient method for the long-term, automated detection and monitoring of stereotypical behaviour, paving the way for its application across various settings and species that can be continuously monitored with cameras. We made the code publicly available at<br
Keywords: Animal welfare; Animal tracking; Behaviour classification; Deep learning; Computer vision; Coping; Machine learning; Pacing; Stereotypical behaviour

Automated Video-Based Analysis Framework for Behavior Monitoring of Individual Animals in Zoos Using Deep Learning—A Study on Polar Bears - Zürl, Matthias; Stoll, Philip; Brehm, Ingrid; Raab, René; Zanca, Dario; Kabri, Samira; Happold, Johanna; Nille, Heiko; Prechtel, Katharina; Wünsch, Sophie; Krause, Marie; Seegerer, Stefan; et al. (Animals, 2022)
The monitoring of animals under human care is a crucial tool for biologists and zookeepers to keep track of the animals’ physical and psychological health. Additionally, it enables the analysis of observed behavioral changes and helps to unravel underlying reasons. Enhancing our understanding of animals ensures and improves ex situ animal welfare as well as in situ conservation. However, traditional observation methods are time- and labor-intensive, as they require experts to observe the animals on-site during long and repeated sessions and manually score their behavior. Therefore, the development of automated observation systems would greatly benefit researchers and practitioners in this domain. We propose an automated framework for basic behavior monitoring of individual animals under human care. Raw video data are processed to continuously determine the position of the individuals within the enclosure. The trajectories describing their travel patterns are presented, along with fundamental analysis, through a graphical user interface (GUI). We evaluate the performance of the framework on captive polar bears (Ursus maritimus). We show that the framework can localize and identify individual polar bears with an F1 score of 86.4%. The localization accuracy of the framework is 19.9±7.6 cm, outperforming current manual observation methods. Furthermore, we provide a bounding-box-labeled dataset of the two polar bears housed in Nuremberg Zoo.<br
Keywords: animal welfare; animal behavior; deep learning; object detection; animal monitoring; behavior observation; Ursus maritimus

Data-Centric Benchmarking of Neural Network Architectures for the Univariate Time Series Forecasting Task - Schlieper, Philipp; Dombrowski, Mischa; Nguyen, An; Zanca, Dario; Eskofier, Björn (Forecasting, 2024)
Time series forecasting has witnessed a rapid proliferation of novel neural network approaches in recent times. However, performances in terms of benchmarking results are generally not consistent, and it is complicated to determine in which cases one approach fits better than another. Therefore, we propose adopting a data-centric perspective for benchmarking neural network architectures on time series forecasting by generating ad hoc synthetic datasets. In particular, we combine sinusoidal functions to synthesize univariate time series data for multi-input-multi-output prediction tasks. We compare the most popular architectures for time series, namely long short-term memory (LSTM) networks, convolutional neural networks (CNNs), and transformers, and directly connect their performance with different controlled data characteristics, such as the sequence length, noise and frequency, and delay length. Our findings suggest that transformers are the best architecture for dealing with different delay lengths. In contrast, for different noise and frequency levels and different sequence lengths, LSTM is the best-performing architecture by a significant amount. Based on our insights, we derive recommendations which allow machine learning (ML) practitioners to decide which architecture to apply, given the dataset’s characteristics.
CRIS-Team Scopus Importer:2024-10-04
Keywords: data synthesis; deep learning; model selection; neural networks; time series; univariate forecasting

Data quality in hospital information systems: Lessons learned from analyzing 30 years of patient data in a regional German hospital - Förstel, Stefan; Förstel, Markus; Gallistl, Markus; Zanca, Dario; Eskofier, Björn; Rothgang, Eva M. (International Journal of Medical Informatics, 2024)
Background: The integration of Hospital Information Systems (HIS) into healthcare delivery has significantly enhanced patient care and operational efficiency. Nonetheless, the rapid acceleration of digital transformation has led to a substantial increase in the volume of data managed by these systems. This emphasizes the need for robust mechanisms for data management and quality assurance. Objective: This study addresses data quality issues related to patient identifiers within the Hospital Information System (HIS) of a regional German hospital, focusing on improving the accuracy and consistency of these administrative data entries. Methods: Employing a combination of data analysis and expert interviews, this study reviews and programmatically cleanses a dataset with over 2,000,000 patient data entries extracted from the HIS. The areas of investigation are patient admissions, discharges, and geographical data. Results: The analysis revealed that roughly 25% of the dataset was rendered unusable by errors and inconsistencies. By implementing a thorough data cleansing process, we significantly enhanced the utility of the dataset. In doing so, we identified the primary issues affecting data quality, including ambiguities among similar variables and a gap between the intended and actual use of the system. Conclusion: The findings highlight the critical importance of enhancing data quality in healthcare information systems. This study shows the necessity of a careful review of data extracted from the HIS before it can be reliably utilized for machine learning tasks, thereby rendering the data more usable for both clinical and analytical purposes.
CRIS-Team Scopus Importer:2024-10-04
Keywords: Data cleansing; Data quality; Digital transformation in healthcare; Healthcare informatics; Hospital information systems

Die Rätselrevolution: Automated German Crossword Solving - Zugarini, Andrea; Röthenbacher, Thomas; Klede, Kai; Ernandes, Marco; Eskofier, Björn; Zanca, Dario (Proceedings of the 9th Italian Conference on Computational Linguistics, 2023)
Crossword puzzles are popular word games played in various languages around the world, with diverse styles across different countries. For this reason, automated crossword solvers designed for a language, may not work well on others. In this paper, we extend Webcrow, an automatic crossword solver, to German, making it the first program for crossword solving in the German language. To address the lack of large clue-answer crossword pairs data, Webcrow combines multiple modules, known as experts, which retrieve potential answers from various resources, including the web, knowledge graphs, and linguistic rules. The system is evaluated on a collection of crosswords from various sources, where it is able to solve perfectly 67% of them. Additional analysis reveals that while our solver achieved commendable results, puzzles with poorly constrained schemas and original clues still presented significant hurdles. These findings shed light on the complexity of the crossword-solving problem and emphasize the need for future research to address and overcome these particular challenges effectivel
Keywords: Crossword Puzzles; Natural Language Processing; Large Language Models; Artificial Intelligence

Discussion and recommendations from the 2023 ”Gaze Meets ML” workshop breakout session - Karargyris, Alexandros; Ahmed, Shuaib; de Belen, Ryan Anthony; Banerjee, Bonny; Ibrayev, Timur; Kashyap, Satyananda; Krupinski, Elizabeth; Kuang, Chenyi; Madu, Paul; Madu, Amarachi Blessing; Makowski, Silvia; Mathew, Athul M.; et al. (Proceedings of The 2nd Gaze Meets ML workshop, 2023)
The Gaze Meets ML (GMML) workshop at NeurIPS aims to bring together diverse machine learning communities to foster research that leverages eye gaze (visual attention) to fulfill synergy between human attention/cognition and machine learning model development and evaluation. Towards this mission, the 2023 GMML workshop ran a breakout session to foster the research community by discussing open challenges. Three focus breakout session areas were identified through a selection process: Datasets, Community, and Vision and Actions for the Future. The findings and discussion points from this session were collected during the meeting and further organized and expanded after the meeting for efficient presentation here. The following sections detail each topic.
CRIS-Team Scopus Importer:2024-12-16
Keywords: eye tracking; gaze; machine learning; reinforcement learning; visual attention

Dynamically Sampled Nonlocal Gradients for Stronger Adversarial Attacks - Schwinn, Leo; Nguyen, An; Raab, René; Zanca, Dario; Eskofier, Björn; Tenbrinck, Daniel; Burger, Martin (International Joint Conference on Neural Networks (IJCNN), 2021)
The vulnerability of deep neural networks to small and even imperceptible perturbations has become a central topic in deep learning research. Although several sophisticated defense mechanisms have been introduced, most were later shown to be ineffective. However, a reliable evaluation of model robustness is mandatory for deployment in safety-critical scenarios. To overcome this problem we propose a simple yet effective modification to the gradient calculation of state-of-the-art first-order adversarial attacks. Normally, the gradient update of an attack is directly calculated for the given data point. This approach is sensitive to noise and small local optima of the loss function. Inspired by gradient sampling techniques from non-convex optimization, we propose Dynamically Sampled Nonlocal Gradient Descent (DSNGD). DSNGD calculates the gradient direction of the adversarial attack as the weighted average over past gradients of the optimization history. Moreover, distribution hyperparameters that define the sampling operation are automatically learned during the optimization scheme. We empirically show that by incorporating this nonlocal gradient information, we are able to give a more accurate estimation of the global descent direction on noisy and non-convex loss surfaces. In addition, we show that DSNGD-based attacks are on average 35% faster while achieving 0.9% to 27.1% higher success rates compared to their gradient descent-based counterpart

Efficient Training of Recurrent Neural Networks for Remaining Time Prediction in Predictive Process Monitoring - Roider, Johannes; Zanca, Dario; Eskofier, Björn (Business Process Management, 22nd International Conference, BPM 2024, Krakow, Poland, September 1–6, 2024, Proceedings, 2024)
An important task in predictive process monitoring is the prediction of the remaining time. The accuracy of methods to solve this task has been steadily improved and verified during the last years. The current state-of-the-art uses Long Short-Term Memory (LSTM) neural networks, represented by the DA-LSTM model. However, training such methods requires substantial amounts of time and memory. This paper addresses specifically these problems. We adjust DA-LSTM and introduce DA-LSTM+ which achieves competitive error levels while reducing the training time significantly. Furthermore, we introduce trace-based sequence encoding as an alternative to prefix encoding, and an approach to use case attributes more efficiently to address time and memory limitations during training. The usage of these is not limited to LSTM’s but they are compatible with any neural network type. We evaluate them together with two alternatives for categorical feature encoding in an extensive benchmark study including eight different model architectures based on DA-LSTM+ and 14 publicly available datasets. The study shows that the training of neural network-based methods can be significantly accelerated with our contributions without affecting model’s performance. Our implementation is memory-efficient and publicly availabl
Keywords: Predictive Process Monitoring; Deep Learning; Process Mining; Remaining Time Prediction

Enhancing unsupervised outlier model selection: A study on ireos algorithms - Schlieper, Philipp; Luft, Hermann; Klede, Kai; Strohmeyer, Christoph; Eskofier, Björn; Zanca, Dario (ACM Transactions on Knowledge Discovery from Data, 2024)
Outlier detection stands as a critical cornerstone in the field of data mining, with a wide range of applications spanning from fraud detection to network security. However, real-world scenarios often lack labeled data for training, necessitating unsupervised outlier detection methods. This study centers on Unsupervised Outlier Model Selection (UOMS), with a specific focus on the family of Internal, Relative Evaluation of Outlier Solutions (IREOS) algorithms. IREOS measures outlier candidate separability by evaluating multiple maximum-margin classifiers and, while effective, it is constrained by its high computational demands. We investigate the impact of several different separation methods in UOMS in terms of ranking quality and runtime. Surprisingly, our findings indicate that different separability measures have minimal impact on IREOS' effectiveness. However, using linear separation methods within IREOS significantly reduces its computation time. These insights hold significance for real-world applications where efficient outlier detection is critical. In the context of this work, we provide the code for the IREOS algorithm and our separability techniques.
CRIS-Team Scopus Importer:2024-07-05
Keywords: anomaly detection; model selection; Outlier detection; unsupervised evaluation

Evaluation of the effectiveness of the decision support algorithm for physicians in retinal dystrophy using machine learning methods - Zhdanov, A. E.; Dolganov, A. Yu; Zanca, Dario; Borisov, V. I.; Luchian, E.; Dorosinsky, L. G. (Computer Optics, 2023)
Electroretinography is a method of electrophysiological testing, which allows diagnosing dis-eases associated with disorders of the vascular structures of the retina. The classical analysis of the electroretinogram is based on assessing four parameters of the amplitude-time representation and often needs to be specified further using alternative diagnostic methods. This study proposes the use of an original physician decision support algorithm for diagnosing retinal dystrophy. The proposed algorithm is based on machine learning methods and uses parameters extracted from the wavelet scalogram of pediatric and adult electroretinogram signals. The study also uses a labeled database of pediatric and adult electroretinogram signals recorded using a computerized electrophysiological workstation EP-1000 (Tomey GmbH) at the IRTC Eye Microsurgery Ekaterinburg Center. The scientific novelty of this study consists in the development of special mathematical and algorithmic software for analyzing a procedure for extracting wavelet scalogram parameters of the electroretinogram signal using the cwt function of the PyWT. The basis function is a Gaussian wavelet of order 8. Also, the scientific novelty includes the development of an algorithm for analyzing electroretinogram signals that implements the classification of adult (pediatric) electro-retinogram signals 19 (20) percent more accurately than classical analysis.
CRIS-Team Scopus Importer:2023-03-03
Keywords: decision support algorithm; decision trees; electrophysiological study; electroretinogram; electroretinography; EPS; ERG; retinal dystrophy; wavelet analysis; wavelet scalogram

Explain to Not Forget: Defending Against Catastrophic Forgetting with XAI - Ede, Sami; Baghdadlian, Serop; Weber, Leander; Nguyen, An; Zanca, Dario; Samek, Wojciech; Lapuschkin, Sebastian (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2022)
The ability to continuously process and retain new information like we do naturally as humans is a feat that is highly sought after when training neural networks. Unfortunately, the traditional optimization algorithms often require large amounts of data available during training time and updates w.r.t. new data are difficult after the training process has been completed. In fact, when new data or tasks arise, previous progress may be lost as neural networks are prone to catastrophic forgetting. Catastrophic forgetting describes the phenomenon when a neural network completely forgets previous knowledge when given new information. We propose a novel training algorithm called Relevance-based Neural Freezing in which we leverage Layer-wise Relevance Propagation in order to retain the information a neural network has already learned in previous tasks when training on new data. The method is evaluated on a range of benchmark datasets as well as more complex data. Our method not only successfully retains the knowledge of old tasks within the neural networks but does so more resource-efficiently than other state-of-the-art solutions.
CRIS-Team Scopus Importer:2022-09-09
Keywords: Catastrophic forgetting; Explainable AI; Layer-wise Relevance Propagation (LRP); Neural network pruning

Exploring Misclassifications of Robust Neural Networks to Enhance Adversarial Attacks - Schwinn, Leo; Raab, René; Nguyen, An; Zanca, Dario; Eskofier, Björn (2021)

Exploring misclassifications of robust neural networks to enhance adversarial attacks - Schwinn, Leo; Raab, René; Nguyen, An; Zanca, Dario; Eskofier, Björn (Applied Intelligence, 2023)
Progress in making neural networks more robust against adversarial attacks is mostly marginal, despite the great efforts of the research community. Moreover, the robustness evaluation is often imprecise, making it challenging to identify promising approaches. We do an observational study on the classification decisions of 19 different state-of-the-art neural networks trained to be robust against adversarial attacks. This analysis gives a new indication of the limits of the robustness of current models on a common benchmark. In addition, our findings suggest that current untargeted adversarial attacks induce misclassification toward only a limited amount of different classes. Similarly, we find that previous attacks under-explore the perturbation space during optimization. This leads to unsuccessful attacks for samples where the initial gradient direction is not a good approximation of the final adversarial perturbation direction. Additionally, we observe that both over- and under-confidence in model predictions result in an inaccurate assessment of model robustness. Based on these observations, we propose a novel loss function for adversarial attacks that consistently improves their efficiency and success rate compared to prior attacks for all 30 analyzed models.
CRIS-Team Scopus Importer:2023-03-31
Keywords: Adversarial attacks; Computer vision; Deep learning; Robustness

FastAMI - A Monte Carlo Approach to the Adjustment for Chance in Clustering Comparison Metrics - Klede, Kai; Schwinn, Leo; Zanca, Dario; Eskofier, Björn (Proceedings of the AAAI Conference on Artificial Intelligence, 37(7), 2023)
Clustering is at the very core of machine learning, and its applications proliferate with the increasing availability of data. However, as datasets grow, comparing clusterings with an adjustment for chance becomes computationally difficult, preventing unbiased ground-truth comparisons and solution selection. We propose FastAMI, a Monte Carlo-based method to efficiently approximate the Adjusted Mutual Information (AMI) and extend it to the Standardized Mutual Information (SMI). The approach is compared with the exact calculation and a recently developed variant of the AMI based on pairwise permutations, using both synthetic and real data. In contrast to the exact calculation our method is fast enough to enable these adjusted information-theoretic comparisons for large datasets while maintaining considerably more accurate results than the pairwise approach.
Keywords: Artificial Intelligence; Machine Learning; Clustering Comparison

FixaTons: A collection of Human Fixations Datasets and Metrics for Scanpath Similarity - Zanca, Dario (2018)
In the last three decades, human visual attention has been a topic of great interest in various disciplines. In computer vision, many models have been proposed to predict the distribution of human fixations on a visual stimulus. Recently, thanks to the creation of large collections of data, machine learning algorithms have obtained state-of-the-art performance on the task of saliency map estimation. On the other hand, computational models of scanpath are much less studied. Works are often only descriptive or task specific. This is due to the fact that the scanpath is harder to model because it must include the description of a dynamic. General purpose computational models are present in the literature, but are then evaluated in tasks of saliency prediction, losing therefore information about the dynamics and the behaviour. In addition, two technical reasons have limited the research. The first reason is the lack of robust and uniformly used set of metrics to compare the similarity between scanpath. The second reason is the lack of sufficiently large and varied scanpath datasets. In this report we want to help in both directions. We present FixaTons, a large collection of datasets human scanpaths (temporally ordered sequences of fixations) and saliency maps. It comes along with a software library for easy data usage, statistics calculation and implementation of metrics for scanpath and saliency prediction evaluation.<br

From patches to objects: exploiting spatial reasoning for better visual representations - Albert, Toni; Eskofier, Björn; Zanca, Dario (Discover Applied Sciences, 2024)
As the field of deep learning steadily transitions from the realm of academic research to practical application, the significance of self-supervised pretraining methods has become increasingly prominent. These methods, particularly in the image domain, offer a compelling strategy to effectively utilize the abundance of unlabeled image data, thereby enhancing downstream tasks’ performance. In this paper, we propose Spatial Reasoning, a novel auxiliary pretraining method that takes advantage of a more flexible formulation of contrastive learning by introducing spatial reasoning as an auxiliary task for discriminative self-supervised methods. Spatial Reasoning works by having the network predict the relative distances between sampled non-overlapping patches. We argue that this forces the network to learn more detailed and intricate internal representations of the objects and the relationships between their constituting parts. Our experiments demonstrate substantial improvement in downstream performance in linear evaluation compared to similar work and provide directions for further research into spatial reasoning.
CRIS-Team Scopus Importer:2024-05-10
Keywords: Contrastive learning; Linear evaluation; Relational reasoning; Self-supervised learning; Spatial reasoning

Gravitational Laws of Focus of Attention - Zanca, Dario; Melacci, Stefano; Gori, Marco (IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020)
The understanding of the mechanisms behind focus of attention in a visual scene is a problem of great interest in visual perception and computer vision. In this paper, we describe a model of scanpath as a dynamic process which can be interpreted as a variational law somehow related to mechanics, where the focus of attention is subject to a gravitational field. The distributed virtual mass that drives eye movements is associated with the presence of details and motion in the video. Unlike most current models, the proposed approach does not estimate directly the saliency map, but the prediction of eye movements allows us to integrate over time the positions of interest. The process of inhibition-of-return is also supported in the same dynamic model with the purpose of simulating fixations and saccades. The differential equations of motion of the proposed model are numerically integrated to simulate scanpaths on both images and videos. Experimental results for the tasks of saliency and scanpath prediction on a wide collection of datasets are presented to support the theory. Top level performances are achieved especially in the prediction of scanpaths, which is the primary purpose of the proposed model.<br
Keywords: Visual attention; eye movements; scanpath prediction; saliency; gravitational laws

Gravitational models explain shifts on human visual attention - Zanca, Dario; Melacci, Stefano; Gori, Marco; Rufa, Alessandra (Scientific Reports, 2020)

HeMoG: A White-Box Model to Unveil the Connection Between Saliency Information and Human Head Motion in Virtual Reality - Rondon, Miguel Fabian Romero; Zanca, Dario; Melacci, Stefano; Gori, Marco; Sassatelli, Lucile (2021 4TH IEEE INTERNATIONAL CONFERENCE ON ARTIFICIAL INTELLIGENCE AND VIRTUAL REALITY (AIVR 2021), 2021)
CRIS-Team WoS Importer:2022-09-09

How Intermodal Interaction Affects the Performance of Deep Multimodal Fusion for Mixed-Type Time Series - Dietz, Simon; Altstidl, Thomas Robert; Zanca, Dario; Eskofier, Björn; Nguyen, An (International Joint Conference on Neural Networks (IJCNN), 2024)
Mixed-type time series (MTTS) is a bimodal data type that is common in many domains, such as healthcare, finance, environmental monitoring, and social media. It consists of regularly sampled continuous time series and irregularly sampled categorical event sequences. The integration of both modalities through multimodal fusion is a promising approach for processing MTTS. However, the question of how to effectively fuse both modalities remains open. In this paper, we present a comprehensive evaluation of several deep multimodal fusion approaches for MTTS forecasting. Our comparison includes different fusion types (early, intermediate, and late) and fusion methods (concatenation, weighted mean, weighted mean with correlation, gating, and feature sharing). We evaluate these fusion approaches on three distinct datasets, one of which was generated using a novel framework. This framework allows for the control of key data properties, such as the strength and direction of intermodal interactions, modality imbalance, and the degree of randomness in each modality, providing a more controlled environment for testing fusion approaches. Our findings show that the performance of different fusion approaches can be substantially influenced by the direction and strength of intermodal interactions. The study reveals that early and intermediate fusion approaches excel at capturing fine-grained and coarse-grained cross-modal features, respectively. These findings underscore the crucial role of intermodal interactions in determining the most effective fusion strategy for MTTS forecasting.<br
Keywords: mixed-type; time series; multimodal; fusion; synthetic data

ICDAR 2021 Competition on On-Line Signature Verification - Tolosana, Ruben; Vera-Rodriguez, Ruben; Gonzalez-Garcia, Carlos; Fierrez, Julian; Rengifo, Santiago; Morales, Aythami; Ortega-Garcia, Javier; Carlos Ruiz-Garcia, Juan; Romero-Tapiador, Sergio; Jiang, Jiajia; Lai, Songxuan; Jin, Lianwen; et al. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2021)
This paper describes the experimental framework and results of the ICDAR 2021 Competition on On-Line Signature Verification (SVC 2021). The goal of SVC 2021 is to evaluate the limits of on-line signature verification systems on popular scenarios (office/mobile) and writing inputs (stylus/finger) through large-scale public databases. Three different tasks are considered in the competition, simulating realistic scenarios as both random and skilled forgeries are simultaneously considered on each task. The results obtained in SVC 2021 prove the high potential of deep learning methods. In particular, the best on-line signature verification system of SVC 2021 obtained Equal Error Rate (EER) values of 3.33% (Task 1), 7.41% (Task 2), and 6.04% (Task 3). SVC 2021 will be established as an on-going competition (https://sites.google.com/view/SVC2021 ), where researchers can easily benchmark their systems against the state of the art in an open common platform using large-scale public databases such as DeepSignDB (https://github.com/BiDAlab/DeepSignDB ) and SVC2021_EvalDB (https://github.com/BiDAlab/SVC2021_EvalDB ), and standard experimental protocols.
CRIS-Team Scopus Importer:2021-10-01
Keywords: Benchmark; Biometrics; Deep learning; DeepSignDB; Handwriting; On-line signature; SVC 2021; SVC2021_EvalDB

Identifying untrustworthy predictions in neural networks by geometric gradient analysis - Schwinn, Leo; Nguyen, An; Raab, René; Bungert, Leon; Tenbrinck, Daniel; Zanca, Dario; Burger, Martin; Eskofier, Björn (Conference on Uncertainty in Artificial Intelligence (UAI), 2021)
The susceptibility of deep neural networks to untrustworthy predictions, including out-of-distribution (OOD) data and adversarial examples, still prevent their widespread use in safety-critical applications. Most existing methods either require a retraining of a given model to achieve robust identification of adversarial attacks or are limited to out-of-distribution sample detection only. In this work, we propose a geometric gradient analysis (GGA) to improve the identification of untrustworthy predictions without retraining of a given model. GGA analyzes the geometry of the loss landscape of neural networks based on the saliency maps of their respective input. We observe considerable differences between the input gradient geometry of trustworthy and untrustworthy predictions. Using these differences, GGA outperforms prior approaches in detecting OOD data and adversarial attacks, including state-of-the-art and adaptive attacks. Code is available at: https://github.com/mad-lab-fau/GGA-Identifying-Untrustworthy-Predictions

Improving Robustness against Real-World and Worst-Case Distribution Shifts through Decision Region Quantification - Schwinn, Leo; Bungert, Leon; Nguyen, An; Raab, René; Pulsmeyer, Falk; Precup, Doina; Eskofier, Björn; Zanca, Dario (162, 2022)
The reliability of neural networks is essential for their use in safety-critical applications. Existing approaches generally aim at improving the robustness of neural networks to either real-world distribution shifts (e.g., common corruptions and perturbations, spatial transformations, and natural adversarial examples) or worst-case distribution shifts (e.g., optimized adversarial examples). In this work, we propose the Decision Region Quantification (DRQ) algorithm to improve the robustness of any differentiable pre-trained model against both real-world and worst-case distribution shifts in the data. DRQ analyzes the robustness of local decision regions in the vicinity of a given data point to make more reliable predictions. We theoretically motivate the DRQ algorithm by showing that it effectively smooths spurious local extrema in the decision surface. Furthermore, we propose an implementation using targeted and untargeted adversarial attacks. An extensive empirical evaluation shows that DRQ increases the robustness of adversarially and non-adversarially trained models against real-world and worst-case distribution shifts on several computer vision benchmark datasets.<br
Keywords: Deep Learning; Robustness; Adversarial examples

Just a Matter of Scale? Reevaluating Scale Equivariance in Convolutional Neural Networks - Altstidl, Thomas Robert; Nguyen, An; Schwinn, Leo; Köferl, Franz; Mutschler, Christopher; Eskofier, Björn; Zanca, Dario (Proc. Intl. Joint Conf. Neural Netw. (IJCNN), 2023)
The widespread success of convolutional neural networks may largely be attributed to their intrinsic property of translation equivariance. However, convolutions are not equivariant to variations in scale and fail to generalize to objects of different sizes. Despite recent advances in this field, it remains unclear how well current methods generalize to unobserved scales on real-world data and to what extent scale equivariance plays a role. To address this, we propose the novel Scaled and Translated Image Recognition (STIR) benchmark based on four different domains. Additionally, we introduce a new family of models that applies many re-scaled kernels with shared weights in parallel and then selects the most appropriate one. Our experimental results on STIR show that both the existing and proposed approaches can improve generalization across scales compared to standard convolutions. We also demonstrate that our family of models is able to generalize well towards larger scales and improve scale equivariance. Moreover, due to their unique design we can validate that kernel selection is consistent with input scale. Even so, none of the evaluated models maintain their performance for large differences in scale, demonstrating that a general understanding of how scale equivariance can improve generalization and robustness is still lacking.<br
Keywords: Image recognition; Neural networks; Benchmark testing; Robustness; Convolutional neural networks; Kernel; Standards

Learning Activation Functions by Means of Kernel Based Neural Networks - Marra, Giuseppe; Zanca, Dario; Betti, Alessandro; Gori, Marco (Springer, 2019)

Local propagation of visual stimuli in focus of attention - Faggi, Lapo; Betti, Alessandro; Zanca, Dario; Melacci, Stefano; Gori, Marco (Neurocomputing, 2023)
Fast reactions to changes in the surrounding visual environment require efficient attention mechanisms to reallocate computational resources to the most relevant locations in the visual field. In this paper, we present a biologically-plausible computational model of focus of attention that exhibits spatiotemporal locality and that is very well-suited for parallel and distributed implementations. Attention emerges as a wave propagation process originated by visual stimuli corresponding to details and motion information. The resulting field obeys the principle of “inhibition of return” so as not to get stuck in potential holes. The proposed model is obtained as a hyperbolic regularization of the Poisson equation to which it reduces in the limit of high speed of propagation. According to the MultiMatch algorithm for scanpaths comparison, the proposed model achieves very competitive results when considering dynamical input stimuli.
CRIS-Team Scopus Importer:2023-10-13
Keywords: Computer vision; Saliency; Scanpath prediction; Visual attention

Neurophthalmologic and Orthoptic Ambulatory Assessments Reveal Ocular and Visual Changes in Patients With Early Alzheimer and Parkinson's Disease - Bargagli, Alessia; Fontanelli, Enrica; Zanca, Dario; Castelli, Ilaria; Rosini, Francesca; Maddii, Silvia; Di Donato, Ilaria; Carluccio, Alessandra; Battisti, Carla; Tosi, Gian Marco; Dotti, Maria Teresa; Rufa, Alessandra (Frontiers in Neurology, 2020)

PolarBearVidID: A Video-Based Re-Identification Benchmark Dataset for Polar Bears - Zürl, Matthias; Dirauf, Richard; Köferl, Franz; Steinlein, Nils; Sueskind, Jonas; Zanca, Dario; Brehm, Ingrid; von Fersen, Lorenzo; Eskofier, Björn (Animals, 2023)
Automated monitoring systems have become increasingly important for zoological institutions in the study of their animals’ behavior. One crucial processing step for such a system is the re-identification of individuals when using multiple cameras. Deep learning approaches have become the standard methodology for this task. Especially video-based methods promise to achieve a good performance in re-identification, as they can leverage the movement of an animal as an additional feature. This is especially important for applications in zoos, where one has to overcome specific challenges such as changing lighting conditions, occlusions or low image resolutions. However, large amounts of labeled data are needed to train such a deep learning model. We provide an extensively annotated dataset including 13 individual polar bears shown in 1431 sequences, which is an equivalent of 138,363 images. PolarBearVidID is the first video-based re-identification dataset for a non-human species to date. Unlike typical human benchmark re-identification datasets, the polar bears were filmed in a range of unconstrained poses and lighting conditions. Additionally, a video-based re-identification approach is trained and tested on this dataset. The results show that the animals can be identified with a rank-1 accuracy of 96.6%. We thereby show that the movement of individual animals is a characteristic feature and it can be utilized for re-identificatio
Keywords: deep learning; re-identification; computer vision; animal identification; animal welfare; automated behavior analysis; motion features; video-based method; dataset

Predictions in Predictive Process Monitoring with Previously Unseen Categorical Values - Roider, Johannes; Wang, Weixin; Zanca, Dario; Matzner, Martin; Eskofier, Björn (Process Mining Workshops, 2024)
Predictive process monitoring (PPM) methods provide users with real-time predictions about ongoing process instances. Machine learning models used for such tasks do not account for data variability, such as the occurrence of previously unseen categorical feature values. Concept drift adaptation solutions are suggested in such scenarios. However, adapting to new feature values requires time and a sample size large enough to train a well-generalizing model. Still, users expect seamless communication during the timeframe between the first occurrence of a new value and the availability of an updated model. Dedicated solutions are needed since encoding techniques like one hot encoding cannot handle previously unseen values by default. In this work, we first introduce and discuss possible solutions from a business perspective, ranging from temporary shutdowns to dedicated manual and technical solutions for an uninterrupted continuation of predictive services. Next, we present five variants for one hot encoding to handle previously unseen categorical values. This is followed by a case study using six real-world event logs and two machine learning models, XGBoost and LSTM, to identify the variants that produce the most reliable remaining time predictions. The study also includes the evaluation of two baseline models as an alternative to the machine learning models. The results show that previously unseen categorical values can be handled on a technical level without severely affecting the remaining time prediction quality. However, future research is required to provide more practical recommendations.<br

p-value Adjustment for Monotonous, Unbiased, and Fast Clustering Comparison - Klede, Kai; Altstidl, Thomas Robert; Zanca, Dario; Eskofier, Björn (Advances in Neural Information Processing Systems 36, 2023)
Popular metrics for clustering comparison, like the Adjusted Rand Index and the Adjusted Mutual Information, are type II biased. The Standardized Mutual Information removes this bias but suffers from counterintuitive non-monotonicity and poor computational efficiency. We introduce the p-value adjusted Rand Index (PMI2), the first cluster comparison method that is type II unbiased and provably monotonous. The PMI2 has fast approximations that outperform the Standardized Mutual information. We demonstrate its unbiased clustering selection, approximation quality, and runtime efficiency on synthetic benchmarks. In experiments on image and social network datasets, we show how the PMI2 can help practitioners choose better clustering and community detection algorithm
Keywords: Clustering; Mutual Information; Machine Learning; Artificial Intelligence

Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen - Wehbi, Mohamad; Luge, Daniel; Hamann, Tim; Barth, Jens; Kaempf, Peter; Zanca, Dario; Eskofier, Björn (Sensors, 2022)
Efficient handwriting trajectory reconstruction (TR) requires specific writing surfaces for detecting movements of digital pens. Although several motion-based solutions have been developed to remove the necessity of writing surfaces, most of them are based on classical sensor fusion methods limited, by sensor error accumulation over time, to tracing only single strokes. In this work, we present an approach to map the movements of an IMU-enhanced digital pen to relative displacement data. Training data is collected by means of a tablet. We propose several pre-processing and data-preparation methods to synchronize data between the pen and the tablet, which are of different sampling rates, and train a convolutional neural network (CNN) to reconstruct multiple strokes without the need of writing segmentation or post-processing correction of the predicted trajectory. The proposed system learns the relative displacement of the pen tip over time from the recorded raw sensor data, achieving a normalized error rate of 0.176 relative to unit-scaled tablet ground truth (GT) trajectory. To test the effectiveness of the approach, we train a neural network for character recognition from the reconstructed trajectories, which achieved a character error rate of 19.51%. Finally, a joint model is implemented that makes use of both the IMU data and the generated trajectories, which outperforms the sensor-only-based recognition approach by 0.75%.<br
Keywords: trajectory reconstruction; inertial measurement unit; sensor-based deep learning; convolutional neural network; handwriting recognition; digital pen

SVC-onGoing: Signature verification competition - Tolosana, Ruben; Vera-Rodriguez, Ruben; Gonzalez-Garcia, Carlos; Fierrez, Julian; Morales, Aythami; Ortega-Garcia, Javier; Carlos Ruiz-Garcia, Juan; Romero-Tapiador, Sergio; Rengifo, Santiago; Caruana, Miguel; Jiang, Jiajia; Lai, Songxuan; et al. (Pattern Recognition, 2022)
This article presents SVC-onGoing1, an on-going competition for on-line signature verification where researchers can easily benchmark their systems against the state of the art in an open common platform using large-scale public databases, such as DeepSignDB2 and SVC2021_EvalDB3, and standard experimental protocols. SVC-onGoing is based on the ICDAR 2021 Competition on On-Line Signature Verification (SVC 2021), which has been extended to allow participants anytime. The goal of SVC-onGoing is to evaluate the limits of on-line signature verification systems on popular scenarios (office/mobile) and writing inputs (stylus/finger) through large-scale public databases. Three different tasks are considered in the competition, simulating realistic scenarios as both random and skilled forgeries are simultaneously considered on each task. The results obtained in SVC-onGoing prove the high potential of deep learning methods in comparison with traditional methods. In particular, the best signature verification system has obtained Equal Error Rate (EER) values of 3.33% (Task 1), 7.41% (Task 2), and 6.04% (Task 3). Future studies in the field should be oriented to improve the performance of signature verification systems on the challenging mobile scenarios of SVC-onGoing in which several mobile devices and the finger are used during the signature acquisition.
Created from Fastlane, Scopus look-up
Keywords: Biometrics; DeepSignDB; Handwriting; Signature verification; SVC 2021; SVC-onGoing; SVC2021_EvalDB

System Design for a Data-Driven and Explainable Customer Sentiment Monitor Using IoT and Enterprise Data - Nguyen, An; Foerstel, Stefan; Kittler, Thomas; Kurzyukov, Andrey; Schwinn, Leo; Zanca, Dario; Hipp, Tobias; Jun, Sun Da; Schrapp, Michael; Rothgang, Eva; Eskofier, Björn (IEEE Access, 2021)
The most important goal of customer service is to keep the customer satisfied. However, service resources are always limited and must prioritize specific customers. Therefore, it is essential to identify customers who potentially become unsatisfied and might lead to escalations. Data science on IoT data (especially log data) for machine health monitoring and analytics on enterprise data for customer relationship management (CRM) have mainly been researched and applied independently. This paper presents a data-driven decision support system framework that combines IoT and enterprise data to model customer sentiment and predicts escalations. The proposed framework includes a fully automated and interpretable machine learning pipeline using state-of-the-art methods. The framework is applied in a real-world case study with a major medical device manufacturer providing data from a fleet of thousands of high-end medical devices. An anonymized version of this industrial benchmark is released for the research community based on the presented case study, which has interesting and challenging properties. In our extensive experiments, we achieve a Recall@50 of 50.0% for the task of predicting customer escalations. In addition, we show that combining IoT and enterprise data can improve prediction results and ease troubleshooting. Additionally, we propose a practical workflow for end-users when applying the proposed framework.
CRIS-Team Scopus Importer:2021-09-10
Keywords: big data; customer service; Data models; decision support system; Decision support systems; explainable AI; Feature extraction; industrial AI; IoT data; Machine learning; machine learning; Medical devices; Monitoring; Predictive models

The Impact of Random Models on Standardized Clustering Similarity - Klede, Kai; Altstidl, Thomas Robert; Zanca, Dario; Eskofier, Björn (IEEE Access, 2024)
Clustering similarity measures are essential for evaluating clustering results and ensuring diversity in multiple clusterings of the same dataset. Common indices like the Mutual Information (MI) and Rand Index (RI) are biased towards smaller clusters and are often adjusted using a random permutation model. Recent advancements have standardized these measures to further correct biases, but the impact of different random models on these standardized measures has not yet been studied. In this work, we introduce equations for standardizing the MI/RI under non-permutation models, specifically focusing on a uniform model over all clusterings and a model that fixes the number of clusterings. Our results show that while standardization improves performance for the fixed number of clusters model, its benefits are limited in the more general uniform model. We validate our findings with gene expression data, highlighting the importance of choosing the right similarity metric for clustering compariso
Keywords: Clustering; Mutual Information; Rand Index; Machine Learning; Artificial Intelligence; Unsupervised learning; External evaluation metrics

Toward Improving the Evaluation of Visual Attention Models: A Crowdsourcing Approach - Zanca, Dario; Melacci, Stefano; Gori, Marco (2020 International Joint Conference on Neural Networks, IJCNN 2020, 2020)
Keywords: crowd-sourcing; fixations; evaluation; scanpath; saliency; Visual attention models

Towards an IMU-based Pen Online Handwriting Recognizer - Wehbi, Mohamad; Hamann, Tim; Barth, Jens; Kaempf, Peter; Zanca, Dario; Eskofier, Björn (International Conference on Document Analysis and Recognition ICDAR 2021, 2021)
Most online handwriting recognition systems require the use of specific writing surfaces to extract positional data. In this paper we present a online handwriting recognition system for word recognition which is based on inertial measurement units (IMUs) for digitizing text written on paper. This is obtained by means of a sensor-equipped pen that provides acceleration, angular velocity, and magnetic forces streamed via Bluetooth. Our model combines convolutional and bidirectional LSTM networks, and is trained with the Connectionist Temporal Classification loss that allows the interpretation of raw sensor data into words without the need of sequence segmentation. We use a dataset of words collected using multiple sensor-enhanced pens and evaluate our model on distinct test sets of seen and unseen words achieving a character error rate of 17.97% and 17.08%, respectively, without the use of a dictionary or language model.
Keywords: Online handwriting recognition; Digital pen; Inertial measurement unit; Time-series data

Tracking the visitor: An optical indoor system for visitor research in museums - Köferl, Franz; Jami, Jitin; Li, Jindong; Zanca, Dario; Eskofier, Björn; Zürl, Matthias (AI in Museums: Reflections, Perspectives and Applications, 2023)
The analysis of the success of any exhibition depends on the visitor experience. The data required for analysis is usually painstakingly collected by hand. We propose a large-scale optical tracking pipeline to estimate visitor data such as visit trajectory, duration, and, potentially, other personal parameters like age, weight, and sex, yet remain ethically acceptable by obtaining visitor consent. We further show, in preliminary results, that the edge device has a localization error of 0.64 meters and an average precision of 0.2. With this work-in-progress, we intend to ensure a viable alternative to current data collection processes in museum research.
CRIS-Team Scopus Importer:2024-12-16

Trends, Applications, and Challenges in Human Attention Modelling - Cartella, Giuseppe; Cornia, Marcella; Cuculo, Vittorio; D'Amelio, Alessandro; Zanca, Dario; Boccignone, Giuseppe; Cucchiara, Rita (IJCAI International Joint Conference on Artificial Intelligence, 2024)
Human attention modelling has proven, in recent years, to be particularly useful not only for understanding the cognitive processes underlying visual exploration, but also for providing support to artificial intelligence models that aim to solve problems in various domains, including image and video processing, vision-and-language applications, and language modelling. This survey offers a reasoned overview of recent efforts to integrate human attention mechanisms into contemporary deep learning models and discusses future research directions and challenges. For a comprehensive overview of the ongoing research, refer to our dedicated repository available at https://github.com/aimagelab/awesome-human-visual-attention.
CRIS-Team Scopus Importer:2024-09-27

Variational laws of visual attention for dynamic scenes - Zanca, Dario; Gori, Marco (31st Annual Conference on Neural Information Processing Systems, NIPS 2017, 2017)