Gesture Classification in Artworks Using Contextual Image Features - Hussian, Azhar; Zinnen, Mathias; Tran, Thi My Hang; Maier, Andreas; Christlein, Vincent (Digital Humanities Conference 2024, 2024)

Recognizing sensory gestures in historical artworks - Zinnen, Mathias; Hussian, Azhar; Maier, Andreas; Christlein, Vincent (Multimedia Tools and Applications, 2024)
The automatic recognition of sensory gestures in artworks provides the opportunity to open up methods of computational humanities to modern paradigms like sensory studies or everyday history. We introduce SensoryArt, a dataset of multisensory gestures in historical artworks, annotated with person boxes, pose estimation key points and gesture labels. We analyze algorithms for each label type and explore their combination for gesture recognition without intermediate supervision. These combined algorithms are evaluated for their ability to recognize and localize depicted persons performing sensory gestures. Our experiments show that direct detection of smell gestures is the most effective method for both detecting and localizing gestures. After applying post-processing, this method outperforms even image-level classification algorithms in image-level classification metrics, despite not being the primary training objective. This work aims to open up the field of sensory history to the computational humanities and provide humanities-based scholars with a solid foundation to complement their methodological toolbox with quantitative methods.
CRIS-Team Scopus Importer:2025-02-04
Keywords: Action recognition; Computational heritage; Digital humanities; Human pose estimation; Person detection

Smelly, dense, and spreaded: The Object Detection for Olfactory References (ODOR) dataset - Zinnen, Mathias; Madhu, Prathmesh; Leemans, Inger; Bell, Peter; Hussian, Azhar; Tran, Thi My Hang; Hürriyetoğlu, Ali; Maier, Andreas; Christlein, Vincent (Expert Systems With Applications, 2024)
Real-world applications of computer vision in the humanities require algorithms to be robust against artistic abstraction, peripheral objects, and subtle differences between fine-grained target classes. Existing datasets provide instance-level annotations on artworks but are generally biased towards the image centre and limited with regard to detailed object classes. The proposed ODOR dataset fills this gap, offering 38,116 object-level annotations across 4712 images, spanning an extensive set of 139 fine-grained categories. Conducting a statistical analysis, we showcase challenging dataset properties, such as a detailed set of categories, dense and overlapping objects, and spatial distribution over the whole image canvas. Furthermore, we provide an extensive baseline analysis for object detection models and highlight the challenging properties of the dataset through a set of secondary studies. Inspiring further research on artwork object detection and broader visual cultural heritage studies, the dataset challenges researchers to explore the intersection of object recognition and smell perception.<br
Keywords: Object detection; Dataset; Computational humanities; Olfaction; Small object detection; Transfer learning

SniffyArt: The Dataset of Smelling Persons - Zinnen, Mathias; Hussian, Azhar; Tran, Thi My Hang; Madhu, Prathmesh; Maier, Andreas; Christlein, Vincent (Proceedings of the 5th Workshop on AnalySis, Understanding and ProMotion of HeritAge Contents, 2023)
Smell gestures play a crucial role in the investigation of past smells in the visual arts yet their automated recognition poses significant challenges. This paper introduces the SniffyArt dataset, consisting of 1941 individuals represented in 441 historical artworks. Each person is annotated with a tightly fitting bounding box, 17 pose keypoints, and a gesture label. By integrating these annotations, the dataset enables the development of hybrid classification approaches for smell gesture recognition. The dataset's high-quality human pose estimation keypoints are achieved through the merging of five separate sets of keypoint annotations per person. The paper also presents a baseline analysis, evaluating the performance of representative algorithms for detection, keypoint estimation, and classification tasks, showcasing the potential of combining keypoint estimation with smell gesture classification. The SniffyArt dataset lays a solid foundation for future research and the exploration of multi-task approaches leveraging pose keypoints and person boxes to advance human gesture and olfactory dimension analysis in historical artworks.<br