ICC++: Explainable Feature Learning for Art History using Image Compositions - Madhu, Prathmesh; Marquart, Tilman; Kosti, Ronak Vijaypal; Suckow, Dirk; Bell, Peter; Maier, Andreas; Christlein, Vincent (Pattern Recognition, 2022)
Image compositions are helpful in the study of image structures and assist in discovering the semantics of the underlying scene portrayed across art forms and styles. With the digitization of artworks in recent years, thousands of images of a particular scene or narrative could potentially be linked together. However, manually linking this data with consistent objectiveness can be a highly challenging and time-consuming task. In this work, we present a novel approach called Image Composition Canvas (ICC) to compare and retrieve images having similar compositional elements. ICC is an improvement over ICC, specializing in generating low and high-level features (compositional elements) motivated by Max Imdahl’s work. To this end, we present a rigorous quantitative and qualitative comparison of our approach with traditional and state-of-the-art (SOTA) methods showing that our proposed method outperforms all of them. In combination with deep features, our method outperforms the best deep learning-based method, opening the research direction for explainable machine learning for digital humanities. We will release the code and the data post-publication.<br

One-Shot Object Detection in Heterogeneous Artwork Datasets - Madhu, Prathmesh; Meyer, Anna; Zinnen, Mathias; Mührenberg, Lara; Suckow, Dirk; Bendschus, Torsten; Reinhardt, Corinna; Bell, Peter; Verstegen, Ute; Kosti, Ronak Vijaypal; Maier, Andreas; Christlein, Vincent (2022 Eleventh International Conference on Image Processing Theory, Tools and Applications (IPTA), 2022)
Christian archeologists face many challenges in understanding visual narration through artwork images. This understanding is essential to access underlying semantic in-formation. Therefore, narrative elements (objects) need to be labeled, compared, and contextualized by experts, which takes an enormous amount of time and effort. Our work aims to reduce labeling costs by using one-shot object detection to generate a labeled database from unannotated images. Novel object categories can be defined broadly and annotated using visual examples of narrative elements without training exclusively for such objects. In this work, we propose two ways of using contextual information as data augmentation to improve the detection performance. Furthermore, we introduce a multi-relation detector to our framework, which extracts global, local, and patch-based relations of the image. Additionally, we evaluate the use of contrastive learning. We use data from Christian archeology (CHA) and art history - IconArt-v2 (IA). Our context encoding approach improves the typical fine-tuning approach in terms of mean average precision (mAP) by about 3.5 % (4 %) at 0.25 intersection over union (IoU) for UnSeen categories, and 6 % (1.5 %) for Seen categories in CHA (IA). To the best of our knowledge, our work is the first to explore few shot object detection on heterogeneous artistic data by investigating evaluation methods and data augmentation strategies. We will release the code and models after acceptance of the wor
Keywords: one-shot; object detection; digital humanities; computational humanities; data augmentation