One-Shot Object Detection in Heterogeneous Artwork Datasets

Madhu P, Meyer A, Zinnen M, Mührenberg L, Suckow D, Bendschus T, Reinhardt C, Bell P, Verstegen U, Kosti RV, Maier A, Christlein V (2022)

Publication Type: Conference contribution, Conference Contribution

Publication year: 2022

Event location: Salzburg, Austria

DOI: 10.1109/IPTA54936.2022.9784141

Abstract

Christian archeologists face many challenges in understanding visual narration through artwork images. This understanding is essential to access underlying semantic in-formation. Therefore, narrative elements (objects) need to be labeled, compared, and contextualized by experts, which takes an enormous amount of time and effort. Our work aims to reduce labeling costs by using one-shot object detection to generate a labeled database from unannotated images. Novel object categories can be defined broadly and annotated using visual examples of narrative elements without training exclusively for such objects. In this work, we propose two ways of using contextual information as data augmentation to improve the detection performance. Furthermore, we introduce a multi-relation detector to our framework, which extracts global, local, and patch-based relations of the image. Additionally, we evaluate the use of contrastive learning. We use data from Christian archeology (CHA) and art history - IconArt-v2 (IA). Our context encoding approach improves the typical fine-tuning approach in terms of mean average precision (mAP) by about 3.5 % (4 %) at 0.25 intersection over union (IoU) for UnSeen categories, and 6 % (1.5 %) for Seen categories in CHA (IA). To the best of our knowledge, our work is the first to explore few shot object detection on heterogeneous artistic data by investigating evaluation methods and data augmentation strategies. We will release the code and models after acceptance of the work.

Authors with CRIS profile

Prathmesh Madhu Lehrstuhl für Informatik 5 (Mustererkennung) Anna Meyer Lehrstuhl für Multimediakommunikation und Signalverarbeitung Mathias Zinnen Lehrstuhl für Informatik 5 (Mustererkennung) Lara Mührenberg Lehrstuhl für Christliche Archäologie Dirk Suckow Juniorprofessur für Digital Humanities mit Schwerpunkt Kunstgeschichte Torsten Bendschus Juniorprofessur für Klassische Archäologie Corinna Reinhardt Juniorprofessur für Klassische Archäologie Ute Verstegen Lehrstuhl für Christliche Archäologie Ronak Vijaypal Kosti Lehrstuhl für Informatik 5 (Mustererkennung) Andreas Maier Lehrstuhl für Informatik 5 (Mustererkennung) Vincent Christlein Lehrstuhl für Informatik 5 (Mustererkennung)

Involved external institutions

Philipps-Universität Marburg DE Germany (DE)

How to cite

APA:

Madhu, P., Meyer, A., Zinnen, M., Mührenberg, L., Suckow, D., Bendschus, T.,... Christlein, V. (2022). One-Shot Object Detection in Heterogeneous Artwork Datasets. In Proceedings of the 2022 Eleventh International Conference on Image Processing Theory, Tools and Applications (IPTA). Salzburg, Austria.

MLA:

Madhu, Prathmesh, et al. "One-Shot Object Detection in Heterogeneous Artwork Datasets." Proceedings of the 2022 Eleventh International Conference on Image Processing Theory, Tools and Applications (IPTA), Salzburg, Austria 2022.

BibTeX: Download