Recognizing Characters in Art History Using Deep Learning

Madhu P, Kosti RV, Mührenberg L, Bell P, Maier A, Christlein V (2019)

Publication Language: English

Publication Type: Conference contribution, Original article

Publication year: 2019

Publisher: Association for Computing Machinery.

Conference Proceedings Title: Recognizing Characters in Art History Using Deep Learning

Event location: Nice

ISBN: 978-1-4503-6910-7/19/10

DOI: 10.1145/3347317.3357242

Abstract

In the field of Art History, images of artworks and their contexts are core to understanding the underlying semantic information. However, the highly complex and sophisticated representation of these artworks makes it difficult, even for the experts, to analyze the scene. From the computer vision perspective, the task of analyzing such artworks can be divided into sub-problems by taking a bottom-up approach. In this paper, we focus on the problem of recognizing the characters in Art History. From the iconography of Annunciation of the Lord (Figure 1), we consider the representation of the main protagonists, Mary and Gabriel, across different artworks and styles. We investigate and present the findings of training a character classifier on features extracted from their face images. The limitations of this method, and the inherent ambiguity in the representation of Gabriel, motivated us to consider their bodies (a bigger context) to analyze in order to recognize the characters. Convolutional Neural Networks (CNN) trained on the bodies of Mary and Gabriel are able to learn person related features and ultimately improve the performance of character recognition. We introduce a new technique that generates more data with similar styles, effectively creating data in the similar domain. We present experiments and analysis on three different models and show that the model trained on domain related data gives the best performance for recognizing character. Additionally, we analyze the localized image regions for the network predictions.

Authors with CRIS profile

Additional Organisation(s)

Related research project(s)

How to cite

APA:

Madhu, P., Kosti, R.V., Mührenberg, L., Bell, P., Maier, A., & Christlein, V. (2019). Recognizing Characters in Art History Using Deep Learning. In Recognizing Characters in Art History Using Deep Learning. Nice, FR: Association for Computing Machinery..

MLA:

Madhu, Prathmesh, et al. "Recognizing Characters in Art History Using Deep Learning." Proceedings of the SUMAC 2019 - The 1st workshop on Structuring and Understanding of Multimedia heritAge Contents, Nice Association for Computing Machinery., 2019.

BibTeX: Download