WordStylist: Styled Verbatim Handwritten Text Generation with Latent Diffusion Models

Nikolaidou K, Retsinas G, Christlein V, Seuret M, Sfikas G, Barney Smith E, Mokayed H, Liwicki M (2023)

Publication Language: English

Publication Type: Conference contribution

Publication year: 2023

Journal

Lecture Notes in Computer Science Springer Verlag

Publisher: Springer

City/Town: Cham

Book Volume: 14188

Pages Range: 384-401

Conference Proceedings Title: Lecture Notes in Computer Science

Event location: San José, CA US

ISBN: 9783031416781

DOI: 10.1007/978-3-031-41679-8_22

Open Access Link: https://arxiv.org/abs/2303.16576

Abstract

Text-to-Image synthesis is the task of generating an image according to a specific text description. Generative Adversarial Networks have been considered the standard method for image synthesis virtually since their introduction. Denoising Diffusion Probabilistic Models are recently setting a new baseline, with remarkable results in Text-to-Image synthesis, among other fields. Aside its usefulness per se, it can also be particularly relevant as a tool for data augmentation to aid training models for other document image processing tasks. In this work, we present a latent diffusion-based method for styled text-to-text-content-image generation on word-level. Our proposed method is able to generate realistic word image samples from different writer styles, by using class index styles and text content prompts without the need of adversarial training, writer recognition, or text recognition. We gauge system performance with the Fréchet Inception Distance, writer recognition accuracy, and writer retrieval. We show that the proposed model produces samples that are aesthetically pleasing, help boosting text recognition performance, and get similar writer retrieval score as real data.

Authors with CRIS profile

Vincent Christlein Lehrstuhl für Informatik 5 (Mustererkennung) Mathias Seuret Lehrstuhl für Informatik 5 (Mustererkennung)

Involved external institutions

Luleå University of Technology SE Sweden (SE) National Technical University of Athens / Εθνικό Μετσόβιο Πολυτεχνείο GR Greece (GR) University of West Attica (UNIWA) / Πανεπιστήμιο Δυτικής Αττικής GR Greece (GR)

How to cite

APA:

Nikolaidou, K., Retsinas, G., Christlein, V., Seuret, M., Sfikas, G., Barney Smith, E.,... Liwicki, M. (2023). WordStylist: Styled Verbatim Handwritten Text Generation with Latent Diffusion Models. In Lecture Notes in Computer Science (pp. 384-401). San José, CA, US: Cham: Springer.

MLA:

Nikolaidou, Konstantina, et al. "WordStylist: Styled Verbatim Handwritten Text Generation with Latent Diffusion Models." Proceedings of the International Conference on Document Analysis and Recognition, San José, CA Cham: Springer, 2023. 384-401.

BibTeX: Download