Multimodal and Multilingual Understanding of Smells using VilBERT and mUNITER

Akdemir K, Hürriyetoğlu A, Troncy R, Paccosi T, Menini S, Zinnen M, Christlein V (2023)

Publication Type: Conference contribution

Publication year: 2023

Publisher: CEUR-WS

Book Volume: 3583

Conference Proceedings Title: CEUR Workshop Proceedings

Event location: Virtual, Online, NOR

Abstract

We evaluate state-of-the-art multimodal models to detect common olfactory references in multilingual text and images in the scope of the Multimodal Understanding of Smells in Texts and Images (MUSTI) Task at Mediaeval 2022. The goal of the MUSTI Subtask 1 is to classify pairs of text and image as to whether they refer to the same smell source or not. We approach this task as a Visual Entailment problem and evaluate the performance of the English model ViLBERT and the multilingual model mUNITER on MUSTI Subtask 1. While base VilBERT and mUNITER models perform worse than a dummy baseline, fine-tuning these models using the training data improve performance significantly in almost all scenarios. We find that fine-tuning mUNITER with SNLI-VE and MUSTI training data performs better than other configurations we implemented. Our experiments demonstrate that the task presents some challenges, but it is by no means impossible. Our code is available at https://github.com/Odeuropa/musti-eval-baselines to encourage reproducibility.

Authors with CRIS profile

Mathias Zinnen Lehrstuhl für Informatik 5 (Mustererkennung) Vincent Christlein Lehrstuhl für Informatik 5 (Mustererkennung)

Involved external institutions

Royal Netherlands Academy of Arts and Sciences / Koninklijke Nederlandse Akademie van Wetenschappen (KNAW) NL Netherlands (NL) Institut Eurécom (EURECOM) FR France (FR) Fondazione Bruno Kessler (FBK) (früher: ITC-irst) IT Italy (IT)

How to cite

APA:

Akdemir, K., Hürriyetoğlu, A., Troncy, R., Paccosi, T., Menini, S., Zinnen, M., & Christlein, V. (2023). Multimodal and Multilingual Understanding of Smells using VilBERT and mUNITER. In Steven Hicks, Alba Garcia Seco De Herrera, Johannes Langguth, Johannes Langguth, Andreas Lommatzsch, Stelios Andreadis, Minh-Son Dao, Pierre-Etienne Martin, Ali Hurriyetoglu, Vajira Thambawita, Tor-Arne Nordmo, Romain Vuillemot, Martha Larson (Eds.), CEUR Workshop Proceedings. Virtual, Online, NOR: CEUR-WS.

MLA:

Akdemir, Kiymet, et al. "Multimodal and Multilingual Understanding of Smells using VilBERT and mUNITER." Proceedings of the 2022 MediaEval Workshop, MediaEval 2022, Virtual, Online, NOR Ed. Steven Hicks, Alba Garcia Seco De Herrera, Johannes Langguth, Johannes Langguth, Andreas Lommatzsch, Stelios Andreadis, Minh-Son Dao, Pierre-Etienne Martin, Ali Hurriyetoglu, Vajira Thambawita, Tor-Arne Nordmo, Romain Vuillemot, Martha Larson, CEUR-WS, 2023.

BibTeX: Download