Are Vision-Language Transformers Learning Multimodal Representations? A Probing Perspective - Salin, Emmanuelle; Farah, Badreddine; Ayache, Stéphane; Favre, Benoit (Proceedings of the AAAI Conference on Artificial Intelligence, 2022)
In recent years, joint text-image embeddings have significantly improved thanks to the development of transformer-based Vision-Language models. Despite these advances, we still need to better understand the representations produced by those models. In this paper, we compare pre-trained and fine-tuned representations at a vision, language and multimodal level. To that end, we use a set of probing tasks to evaluate the performance of state-of-the-art Vision-Language models and introduce new datasets specifically for multimodal probing. These datasets are carefully designed to address a range of multimodal capabilities while minimizing the potential for models to rely on bias. Although the results confirm the ability of Vision-Language models to understand color at a multimodal level, the models seem to prefer relying on bias in text data for object position and size. On semantically adversarial examples, we find that those models are able to pinpoint fine-grained multimodal differences. Finally, we also notice that fine-tuning a Vision-Language model on multimodal tasks does not necessarily improve its multimodal ability. We make all datasets and code available to replicate experiments.
Keywords: Speech & Natural Language Processing (SNLP); Computer Vision (CV); Machine Learning (ML)

Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies? - Nikolaus, Mitja; Salin, Emmanuelle; Ayache, Stéphane; Fourtassi, Abdellah; Favre, Benoit (Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022)
Recent advances in vision-and-language modeling have seen the development of Transformer architectures that achieve remarkable performance on multimodal reasoning tasks.Yet, the exact capabilities of these black-box models are still poorly understood. While much of previous work has focused on studying their ability to learn meaning at the word-level, their ability to track syntactic dependencies between words has received less attention.We take a first step in closing this gap by creating a new multimodal task targeted at evaluating understanding of predicate-noun dependencies in a controlled setup.We evaluate a range of state-of-the-art models and find that their performance on the task varies considerably, with some models performing relatively well and others at chance level. In an effort to explain this variability, our analyses indicate that the quality (and not only sheer quantity) of pretraining data is essential. Additionally, the best performing models leverage fine-grained multimodal pretraining objectives in addition to the standard image-text matching objectives.This study highlights that targeted and controlled evaluations are a crucial step for a precise and rigorous test of the multimodal knowledge of vision-and-language model

Towards an Exhaustive Evaluation of Vision-Language Foundation Models - Salin, Emmanuelle; Ayache, Stéphane; Favre, Benoit (Proceedings - 2023 IEEE/CVF International Conference on Computer Vision Workshops, ICCVW 2023, 2023)
Vision-language foundation models have had considerable increase in performances in the last few years. However, there is still a lack comprehensive evaluation methods able to clearly explain their performances. We argue that a more systematic approach to foundation model evaluation would be beneficial to their use in real-world applications. In particular, we think that those models should be evaluated on a broad range of precise capabilities, in order to bring awareness to the width of their scope and their potential weaknesses. To that end, we propose a methodology to build a taxonomy of multimodal capabilities for vision-language foundation models. The proposed taxonomy is intended as a first step towards an exhaustive evaluation of vision-language foundation models.
Created from Fastlane, Scopus look-up
Keywords: Foundation models; Language; Transformers; Vision