FovEx: Human-Inspired Explanations for Vision Transformers and Convolutional Neural Networks

Panda MP, Tiezzi M, Vilas M, Roig G, Eskofier B, Zanca D (2025)

Publication Type: Journal article

Publication year: 2025

Journal

International Journal of Computer Vision Springer Verlag (Germany)

Book Volume: 133

Pages Range: 7437-7459

Journal Issue: 10

DOI: 10.1007/s11263-025-02543-y

Abstract

Explainability in artificial intelligence (XAI) remains a crucial aspect for fostering trust and understanding in machine learning models. Current visual explanation techniques, such as gradient-based or class-activation-based methods, often exhibit a strong dependence on specific model architectures. Conversely, perturbation-based methods, despite being model-agnostic, are computationally expensive as they require evaluating models on a large number of forward passes. We introduce Foveation-based Explanations (FovEx), a novel XAI method inspired by human vision, which combines biologically inspired foveation-based transformations with gradient-driven overt attention to iteratively select locations of interest. These locations are selected to maximize the performance of the model to be explained with respect to the downstream task and then combined to generate an attribution map. We provide a thorough evaluation with qualitative and quantitative assessments on established benchmarks. Our method achieves state-of-the-art performance on both transformers (on 4 out of 5 metrics) and convolutional models (on 3 out of 5 metrics), demonstrating its versatility among various architectures. Furthermore, we show the alignment between the explanation map produced by FovEx and human gaze patterns (+14% in NSS compared to RISE, +203% in NSS compared to GradCAM). This comparison enhances our confidence in FovEx’s ability to close the interpretation gap between humans and machines.

Authors with CRIS profile

Björn Eskofier Department Artificial Intelligence in Biomedical Engineering (AIBE) Dario Zanca Machine Learning and Data Analytics Lab

Involved external institutions

Italian Institute of Technology / Istituto Italiano di Tecnologia (IIT) IT Italy (IT) Ernst Strüngmann Institut (ESI) / Ernst Strüngmann Institute for Neuroscience DE Germany (DE) Goethe-Universität Frankfurt am Main DE Germany (DE)

How to cite

APA:

Panda, M.P., Tiezzi, M., Vilas, M., Roig, G., Eskofier, B., & Zanca, D. (2025). FovEx: Human-Inspired Explanations for Vision Transformers and Convolutional Neural Networks. International Journal of Computer Vision, 133(10), 7437-7459. https://doi.org/10.1007/s11263-025-02543-y

MLA:

Panda, Mahadev Prasad, et al. "FovEx: Human-Inspired Explanations for Vision Transformers and Convolutional Neural Networks." International Journal of Computer Vision 133.10 (2025): 7437-7459.

BibTeX: Download