Quantifying Divergence for Human-AI Collaboration and Cognitive Trust
Gebeşçe A, Kural M, Chubakov T, Şahin GG (2025)
Publication Type: Conference contribution
Publication year: 2025
Publisher: Association for Computing Machinery
Conference Proceedings Title: Conference on Human Factors in Computing Systems - Proceedings
Event location: Yokohama, JPN
ISBN: 9798400713958
Abstract
Predicting the collaboration likelihood and measuring cognitive trust to AI systems is more important than ever. To do that, previous research mostly focused solely on model features (e.g., accuracy, confidence) and ignored the human factor. To address that, we propose several decision-making similarity measures based on divergence metrics (e.g., KL, JSD) calculated over the labels acquired from humans and a wide range of models. We conduct a user study (N=100) on a textual entailment task, where the users are provided with soft labels from various models and asked to pick the closest option to them. The users are then shown the similarities/differences to their most similar model and are surveyed for their likelihood of collaboration and cognitive trust to the selected system. Finally, we qualitatively and quantitatively analyze the relation between the proposed decision-making similarity measures and the survey results. We find that people tend to collaborate with their most similar models—measured via JSD—yet this collaboration does not necessarily imply a similar level of cognitive trust. We release all resources related to the user study (e.g., design, outputs), models, and metrics at our repo.
Involved external institutions
Koç-Universität (KU)
Turkey (TR)
University of California, Berkeley (UC Berkeley)
United States (USA) (US)
Turkey (TR)
University of California, Berkeley (UC Berkeley)
United States (USA) (US)
How to cite
APA:
Gebeşçe, A., Kural, M., Chubakov, T., & Şahin, G.G. (2025). Quantifying Divergence for Human-AI Collaboration and Cognitive Trust. In Conference on Human Factors in Computing Systems - Proceedings. Yokohama, JPN: Association for Computing Machinery.
MLA:
Gebeşçe, Ali, et al. "Quantifying Divergence for Human-AI Collaboration and Cognitive Trust." Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI EA 2025, Yokohama, JPN Association for Computing Machinery, 2025.
BibTeX: Download