Exploring the Role of Co-Speech Gestures: An In-the-Wild Study with a Virtual Agent in a Museum
Chojnowski O, Kirschbaum A, Neef C, Jeschke S, Richert A (2026)
Publication Type: Conference contribution
Publication year: 2026
Publisher: Association for Computing Machinery, Inc
Pages Range: 133-140
Conference Proceedings Title: HAI 2025 - Proceedings of the 13th International Conference on Human-Agent Interaction
Event location: Yokohama, JPN
ISBN: 9798400721786
Abstract
Human-like nonverbal behaviors, such as gestures or facial expressions, play a crucial role in face-to-face communication, yet their impact on user interactions with virtual agents (VAs) in real-world environments remains underexplored. This study examines how co-speech gestures influence engagement and perception in interactions with a VA deployed in a museum. Using a mixed-methods approach, we conducted an A/B test comparing a gesturing VA to a nongesturing VA. Quantitative analysis of system logs revealed no significant difference in the number of conversational turns between conditions. Qualitative findings revealed that gestures may positively influence usability by making the interaction more intuitive and reducing user confusion regarding the VA's internal states. However, perceptions of nonverbal behavior varied, with some users finding gestures engaging while others considered them uncanny. Regardless of condition, system response times, verbal communication errors, and answer quality were key drivers of user dissatisfaction. These findings highlight both the benefits and challenges of incorporating and studying nonverbal behavior in VAs and underscore the need for robust verbal communication to support meaningful interactions.
Involved external institutions
Karlsruhe Institute of Technology (KIT)
Germany (DE)
KI Park e.V.
Germany (DE)
TH Köln / Technische Hochschule Köln / Fachhochschule Köln
Germany (DE)
Germany (DE)
KI Park e.V.
Germany (DE)
TH Köln / Technische Hochschule Köln / Fachhochschule Köln
Germany (DE)
How to cite
APA:
Chojnowski, O., Kirschbaum, A., Neef, C., Jeschke, S., & Richert, A. (2026). Exploring the Role of Co-Speech Gestures: An In-the-Wild Study with a Virtual Agent in a Museum. In Hirotaka Osawa, Helena Lindgren, Aaron Steinfeld, Mary Ellen Foster, Shogo Okada, Haiyi Zhu (Eds.), HAI 2025 - Proceedings of the 13th International Conference on Human-Agent Interaction (pp. 133-140). Yokohama, JPN: Association for Computing Machinery, Inc.
MLA:
Chojnowski, Oliver, et al. "Exploring the Role of Co-Speech Gestures: An In-the-Wild Study with a Virtual Agent in a Museum." Proceedings of the 13th International Conference on Human-Agent Interaction, HAI 2025, Yokohama, JPN Ed. Hirotaka Osawa, Helena Lindgren, Aaron Steinfeld, Mary Ellen Foster, Shogo Okada, Haiyi Zhu, Association for Computing Machinery, Inc, 2026. 133-140.
BibTeX: Download