Predicting Request Success with Objective Features in German Multimodal Speech Assistants

Weber M, Halimeh MM, Kellermann W, Popp B (2022)

Publication Type: Conference contribution

Publication year: 2022

Journal

Lecture Notes in Computer Science Springer Verlag

Publisher: Springer Science and Business Media Deutschland GmbH

Book Volume: 13336 LNAI

Pages Range: 594-609

Conference Proceedings Title: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Event location: Virtual, Online

ISBN: 9783031056420

DOI: 10.1007/978-3-031-05643-7_39

Abstract

We investigate whether objective features, like occurrence of an error and number of turns, can automatically predict success in interactions with multimodal speech assistants. We used interactions from the SmartKom corpus, a data set on multimodal interactions with virtual assistants in German. In a first step, we segmented the interactions into requests and labeled them as successful or unsuccessful. Afterwards, we defined task success as the average of request success rate. Next, we investigated whether subjective features such as emotions expressed by users show a relation to task success. We find no significant correlation. Finally, we exploited objective features, e.g., number of turns to predict request success. We find that objective features suffice to reach F1 scores over 0.9 (prediction of successful requests) and F0 scores above 0.83 (prediction of unsuccessful requests). Finally, we discuss implications of our findings for automatic evaluation of pragmatic aspects of user experience.

Authors with CRIS profile

Mareike Weber Lehrstuhl für Technische Elektronik Mhd Modar Halimeh Professur für Signalverarbeitung Walter Kellermann Professur für Signalverarbeitung

Involved external institutions

Fraunhofer-Institut für Integrierte Systeme und Bauelementetechnologie (IISB) DE Germany (DE)

How to cite

APA:

Weber, M., Halimeh, M.M., Kellermann, W., & Popp, B. (2022). Predicting Request Success with Objective Features in German Multimodal Speech Assistants. In Helmut Degen, Stavroula Ntoa (Eds.), Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (pp. 594-609). Virtual, Online: Springer Science and Business Media Deutschland GmbH.

MLA:

Weber, Mareike, et al. "Predicting Request Success with Objective Features in German Multimodal Speech Assistants." Proceedings of the 3rd International Conference on Artificial Intelligence in HCI, AI-HCI 2022 Held as Part of the 24th HCI International Conference, HCII 2022, Virtual, Online Ed. Helmut Degen, Stavroula Ntoa, Springer Science and Business Media Deutschland GmbH, 2022. 594-609.

BibTeX: Download