Multi-Modal Biomarker Extraction Framework for Therapy Monitoring of Social Anxiety and Depression Using Audio and Video

Yang SH, Pérez Toro PA, Weise T, Hoffmann B, Demir K, Nöth E, Kallert T, Maier A (2023)

Publication Language: English

Publication Type: Conference contribution, Conference Contribution

Publication year: 2023

Pages Range: 9

Conference Proceedings Title: International Conference on Machine Learning (Workshop on Machine Learning for Multimodal Healthcare)

Event location: Hawaii Convention Center, 1801 Kalākaua Ave, Honolulu, HI 96815, United States US

Abstract

This paper introduces a framework that can be used for feature extraction, relevant to monitoring the speech therapy progress of individuals suffering from social anxiety or depression. It operates multi-modal (decision fusion) by incorporating audio and video recordings of a patient and the corresponding interviewer, at two separate test assessment sessions. The used data is provided by an ongoing project in a day-hospital and outpatient setting in Germany, with the goal of investigating whether an established speech therapy group program for adolescents, which is implemented in a stationary and semi-stationary setting, can be successfully carried out via telemedicine. The features proposed in this multi-modal approach could form the basis for interpretation and analysis by medical experts and therapists, in addition to acquired data in the form of questionnaires. Extracted audio features focus on prosody (intonation, stress, rhythm, and timing), as well as predictions from a deep neural network model, which is inspired by the Pleasure, Arousal, Dominance (PAD) emotional model space. Video features are based on a pipeline that is designed to enable visualization of the interaction between the patient and the interviewer in terms of Facial Emotion Recognition (FER), utilizing the mini-Xception network architecture.

Authors with CRIS profile

Seung Hee Yang Professur für Artificial Intelligence in Biomedical Speech Processing (Stiftungsprofessur) Paula Andrea Pérez Toro Lehrstuhl für Informatik 5 (Mustererkennung) Tobias Weise Professur für Artificial Intelligence in Biomedical Speech Processing (Stiftungsprofessur) Bettina Hoffmann Medizinische Fakultät Kubilay Demir Lehrstuhl für Informatik 5 (Mustererkennung) Elmar Nöth Professur für Informatik (Mustererkennung) Thomas Kallert Institut für Lehre und Forschung am Medizincampus Oberfranken Andreas Maier Lehrstuhl für Informatik 5 (Mustererkennung)

How to cite

APA:

Yang, S.H., Pérez Toro, P.A., Weise, T., Hoffmann, B., Demir, K., Nöth, E.,... Maier, A. (2023). Multi-Modal Biomarker Extraction Framework for Therapy Monitoring of Social Anxiety and Depression Using Audio and Video. In International Conference on Machine Learning (Workshop on Machine Learning for Multimodal Healthcare) (pp. 9). Hawaii Convention Center, 1801 Kalākaua Ave, Honolulu, HI 96815, United States, US.

MLA:

Yang, Seung Hee, et al. "Multi-Modal Biomarker Extraction Framework for Therapy Monitoring of Social Anxiety and Depression Using Audio and Video." Proceedings of the International Conference on Machine Learning (Workshop on Machine Learning for Multimodal Healthcare), Hawaii Convention Center, 1801 Kalākaua Ave, Honolulu, HI 96815, United States 2023. 9.

BibTeX: Download