Bio-Signal Based Multimodal Fusion with Bilinear Model for Emotion Recognition
Singh A, Holzer N, Gotz T, Wittenberg T, Gob S, Sawant S, Salman MM, Pahl J (2023)
Publication Type: Conference contribution
Publication year: 2023
Publisher: Institute of Electrical and Electronics Engineers Inc.
Pages Range: 4834-4839
Conference Proceedings Title: Proceedings - 2023 2023 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2023
Event location: Istanbul, TUR
ISBN: 9798350337488
DOI: 10.1109/BIBM58861.2023.10385273
Abstract
In this paper, we propose a feature-based multimodal fusion deep-learning approach to combine biosignals, such as electrocardiogram (ECG) and galvanic skin response (GSR) for effective emotion recognition. We make use of a bilinear fusion method to fuse ECG and GSR based features and to jointly capture and complement the complex information from both modalities to enhance the performance of an emotion recognition system. The performance of the proposed fusion approach is evaluated on the MAHNOB-HCI dataset and compared with existing multimodal fusion approaches. Experimental results show that the combination of ECG and GSR features have a high discriminating ability in classifying (or predicting) the emotional states 'Arousal' and 'Valence' better than that of a single modality based emotion recognition. We also provide the model's uncertainty analysis to understand how confident the obtained predictions are.
Authors with CRIS profile
Involved external institutions
Germany (DE)How to cite
APA:
Singh, A., Holzer, N., Gotz, T., Wittenberg, T., Gob, S., Sawant, S.,... Pahl, J. (2023). Bio-Signal Based Multimodal Fusion with Bilinear Model for Emotion Recognition. In Xingpeng Jiang, Haiying Wang, Reda Alhajj, Xiaohua Hu, Felix Engel, Mufti Mahmud, Nadia Pisanti, Xuefeng Cui, Hong Song (Eds.), Proceedings - 2023 2023 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2023 (pp. 4834-4839). Istanbul, TUR: Institute of Electrical and Electronics Engineers Inc..
MLA:
Singh, Ashutosh, et al. "Bio-Signal Based Multimodal Fusion with Bilinear Model for Emotion Recognition." Proceedings of the 2023 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2023, Istanbul, TUR Ed. Xingpeng Jiang, Haiying Wang, Reda Alhajj, Xiaohua Hu, Felix Engel, Mufti Mahmud, Nadia Pisanti, Xuefeng Cui, Hong Song, Institute of Electrical and Electronics Engineers Inc., 2023. 4834-4839.
BibTeX: Download