Bio-Signal Based Multimodal Fusion with Bilinear Model for Emotion Recognition

Singh A, Holzer N, Gotz T, Wittenberg T, Gob S, Sawant S, Salman MM, Pahl J (2023)


Publication Type: Conference contribution

Publication year: 2023

Publisher: Institute of Electrical and Electronics Engineers Inc.

Pages Range: 4834-4839

Conference Proceedings Title: Proceedings - 2023 2023 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2023

Event location: Istanbul, TUR

ISBN: 9798350337488

DOI: 10.1109/BIBM58861.2023.10385273

Abstract

In this paper, we propose a feature-based multimodal fusion deep-learning approach to combine biosignals, such as electrocardiogram (ECG) and galvanic skin response (GSR) for effective emotion recognition. We make use of a bilinear fusion method to fuse ECG and GSR based features and to jointly capture and complement the complex information from both modalities to enhance the performance of an emotion recognition system. The performance of the proposed fusion approach is evaluated on the MAHNOB-HCI dataset and compared with existing multimodal fusion approaches. Experimental results show that the combination of ECG and GSR features have a high discriminating ability in classifying (or predicting) the emotional states 'Arousal' and 'Valence' better than that of a single modality based emotion recognition. We also provide the model's uncertainty analysis to understand how confident the obtained predictions are.

Authors with CRIS profile

Involved external institutions

How to cite

APA:

Singh, A., Holzer, N., Gotz, T., Wittenberg, T., Gob, S., Sawant, S.,... Pahl, J. (2023). Bio-Signal Based Multimodal Fusion with Bilinear Model for Emotion Recognition. In Xingpeng Jiang, Haiying Wang, Reda Alhajj, Xiaohua Hu, Felix Engel, Mufti Mahmud, Nadia Pisanti, Xuefeng Cui, Hong Song (Eds.), Proceedings - 2023 2023 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2023 (pp. 4834-4839). Istanbul, TUR: Institute of Electrical and Electronics Engineers Inc..

MLA:

Singh, Ashutosh, et al. "Bio-Signal Based Multimodal Fusion with Bilinear Model for Emotion Recognition." Proceedings of the 2023 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2023, Istanbul, TUR Ed. Xingpeng Jiang, Haiying Wang, Reda Alhajj, Xiaohua Hu, Felix Engel, Mufti Mahmud, Nadia Pisanti, Xuefeng Cui, Hong Song, Institute of Electrical and Electronics Engineers Inc., 2023. 4834-4839.

BibTeX: Download