How Intermodal Interaction Affects the Performance of Deep Multimodal Fusion for Mixed-Type Time Series

Dietz S, Altstidl TR, Zanca D, Eskofier B, Nguyen A (2024)

Publication Language: English

Publication Type: Conference contribution, Conference Contribution

Publication year: 2024

Event location: Yokohama

ISBN: 979-8-3503-5931-2

URI: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10650421

DOI: 10.1109/IJCNN60899.2024.10650421

Abstract

Mixed-type time series (MTTS) is a bimodal data type that is common in many domains, such as healthcare, finance, environmental monitoring, and social media. It consists of regularly sampled continuous time series and irregularly sampled categorical event sequences. The integration of both modalities through multimodal fusion is a promising approach for processing MTTS. However, the question of how to effectively fuse both modalities remains open. In this paper, we present a comprehensive evaluation of several deep multimodal fusion approaches for MTTS forecasting. Our comparison includes different fusion types (early, intermediate, and late) and fusion methods (concatenation, weighted mean, weighted mean with correlation, gating, and feature sharing). We evaluate these fusion approaches on three distinct datasets, one of which was generated using a novel framework. This framework allows for the control of key data properties, such as the strength and direction of intermodal interactions, modality imbalance, and the degree of randomness in each modality, providing a more controlled environment for testing fusion approaches. Our findings show that the performance of different fusion approaches can be substantially influenced by the direction and strength of intermodal interactions. The study reveals that early and intermediate fusion approaches excel at capturing fine-grained and coarse-grained cross-modal features, respectively. These findings underscore the crucial role of intermodal interactions in determining the most effective fusion strategy for MTTS forecasting.

Authors with CRIS profile

Additional Organisation(s)

How to cite

APA:

Dietz, S., Altstidl, T.R., Zanca, D., Eskofier, B., & Nguyen, A. (2024). How Intermodal Interaction Affects the Performance of Deep Multimodal Fusion for Mixed-Type Time Series. In IEEE Computational Intelligence Society (Eds.), Proceedings of the International Joint Conference on Neural Networks (IJCNN). Yokohama, JP.

MLA:

Dietz, Simon, et al. "How Intermodal Interaction Affects the Performance of Deep Multimodal Fusion for Mixed-Type Time Series." Proceedings of the International Joint Conference on Neural Networks (IJCNN), Yokohama Ed. IEEE Computational Intelligence Society, 2024.

BibTeX: Download