Dietz S, Altstidl TR, Zanca D, Eskofier B, Nguyen A (2024)
Publication Language: English
Publication Type: Conference contribution, Conference Contribution
Publication year: 2024
ISBN: 979-8-3503-5931-2
URI: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10650421
DOI: 10.1109/IJCNN60899.2024.10650421
Mixed-type time series (MTTS) is a bimodal data
type that is common in many domains, such as healthcare, finance, environmental monitoring, and social media. It consists
of regularly sampled continuous time series and irregularly
sampled categorical event sequences. The integration of both
modalities through multimodal fusion is a promising approach
for processing MTTS. However, the question of how to effectively
fuse both modalities remains open. In this paper, we present
a comprehensive evaluation of several deep multimodal fusion
approaches for MTTS forecasting. Our comparison includes
different fusion types (early, intermediate, and late) and fusion
methods (concatenation, weighted mean, weighted mean with
correlation, gating, and feature sharing). We evaluate these fusion
approaches on three distinct datasets, one of which was generated using a novel framework. This framework allows for the control
of key data properties, such as the strength and direction of
intermodal interactions, modality imbalance, and the degree
of randomness in each modality, providing a more controlled
environment for testing fusion approaches. Our findings show
that the performance of different fusion approaches can be substantially influenced by the direction and strength of intermodal interactions. The study reveals that early and intermediate fusion
approaches excel at capturing fine-grained and coarse-grained
cross-modal features, respectively. These findings underscore the
crucial role of intermodal interactions in determining the most
effective fusion strategy for MTTS forecasting.
APA:
Dietz, S., Altstidl, T.R., Zanca, D., Eskofier, B., & Nguyen, A. (2024). How Intermodal Interaction Affects the Performance of Deep Multimodal Fusion for Mixed-Type Time Series. In IEEE Computational Intelligence Society (Eds.), Proceedings of the International Joint Conference on Neural Networks (IJCNN). Yokohama, JP.
MLA:
Dietz, Simon, et al. "How Intermodal Interaction Affects the Performance of Deep Multimodal Fusion for Mixed-Type Time Series." Proceedings of the International Joint Conference on Neural Networks (IJCNN), Yokohama Ed. IEEE Computational Intelligence Society, 2024.
BibTeX: Download