Multi-Task Edge Prediction in Temporally-Dynamic Video Graphs

Ülger O, Wiederer J, Ghafoorian M, Belagiannis V, Mettes P (2022)

Publication Type: Conference contribution

Publication year: 2022

Publisher: British Machine Vision Association, BMVA

Conference Proceedings Title: BMVC 2022 - 33rd British Machine Vision Conference Proceedings

Event location: London, GBR

Abstract

Graph neural networks have shown to learn effective node representations, enabling node-, link-, and graph-level inference. Conventional graph networks assume static relations between nodes, while relations between entities in a video often evolve over time, with nodes entering and exiting dynamically. In such temporally-dynamic graphs, a core problem is inferring the future state of spatio-temporal edges, which can constitute multiple types of relations. To address this problem, we propose MTD-GNN, a graph network for predicting temporally-dynamic edges for multiple types of relations. We propose a factorized spatio-temporal graph attention layer to learn dynamic node representations and present a multi-task edge prediction loss that models multiple relations simultaneously. The proposed architecture operates on top of scene graphs that we obtain from videos through object detection and spatio-temporal linking. Experimental evaluations on ActionGenome and CLEVRER show that modeling multiple relations in our temporally-dynamic graph network can be mutually beneficial, outperforming existing static and spatio-temporal graph neural networks, as well as state-of-the-art predicate classification methods. Code is available at https://github.com/ozzyou/MTD-GNN.

Authors with CRIS profile

Vasileios Belagiannis Professur für Maschinelles Lernen in der Signalverarbeitung

Involved external institutions

University of Amsterdam NL Netherlands (NL) Daimler AG DE Germany (DE)

How to cite

APA:

Ülger, O., Wiederer, J., Ghafoorian, M., Belagiannis, V., & Mettes, P. (2022). Multi-Task Edge Prediction in Temporally-Dynamic Video Graphs. In BMVC 2022 - 33rd British Machine Vision Conference Proceedings. London, GBR: British Machine Vision Association, BMVA.

MLA:

Ülger, Osman, et al. "Multi-Task Edge Prediction in Temporally-Dynamic Video Graphs." Proceedings of the 33rd British Machine Vision Conference Proceedings, BMVC 2022, London, GBR British Machine Vision Association, BMVA, 2022.

BibTeX: Download