Video Summarization Through Reinforcement Learning with a 3D Spatiooral U-Net

Liu T, Meng Q, Huang JJ, Vlontzos A, Rueckert D, Kainz B (2022)

Publication Type: Journal article

Publication year: 2022

Journal

IEEE Transactions on Image Processing Institute of Electrical and Electronics Engineers (IEEE)

Book Volume: 31

Pages Range: 1573-1586

DOI: 10.1109/TIP.2022.3143699

Abstract

Intelligent video summarization algorithms allow to quickly convey the most relevant information in videos through the identification of the most essential and explanatory content while removing redundant video frames. In this paper, we introduce the 3DST-UNet-RL framework for video summarization. A 3D spatiooral U-Net is used to efficiently encode spatiooral information of the input videos for downstream reinforcement learning (RL). An RL agent learns from spatiooral latent scores and predicts actions for keeping or rejecting a video frame in a video summary. We investigate if real/inflated 3D spatiooral CNN features are better suited to learn representations from videos than commonly used 2D image features. Our framework can operate in both, a fully unsupervised mode and a supervised training mode. We analyse the impact of prescribed summary lengths and show experimental evidence for the effectiveness of 3DST-UNet-RL on two commonly used general video summarization benchmarks. We also applied our method on a medical video summarization task. The proposed video summarization method has the potential to save storage costs of ultrasound screening videos as well as to increase efficiency when browsing patient video data during retrospective analysis or audit without loosing essential information.

Authors with CRIS profile

Bernhard Kainz Professur für Image Data Exploration and Analysis

Involved external institutions

Imperial College London / The Imperial College of Science, Technology and Medicine GB United Kingdom (GB) National University of Defense Technology CN China (CN)

How to cite

APA:

Liu, T., Meng, Q., Huang, J.-J., Vlontzos, A., Rueckert, D., & Kainz, B. (2022). Video Summarization Through Reinforcement Learning with a 3D Spatiooral U-Net. IEEE Transactions on Image Processing, 31, 1573-1586. https://dx.doi.org/10.1109/TIP.2022.3143699

MLA:

Liu, Tianrui, et al. "Video Summarization Through Reinforcement Learning with a 3D Spatiooral U-Net." IEEE Transactions on Image Processing 31 (2022): 1573-1586.

BibTeX: Download