Video Summarization Through Reinforcement Learning with a 3D Spatiooral U-Net

Liu T, Meng Q, Huang JJ, Vlontzos A, Rueckert D, Kainz B (2022)

Publication Type: Journal article

Publication year: 2022


Book Volume: 31

Pages Range: 1573-1586

DOI: 10.1109/TIP.2022.3143699


Intelligent video summarization algorithms allow to quickly convey the most relevant information in videos through the identification of the most essential and explanatory content while removing redundant video frames. In this paper, we introduce the 3DST-UNet-RL framework for video summarization. A 3D spatiooral U-Net is used to efficiently encode spatiooral information of the input videos for downstream reinforcement learning (RL). An RL agent learns from spatiooral latent scores and predicts actions for keeping or rejecting a video frame in a video summary. We investigate if real/inflated 3D spatiooral CNN features are better suited to learn representations from videos than commonly used 2D image features. Our framework can operate in both, a fully unsupervised mode and a supervised training mode. We analyse the impact of prescribed summary lengths and show experimental evidence for the effectiveness of 3DST-UNet-RL on two commonly used general video summarization benchmarks. We also applied our method on a medical video summarization task. The proposed video summarization method has the potential to save storage costs of ultrasound screening videos as well as to increase efficiency when browsing patient video data during retrospective analysis or audit without loosing essential information.

Authors with CRIS profile

Involved external institutions

How to cite


Liu, T., Meng, Q., Huang, J.-J., Vlontzos, A., Rueckert, D., & Kainz, B. (2022). Video Summarization Through Reinforcement Learning with a 3D Spatiooral U-Net. IEEE Transactions on Image Processing, 31, 1573-1586.


Liu, Tianrui, et al. "Video Summarization Through Reinforcement Learning with a 3D Spatiooral U-Net." IEEE Transactions on Image Processing 31 (2022): 1573-1586.

BibTeX: Download