The Bjøntegaard Bible - Why your Way of Comparing Video Codecs May Be Wrong

Herglotz C, Och H, Meyer A, Ramasubbu G, Eichermüller L, Kränzler M, Brand F, Fischer K, Nguyen DT, Regensky A, Kaup A (2024)


Publication Language: English

Publication Type: Journal article, Original article

Publication year: 2024

Journal

Book Volume: 33

Pages Range: 987 - 1001

DOI: 10.1109/TIP.2023.3346695

Open Access Link: https://arxiv.org/abs/2304.12852

Abstract

In this paper, we provide an in-depth assessment on the Bjøntegaard Delta. We construct a large data set of video compression performance comparisons using a diverse set of metrics including PSNR, VMAF, bitrate, and processing energies. These metrics are evaluated for visual data types such as classic perspective video, 360° video, point clouds, and screen content. As compression technology, we consider multiple hybrid video codecs as well as state-of-the-art neural network based compression methods. Using additional supporting points inbetween standard points defined by parameters such as the quantization parameter, we assess the interpolation error of the Bjøntegaard-Delta (BD) calculus and its impact on the final BD value. From the analysis, we find that the BD calculus is most accurate in the standard application of rate-distortion comparisons with mean errors below 0.5 percentage points. For other applications and special cases, e.g., VMAF quality, energy considerations, or inter-codec comparisons, the errors are higher (up to 5 percentage points), but can be halved by using a higher number of supporting points. We finally come up with recommendations on how to use the BD calculus such that the validity of the resulting BD-values is maximized. Main recommendations are as follows: First, relative curve differences should be plotted and analyzed. Second, the logarithmic domain should be used for saturating metrics such as SSIM and VMAF. Third, BD values below a certain threshold indicated by the subset error should not be used to draw recommendations. Fourth, using two supporting points is sufficient to obtain rough performance estimates.

Authors with CRIS profile

How to cite

APA:

Herglotz, C., Och, H., Meyer, A., Ramasubbu, G., Eichermüller, L., Kränzler, M.,... Kaup, A. (2024). The Bjøntegaard Bible - Why your Way of Comparing Video Codecs May Be Wrong. IEEE Transactions on Image Processing, 33, 987 - 1001. https://dx.doi.org/10.1109/TIP.2023.3346695

MLA:

Herglotz, Christian, et al. "The Bjøntegaard Bible - Why your Way of Comparing Video Codecs May Be Wrong." IEEE Transactions on Image Processing 33 (2024): 987 - 1001.

BibTeX: Download