Comparing Direct and Indirect Methods of Audio Quality Evaluation in Virtual Reality Scenes of Varying Complexity

Robotham T, Rummukainen OS, Kurz M, Eckert M, Habets E (2022)

Publication Type: Journal article

Publication year: 2022

Journal

IEEE Transactions on Visualization and Computer Graphics Institute of Electrical and Electronics Engineers (IEEE)

Book Volume: 28

Pages Range: 2091-2101

Journal Issue: 5

DOI: 10.1109/TVCG.2022.3150491

Abstract

Many quality evaluation methods are used to assess uni-modal audio or video content without considering perceptual, cognitive, and interactive aspects present in virtual reality (VR) settings. Consequently, little is known regarding the repercussions of the employed evaluation method, content, and subject behavior on the quality ratings in VR. This mixed between- and within-subjects study uses four subjective audio quality evaluation methods (viz. multiple-stimulus with and without reference for direct scaling, and rank-order elimination and pairwise comparison for indirect scaling) to investigate the contributing factors present in multi-modal 6-DoF VR on quality ratings of real-time audio rendering. For each between-subjects employed method, two sets of conditions in five VR scenes were evaluated within-subjects. The conditions targeted relevant attributes for binaural audio reproduction using scenes with various amounts of user interactivity. Our results show all referenceless methods produce similar results using both condition sets. However, rank-order elimination proved to be the fastest method, required the least amount of repetitive motion, and yielded the highest discrimination between spatial conditions. Scene complexity was found to be a main effect within results, with behavioral and task load index results implying more complex scenes and interactive aspects of 6-DoF VR can impede quality judgments.

Authors with CRIS profile

Thomas Robotham International Audio Laboratories Erlangen (AudioLabs) Emanuël Habets Lehrstuhl für wahrnehmungsbasierte räumliche Audiosignalverarbeitung (Stiftungslehrstuhl)

Involved external institutions

Fraunhofer-Institut für Integrierte Schaltungen (IIS) DE Germany (DE)

How to cite

APA:

Robotham, T., Rummukainen, O.S., Kurz, M., Eckert, M., & Habets, E. (2022). Comparing Direct and Indirect Methods of Audio Quality Evaluation in Virtual Reality Scenes of Varying Complexity. IEEE Transactions on Visualization and Computer Graphics, 28(5), 2091-2101. https://doi.org/10.1109/TVCG.2022.3150491

MLA:

Robotham, Thomas, et al. "Comparing Direct and Indirect Methods of Audio Quality Evaluation in Virtual Reality Scenes of Varying Complexity." IEEE Transactions on Visualization and Computer Graphics 28.5 (2022): 2091-2101.

BibTeX: Download