Audiovisual Database with 360° Video and Higher-Order Ambisonics Audio for Perception, Cognition, Behavior, and QoE Evaluation Research - Robotham, Thomas; Singla, Ashutosh; Rummukainen, Olli S.; Raake, Alexander; Habets, Emanuël (2022 14th International Conference on Quality of Multimedia Experience, QoMEX 2022, 2022)
Research into multi-modal perception, human cog-nition, behavior, and attention can benefit from high-fidelity content that may recreate real-life-like scenes when rendered on head-mounted displays. Moreover, aspects of audiovisual perception, cognitive processes, and behavior may complement questionnaire-based Quality of Experience (QoE) evaluation of interactive virtual environments. Currently, there is a lack of high-quality open-source audiovisual databases that can be used to evaluate such aspects or systems capable of reproducing high-quality content. With this paper, we provide a publicly available audiovisual database consisting of twelve scenes capturing real-life nature and urban environments with a video resolution of 7680×3840 at 60 frames-per-second and with 4th-order Ambison-ics audio. These 360° video sequences, with an average duration of 60 seconds, represent real-life settings for systematically evaluating various dimensions of uni-/multi-modal perception, cognition, behavior, and QoE. The paper provides details of the scene requirements, recording approach, and scene descriptions. The database provides high-quality reference material with a balanced focus on auditory and visual sensory information. The database will be continuously updated with additional scenes and further metadata such as human ratings and saliency information.
CRIS-Team Scopus Importer:2022-11-25
Keywords: Ambison-ics; cognition; database; Omni-directional video; perception; quality; spatial audio

Comparing Direct and Indirect Methods of Audio Quality Evaluation in Virtual Reality Scenes of Varying Complexity - Robotham, Thomas; Rummukainen, Olli S.; Kurz, Miriam; Eckert, Marie; Habets, Emanuël (IEEE Transactions on Visualization and Computer Graphics, 2022)
Many quality evaluation methods are used to assess uni-modal audio or video content without considering perceptual, cognitive, and interactive aspects present in virtual reality (VR) settings. Consequently, little is known regarding the repercussions of the employed evaluation method, content, and subject behavior on the quality ratings in VR. This mixed between- and within-subjects study uses four subjective audio quality evaluation methods (viz. multiple-stimulus with and without reference for direct scaling, and rank-order elimination and pairwise comparison for indirect scaling) to investigate the contributing factors present in multi-modal 6-DoF VR on quality ratings of real-time audio rendering. For each between-subjects employed method, two sets of conditions in five VR scenes were evaluated within-subjects. The conditions targeted relevant attributes for binaural audio reproduction using scenes with various amounts of user interactivity. Our results show all referenceless methods produce similar results using both condition sets. However, rank-order elimination proved to be the fastest method, required the least amount of repetitive motion, and yielded the highest discrimination between spatial conditions. Scene complexity was found to be a main effect within results, with behavioral and task load index results implying more complex scenes and interactive aspects of 6-DoF VR can impede quality judgments.
CRIS-Team WoS Importer:2022-06-17

Dialogue Enhancement and Listening Effort in Broadcast Audio: A Multimodal Evaluation - Torcoli, Matteo; Robotham, Thomas; Habets, Emanuël (2022 14th International Conference on Quality of Multimedia Experience, QoMEX 2022, 2022)
Dialogue enhancement (DE) plays a vital role in broadcasting, enabling the personalization of the relative level between foreground speech and background music and effects. DE has been shown to improve the quality of experience, intel-ligibility, and self-reported listening effort (LE). A physiological indicator of LE known from audiology studies is pupil size. The relation between pupil size and LE is typically studied using artificial sentences and background noises not encountered in broadcast content. This work evaluates the effect of DE on LE in a multimodal manner that includes pupil size (tracked by a VR headset) and real-world audio excerpts from TV. Under ideal listening conditions, 28 normal-hearing participants listened to 30 audio excerpts presented in random order and processed by conditions varying the relative level between foreground and background audio. One of these conditions employed a recently proposed source separation system to attenuate the background given the original mixture as the sole input. After listening to each excerpt, subjects were asked to repeat the heard sentence and self-report the LE. Mean pupil dilation and peak pupil dilation were analyzed and compared with the self-report and the word recall rate. The multimodal evaluation shows a consistent trend of decreasing LE along with decreasing background level. DE, also when enabled by source separation, significantly reduces the pupil size as well as the self-reported LE. This highlights the benefit of personalization functionalities at the user's end.
CRIS-Team Scopus Importer:2022-11-25
Keywords: broadcast; dialogue; listening effort; pupil

Head-Related Transfer Functions for Dynamic Listeners in Virtual Reality - Rummukainen, Olli S.; Robotham, Thomas; Habets, Emanuël (Applied Sciences, 2021)
In dynamic virtual reality, visual cues and motor actions aid auditory perception. With multimodal integration and auditory adaptation effects, generic head-related transfer functions (HRTFs) may yield no significant disadvantage to individual HRTFs regarding accurate auditory perception. This study compares two individual HRTF sets against a generic HRTF set by way of objective analysis and two subjective experiments. First, auditory-model-based predictions examine the objective deviations in localization cues between the sets. Next, the HRTFs are compared in a static subjective (N = 8) localization experiment. Finally, the localization accuracy, timbre, and overall quality of the HRTF sets are evaluated subjectively (N = 12) in a six-degrees-of-freedom audio-visual virtual environment. The results show statistically significant objective deviations between the sets, but no perceived localization or overall quality differences in the dynamic virtual reality.
CRIS-Team WoS Importer:2021-08-06

Listening Tests with Individual verses Generic Head-Related Transfer Functions in Six-Degrees-of-Freedom Virtual Reality - Rummukainen, Olli S.; Robotham, Thomas; Plinge, Axel; Wefers, Frank; Herre, Jürgen; Habets, Emanuël (5th International Conference on Spatial Audio ICSA, 2019)
Vorträge und Veröffentlichungen auf Konferenzen_Virtual and Augmented Reality

Perceptual Evaluation of Interior Panning Algorithms Using Static Auditory Events - Robotham, Thomas; Silzle, Andreas; Nastasa, Anamaria; Pawlak, Alan; Herre, Jürgen (151st AES Convention, 2021)
herre_papers_virtual_and_augmented_reality_rendering
Keywords: Best Peer Reviewed Paper Award

Pupillary Light Reflex Correction for Robust Pupillometry in Virtual Reality - Eckert, Mirjam; Robotham, Thomas; Habets, Emanuël; Rummukainen, Olli S. (Proceedings of the ACM on Computer Graphics and Interactive Techniques, 2022)
Virtual reality (VR) headsets with an integrated eye tracker enable the measurement of pupil size fluctuations correlated with cognition during a VR experience. We present a method to correct for the light-induced pupil size changes, otherwise masking the more subtle cognitively-driven effects, such as cognitive load and emotional state. We explore multiple calibration sequences to find individual mapping functions relating the luminance to pupil dilation that can be employed in real-time during a VR experience. The resulting mapping functions are evaluated in a VR-based n-back task and in free exploration of a six-degrees-of-freedom VR scene. Our results show estimating luminance from a weighted average of the fixation area and the background yields the best performance. Calibration sequence composed of either solid gray or realistic scene brightness levels shown for 6 s in a pseudo-random order proved most robust.
CRIS-Team WoS Importer:2022-06-03