Hearing Your Way Through Music Recordings: A Text Alignment and Synthesis Approach
Strahl S, Özer Y, Berendes HU, Müller M (2025)
Publication Language: English
Publication Type: Conference contribution, Conference Contribution
Publication year: 2025
Pages Range: 65-72
Conference Proceedings Title: Proceedings of the Sound and Music Computing Conference (SMC)
Event location: Graz
Abstract
Annotations related to musical events such as chord labels, measure numbers, or structural descriptions are typically provided in textual format within or alongside a score-based representation of a piece. However, following these annotations while listening to a recording can be challenging without additional visual or auditory display. In this paper, we introduce an approach for enriching the listening experience by mixing music recordings with synthesized text annotations. Our approach aligns text annotations from a score-based timeline to the timeline of a specific recording and then utilizes text-to-speech synthesis to acoustically superimpose them with the recording. We describe a processing pipeline for implementing this approach, allowing users to customize settings such as speaking language, speed, speech positioning, and loudness. Case studies include synthesizing text comments on measure positions in Schubert songs, chord annotations for Beatles songs, structural elements of Beethoven piano sonatas, and leitmotif occurrences in Wagner operas. Beyond these specific examples, our aim is to highlight the broader potential of speech-based auditory display. This approach offers valuable tools for researchers seeking a deeper understanding of datasets and their annotations, for evaluating music information retrieval algorithms, or for educational purposes in instrumental training, music-making, and aural training.
Authors with CRIS profile
Sebastian Strahl
International Audio Laboratories Erlangen (AudioLabs)
Yigitcan Özer
International Audio Laboratories Erlangen (AudioLabs)
Hans-Ulrich Berendes
International Audio Laboratories Erlangen (AudioLabs)
Meinard Müller
Lehrstuhl für Semantische Audiosignalverarbeitung (AudioLabs)
How to cite
APA:
Strahl, S., Özer, Y., Berendes, H.-U., & Müller, M. (2025). Hearing Your Way Through Music Recordings: A Text Alignment and Synthesis Approach. In Proceedings of the Sound and Music Computing Conference (SMC) (pp. 65-72). Graz, AT.
MLA:
Strahl, Sebastian, et al. "Hearing Your Way Through Music Recordings: A Text Alignment and Synthesis Approach." Proceedings of the Sound and Music Computing Conference (SMC), Graz 2025. 65-72.
BibTeX: Download