A Cross-Version Approach to Audio Representation Learning for Orchestral Music - Krause, Michael; Weiß, Christof; Müller, Meinard (Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2023)
Deep learning systems have become popular for tackling a variety of music information retrieval tasks. However, these systems often require large amounts of labeled data for supervised training, which can be very costly to obtain. To alleviate this problem, recent papers on learning music audio representations employ alternative training strategies that utilize unannotated data. In this paper, we introduce a novel cross-version approach to audio representation learning that can be used with music datasets containing several versions (performances) of a musical work. Our method exploits the correspondences that exist between two versions of the same musical section. We evaluate our proposed cross-version approach qualitatively and quantitatively on complex orchestral music recordings and show that it can better capture aspects of instrumentation compared to techniques that do not use cross-version information.<br

A Dataset of Larynx Microphone Recordings for Singing Voice Reconstruction - Schwär, Simon; Krause, Michael; Fast, Michael; Rosenzweig, Sebastian; Scherbaum, Frank; Müller, Meinard (Transactions of the International Society for Music Information Retrieval, 2024)
ading: A Dataset of Larynx Microphone Recordings for Singing Voice Reconstruction Download A- Decrease article font size A+ Increase article font size Cite this article Print this article Alt. DisplayAlternative display Share: Share on Facebook Share on X Share on LinkedIn Share as email Dataset articles A Dataset of Larynx Microphone Recordings for Singing Voice Reconstruction Authors: Simon SchwärEmail Simon Schwär Michael Krause Michael Fast Sebastian Rosenzweig Frank Scherbaum Meinard Müller Toggle author information panel Abstract Larynx microphones (LMs) make it possible to obtain practically crosstalk-free recordings of the human voice by picking up vibrations directly from the throat. This can be useful in a multitude of music information retrieval scenarios related to singing, e.g., the analysis of individual voices recorded in environments with lots of interfering noise. However, LMs have a limited frequency range and barely capture the effects of the vocal tract, which makes the recorded signal unsuitable for downstream tasks that require high-quality recordings. In this paper, we introduce the task of reconstructing a natural sounding, high-quality singing voice recording from an LM signal. With an explicit focus on the singing voice, the problem lies at the intersection of speech enhancement and singing voice synthesis with the additional requirement of faithful reproduction of expressive parameters like intonation. In this context, we make three main contributions. First, we publish a dataset with over 4 hours of popular music we recorded with four amateur singers accompanied by a guitar, where both LM and clean close-up microphone signals are available. Second, we propose a data-driven baseline approach for singing voice reconstruction from LM signals using differentiable signal processing, inspired by a source-filter model that emulates the missing vocal tract effects. Third, we evaluate the baseline with a listening test and further show that it can improve the accuracy of lyrics transcription as an exemplary downstream tas
Keywords: Larynx Microphone; Singing Voice Reconstruction; Dataset; Differentiable Signal Processing; Singing Analysis

Classifying Leitmotifs in Recordings of Operas by Richard Wagner - Krause, Michael; Zalkow, Frank; Zalkow, Julia; Weiß, Christof; Müller, Meinard (Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2020)

Deep Learning and Knowledge Integration for Music Audio Analysis (Dagstuhl Seminar 22082) - Müller, Meinard; Bittner, Rachel; Nam, Juhan; Krause, Michael; Özer, Yigitcan (2022)
Given the increasing amount of digital music, the development of computational tools that allow users to find, organize, analyze, and interact with music has become central to the research field known as Music Information Retrieval (MIR). As in general multimedia processing, many of the recent advances in MIR have been driven by techniques based on deep learning (DL). There is a growing trend to relax problem-specific modeling constraints from MIR systems and instead apply relatively generic DL-based approaches that rely on large quantities of data. In the Dagstuhl Seminar 22082, we critically examined this trend, discussing the strengths and weaknesses of these approaches using music as a challenging application domain. We mainly focused on music analysis tasks applied to audio representations (rather than symbolic music representations) to give the seminar cohesion. In this context, we systematically explored how musical knowledge can be integrated into or relaxed from computational pipelines. We then discussed how this choice could affect the explainability of models or the vulnerability to data biases and confounding factors. Furthermore, besides explainability and generalization, we also addressed efficiency, ethical and educational aspects considering traditional model-based and recent data-driven methods. In this report, we give an overview of the various contributions and results of the seminar. We start with an executive summary describing the main topics, goals, and group activities. Then, we give an overview of the participants' stimulus talks and subsequent discussions (listed alphabetically by the main contributor’s last name) and summarize further activities, including group discussions and music sessions. <br

Hierarchical Classification for Instrument Activity Detection in Orchestral Music Recordings - Krause, Michael; Müller, Meinard (IEEE/ACM Transactions on Audio, Speech and Language Processing, 2023)
Instrument activity detection is a fundamental task in music information retrieval, serving as a basis for many applications, such as music recommendation, music tagging, or remixing. Most published works on this task cover popular music and music for smaller ensembles. In this paper, we embrace orchestral and opera music recordings as a rarely considered scenario for automated instrument activity detection. Orchestral music is particularly challenging since it consists of intricate polyphonic and polytimbral sound mixtures where multiple instruments are playing simultaneously. Orchestral instruments can naturally be arranged in hierarchical taxonomies, according to instrument families. As the main contribution of this paper, we show that a hierarchical classification approach can be used to detect instrument activity in our scenario, even if only few fine-grained, instrument-level annotations are available. We further consider additional loss terms for improving the hierarchical consistency of predictions. For our experiments, we collect a dataset containing 14 hours of orchestral music recordings with aligned instrument activity annotations. Finally, we perform an analysis of the behavior of our proposed approach with regard to potential confounding errors.
CRIS-Team Scopus Importer:2023-07-21
Keywords: Analytical models; Annotations; hierarchical classification; instrument activity detection; Instruments; Music; music information retrieval; music processing; orchestral music; Recording; Speech processing; Task analysis

HIERARCHICAL CLASSIFICATION OF SINGING ACTIVITY, GENDER, AND TYPE IN COMPLEX MUSIC RECORDINGS - Krause, Michael; Müller, Meinard (2022 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 2022)
CRIS-Team WoS Importer:2023-01-13

Singing Voice Detection in Opera Recordings: A Case Study on Robustness and Generalization - Krause, Michael; Müller, Meinard; Weiß, Christof (Electronics, 2021)
Automatically detecting the presence of singing in music audio recordings is a central task within music information retrieval. While modern machine-learning systems produce high-quality results on this task, the reported experiments are usually limited to popular music and the trained systems often overfit to confounding factors. In this paper, we aim to gain a deeper understanding of such machine-learning methods and investigate their robustness in a challenging opera scenario. To this end, we compare two state-of-the-art methods for singing voice detection based on supervised learning: A traditional approach relying on hand-crafted features with a random forest classifier, as well as a deep-learning approach relying on convolutional neural networks. To evaluate these algorithms, we make use of a cross-version dataset comprising 16 recorded performances (versions) of Richard Wagner's four-opera cycle Der Ring des Nibelungen. This scenario allows us to systematically investigate generalization to unseen versions, musical works, or both. In particular, we study the trained systems' robustness depending on the acoustic and musical variety, as well as the overall size of the training dataset. Our experiments show that both systems can robustly detect singing voice in opera recordings even when trained on relatively small datasets with little variety.
CRIS-Team WoS Importer:2021-06-04

Soft Dynamic Time Warping for Multi-Pitch Estimation and Beyond - Krause, Michael; Weiß, Christof; Müller, Meinard (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 2023)
Many tasks in music information retrieval (MIR) involve weakly aligned data, where exact temporal correspondences are unknown. The connectionist temporal classification (CTC) loss is a standard technique to learn feature representations based on weakly aligned training data. However, CTC is limited to discrete-valued target sequences and can be difficult to extend to multi-label problems. In this article, we show how soft dynamic time warping (SoftDTW), a differentiable variant of classical DTW, can be used as an alternative to CTC. Using multi-pitch estimation as an example scenario, we show that SoftDTW yields results on par with a state-of-the-art multi-label extension of CTC. In addition to being more elegant in terms of its algorithmic formulation, SoftDTW naturally extends to real-valued target sequences.
Created from Fastlane, Scopus look-up
Keywords: dynamic time warping; multi-pitch estimation; music information retrieval; music processing; music transcription

Soft Dynamic Time Warping with Variable Step Weights - Zeitler, Johannes; Krause, Michael; Müller, Meinard (Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2024)

Stabilizing Training with Soft Dynamic Time Warping: A Case Study for Pitch Class Estimation with Weakly Aligned Targets - Zeitler, Johannes; Deniffel, Simon; Krause, Michael; Müller, Meinard (Proceedings of the 24th International Society for Music Information Retrieval Conference, 2023)
Soft dynamic time warping (SDTW) is a differentiable loss function that allows for training neural networks from weakly aligned data. Typically, SDTW is used to iteratively compute and refine soft alignments that compensate for temporal deviations between the training data and its weakly annotated targets. One major problem is that a mismatch between the estimated soft alignments and the reference alignments in the early training stage leads to incorrect parameter updates, making the overall training procedure unstable. In this paper, we investigate such stability issues by considering the task of pitch class estimation from music recordings as an illustrative case study. In particular, we introduce and discuss three conceptually different strategies (a hyperparameter scheduling, a diagonal prior, and a sequence unfolding strategy) with the objective of stabilizing intermediate soft alignment results. Finally, we report on experiments that demonstrate the effectiveness of the strategies and discuss efficiency and implementation issues.<br

Sync Toolbox: A Python Package for Efficient, Robust, and Accurate Music Synchronization - Müller, Meinard; Özer, Yigitcan; Krause, Michael; Prätzlich, Thomas; Driedger, Jonathan (Journal of Open Source Software, 2021)
Music can be described and represented in many different ways, including as sheet music, symbolic representations, and audio recordings (Müller, 2015). For each of these representations, different versions (e.g., recordings performed by different orchestras and conductors) that correspond to the same musical work may exist. Music information retrieval (MIR) aims at developing techniques and tools for organizing, understanding, and searching this information in a robust, efficient, and intelligent manner. In this context, various alignment and synchronization procedures have been developed with the common goal of automatically linking several types of music representations, thus coordinating the multiple information sources related to a given musical work. In the design and implementation of synchronization algorithms, one has to deal with a delicate tradeoff between efficiency, robustness, and accuracy—requirements leading to various approaches with many design choices. In this contribution, we introduce a Python package called Sync Toolbox that provides open-source reference implementations for full-fledged music synchronization pipelines and yields state-of-the-art alignment results for a wide range of Western music. Using suitable feature representations and cost measures, the toolbox’s core technology is based on dynamic time warping (DTW), which brings the feature sequences into temporal correspondence. To account for efficiency, robustness and, accuracy, our toolbox integrates and combines techniques such as multiscale DTW (MsDTW) (Müller et al., 2006; Salvador & Chan, 2004), memory-restricted MsDTW (MrMsDTW) (Prätzlich et al., 2016), and high-resolution music synchronization (Ewert et al., 2009). While realizing a complete system with presets that allow users to reproduce research results from the literature, our toolbox also provides well-documented functions for all basic building blocks required for feature extraction and alignment. Furthermore, the toolbox contains example code for visualizing, sonifying, and evaluating synchronization results, thus deepening the understanding of the techniques and dat

Towards Leitmotif Activity Detection in Opera Recordings - Krause, Michael; Müller, Meinard; Weiß, Christof (Transactions of the International Society for Music Information Retrieval, 2021)
This paper approaches the automatic detection of musical patterns in audio recordings with a particular focus on leitmotifs, which are specific types of patterns associated with certain characters, places, items, or feelings occurring in an opera or movie soundtrack. The detection of such leitmotifs is particularly challenging since their appearance can change substantially over the course of a musical work. In our case study, we consider a self-contained yet comprehensive scenario comprising 16 recorded performances of Richard Wagner’s four-opera cycle Der Ring des Nibelungen, which is a prime example for the use of leitmotifs. Within this scenario, we introduce and formalize the novel task of leitmotif activity detection. Based on a dataset of 200 hours of audio with over 50 000 annotated leitmotif instances, we explore the benefits and limitations of deep-learning techniques for detecting leitmotifs. To this end, we adapt two common deep-learning strategies based on recurrent and convolutional neural networks, respectively. To investigate the robustness of the trained systems, we test their sensitivity to different modifications of the input. We find that our deep-learning systems work well in general but capture confounding factors, such as pitch distributions in leitmotif regions, instead of characteristic musical properties, such as rhythm and melody. Thus, our in-depth analysis demonstrates some challenges that may arise from applying deep-learning approaches for detecting complex musical patterns in audio recordings.<br
Keywords: leitmotifs; opera; musical patterns; deep neural networks; sound event detection

Using the Sync Toolbox for an Experiment on High-Resolution Music Alignment - Özer, Yigitcan; Krause, Michael; Müller, Meinard (Demos and Late Breaking News of the International Society for Music Information Retrieval Conference (ISMIR), 2021)

Wagner Ring Dataset: A Complex Opera Scenario for Music Processing and Computational Musicology - Weiß, Christof; Arifi-Müller, Vlora; Krause, Michael; Zalkow, Frank; Klauk, Stephanie; Kleinertz, Rainer; Müller, Meinard (Transactions of the International Society for Music Information Retrieval, 2023)
This paper introduces the Wagner Ring Dataset (WRD), a multi-modal and multi-version resource on the large-scale opera cycle Der Ring des Nibelungen by Richard Wagner. The Ring comprises four music dramas organized into eleven acts and 21 939 measures in total. Concerning sheet music, we processed a publicly available piano reduction (822 pages) of the full score with optical music recognition followed by extensive manual corrections to create a high-quality, machine-readable symbolic score. Concerning audio data, our corpus covers 16 recorded performances of the full Ring (three of which are publicly available thanks to copyright expiry), each lasting about 14–15 hours. To musically synchronize these versions among each other, we manually annotated all measure positions for three performances, which we transferred to the remaining performances via automated synchronization techniques. The dataset further comprises annotations of key and time signatures, scenes, and singing voice regions (libretto). Moreover, we provide note event annotations for all performances derived from the piano score. The WRD thus constitutes a comprehensive resource for developing algorithms for various music information retrieval tasks, complementing existing datasets with a complex opera scenario. For computational musicology, the WRD serves as a structured dataset that allows for studying the composition and performances of the Ring.<br
Keywords: Dataset; Multi-Modal; Cross-Version; Music Processing; Computational Musicology

Weakly Supervised Multi-Pitch Estimation Using Cross-Version Alignment - Krause, Michael; Strahl, Sebastian; Müller, Meinard (Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2023)
Multi-pitch estimation (MPE), the task of detecting active pitches within a polyphonic music recording, has garnered significant research interest in recent years. Most state-of-the-art approaches for MPE are based on deep networks trained using pitch annotations as targets. The success of current methods is therefore limited by the difficulty of obtaining large amounts of accurate annotations. In this paper, we propose a novel technique for learning MPE without any pitch annotations at all. Our approach exploits multiple recorded versions of a musical piece as surrogate targets. Given one version of a piece as input, we train a network to minimize the distance between its output and time-frequency representations of other versions of that piece. Since all versions are based on the same musical score, we hypothesize that the learned output corresponds to pitch estimates. To further ensure that this hypothesis holds, we incorporate domain knowledge about overtones and noise levels into the network. Overall, our method replaces strong pitch annotations with weaker and easier-to-obtain cross-version targets. In our experiments, we show that our proposed approach yields viable multi-pitch estimates and outperforms two baselines.<br