A Cross-Version Approach to Audio Representation Learning for Orchestral Music - Krause, Michael; Weiß, Christof; Müller, Meinard (Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2023)
Deep learning systems have become popular for tackling a variety of music information retrieval tasks. However, these systems often require large amounts of labeled data for supervised training, which can be very costly to obtain. To alleviate this problem, recent papers on learning music audio representations employ alternative training strategies that utilize unannotated data. In this paper, we introduce a novel cross-version approach to audio representation learning that can be used with music datasets containing several versions (performances) of a musical work. Our method exploits the correspondences that exist between two versions of the same musical section. We evaluate our proposed cross-version approach qualitatively and quantitatively on complex orchestral music recordings and show that it can better capture aspects of instrumentation compared to techniques that do not use cross-version information.<br

A Multi-Version Approach for Transferring Measure Annotations Between Music Recordings - Zalkow, Frank; Weiß, Christof; Prätzlich, Thomas; Arifi-Müller, Vlora; Müller, Meinard (Proceedings of the AES Conference on Semantic Audio, 2017)

Analyzing Measure Annotations for Western Classical Music Recordings - Weiß, Christof; Arifi-Müller, Vlora; Prätzlich, Thomas; Kleinertz, Rainer; Müller, Meinard (Proceedings of the International Conference on Music Information Retrieval (ISMIR), 2016)

Book Review: Gareth E. Roberts, From music to mathematics: Exploring the connections - Müller, Meinard; Weiß, Christof (Musicae Scientiae, 2016)

Classifying Leitmotifs in Recordings of Operas by Richard Wagner - Krause, Michael; Zalkow, Frank; Zalkow, Julia; Weiß, Christof; Müller, Meinard (Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2020)

Computational Corpus Analysis: A Case Study on Jazz Solos - Weiß, Christof; Balke, Stefan; Abeßer, Jakob; Müller, Meinard (Proceedings of the International Conference on Music Information Retrieval (ISMIR), 2018)

Computergestützte Visualisierung Harmonischer Verläufe: Eine Fallstudie zu Wagners Ring - Weiß, Christof; Zalkow, Frank; Müller, Meinard; Klauk, Stephanie; Kleinertz, Rainer (Proceedings of the GI Jahrestagung, 2017)

Computergestützte Visualisierung von Tonalitätsverläufen in Musikaufnahmen. Möglichkeiten für die Korpusanalyse - Weiß, Christof; Müller, Meinard (Instrumentalmusik neben Haydn und Mozart. Analyse, Aufführungspraxis und Edition, 2021)

Cross-Version Singing Voice Detection in Opera Recordings: Challenges for Supervised Learning - Mimilakis, Stylianos Ioannis; Weiß, Christof; Arifi-Müller, Vlora; Abeßer, Jakob; Müller, Meinard (Proceedings of the International Workshop on Machine Learning and Music (MML), 2019)

Dagstuhl ChoirSet: A Multitrack Dataset for MIR Research on Choral Singing - Rosenzweig, Sebastian; Cuesta, Helena; Weiß, Christof; Scherbaum, Frank; Gómez, Emilia; Müller, Meinard (Transactions of the International Society for Music Information Retrieval, 2020)
Choral singing is a central part of musical cultures across the world, yet many facets of this widespread form of polyphonic singing are still to be explored. Music information retrieval (MIR) research on choral singing benefits from multitrack recordings of the individual singing voices. However, there exist only few publicly available multitrack datasets on polyphonic singing. In this paper, we present Dagstuhl ChoirSet (DCS), a multitrack dataset of a cappella choral music designed to support MIR research on choral singing. The dataset includes recordings of an amateur vocal ensemble performing two choir pieces in full choir and quartet settings. The audio data was recorded during an MIR seminar at Schloss Dagstuhl using different close-up microphones to capture the individual singers’ voices. In this article, we give detailed insights into all stages of creating DCS: recording process, data preparation, generation of annotations as well as development of suitable interfaces for publicly accessing and reusing the data. Furthermore, we demonstrate the potential of the dataset for MIR research by discussing case studies on choral intonation assessment and multiple fundamental frequency (F0) estimation.<br
Keywords: Dagstuhl Choir; Set; Choral Music; Multitrack Recording; Choral Singing Analysis; F0

Discourse Not Dualism: An Interdisciplinary Dialogue On Sonata Form in Beethoven's Early Piano Sonatas - Weiß, Christof; Klauk, Stephanie; Gotham, Mark; Müller, Meinard; Kleinertz, Rainer (Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2020)

Exploring Tonal-Dramatic Relationships in Richard Wagner's Ring Cycle - Zalkow, Frank; Weiß, Christof; Müller, Meinard (Proceedings of the International Conference on Music Information Retrieval (ISMIR), 2017)

Investigating Style Evolution of Western Classical Music: A Computational Approach - Weiß, Christof; Mauch, Matthias; Dixon, Simon; Müller, Meinard (Musicae Scientiae, 2018)

JSD: A Dataset for Structure Analysis in Jazz Music - Balke, Stefan; Reck, Julian; Weiß, Christof; Abeßer, Jakob; Müller, Meinard (Transactions of the International Society for Music Information Retrieval, 2022)
Given a music recording, music structure analysis aims at identifying important structural elements and segmenting the recording according to these elements. In jazz music, a performance is often structured by repeating harmonic schemata (known as choruses), which lay the foundation for improvisation by soloists. Within the fields of music information retrieval (MIR) and computational musicology, the Weimar Jazz Database (WJD) has turned out to be an extremely valuable resource for jazz research. Containing high-quality solo transcriptions for 456 solo sections, the dataset opened up new avenues for the understanding of creative processes in jazz improvisation using computational methods. In this paper, we complement this dataset by introducing the Jazz Structure Dataset (JSD), which provides annotations on structure and instrumentation of entire recordings. The JSD comprises 340 recordings with more than 3000 annotated segments, along with a segment-wise encoding of the solo and accompanying instruments. These annotations provide the basis for training, testing, and evaluating models for various important MIR tasks, including structure analysis, solo detection, or instrument recognition. As an example application, we consider the task of structure boundary detection. Based on a traditional novelty-based as well as a more recent data-driven approach using deep learning, we indicate the potential of the JSD while critically reflecting on some evaluation aspects of structure analysis. In this context, we also demonstrate how the JSD annotations and analysis results can be made accessible in a user-friendly way via web-based interfaces for data inspection and visualization. All annotations, experimental results, and code for reproducibility are made publicly available for research purposes.<br
Keywords: Dataset; Music Structure Analysis; Boundary Detection; Instrument Recognition; Jazz

Learning Pitch-Class Representations from Score-Audio Pairs of Classical Music - Weiß, Christof; Zeitler, Johannes; Zunner, Tim; Schuberth, Florian; Müller, Meinard (Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2021)
Chroma or pitch-class representations of audio recordings are an essential tool in music information retrieval. Traditional chroma features relying on signal processing are often influenced by timbral properties such as overtones or vibrato and, thus, only roughly correspond to the pitch classes indicated by a score. Deep learning provides a promising possibility to overcome such problems but requires large annotated datasets. Previous approaches therefore use either synthetic audio, MIDI-piano recordings, or chord annotations for training. Since these strategies have different limitations, we propose to learn transcription-like pitch-class representations using pre-synchronized score-audio pairs of classical music. We train several CNNs with musically inspired architectures and evaluate their pitch-class estimates for various instrumentations including orchestra, piano, chamber music, and singing. Moreover, we illustrate the learned features' behavior when used as input to a chord recognition system. In all our experiments, we compare cross-validation with cross-dataset evaluation. Obtaining promising results, our strategy shows how to leverage the power of deep learning for constructing robust but interpretable tonal representation

Local Key Estimation In Classical Music Recordings: A Cross-Version Study on Schubert’s Winterreise - Schreiber, Hendrik; Weiß, Christof; Müller, Meinard (ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020)
While global key and chord estimation for both popular and classical music recordings have received a lot of attention, little research has been devoted to estimating the local key for classical music. In this work, we approach local key estimation on a unique cross-version dataset comprising nine performances (versions) of Schubert's song cycle Winterreise-a challenging scenario of high musical ambiguity and subjectivity. We compare an HMM-based system with a CNN-based approach. For both models, we employ a similar training procedure including the optimization of hyperparameters on a validation split. We systematically evaluate the model predictions and provide musical explanations for key confusions. As our main contribution, we explore how different training-test splits affect the models' efficacy. Splitting along the song axis, we find that both methods perform similarly well. Splitting along the version axis leads to clearly higher results, especially for the CNN, which seems to effectively learn the harmonic progressions of the songs ("cover song effect") and successfully generalizes to unseen versions.<br
Keywords: Training; Hidden Markov models; Estimation; Music; Training data; Harmonic analysis; Task analysis; music information retrieval; local key estimation; harmony analysis; evaluation; deep neural networks

Local Key Estimation in Music Recordings: A Case Study across Songs, Versions, and Annotators - Weiß, Christof; Schreiber, Hendrik; Müller, Meinard (IEEE/ACM Transactions on Audio, Speech and Language Processing, 2020)
While global key and chord estimation for both popular and classical music recordings have received a lot of attention, little research has been devoted to estimating the local key for classical music. Partly, this may be due to its inherent ambiguity and subjectivity, which makes annotating local keys a challenging task. In this article, we approach local key estimation with a cross-version dataset comprising nine performances (versions) of Schubert's song cycle Winterreise annotated by three different music theory experts. We consider two baseline methods that are representative for common types of signal processing algorithms: an HMM-based system and a CNN-based approach. For both models, we employ a similar training procedure including the optimization of hyperparameters on a validation split. We systematically evaluate the model predictions and provide musical explanations for key confusions. As a central contribution, we explore how different training-test splits affect the models' efficacy. Splitting along the song axis, we find that both methods perform similarly well. Splitting along the version axis, we obtain substantially higher accuracies, especially for the CNN, which seems to effectively learn the harmonic progressions of the songs ('cover song effect') and successfully generalizes to unseen versions. We further discuss the results for several songs in detail and assess our results from the perspective of multiple annotators. This cross-annotator study reveals that a substantial part of the systems' errors coincides with annotator disagreement and an even larger part can be traced back to musically explainable relationships among different keys.
CRIS-Team Scopus Importer:2020-11-27
Keywords: Deep neural networks; evaluation; harmony analysis; local key estimation; music information retrieval

Mid-Level Chord Transition Features for Musical Style Analysis - Müller, Meinard; Weiß, Christof; Brand, Fabian (Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2019)

Möglichkeiten der computergestützten Erkennung und Visualisierung harmonischer Strukturen - eine Fallstudie zu Richard Wagners "Die Walküre'' - Weiß, Christof; Kleinertz, Rainer; Müller, Meinard (Bericht zur Jahrestagung der Gesellschaft für Musikforschung (GfM) 2015 in Halle/Saale, 2016)

Neue Wege für die Musikforschung: Computergestützte Analyse harmonischer Strukturen - Weiß, Christof; Müller, Meinard; Klauk, Stephanie; Kleinertz, Rainer (INFORMATIK 2023 — Designing Futures: Zukünfte gestalten, 2023)

Perspektiven computergestützter harmonischer Analyse: Beethovens op. 14 Nr. 1 als Gegenstand gattungsübergreifender Korpusanalyse - Klauk, Stephanie; Kleinertz, Rainer; Schmolenzky, Pascal; Weiß, Christof; Müller, Meinard (INFORMATIK 2023 - Designing Futures: Zukünfte gestalten, 2023)

Quantifying and Visualizing Tonal Complexity - Weiß, Christof; Müller, Meinard (Proceedings of the Conference on Interdisciplinary Musicology (CIM), 2014)

Schubert Winterreise Dataset: A Multimodal Scenario for Music Analysis - Weiß, Christof; Zalkow, Frank; Arifi-Müller, Vlora; Müller, Meinard; Koops, Hendrik Vincent; Volk, Anja; Grohganz, Harald G. (Journal of Computing and Cultural Heritage, 2021)
This article presents a multimodal dataset comprising various representations and annotations of Franz Schubert's song cycle Winterreise. Schubert's seminal work constitutes an outstanding example of the Romantic song cycle-a central genre within Western classical music. Our dataset unifies several public sources and annotations carefully created by music experts, compiled in a comprehensive and consistent way. The multimodal representations comprise the singer's lyrics, sheet music in different machine-readable formats, and audio recordings of nine performances, two of which are freely accessible for research purposes. By means of explicit musical measure positions, we establish a temporal alignment between the different representations, thus enabling a detailed comparison across different performances and modalities. Using these alignments, we provide for the different versions various musicological annotations describing tonal and structural characteristics. This metadata comprises chord annotations in different granularities, local and global annotations of musical keys, and segmentations into structural parts. From a technical perspective, the dataset allows for evaluating algorithmic approaches to tasks such as automated music transcription, cross-modal music alignment, or tonal analysis, and for testing these algorithms' robustness across songs, performances, and modalities. From a musicological perspective, the dataset enables the systematic study of Schubert's musical language and style in Winterreise and the comparison of annotations regarding different annotators and granularities. Beyond the research domain, the data may serve further purposes such as the didactic preparation of Schubert's work and its presentation to a wider public by means of an interactive multimedia experience. With this article, we provide a detailed description of the dataset, indicate its potential for computational music analysis by means of several studies, and point out possibilities for future research.
CRIS-Team WoS Importer:2021-07-09

Seitensatz versus Mittelsatz: Expositionen in Beethovens frühen Klaviersonaten zwischen zeitgenössischer Theorie und computergestützter Analyse - Klauk, Stephanie; Kleinertz, Reiner; Weiß, Christof; Müller, Meinard (Jahrbuch 2017 des Staatlichen Instituts für Musikforschung (SIM) — Preußischer Kulturbesitz, 2021)

Singing Voice Detection in Opera Recordings: A Case Study on Robustness and Generalization - Krause, Michael; Müller, Meinard; Weiß, Christof (Electronics, 2021)
Automatically detecting the presence of singing in music audio recordings is a central task within music information retrieval. While modern machine-learning systems produce high-quality results on this task, the reported experiments are usually limited to popular music and the trained systems often overfit to confounding factors. In this paper, we aim to gain a deeper understanding of such machine-learning methods and investigate their robustness in a challenging opera scenario. To this end, we compare two state-of-the-art methods for singing voice detection based on supervised learning: A traditional approach relying on hand-crafted features with a random forest classifier, as well as a deep-learning approach relying on convolutional neural networks. To evaluate these algorithms, we make use of a cross-version dataset comprising 16 recorded performances (versions) of Richard Wagner's four-opera cycle Der Ring des Nibelungen. This scenario allows us to systematically investigate generalization to unseen versions, musical works, or both. In particular, we study the trained systems' robustness depending on the acoustic and musical variety, as well as the overall size of the training dataset. Our experiments show that both systems can robustly detect singing voice in opera recordings even when trained on relatively small datasets with little variety.
CRIS-Team WoS Importer:2021-06-04

Soft Dynamic Time Warping for Multi-Pitch Estimation and Beyond - Krause, Michael; Weiß, Christof; Müller, Meinard (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 2023)
Many tasks in music information retrieval (MIR) involve weakly aligned data, where exact temporal correspondences are unknown. The connectionist temporal classification (CTC) loss is a standard technique to learn feature representations based on weakly aligned training data. However, CTC is limited to discrete-valued target sequences and can be difficult to extend to multi-label problems. In this article, we show how soft dynamic time warping (SoftDTW), a differentiable variant of classical DTW, can be used as an alternative to CTC. Using multi-pitch estimation as an example scenario, we show that SoftDTW yields results on par with a state-of-the-art multi-label extension of CTC. In addition to being more elegant in terms of its algorithmic formulation, SoftDTW naturally extends to real-valued target sequences.
Created from Fastlane, Scopus look-up
Keywords: dynamic time warping; multi-pitch estimation; music information retrieval; music processing; music transcription

Tonal Complexity Features for Style Classification of Classical Music - Weiß, Christof; Müller, Meinard (Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2015)

Towards Computational Feedback for Western Choral Singing Assessment - Weiß, Christof; Schlecht, Sebastian; Rosenzweig, Sebastian; Müller, Meinard (Proceedings of the International Conference on Music Information Retrieval (ISMIR), 2019)

Towards Improving CNN-Based Cross-Dataset Instrument Family Recognition In Classical Music - Tänzer, Michael; Abeßer, Jakob; Mimilakis, Stylianos Ioannis; Weiß, Christof; Lukashevich, Hanna M.; Müller, Meinard (Proceedings of the International Conference on Music Information Retrieval (ISMIR), 2019)

Towards Leitmotif Activity Detection in Opera Recordings - Krause, Michael; Müller, Meinard; Weiß, Christof (Transactions of the International Society for Music Information Retrieval, 2021)
This paper approaches the automatic detection of musical patterns in audio recordings with a particular focus on leitmotifs, which are specific types of patterns associated with certain characters, places, items, or feelings occurring in an opera or movie soundtrack. The detection of such leitmotifs is particularly challenging since their appearance can change substantially over the course of a musical work. In our case study, we consider a self-contained yet comprehensive scenario comprising 16 recorded performances of Richard Wagner’s four-opera cycle Der Ring des Nibelungen, which is a prime example for the use of leitmotifs. Within this scenario, we introduce and formalize the novel task of leitmotif activity detection. Based on a dataset of 200 hours of audio with over 50 000 annotated leitmotif instances, we explore the benefits and limitations of deep-learning techniques for detecting leitmotifs. To this end, we adapt two common deep-learning strategies based on recurrent and convolutional neural networks, respectively. To investigate the robustness of the trained systems, we test their sensitivity to different modifications of the input. We find that our deep-learning systems work well in general but capture confounding factors, such as pitch distributions in leitmotif regions, instead of characteristic musical properties, such as rhythm and melody. Thus, our in-depth analysis demonstrates some challenges that may arise from applying deep-learning approaches for detecting complex musical patterns in audio recordings.<br
Keywords: leitmotifs; opera; musical patterns; deep neural networks; sound event detection

Towards measuring intonation quality of choir recordings: A case study on bruckner's locus iste - Weiß, Christof; Schlecht, Sebastian; Rosenzweig, Sebastian; Müller, Meinard (Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019, 2019)
Unaccompanied vocal music is a central part of Western art music, yet it requires excellent skills for singers to achieve proper intonation. In this paper, we analyze intonation deficiencies by introducing an intonation cost measure that can be computed from choir recordings and may help to assess the singers' intonation quality. With our approach, we measure the deviation between the recording's local salient frequency content and an adaptive reference grid based on the equal-tempered scale. The adaptivity introduces invariance of the local intonation measure to global intonation drifts. In our experiments, we compute this measure for several recordings of Anton Bruckner's choir piece Locus Iste. We demonstrate the robustness of the proposed measure by comparing scenarios of different complexity regarding the availability of aligned scores and multi-track recordings, as well as the number of singers per part. Even without using score information, our cost measure shows interesting trends, thus indicating the potential of our method for real-world applications.
CRIS-Team Scopus Importer:2023-11-06

Wagner Ring Dataset: A Complex Opera Scenario for Music Processing and Computational Musicology - Weiß, Christof; Arifi-Müller, Vlora; Krause, Michael; Zalkow, Frank; Klauk, Stephanie; Kleinertz, Rainer; Müller, Meinard (Transactions of the International Society for Music Information Retrieval, 2023)
This paper introduces the Wagner Ring Dataset (WRD), a multi-modal and multi-version resource on the large-scale opera cycle Der Ring des Nibelungen by Richard Wagner. The Ring comprises four music dramas organized into eleven acts and 21 939 measures in total. Concerning sheet music, we processed a publicly available piano reduction (822 pages) of the full score with optical music recognition followed by extensive manual corrections to create a high-quality, machine-readable symbolic score. Concerning audio data, our corpus covers 16 recorded performances of the full Ring (three of which are publicly available thanks to copyright expiry), each lasting about 14–15 hours. To musically synchronize these versions among each other, we manually annotated all measure positions for three performances, which we transferred to the remaining performances via automated synchronization techniques. The dataset further comprises annotations of key and time signatures, scenes, and singing voice regions (libretto). Moreover, we provide note event annotations for all performances derived from the piano score. The WRD thus constitutes a comprehensive resource for developing algorithms for various music information retrieval tasks, complementing existing datasets with a complex opera scenario. For computational musicology, the WRD serves as a structured dataset that allows for studying the composition and performances of the Ring.<br
Keywords: Dataset; Multi-Modal; Cross-Version; Music Processing; Computational Musicology

What if the 'When' Implies the 'What'?: Human Harmonic Analysis Datasets Clarify the Relative Role of Separate Steps in Automatic Tonal Analysis - Gotham, Mark; Kleinertz, Rainer; Weiß, Christof; Müller, Meinard; Klauk, Stephanie (Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2021)