Third party funded individual grant
Acronym: MU 2686/17-1 (No. 521420645)
Start date : 01.01.2024
End date : 31.12.2026
Website: https://www.audiolabs-erlangen.de/fau/professor/mueller/projects/dat4mir
The research field known as Music Information Retrieval (MIR) aims to develop computational tools that allow users to find, organize, analyze, and interact with music in all its different forms and facets. From a multimedia perspective, music is a challenging domain due to the many time-dependent musical concepts such as melody, harmony, pitch and instrumentation activity, loudness, rhythm, and lyrics. Given data-driven deep learning approaches to capture these concepts, one typically requires fine-grained target annotations that reflect the local properties of the underlying music recordings. However, such strongly aligned (frame-level) annotations are generally difficult to obtain or generate. Recent years have seen major advances in general time series analysis by developing differentiable alignment techniques that can be used in loss functions for deep learning pipelines. Since the alignment process can then be part of the differentiable model, such techniques make it possible to train a neural network based on weakly aligned target annotations where only global correspondences need to be known. In this project, our primary goal is to adapt, explore, and develop differentiable alignment techniques in the context of challenging music analysis and retrieval applications. Building upon recently proposed differentiable versions of dynamic time warping, we will systematically study efficiency and approximation properties from a theoretical and practical perspective. Furthermore, we will investigate the role of temporal constraints to better handle confounding factors and improve the explainability of models and learned representations. From an MIR perspective, we want to achieve substantial advances in analyzing music signals by exploiting weakly annotated training data. To this end, we will consider concrete MIR tasks with many yet unsolved problems, including multi-pitch estimation, cross-version music retrieval, and score-audio matching of musical patterns such as themes and leitmotifs. In summary, while making substantial progress for various MIR tasks, we want to gain a better understanding and advance research of modern alignment techniques using music as a challenging multimedia domain.