Theme Transformer: Symbolic Music Generation with Theme-Conditioned Transformer

Shih YJ, Wu SL, Zalkow F, Müller M, Yang YH (2022)

Publication Type: Journal article

Publication year: 2022

Journal

IEEE Transactions on Multimedia Institute of Electrical and Electronics Engineers (IEEE)

DOI: 10.1109/TMM.2022.3161851

Abstract

Attention-based Transformer models have been increasingly employed for automatic music generation. To condition the generation process of such a model with a user-specified sequence, a popular approach is to take that conditioning sequence as a priming sequence and ask a Transformer decoder to generate a continuation. However, this <i>prompt-based</i> conditioning cannot guarantee that the conditioning sequence would develop or even simply repeat itself in the generated continuation. In this paper, we propose an alternative conditioning approach, called <i>theme-based</i> conditioning, that explicitly trains the Transformer to treat the conditioning sequence as a thematic material that has to manifest itself multiple times in its generation result. This is achieved with two main technical contributions. First, we propose a deep learning-based approach that uses contrastive representation learning and clustering to automatically retrieve thematic materials from music pieces in the training data. Second, we propose a novel gated parallel attention module to be used in a sequence-to-sequence (seq2seq) encoder/decoder architecture to more effectively account for a given conditioning thematic material in the generation process of the Transformer decoder. We report on objective and subjective evaluations of variants of the proposed Theme Transformer and the conventional prompt-based baseline, showing that our best model can generate, to some extent, polyphonic pop piano music with repetition and plausible variations of a given condition.

Authors with CRIS profile

Frank Zalkow International Audio Laboratories Erlangen (AudioLabs) Meinard Müller International Audio Laboratories Erlangen (AudioLabs)

Involved external institutions

National Taiwan University (NTU) TW Taiwan (TW) Academia Sinica / 中央研究院 TW Taiwan (TW)

How to cite

APA:

Shih, Y.J., Wu, S.L., Zalkow, F., Müller, M., & Yang, Y.H. (2022). Theme Transformer: Symbolic Music Generation with Theme-Conditioned Transformer. IEEE Transactions on Multimedia. https://doi.org/10.1109/TMM.2022.3161851

MLA:

Shih, Yi Jen, et al. "Theme Transformer: Symbolic Music Generation with Theme-Conditioned Transformer." IEEE Transactions on Multimedia (2022).

BibTeX: Download