Neural ODE Control for Classification, Approximation, and Transport

Ruiz-Balet D, Zuazua Iriondo E (2023)

Publication Type: Journal article

Publication year: 2023

Journal

SIAM Review Society for Industrial and Applied Mathematics

Book Volume: 65

Pages Range: 735-773

Journal Issue: 3

DOI: 10.1137/21M1411433

Abstract

We analyze neural ordinary differential equations (NODEs) from a control theoretical perspective to address some of the main properties and paradigms of deep learning (DL), in particular, data classification and universal approximation. These objectives are tackled and achieved from the perspective of the simultaneous control of systems of NODEs. For instance, in the context of classification, each item to be classified corresponds to a different initial datum for the control problem of the NODE, to be classified, all of them by the same common control, to the location (a subdomain of the Euclidean space) associated to each label. Our proofs are genuinely nonlinear and constructive, allowing us to estimate the complexity of the control strategies we develop. The nonlinear nature of the activation functions governing the dynamics of NODEs under consideration plays a key role in our proofs, since it allows deforming half of the phase space while the other half remains invariant, a property that classical models in mechanics do not fulfill. This very property allows us to build elementary controls inducing specific dynamics and transformations whose concatenation, along with properly chosen hyperplanes, allows us to achieve our goals in finitely many steps. The nonlinearity of the dynamics is assumed to be Lipschitz. Therefore, our results apply also in the particular case of the ReLU activation function. We also present the counterparts in the context of the control of neural transport equations, establishing a link between optimal transport and deep neural networks.

Authors with CRIS profile

Enrique Zuazua Iriondo Lehrstuhl für Dynamics, Control, Machine Learning and Numerics (Alexander von Humboldt-Professur)

Involved external institutions

Imperial College London / The Imperial College of Science, Technology and Medicine GB United Kingdom (GB)

How to cite

APA:

Ruiz-Balet, D., & Zuazua Iriondo, E. (2023). Neural ODE Control for Classification, Approximation, and Transport. SIAM Review, 65(3), 735-773. https://doi.org/10.1137/21M1411433

MLA:

Ruiz-Balet, Domènec, and Enrique Zuazua Iriondo. "Neural ODE Control for Classification, Approximation, and Transport." SIAM Review 65.3 (2023): 735-773.

BibTeX: Download