Interplay between depth and width for interpolation in neural ODEs

Álvarez-López A, Slimane AH, Zuazua Iriondo E (2024)

Publication Type: Journal article

Publication year: 2024

Journal

Neural Networks Elsevier

Book Volume: 180

Article Number: 106640

DOI: 10.1016/j.neunet.2024.106640

Abstract

Neural ordinary differential equations have emerged as a natural tool for supervised learning from a control perspective, yet a complete understanding of the role played by their architecture remains elusive. In this work, we examine the interplay between the width p and the number of transitions between layers L (corresponding to a depth of L+1). Specifically, we construct explicit controls interpolating either a finite dataset D, comprising N pairs of points in R^d, or two probability measures within a Wasserstein error margin ɛ>0. Our findings reveal a balancing trade-off between p and L, with L scaling as 1+O(N/p) for data interpolation, and as 1+Op⁻¹+(1+p)⁻¹ɛ^−d for measures. In the high-dimensional and wide setting where d,p>N, our result can be refined to achieve L=0. This naturally raises the problem of data interpolation in the autonomous regime, characterized by L=0. We adopt two alternative approaches: either controlling in a probabilistic sense, or by relaxing the target condition. In the first case, when p=N we develop an inductive control strategy based on a separability assumption whose probability increases with d. In the second one, we establish an explicit error decay rate with respect to p which results from applying a universal approximation theorem to a custom-built Lipschitz vector field interpolating D.

Authors with CRIS profile

Enrique Zuazua Iriondo Lehrstuhl für Dynamics, Control, Machine Learning and Numerics (Alexander von Humboldt-Professur)

Involved external institutions

École Normale Supérieure de Cachan

France (FR) Universidad Autónoma de Madrid (UAM)

Spain (ES)

How to cite

APA:

Álvarez-López, A., Slimane, A.H., & Zuazua Iriondo, E. (2024). Interplay between depth and width for interpolation in neural ODEs. Neural Networks, 180. https://doi.org/10.1016/j.neunet.2024.106640

MLA:

Álvarez-López, Antonio, Arselane Hadj Slimane, and Enrique Zuazua Iriondo. "Interplay between depth and width for interpolation in neural ODEs." Neural Networks 180 (2024).

BibTeX: Download