Interplay between depth and width for interpolation in neural ODEs
Alvarez-Lopez A, Hadj Slimane A, Zuazua Iriondo E (2024)
Publication Language: English
Publication Status: Submitted
Publication Type: Unpublished / Preprint
Future Publication Type: Journal article
Publication year: 2024
DOI: 10.48550/arXiv.2401.09902
Open Access Link: https://dcn.nat.fau.eu/wp-content/uploads/DepthVsWidth.pdf
Abstract
Neural ordinary differential equations (neural ODEs) have emerged as a natural tool for supervised learning from a control perspective, yet a complete understanding of their optimal architecture remains elusive. In this work, we examine the interplay between their width [katex]p[/katex] and number of layer transitions [katex]L[/katex] (effectively the depth [katex]L+1[/katex]). Specifically, we assess the model expressivity in terms of its capacity to interpolate either a finite dataset [katex]\mathcal{D}[/katex] comprising [katex]N[/katex] pairs of points or two probability measures in [katex]\mathbb{R}^d[/katex] within a Wasserstein error margin [katex]\varepsilon>0[/katex]. Our findings reveal a balancing trade-off between [katex]p[/katex] and [katex]L[/katex], with [katex]L[/katex] scaling as [katex]O(1+N/p)[/katex] for dataset interpolation, and [katex]L=O\left(1+(p\varepsilon^d)^{-1}\right)[/katex] for measure interpolation. In the autonomous case, where [katex]L=0[/katex], a separate study is required, which we undertake focusing on dataset interpolation. We address the relaxed problem of [katex]\varepsilon[/katex]-approximate controllability and establish an error decay of [katex]\varepsilon\sim O(\log(p)p^{-1/d})[/katex]. This decay rate is a consequence of applying a universal approximation theorem to a custom-built Lipschitz vector field that interpolates [katex]\mathcal{D}[/katex]. In the high-dimensional setting, we further demonstrate that [katex]p=O(N)[/katex] neurons are likely sufficient to achieve exact control.
Authors with CRIS profile
Enrique Zuazua Iriondo
Lehrstuhl für Dynamics, Control, Machine Learning and Numerics (Alexander von Humboldt-Professur)
Involved external institutions
Spain (ES)
France (FR)How to cite
APA:
Alvarez-Lopez, A., Hadj Slimane, A., & Zuazua Iriondo, E. (2024). Interplay between depth and width for interpolation in neural ODEs. (Unpublished, Submitted).
MLA:
Alvarez-Lopez, Antonio, Arselane Hadj Slimane, and Enrique Zuazua Iriondo. Interplay between depth and width for interpolation in neural ODEs. Unpublished, Submitted. 2024.
BibTeX: Download