Interplay between depth and width for interpolation in neural ODEs

Alvarez-Lopez A, Hadj Slimane A, Zuazua Iriondo E (2024)

Publication Language: English

Publication Status: Accepted

Publication Type: Unpublished / Preprint

Future Publication Type: Journal article

Publication year: 2024

Journal

Neural Networks Elsevier

Publisher: NEUNET

DOI: 10.48550/arXiv.2401.09902

Open Access Link: https://dcn.nat.fau.eu/wp-content/uploads/DepthVsWidth.pdf

Abstract

Neural ordinary differential equations (neural ODEs) have emerged as a natural tool for supervised learning from a control perspective, yet a complete understanding of their optimal architecture remains elusive. In this work, we examine the interplay between their width [katex]p[/katex] and number of layer transitions [katex]L[/katex] (effectively the depth [katex]L+1[/katex]).  Specifically, we assess the model expressivity in terms of its capacity to interpolate either a finite dataset [katex]\mathcal{D}[/katex] comprising [katex]N[/katex] pairs of points or  two probability measures in [katex]\mathbb{R}^d[/katex] within a Wasserstein error margin [katex]\varepsilon>0[/katex]. Our findings reveal a balancing trade-off between [katex]p[/katex] and [katex]L[/katex], with [katex]L[/katex] scaling as [katex]O(1+N/p)[/katex] for dataset interpolation, and [katex]L=O\left(1+(p\varepsilon^d)^{-1}\right)[/katex] for measure interpolation. In the autonomous case, where [katex]L=0[/katex], a separate study is required, which we undertake focusing on dataset interpolation. We address the relaxed problem of [katex]\varepsilon[/katex]-approximate controllability and establish an error decay of [katex]\varepsilon\sim O(\log(p)p^{-1/d})[/katex]. This decay rate is a consequence of applying a universal approximation theorem to a custom-built Lipschitz vector field that interpolates [katex]\mathcal{D}[/katex]. In the high-dimensional setting, we further demonstrate that [katex]p=O(N)[/katex] neurons are likely sufficient to achieve exact control.

Authors with CRIS profile

Enrique Zuazua Iriondo Lehrstuhl für Dynamics, Control, Machine Learning and Numerics (Alexander von Humboldt-Professur)

Involved external institutions

Universidad Autónoma de Madrid (UAM) ES Spain (ES) École Normale Supérieure - PSL (ENS) FR France (FR)

How to cite

APA:

Alvarez-Lopez, A., Hadj Slimane, A., & Zuazua Iriondo, E. (2024). Interplay between depth and width for interpolation in neural ODEs. (Unpublished, Accepted).

MLA:

Alvarez-Lopez, Antonio, Arselane Hadj Slimane, and Enrique Zuazua Iriondo. Interplay between depth and width for interpolation in neural ODEs. Unpublished, Accepted. 2024.

BibTeX: Download