Large-time asymptotics in deep learning

Esteve Yagüe C, Geshkovski B, Pighin D, Zuazua Iriondo E (2024)

Publication Language: English

Publication Status: Submitted

Publication Type: Unpublished / Preprint

Future Publication Type: Journal article

Publication year: 2024

DOI: 10.48550/arXiv.2008.02491

Open Access Link: https://doi.org/10.48550/arXiv.2008.02491

Abstract

We consider the neural ODE perspective of supervised learning and study the impact of the final time T (which may indicate the depth of a corresponding ResNet) in training. For the classical L2--regularized empirical risk minimization problem, whenever the neural ODE dynamics are homogeneous with respect to the parameters, we show that the training error is at most of the order (1T). Furthermore, if the loss inducing the empirical risk attains its minimum, the optimal parameters converge to minimal L2--norm parameters which interpolate the dataset. By a natural scaling between T and the regularization hyperparameter λ we obtain the same results when λ↘0 and T is fixed. This allows us to stipulate generalization properties in the overparametrized regime, now seen from the large depth, neural ODE perspective. To enhance the polynomial decay, inspired by turnpike theory in optimal control, we propose a learning problem with an additional integral regularization term of the neural ODE trajectory over [0,T]. In the setting of ℓp--distance losses, we prove that both the training error and the optimal parameters are at most of the order (e−μt) in any t∈[0,T]. The aforementioned stability estimates are also shown for continuous space-time neural networks, taking the form of nonlinear integro-differential equations. By using a time-dependent moving grid for discretizing the spatial variable, we demonstrate that these equations provide a framework for addressing ResNets with variable widths.

Authors with CRIS profile

Enrique Zuazua Iriondo Lehrstuhl für Dynamics, Control, Machine Learning and Numerics (Alexander von Humboldt-Professur)

Involved external institutions

Universidad Autónoma de Madrid (UAM) ES Spain (ES)

How to cite

APA:

Esteve Yagüe, C., Geshkovski, B., Pighin, D., & Zuazua Iriondo, E. (2024). Large-time asymptotics in deep learning. (Unpublished, Submitted).

MLA:

Esteve Yagüe, Carlos, et al. Large-time asymptotics in deep learning. Unpublished, Submitted. 2024.

BibTeX: Download