Activation Saturation and Floquet Spectrum Collapse in Neural ODEs - Matzakos, Nikolaos; et al. (2026)
We prove that activation saturation imposes a structural dynamical limitation on autonomous Neural ODEs h˙=fθ(h) with saturating activations (tanh, sigmoid, etc.): if q hidden layers of the MLP fθ satisfy |σ′|≤δ on a region U, the input Jacobian is attenuated as Df_θ(x)C(U) (for activations with supx|σ′(x)|≤1, e.g. tanh and sigmoid, this reduces to CWδq), forcing every Floquet (Lyapunov) exponen along any T-periodic orbit γ⊂U into the interval [−C(U),C(U)]. This is a collapse of the Floquet spectrum: as saturation deepens (δ→0), all exponents are driven to zero, limiting both strong contraction and chaotic sensitivity. The obstruction is structural – it constrains the learned vector field at inference time, independent of training quality. As a secondary contribution, for activations with σ′>0, a saturation-weighted spectral factorisation yields a refined bound C˜(U)≤C(U) whose improvement is amplified exponentially in T at the flow level. All results are numerically illustrated on the Stuart–Landau oscillator; the bounds provide a theoretical explanation for the empirically observed failure of tanh-NODEs on the Morris–Lecar neuron mode
https://cris.fau.de/converis/publicweb/Publication/361846623