Neural networks reach a limit for approximating smooth functions
The proof covered smooth functions in any number of dimensions, with no required relationship between the networks' width and depth.
Edited by Boris Djordjevic · Paperfoot
The paper proved an optimal uniform approximation rate for fully connected neural networks using rectified linear units on a class of smooth functions in any number of dimensions. Rectified linear units leave positive values unchanged and set negative values to zero. Existing mathematical constructions had remained separated from the information-theoretic efficiency limit, except for one-dimensional Lipschitz functions, whose output changes are bounded in proportion to input changes. The authors' construction closed this gap for all positive smoothness levels and any number of dimensions. The networks attained a uniform approximation error determined by their width and depth, with bounded parameter sizes and no required relationship between width and depth. The construction used new encoding and decoding mechanisms that exploit bounds on local Taylor coefficients—numbers describing the target function near a point.
Why it matters
The proof closed a gap between earlier network constructions and the theoretical efficiency limit. It established what these networks can mathematically represent, rather than how readily training algorithms can find them.
Caveats
The results are mathematical proofs for a specific class of smooth functions, rather than tests of neural networks trained on real-world data. The paper is a preprint and has not yet been peer-reviewed.
In this paper
uniform approximation of the Hölder ball by ReLU networks
- Methods
- fully connected ReLU networks (this paper)
What it shows
- Up to constant factors, no function class attaining accuracy C epsilon_{N,L} has smaller metric entropy than the proposed network class.
Hence, up to constant factors, no class that attains accuracy C\epsilon_{N,L} has smaller metric entropy than \mathcal{F}^{N,L}.
Corollary 3.3 (Entropy optimality).
- Empirical risk minimizers over the corresponding clipped network classes attain the minimax regression rate n^(-2s/(2s+D)) without additional logarithmic factors.
As a statistical consequence, empirical risk minimizers over the corresponding clipped network classes attain the minimax regression rate n^(-2s/(2s+D)) without additional logarithmic factors.
Abstract
- Even allowing parameters that grow polynomially in the width does not improve the exact uniform approximation rate of the Hölder ball.
In fact, even allowing parameters that grow polynomially in the width does not improve the rate.
Theorem 3.2 (Lower bound).
Quoted word for word from the full text on arXiv.
The paper
Optimal approximation of smooth functions by ReLU neural networks
arXiv, Oct 15, 2026, Open access, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.22693
arXiv:2610.22693v1 in stat.ML, also cs.LG, cs.NA, math.NA
Cite as BibTeX
@misc{nagler2026optimal,
title = {Optimal approximation of smooth functions by ReLU neural networks},
author = {Nagler, Thomas},
year = {2026},
eprint = {2610.22693},
archivePrefix = {arXiv},
primaryClass = {stat.ML},
doi = {10.48550/arxiv.2610.22693},
url = {https://arxiv.org/abs/2610.22693}
}