Neural networks, despite being theoretically capable of fitting as many data samples as they have parameters, often fall short in reality due to limitations in training procedures. This creates a gap between their potential and their practical performance, which can be an obstacle for applications that require precise data fitting, such as medical diagnoses, autonomous driving, and large-scale language models.
Currently, methods to enhance the flexibility of neural networks involve techniques like overparameterization, convolutional architectures, using various optimizers, and activation functions like ReLU. However, these methods come with their own drawbacks. Convolutional networks, which are more efficient in parameter usage than MLPs and ViTs, do not fully maximize their potential on randomly labelled data. Optimizers like SGD and Adam that are traditionally used for regularization may actually limit the network’s capacity to fit data.
Attempting to address these challenges, a team of researchers from New York University, University of Maryland, and Capital One proposed a comprehensive empirical examination of neural networks’ data-fitting capacity using the Effective Model Complexity (EMC) metric. This new metric quantifies the maximum sample size a model can perfectly fit, considering realistic training scenarios and different types of data.
The EMC metric is calculated through an iterative process, where the model starts with a smaller training set and gradually expands it until it no longer achieves 100% accuracy. This method was applied across several datasets, with key technical factors being the use of different neural network architectures and optimizers. Each training run was also ensured to reach a minimum of the loss function.
The researchers’ findings reveal that standard optimizers tend to limit data-fitting capacity, while CNNs are more parameter-efficient, even with random data. Additionally, ReLU activation functions perform better for data fitting than sigmoidal activations. CNNs were found to have superior data fitting capacity than MLPs and ViTs, especially with datasets with semantically consistent labels.
CNNs trained with stochastic gradient descent (SGD) could fit more training samples than those trained with full-batch gradient descent, indicating better generalization. The effectiveness of CNNs is further exhibited in their ability to fit more correctly labelled samples than incorrectly labelled ones.
In conclusion, the research provides substantial insights into the practical capacity of neural networks, revealing the significant influence of optimizers and activation functions on data fitting. These findings, driven by the proposed EMC metric, can significantly aid in improving neural network training and designing better architectures, thereby helping to address a critical challenge in AI research. This empirical approach to measuring complexity and identifying impacting factors offers a new understanding of neural network flexibility.
