Artificial neural networks (ANNs) have remarkable capabilities when trained on natural data. Regardless of exact initialization, dataset, or training objective, neural networks trained on the same data domain tend to converge to similar patterns. For different image models, the initial layer weights typically converge to Gabor filters and color-contrast detectors, underlying a sort of “universal” learning pattern that extends beyond biological and artificial systems. However, these phenomena are not yet fully understood from a theoretical perspective.
Researchers from the KTH Royal Institute of Technology, the Redwood Center for Theoretical Neuroscience and the University of California, Santa Barbara, have discovered a mathematical explanation for this universal learning pattern. They found that Fourier features, a commonly observed universal feature in image models, emerge due to the downstream invariance of the learner, which learns to become insensitive to specific transformations like planar translation or rotation. This discovery is based on the concept that ‘invariance,’ or the resistance to change in different conditions, is a fundamental bias introduced into learning systems because of the symmetries in natural data.
The researchers put forth two main theorems to explain these phenomena. Their first theorem posits that if a parametric function is unchanged by the action of a finite group, its weights will match a harmonic of the group, subject to a linear transformation. Their second theorem suggests that if a parametric function is almost invariant to a finite group, the structure of the finite group can be derived from the weights of the invariant model.
The theoretical constructs proposed by the team relate to various situations and neural network architectures, laying a foundational understanding of how learning is represented in both artificial and biological neural systems. The researchers also demonstrated that the algebraic structure of an unknown group can be discovered from an invariant model, indicating that specific patterns of learning can be inferred from the model weights.
The findings point to exciting possibilities for future work. The researchers indicated an interest in exploring the analogues of their theory in respect to real numbers, which could align more closely with current practices in machine learning and neural network training. The introduction of a mathematical theory to explain learning patterns in neural networks is a significant step towards understanding machine learning from a deeply theoretical standpoint, potentially informing more effective designs and operational mechanisms for artificial neural networks.
