Deep learning’s exceptional performance across a wide range of scientific fields and its utilization in various applications have been proven. However, these models often come with many parameters that require a substantial amount of computational power for training and testing. The improvement of these models has been a primary focus of advancement in the field, as researchers strive to reduce their size without compromising their performance. The concept of sparsity in neural networks has been a significant area explored that could potentially increase the efficiency and manageability of these models.
A considerable challenge with neural networks is their enormous consumption of computational power and memory due to the large number of parameters. Classic compression techniques like pruning, which reduce the size of the model by removing a percentage of the weights based on preset conditions, often fail to achieve the desired efficiency. They retain zeroed weights within memory, subsequently limiting how much sparsity can contribute to the improvement of these models. This limitation thus underscores the need for truly sparse implementations that could fully optimize memory and computational resources.
To create sparse neural networks, the technique relies on the use of binary masks, which only partially employ the benefits of sparse computations. Zeroed weights are saved in memory and factored into computations. For instance, Dynamic Sparse Training, a technique that modifies network topology during training, still heavily depends on dense matrix operations. Libraries like PyTorch and Keras provide some extent of support for sparse models, but they fail to achieve genuine reductions in memory and computation time due to their reliance on binary masks. Thus, the full potential of sparse neural networks is yet to be discovered.
To provide a truly sparse implementation, researchers from the Eindhoven University of Technology have introduced Nerva, a unique neural network library coded in C++. The utilization of Intel’s Math Kernel Library (MKL) for sparse matrix operations by Nerva bypasses the need for binary masks and optimizes training time and memory usage. This library’s design focuses on runtime efficiency, energy efficiency, and memory efficiency, and is accessible to a wide range of researchers who are already familiar with popular frameworks like PyTorch and Keras.
Besides, Nerva leverages sparse matrix operations to significantly reduce the computational burden associated with neural networks. Unlike traditional methods that retain zeroed weights, Nerva only keeps the non-zero entries, thereby providing considerable memory savings. It’s optimized for CPU performance and has plans to support GPU operations in the future. Crucial operations on sparse matrices are implemented efficiently, ensuring that Nerva can manage large-scale models while maintaining optimal performance.
When tested against PyTorch using the CIFAR-10 dataset, Nerva showed a linear decrease in runtime with increasing levels of sparsity, outperforming PyTorch in high sparsity regimes. Nerva achieved accuracy that was comparable to that of PyTorch while significantly reducing the training and inference times. Memory usage was optimized, with a 49-fold reduction observed for models with 99% sparsity compared to fully dense models.
In conclusion, Nerva provides a truly sparse implementation that addresses traditional methods’ inefficiencies and delivers considerable improvements in runtime and memory usage. Researchers contend that Nerva can equal the accuracy of frameworks like PyTorch but with greater efficiency, especially in scenarios of high sparsity. With the inclusion of dynamic sparse training and GPU operations in development plans, Nerva is positioned to become a useful tool for researchers keen on optimizing neural network models.
