In recent years, natural language processing (NLP) has seen significant advancements due to the transformer architecture. However, as these models grow in size, so do their computational costs and memory requirements, limiting their practical use to a select few corporations. Increasing model depths also present challenges, as deeper models need larger datasets for training, which are not always available.
In response to these challenges, researchers at EPFL and the University of Geneva have developed DenseFormer, a modification to the standard transformer architecture that improves the model’s comprehension without increasing its size. The DenseFormer model incorporates Depth-Weighted-Average (DWA) steps after each transformer block, ensuring a coherent data flow and subsequent improvements to efficiency. It also employs a weighted average of previous block outputs as new inputs, which enhances the model’s compactness and speed whilst preserving memory during inference.
Contrary to traditional transformer models that focus on internal changes, DenseFormer operates between blocks and is compatible with existing models. This operation also makes it adaptable to multiple model methods, such as the mixtures of experts, and emphasizes communication between these models.
DenseFormer enhances the standard transformer model by initializing with DWA modules, thus maintaining its compatibility with the standard Transformer. To reduce computational costs further, researchers have also introduced Dilated DenseFormer, a model that specifies DWA weights by periodically zeroing them out. The study also explores Periodic DenseFormer, another model that varies the frequency of the DWA module, which demonstrates significant computational savings without any evident performance degradation.
Subsequent experiments evaluating DenseFormer’s performance in language modeling tasks saw it consistently outperforming standard transformer architectures in all significant metrics. Additionally, it was found to match or exceed models’ deeper counterparts in terms of perplexity while being quicker during inference.
In summation, DenseFormer represents an exciting opportunity to improve efficiency in natural language processing tasks. The development of scalable, distributed training methods, more efficient implementation of DenseFormer, and efficient sparsity patterns is the focus of future research in this domain.
These findings have been published in a research paper and are available on Github. This breakthrough is credited to the researchers involved in this project, whose information and updates can be tracked on Twitter, the Telegram Channel, Discord Channel as well as LinkedIn Group.
