The increase in the hidden layer width of feedforward (FFW) layers results in linear growth in computational costs and activation memory in transformer architectures. This causes a significant issue in scaling, especially with increasingly complex models. These challenges affect the deployment of large-scale models in real-world applications, including language modeling and natural language processing.
Previously, Mixture…
