Large-scale multilingual language models, pivotal in numerous cross-lingual and non-English Natural Language Processing (NLP) applications, pose possible limitations due to the concept of inter-language competition for model parameters, a phenomenon known as the curse of multilingualism. To address this, a team of researchers from the University of Washington, Charles University in Prague, and the Allen Institute for Artificial Intelligence proposed Cross-lingual Expert Language Models (X-ELM).
The X-ELM technique involves training language models independently on different segments of a multilingual data set, allowing each model in the ensemble to specialize in a specific subset of the multilingual data, thus reducing inter-language conflict. This method strives to maintain the ensemble’s effectiveness while adapting each model’s proficiency to a particular language. This independent training reflects the corpus’ languages more accurately and accommodates even languages with fewer resources.
The researchers further enhanced these efforts with x-BTM, an expansion of the Branch-Train-Merge (BTM) paradigm designed for a more diverse multilingual environment. This improvement incorporated a balanced multilingual data clustering approach based on typological similarity and Hierarchical Multi-Round training (HMR) that enables specialized knowledge training on new languages or multilingual data distributions.
With a twenty-language experiment and the introduction of four new languages, it was shown that X-ELMs produced better performances under varying conditions than other language models. The distribution of improvement across languages was even, and HMR training proved to be more effective than traditional pretraining techniques in terms of adapting models to new languages.
The studies revealed that X-ELM outperformed jointly trained, multilingual models in all considered languages when provided with the same computational resources. X-ELM’s performance enhancements extend to downstream tasks, establishing its practical applicability in real-world scenarios. Moreover, the method demonstrated adaptability to new languages without forgetting previous learning.
Thus, the research provides a robust solution to the challenges encountered with multilingual language models by introducing X-ELM. The research paper underpins the project, and all credit goes to the participating researchers. Continuous updates on this field can be accessed through following Twitter, subscribing to the newsletter, and joining platforms on ML SubReddit, Facebook Community, Discord Channel, LinkedIn Group, and Telegram Channel.
