CroissantLLM, an innovative language model offering robust bilingual capabilities in both English and French, is bridging linguistic divides. Developed through a collaborative effort involving multiple prestigious institutions and firms, including Illumina Technology, Unbabel and INESC-ID Lisboa, this initiative represents a dramatic shift from the English-focused bias of traditional models. CroissantLLM was borne out of the realization that English-centric methods limit language model training and inhibit performance in non-English settings.
CroissantLLM addresses these issues by employing a balanced training methodology with English and French data. The model is pre-trained on 3 trillion English and French tokens and boasts a 1:1 English-to-French pre-training data ratio. This approach is supplemented by a custom tokenizer and bilingual fine-tuning datasets, further differentiating this model from its predecessors.
CroissantLLM’s effectiveness is highlighted by its high performance metrics, setting new standards in bilingual language processing. Its capabilities have been validated through FrenchBench, a novel benchmark. The model successfully leverages a specially curated dataset that features a French split with manually selected, high-quality and diverse data sources. This methodology enables the model to function equally well in English and French, which is an accomplishment currently unmatched by other models in the field.
The success of CroissantLLM has implications far beyond academic research. It spearheads greater linguistic inclusivity in NLP applications by tackling the language bias inherent in preceding language models. This transformative development enhances the NLP environment by moving away from an English-dominated model, bolstering our knowledge of multilingualism in language models.
In conclusion, CroissantLLM heralds a sea change in bilingual language model training, espousing the principles of diversity and inclusivity. Its balanced approach to English and French training, combined with the release of a robust training dataset and performance benchmarks, underscores the potential of bilingual models in overcoming linguistic boundaries. This will certainly stimulate future multilingual NLP initiatives, thereby facilitating the advent of more universally accessible and equitable language technologies.
