The increasing sophistication in Artificial Intelligence (AI), specifically the Large Language Models (LLMs), has made significant progress in text generation, language translation, text summarization, and code completion. Yet, the most advanced models are often private; this restricts accessibility to their vital training procedures, making it challenging to comprehensively understand, evaluate and improve them, especially in terms of bias identification and hazard assessment.
Addressing these challenges, researchers from the Allen Institute for AI (AI2) have developed OLMo (Open Language Model), a framework aimed at fostering transparency in Natural Language Processing (NLP). Instead of being just another language model, OLMo serves as a comprehensive framework for creating, analysing and refining language models. It not only provides access to the model’s weights and inference capabilities but also to the entire set of tools used in its formation, including the training and evaluation code, training data sets and complete documentation of the architecture and development processes.
OLMo features several notable characteristics. It is based on AI2’s Dolma set and has access to a large open corpus, facilitating strong model pretraining. It also promotes openness and further research by providing necessary resources to replicate the model’s training process. The framework includes comprehensive evaluation tools for meticulous assessment of the model’s performance to improve its capabilities scientifically. Available in different versions including 1B, 7B and an in-progress 65B parameter models, OLMo’s complexity and power can be scaled up to accommodate a range of applications.
The framework has undergone an exhaustive evaluation procedure that involves online and offline phases. Offline evaluation uses the Catwalk framework, including intrinsic and downstream language modelling assessments through the Paloma perplexity benchmark. In-loop online assessments have been employed during training to help shape decisions on initialization, architecture, etc.
Results from the downstream evaluation depict zero-shot performance on nine core tasks linked with commonsense reasoning. The largest model for perplexity evaluations, OLMo-7B, has been used for intrinsic language modeling evaluation, leveraging Paloma’s extensive dataset covering 585 different text domains.
In summary, OLMo represents a significant leap towards facilitating an ecosystem for transparent research. It aims to boost the technological capabilities of language models, ensuring that these advancements are made in an inclusive, transparent, and ethical manner. The researchers behind this project deserve all the credit for this research.
