The BigCode project has successfully developed StarCoder2, the second iteration of an advanced large language model designed to revolutionise the field of software development. A collaboration between over 30 top universities and institutions, StarCoder2 uses machine learning to optimise code generation, making it easier to fix bugs and automate routine coding tasks.
Training StarCoder2 on a vast dataset encompassing Software Heritage repositories and Github pull requests, the BigCode project managed to expand its training set capacity by four times. Different variants of StarCoder2 range from 3B, 7B to 15B in size. The largest 15B model outshines the others in performance, proving the success of the project in enhancing code generation capabilities.
The BigCode project has demonstrated a commitment to ethical development and transparency in launching StarCoder2. The weights of the StarCoder2 model are freely accessible under an OpenRAIL license and the Software Heritage persistent IDs for its training dataset have also been published in an effort to promote trust, collaboration and innovation.
The success of StarCoder2 can be attributed to The Stack v2, a carefully curated dataset which is a staggering ten times larger than its predecessor. Encompassing diverse data sources like Software Heritage repositories, GitHub pull requests, and extensive code documentation, The Stack v2 enables StarCoder2 to understand and generate sophisticated code in various programming languages.
The training of StarCoder2 was a complex process. The preparation stages required intensive data cleansing, filtering and subsampling, consequently transforming a massive 67.5TB raw dataset to a manageable 3TB set. This optimisation process was essential to boost the performance of the model by enabling it to learn from high-quality and relevant code examples.
Reputable evaluations have commended the performance of the StarCoder2 models, particularly praising the models’ capabilities in code completion, editing, and reasoning. Small or large, the StarCoder2 models consistently exhibit high performance in comparison to similar sized models in the field.
Summarily, StarCoder2 is a state-of-the-art code generation large language model developed by The BigCode project. The model relies on The Stack V2, a 3TB training dataset obtained from a 67.5TB Software Heritage archive, which is ten times the size of its predecessor. StarCoder2 models range from 3B, 7B, to 15B parameters and supersede others in code completion, editing and reasoning. The BigCode project’s commitment to openness and transparency is highlighted in their decision to publicly share the model’s weights and training data sources. Such transparency aims to foster trust and stimulate further advancement in the field of software development.
