Natural evolution has meticulously shaped proteins over more than three billion years. Modern-day research is closely studying these proteins to understand their structures and functions. Large language models are increasingly being employed to interpret the complexities of these protein structures. Such models demonstrate a solid capacity, even without specific training on biological functions, to naturally understand protein structures and functions, especially as they grow more complex.
A team of researchers from Evolutionary Scale PBC, Arc Institute, and the University of California has developed an advanced generative language model for proteins named ESM3. This model’s unique capability is that it can simulate evolutionary processes to develop functional proteins that are considerably different from known ones. It incorporates sequence, structure, and function to create proteins following complex prompts. In a remarkable display of its capability, ESM3 developed a new fluorescent protein (esmGFP) that is 58% different from known ones – a variation as significant as 500 million years of natural evolution. This indicates ESM3’s potential in protein engineering, offering innovative solutions to biological challenges.
ESM3 employs a masked language modeling approach and utilizes tokenized data to understand and predict protein sequence, structure, and function. It uses transformer blocks with geometric attention to process these aspects. The model was trained on vast datasets, including 2.78 billion proteins and 236 million structures, and builds capability up to 98 billion parameters. ESM3 excels at following various inputs, like sequence or structural details, letting it innovate within constraints and generate novel protein designs.
The model’s capability for generating proteins that align with specific prompts improves significantly when scaled and fine-tuned. Although the primary models perform well, fine-tuning them with preference data leads to more successful and diverse solutions. This endorses the idea that larger models have greater inherent adaptability to challenging tasks and show improved performances when aligned with specific goals.
Apart from generating novel proteins, ESM3 has shown a capacity to create proteins with a minimum resemblance to known ones. One notable achievement was the development of esmGFP, a green fluorescent protein (GFP), which was achieved by prompting the model with structures and residues critical for GFP function. The protein developed by ESM3 exhibited natural GFP-like fluorescence, displaying evolution’s path in the process. It implies that the model can explore protein areas that evolution hasn’t, allowing it to simulate potentially millions of years of evolution in creating new functional proteins.
These advancements reflect the potential of advanced computational models to generate novel, functional proteins, pushing the boundaries of biological and medical research. The application of such models could potentially revolutionize fields such as drug discovery and genetic engineering, among others.
