Current progress, challenges, and future perspectives of language models for protein representation and protein design

The scheme for protein representation and protein design

With the rapid growth of ChatGPT (https://chat.openai.com/), which has over 100 million active users in just two months, people hear language model over and over. However, many people are unaware that the language models have been used extensively in protein research.1 We do know that the famous software AlphaFold2 can accurately predict the protein structure based only on the protein sequence with attention and transformer architecture.2 The attention mechanism proposed by Google in 2017 is a revolutionary innovation for natural language processing. The basis of ChatGPT and GPT-4 are large language models, which essentially belong to natural language processing. Protein sequences can be treated as sentences and the 20 amino acids are the words. Since the sequence-structure-function paradigm of protein is fundamental to molecular biology, we can use language models borrowed from computer science to investigate the underlying mechanisms of this relationship. Language models have two primary applications in protein sciences: protein representation and protein design.