PAPER KEY: A23BHHS7
TITLE: Simulating 500 million years of evolution with a language model
AUTHORS: Lin, Zeming; Akin, Halil; Rao, Roshan; Verkuil, Robert; Sercu, Tom; Candido, Salvatore; Rives, Alexander; Hayes, Thomas; Sofroniew, Nicholas James; Oktay, Deniz; Tran, Vincent Quy; Deaton, Jonathan; Wiggert, Marius; Badkundri, Rohil; Shafkat, Irhum; Gong, Jun; Derry, Alexander; Molina, Raul Santiago; Thomas, Neil; Khan, Yousuf; Mishra, Chetan; Kim, Carolyn; Bartie, Liam J; Hsu, Patrick D

Simulating 500 million years of evolution with a language model

Thomas Hayes 1 * Roshan Rao 1 * Halil Akin 1 * Nicholas James Sofroniew 1 * Deniz Oktay 1 * Zeming Lin 1 *

Robert Verkuil 1 * Vincent Quy Tran 2 3 Jonathan Deaton 1 Marius Wiggert 1 Rohil Badkundri 1

Irhum Shafkat 1 Jun Gong 1 Alexander Derry 1 Raul Santiago Molina 1 Neil Thomas 1 Yousuf Khan 4

Chetan Mishra 1 Carolyn Kim 1 Liam J. Bartie 2 Patrick D. Hsu 2 3 Tom Sercu 1 Salvatore Candido 1 Alexander Rives 1 †

Abstract
More than three billion years of evolution have produced an image of biology encoded into the space of natural proteins. Here we show that language models trained on tokens generated by evo-
W lution can act as evolutionary simulators to gen-
erate functional proteins that are far away from known proteins. We present ESM3, a frontier multimodal generative language model that reasons over the sequence, structure, and function
E of proteins. ESM3 can follow complex prompts
combining its modalities and is highly responsive to biological alignment. We have prompted ESM3
I to generate ﬂuorescent proteins with a chain of
thought. Among the generations that we synthesized, we found a bright ﬂuorescent protein at far distance (58% identity) from known ﬂuorescent proteins. Similarly distant natural ﬂuorescent proteins are separated by over ﬁve hundred million years of evolution.
V Introduction
The proteins that exist today have developed into their present forms over the course of billions of years of nat-
E ural evolution, passing through a vast evolutionary sieve. In
parallel experiments conducted over geological time, nature creates random mutations and applies selection, ﬁltering proteins by their myriad sequences, structures, and functions.
R As a result, the patterns in the proteins we observe reﬂect the
action of the deep hidden variables of the biology that have shaped their evolution across time. Gene sequencing surveys
P*Equal contribution 1EvolutionaryScale, PBC 2Arc Insti-

of Earth’s natural diversity are cataloging the sequences (1–3) and structures (4, 5) of proteins, containing billions of sequences and hundreds of millions of structures that illuminate patterns of variation across life. A consensus is building that underlying these sequences is a fundamental language of protein biology that can be understood using large language models (6–10).
A number of language models of protein sequences have now been developed and evaluated (9, 11–14). It has been found that the representations that emerge within language models reﬂect the biological structure and function of proteins (6, 15, 16), and are learned without any supervision on those properties, improving with scale (5, 17, 18). In artiﬁcial intelligence, scaling laws have been found that predict the growth in capabilities with increasing scale, describing a frontier in compute, parameters and data (19–21).
We present ESM3, a frontier multimodal generative model, that reasons over the sequences, structures, and functions of proteins. ESM3 is trained as a generative masked language model over discrete tokens for each modality. Structural reasoning is achieved by encoding three-dimensional atomic structure as discrete tokens rather than with the complex architecture and diffusion in three-dimensional space employed in recent predictive (22) and generative models (14, 23–25) of proteins. All-to-all modeling of discrete tokens is scalable, and allows ESM3 to be prompted with any combination of its modalities, enabling controllable generation of new proteins that respect combinations of prompts.
ESM3 at its largest scale was trained with 1.07×1024 FLOPs on 2.78 billion proteins and 771 billion unique tokens, and has 98 billion parameters. Scaling ESM3 to this 98 billion parameter size results in improvements in the representation of sequence, structure, and function, as well as on generative evaluations. We ﬁnd that ESM3 is highly responsive to prompts, and ﬁnds creative solutions to com-

tute 3University of California, Berkeley 4Work done dur- plex combinations of prompts, including solutions for which

ing internship at EvolutionaryScale, PBC †Correspondence to we can ﬁnd no matching structure in nature. We ﬁnd that

<arives@evolutionaryscale.ai>.

models at all scales can be aligned to better follow prompts.

Preview 2024-06-25. Pending submission to bioRxiv. Copyright Larger models are far more responsive to alignment, and

2024 by the authors.

Simulating 500 million years of evolution with a language model

show greater capability to solve the hardest prompts after To generate from ESM3, tokens are iteratively sampled.

alignment.

Starting from a sequence of all mask tokens, tokens can be

We report the generation of a new green ﬂuorescent protein (GFP) with ESM3. Fluorescent proteins are responsible for the glowing colors of jellyﬁsh and corals (26) and are important tools in modern biotechnology (27). They share an elegant structure: an eleven stranded beta barrel with a helix that threads its center, which scaffolds the formation of a light-emitting chromophore out of the protein’s own atoms. This mechanism is unique in nature—no other protein spontaneously forms a ﬂuorescent chromophore out of its own structure—suggesting that producing ﬂuorescence is hard even for nature.
Our new protein, which we have named esmGFP, has 36% sequence identity to Aequorea victoria GFP, and 58% sequence identity to the most similar known ﬂuorescent protein. Despite GFP’s intense focus as a target for protein
W engineering over several decades, as far as we are aware,
proteins this distant have only been found through the discovery of new GFPs in nature.
Similar amounts of diversiﬁcation among natural GFPs have occurred over predictable timescales. Understood in these
E terms, the generation of a new ﬂuorescent protein at this
distance from existing proteins appears to be equivalent to simulating over 500 million years of evolution.
I ESM3
ESM3 reasons over the sequence, structure, and function of proteins. All three modalities are represented by tokens, and are input and output as separate tracks that are fused into a single latent space within the model. ESM3 is trained
V with a generative masked language modeling objective:

L = −Ex,m

1 |m|

log p(xi|x\m)

i∈m

E A random mask m is applied to the tokens x describing the
protein, and the model is supervised to predict the identity of the tokens that have been masked. During training, the mask is sampled from a noise schedule so that ESM3 sees
R many different combinations of masked sequence, structure,
and function, and predicts completions of any combination of the modalities from any other. This differs from the classical masked language modeling (28) in that the supervision is applied across all possible masking rates rather than a
Psingle ﬁxed masking rate. This supervision factorizes the

sampled one at a time, or in parallel, in any order, until all tokens are fully unmasked (Fig. 1A). Masking is applied independently to sequence, structure, and function tracks, which enables generation from any combination of empty, partial, or complete inputs. ESM3’s training objective is also effective for representation learning. We choose a noise schedule that balances generative capabilities with representation learning (Appendix A.2.2).
Tokenization enables efﬁcient reasoning over structure. Protein structures are tokenized by a discrete auto-encoder (32), which is trained to compress the high dimensional space of three-dimensional structure into discrete tokens (Fig. 1C). We propose an invariant geometric attention mechanism to efﬁciently process three-dimensional structure. The mechanism operates in local reference frames deﬁned by the bond geometry at each amino acid, and allows local frames to interact globally through a transformation into the global frame (Appendix A.1.6). This mechanism can be efﬁciently realized through the same computational primitives as attention (33), and is readily scalable. The local structural neighborhoods around each amino acid are encoded into a sequence of discrete tokens, one for each amino acid.
When predicting or generating protein structure, structure tokens output by ESM3 are passed to the decoder, which reconstructs the all-atom structure. The autoencoder is trained to encode and reconstruct atomic coordinates with a geometric loss that supervises the pairwise distances and relative orientations of bond vectors and normals (Appendix A.1.7.3.1). This tokenization delivers nearperfect reconstruction of protein structure (<0.3A˚ RMSD on CAMEO, Fig. S3), enabling representation of structure at the input and output with atomic accuracy.
We also ﬁnd that providing ESM3 direct access to atomic coordinates in the input via a geometric attention projection into the transformer improves the response to atomic coordinate prompts. ESM3 can be conditioned on either or both of tokenized structure and atomic coordinates. We supplement these structure representations with coarse grained tokens encoding secondary structure state (SS8) and solvent accessible surface area (SASA). Function is presented to the model in the form of tokenized keyword sets for each position in the sequence.
ESM3 is a bidirectional transformer. While extensive research has gone into creating specialized architectures and training objectives for proteins, we ﬁnd that tokenization paired with a standard masked language modeling objective

probability distribution over all possible predictions of the and the basic transformer architecture is highly effective

next token given any combination of previous tokens, en- for both representation learning and generative modeling.

suring that tokens can be generated in any order from any Sequence, structure, and function tracks are input as tokens,

starting point (29–31).

which are embedded and fused, then processed through a

2

Simulating 500 million years of evolution with a language model
VIEW Figure 1. ESM3 is a generative language model that reasons over the sequence, structure, and function of proteins. (A) Iterative sampling
with ESM3. Sequence, structure, and function can all be used to prompt the model. At each timestep t, a fraction of the masked positions
E are sampled until all positions are unmasked. (B) ESM3 architecture. Sequence, structure, and function are represented as tracks of
discrete tokens at the input and output. The model is a series of transformer blocks, where all tracks are fused within a single latent space; geometric attention in the ﬁrst block allows conditioning on atomic coordinates. ESM3 is supervised to predict masked tokens. (C) Structure tokenization. Local atomic structure around each amino acid is encoded into tokens. (D) Models are trained at three scales: 1.4B, 7B, and 98B parameters. Negative log likelihood on test set as a function of training FLOPs shows response to conditioning on each of the input tracks, improving with increasing FLOPs. (E) Unconditional generations from ESM3 98B (colored by sequence identity to the nearest sequence in the training set), embedded by ESM3, and projected by UMAP alongside randomly sampled sequences from
PR UniProt (in gray). Generations are diverse, high quality, and cover the distribution of natural sequences.
3

Simulating 500 million years of evolution with a language model

stack of transformer blocks. The ﬁrst transformer block also ing the function and fold topology, using the learned gen-

includes a geometric attention layer for atomic structure co- erative model to ﬁnd a coherent solution that respects the

ordinate conditioning. At the output of the model, shallow prompt.

MLP heads project the ﬁnal layer representation into token probabilities for each of the tracks.

We evaluate ESM3’s ability to follow prompts in each of the tracks independently. A set of prompts are constructed for

The largest ESM3 model is trained on 2.78 billion natu- each of the tracks using a temporally held out test set of nat-

ral proteins derived from sequence and structure databases ural proteins (Appendix A.3.7). We evaluated the resulting

(2, 34–37). As a small fraction of structures have been generations for consistency with the prompt and foldabil-

experimentally determined relative to sequences, we lever- ity, the conﬁdence of the structure prediction TM-score

age predicted structures (4, 5). We also generate synthetic sequences with an inverse folding model (described in Appendix A.2.1.3) for all structures, including predicted ones. Function keywords are derived by predicting functional annotations from sequence using a library of hidden markov models (38). Overall this increased training data to 3.15 billion protein sequences, 236 million protein structures, and 539 million proteins with function annotations, totaling 771 billion unique tokens. Full details of the training dataset
W are described in Appendix A.2.1.8.
We train ESM3 models at three scales: 1.4 billion, 7 billion, and 98 billion parameters. In an initial series of experiments to evaluate representation learning performance in response to architecture hyperparameters, we ﬁnd a greater
E response to increasing depth than to width. This informed
the choice of relatively deep networks for the ﬁnal architectures, with the 98 billion parameter model incorporating
I 216 Transformer blocks (Appendix A.1.5).
Scaling ESM3 from 1.4 billion to 98 billion parameters results in substantial improvements in the validation loss for all tracks, with the greatest improvements observed in sequence loss (Fig. 1D, Fig. S11). These gains in validation loss lead to better representation learning (Ta-
V ble S7 and Fig. S8). In single sequence structure pre-
diction (Table S8) on CAMEO, ESM3 98B obtains 0.895 mean local distance difference test (LDDT) and surpasses ESMFold (0.865 LDDT). Unconditional generation produces high-quality proteins—with a mean predicted LDDT
E (pLDDT) 0.84 and predicted template modeling score
(pTM) 0.52—that are diverse in both sequence (mean pairwise sequence identity 0.155) and structure (mean pairwise TM score 0.48), spanning the distribution of known proteins (Fig. 1E, Fig. S13).
R Programmable design with ESM3
We explore the ability of ESM3 to follow complex prompts with different compositions. ESM3 can be prompted with in-
Pstructions from each of its input tracks: sequence, structure

(pTM) under ESMFold. We deﬁne consistency metrics for each track: constrained site RMSD (cRMSD) is the RMSD between the prompt coordinates and the corresponding coordinates in the generation; SS3 accuracy is the fraction of residues where three-class secondary structure between the prompt and generations match; SASA spearman ρ is the correlation between the SASA prompt and the corresponding region of the generation; keyword recovery is the fraction of prompt keywords recovered by InterProScan (38). Across all tracks, ESM3 ﬁnds solutions that follow the prompt, and have conﬁdently predicted structures by ESMFold (pTM > 0.8) (Fig. 2A).
Unconditional generations reﬂect the distribution of natural proteins. Since we observed ESM3 can faithfully follow prompts, we reasoned that prompting could steer the model to generate proteins that differ from natural proteins. First we test the ability of the model to follow out-of-distribution prompts. We construct a set of prompts combining SS8 and SASA from held out structures (TM < 0.7 to training set). Under these prompts, while the model continues to generate coherent globular structures (mean pTM 0.85 ± 0.03), the distribution of similarities to the training set (as measured by TM-score and sequence identity) shifts to be more novel (average sequence identity to nearest training set protein < 20% and mean TM-score 0.48 ± 0.09; Fig. 2B top). To test the ability to generalize to structures beyond the distribution of natural proteins, we use secondary structure prompts derived from a dataset of artiﬁcial symmetric protein designs distinct from the natural proteins found in the training dataset (Appendix A.3.8). Similarly, ESM3 produces high conﬁdence generations (pTM > 0.8, pLDDT > 0.8) with low sequence and structure similarity to proteins in the training set (sequence identity < 20% and TM-score 0.52±0.10; Fig. 2B bottom), indicating that the model can be used to generate protein sequences and structures highly distinct from those that exist in nature.
ESM3 is able to follow complex prompts, and has the ability to compose prompts from different tracks, and at different levels of abstraction. To evaluate this ability, we prompt

coordinates, secondary structure (SS8), solvent-accessible ESM3 with motifs that require the model to solve for spatial

surface area (SASA), and function keywords. This allows coordination of individual atoms, including ones requiring

prompts to be speciﬁed at multiple levels of abstraction, tertiary coordination between residues far apart in the se-

from atomic level structure to high level keywords describ- quence, such as catalytic centers and ligand binding sites.

4

Simulating 500 million years of evolution with a language model
VIEW Figure 2. Generative programming with ESM3. (A) ESM3 can follow prompts from each of its input tracks. Density of faithfulness to
prompting for each of the tracks is shown. Generations achieve consistency with the prompt and high foldability (pTM). (B) ESM3 can be prompted to generate proteins that differ in structure (left) and sequence (right) from natural proteins. Prompted generations (blue) shift
E toward a more novel space vs. unconditional generations (red), in response to prompts derived from out-of-distribution natural structures
(upper panel) and computationally designed symmetric proteins (lower panel). (C) ESM3 generates creative solutions to a variety of combinations of complex prompts. We show compositions of atomic level motifs with high level instructions speciﬁed through keyword or secondary structure. Fidelity to the prompt is shown via similarity to reference structure (for keyword prompts) and all-atom RMSD to the prompted structure (for atomic coordination prompts). Solutions differ from the scaffolds where the motif was derived (median TM-score 0.36 ± 0.14), and for many motifs (e.g. serotonin, calcium, protease inhibitor, and Mcl-1 inhibitor binding sites), we could ﬁnd no signiﬁcant similarity to other proteins that contain the same motif. (D) An example of especially creative behavior. ESM3 compresses
PR a serine protease by 33% while maintaining the active site structure.
5

Simulating 500 million years of evolution with a language model

We combine these with prompts that specify the fold archi- Biological alignment

tecture. For each unique combination of motif and scaffold,
we generate samples until the prompt is satisﬁed (cRMSD < 1.5A˚ for coordinates; TM > 0.6 to a representative structure for fold level prompts; and SS3 accuracy > 80% for secondary structure prompts) with high conﬁdence (pTM > 0.8, pLDDT > 0.8).

While we have observed meaningful increases in performance in the base models with scale, larger models could have even greater latent capabilities that we do not observe. The base ESM3 models can be prompted to perform difﬁcult tasks such as atomic coordination and composition of prompts, despite the fact that the models have not been

We ﬁnd that ESM3 is able to solve a wide variety of such explicitly optimized for these objectives. Likewise, the prop-

tasks (Fig. 2C). It does so without retrieving the motif’s orig- erties we evaluate generative outputs on—such as high pTM,

inal scaffold (median TM-score of 0.40 ± 0.10 to reference protein; Appendix A.3.9). In some cases, the scaffolds are transferred from existing proteins which have similar motifs (for example, the ESM3-designed alpha-helical scaffold for the zinc-binding motif has high similarity to Ni2+-binding proteins, PDB: 5DQW, 5DQY; Fig. 2C, row 3 column 1). 