Chapter 8: Generating protein structures
This introduction previews examples of specialized small language models for drug R&D and chemistry, including structure generation and inference optimization.
After this chapter you will be able to
- Chemistry as sequence modeling. Explain how researchers reframe chemistry tasks as text-sequence problems and distinguish early molecule-only models from newer multimodal work.
- Molecular Transformer reaction prediction. Describe how the Molecular Transformer treats product prediction as translation between SMILES strings and what it learns instead of using handcrafted rules.
- Specialized models for drug discovery. Justify the move toward smaller open-source pretrained language models for drug discovery in light of the limitations described for large commercial models.
- ProtGPT2 protein language model. Explain what ProtGPT2 generates and how its training enables protein-sequence generation.
- Protein sequence generation. Describe how the section uses ProtGPT2 to generate sequences and how a starting prompt affects generation.
- AntibodyGPT model origins. Explain that AntibodyGPT is a specialized antibody-sequence generator built by fine-tuning ProGen2 models on experimentally resolved antibody–antigen crystal structures.
- Antibody sequence generation workflow. Trace the example’s inference workflow from loading the model and tokenizer through preparing a target antigen sequence, generating outputs, and decoding antibody sequences.
- Generated antibody validation. Justify using ANARCI and subject-matter review to assess generated antibody sequences rather than relying on perplexity alone.
- CIF next-token training. Explain how CrystaLLM learns from textual CIF files rather than traditional chemical representations: it predicts each next token and compares its predictions with shifted target tokens using cross-entropy loss.
- Conditional CIF generation. Trace how a trained CrystaLLM generates a CIF file from a prompt containing symbol data and a target cell composition, sampling and appending one token at a time until a stopping condition is reached.
- Generated CIF validation workflow. Explain why the raw generated CIF files must be postprocessed and how the section then checks them: postprocess.py produces postprocessed files, which evaluate_cifs.py validates from a .tar.gz archive.
What it builds on
- Risks and challenges with generalist LLMs (chapter 1) · Earlier discussion of risks and challenges with generalist LLMs.
- Domain-specific vs. generalist LLMs for business value (chapter 1) · Earlier comparison of domain-specific and generalist models for business value.
- The Transformer architecture (chapter 1) · The Transformer architecture underlying a decoder-only GPT-2 model
- Text completion (chapter 4) · Generating a continuation with a language model
- Evaluating the generated content (chapter 4) · The idea of evaluating generated content
- Fine-tuning (chapter 2) · The idea of fine-tuning a pretrained model for a specific domain
- 8-bit quantization (chapter 6) · The concept of 8-bit model quantization
Section by section
Introduction
This introduction previews examples of specialized small language models for drug R&D and chemistry, including structure generation and inference optimization.
8.1 Applying Transformers in chemistry
This section explains how chemistry tasks can be represented for Transformer models and why specialized open-source language models are valuable in drug discovery.
- Chemistry as sequence modeling: Explain how researchers reframe chemistry tasks as text-sequence problems and distinguish early molecule-only models from newer multimodal work.
Why it matters: This framing explains why language-model techniques can be useful for chemistry data. - Molecular Transformer reaction prediction: Describe how the Molecular Transformer treats product prediction as translation between SMILES strings and what it learns instead of using handcrafted rules.
Why it matters: It is a concrete example of turning a specialized chemistry problem into a language-model task. - Specialized models for drug discovery: Justify the move toward smaller open-source pretrained language models for drug discovery in light of the limitations described for large commercial models.
Why it matters: It explains the practical motivation for using domain-specific SLMs rather than assuming a general-purpose model will suffice.
Key terms: text-sequence problems, single-modal, multimodal, spectra, Molecular Transformer, product prediction, SMILES, reactants, reagents, chemical motifs
8.2 From natural language to protein structures
The section uses ProtGPT2 to show how a small language model can generate novel protein sequences and how to assess its outputs.
- ProtGPT2 protein language model: Explain what ProtGPT2 generates and how its training enables protein-sequence generation.
Why it matters: It connects ordinary next-token language modeling to the domain-specific task of designing protein sequences. - Protein sequence generation: Describe how the section uses ProtGPT2 to generate sequences and how a starting prompt affects generation.
Why it matters: It shows how a familiar text-generation workflow applies to protein sequences. - Generation cost and quality: Explain the section’s distinction between practical generation cost and assessing generated-sequence quality.
Why it matters: It explains why the example emphasizes output quality rather than faster inference.
Key terms: ProtGPT2, decoder-only GPT-2, self-supervised, oligomer, UniRef50, Transformers, inference pipeline, prompt, protein sequences, zero-shot
8.3 Antibody generation with an SLM
This section shows how to generate antibody sequences with AntibodyGPT, validate them, and improve inference efficiency.
- AntibodyGPT model origins: Explain that AntibodyGPT is a specialized antibody-sequence generator built by fine-tuning ProGen2 models on experimentally resolved antibody–antigen crystal structures.
Why it matters: The model’s training explains why it is suited to generating antibody sequences rather than general protein sequences. - Antibody sequence generation workflow: Trace the example’s inference workflow from loading the model and tokenizer through preparing a target antigen sequence, generating outputs, and decoding antibody sequences.
Why it matters: These steps connect a chosen antigen to readable generated antibody sequences. - Generated antibody validation: Justify using ANARCI and subject-matter review to assess generated antibody sequences rather than relying on perplexity alone.
Why it matters: A generated sequence needs domain-specific scrutiny before it can be treated as potentially useful. - Antibody inference efficiency: Describe the section’s two efficiency adjustments: testing expandable CUDA allocator segments for latency and applying 8-bit Bitsandbytes quantization to reduce model size and benchmark latency.
Why it matters: The example shows that practical antibody generation depends on inference settings as well as model choice.
Key terms: Antibody Generator, AntibodyGPT, ProGen2, fine-tuned, ProGenForCausalLM, target antigen sequence, tokenizer, PyTorch tensor, decode, perplexity
8.4 From CIF files to crystal structures
The section explains how CrystaLLM learns from CIF text and walks through generating, validating, and optionally sharing crystal-structure models.
- CIF next-token training: Explain how CrystaLLM learns from textual CIF files rather than traditional chemical representations: it predicts each next token and compares its predictions with shifted target tokens using cross-entropy loss.
Why it matters: This explains how a language model can learn to produce crystal-structure files. - Conditional CIF generation: Trace how a trained CrystaLLM generates a CIF file from a prompt containing symbol data and a target cell composition, sampling and appending one token at a time until a stopping condition is reached.
Why it matters: It connects a requested crystal composition to the model’s generated CIF output. - Composition prompt and sampling: Describe how the example prepares a cell-composition prompt and runs the pretrained model: Na2Cl2 becomes a prompt in the required data_ format, and sample.py generates the requested CIF samples.
Why it matters: Correct prompt formatting lets the pretrained model generate structures for the requested composition. - Generated CIF validation workflow: Explain why the raw generated CIF files must be postprocessed and how the section then checks them: postprocess.py produces postprocessed files, which evaluate_cifs.py validates from a .tar.gz archive.
Why it matters: Generating a file is not the same as obtaining a checked, usable crystal structure. - MCTS generation option: Distinguish standard sampling from the section’s alternative of running the same CrystaLLM model inside an MCTS loop, which takes settings such as tree width, maximum depth, simulations, and rollouts.
Why it matters: It identifies the alternative decoding route without confusing it with a separately trained model. - Transformers model packaging: Describe the main steps for making a CrystaLLM checkpoint available through the Transformers library: create config.json, convert the .pt checkpoint to .bin using weights under the model key, wrap and save the model, and optionally push it to the Hugging Face Hub.
Why it matters: Packaging makes the pretrained or fine-tuned model usable through familiar Transformers interfaces.
Key terms: CrystaLLM, CIF, next token, cross-entropy loss, prompt, target cell composition, cell composition, data_ prefix, make_prompt_file.py, sample.py
8.5 Summary
Where this is used later
- BitNet and Python (chapter 9) · The earlier discussion of ProtGPT2, used here as a pretrained-model example.
Concepts explained in this chapter
- CrystaLLMA decoder-only Transformer language model in this section that is trained on CIF text to generate crystal structures.
- ANARCIAn antibody-numbering and receptor-classification tool used here to validate whether generated antibody sequences are biologically plausible.
- AntibodyGPTA specialized protein generation model in this chapter for generating therapeutic antibody sequences from an antigen context.
- fine-tuningTraining a pretrained model further on task- or domain-specific data to adapt it to a narrower use case.
- quantizationMethods that reduce numeric precision or representational size of model values to improve efficiency, especially for inference; here the absmax method is a specific quantization procedure.
- generationThe task of producing output sequences from a model given an input context or prompt, here applied to proteins rather than natural language.
- GPT-2A decoder-only Transformer language model used for text generation, here serving as the architecture underlying ProtGPT2.
- int4 quantizationA quantization method that represents model weights or activations with 8-bit numbers to reduce model size and improve efficiency.
- perplexityA measure of how well a language model predicts tokens in sequence, with lower values generally indicating better predictive fit.
- language modelA very large neural language model trained on broad text corpora to predict or generate language, used here for code generation and assistance.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.