How deeply does Learn Generative AI with PyTorch cover transformers?

Explains installation, tokenizer calls, vocabulary indexing, state-dictionary extraction, and selected weight transpositions.

Transformers

As explained in Learn Generative AI with PyTorch

A software library for working with pretrained transformer models and tokenizers from Hugging Face.

Transformers is a Hugging Face software library for using pretrained transformer models and tokenizers. It can turn multilingual text into subword tokens or provide pretrained parameters for a model implemented in PyTorch. section 10.1.1 section 11.3.1

Common applications include:

Tokenize English and French with XLMTokenizer, including subword pieces such as “fr” and “ench,” then map tokens to vocabulary indexes. section 10.1.1
Load GPT-2XL through GPT2LMHeadModel.from_pretrained and extract its state dictionary for a compatible model. section 11.3.1

Loading a pretrained tokenizer

Install the library, create an XLM tokenizer with XLMTokenizer.from_pretrained("xlm-clm-enfr-1024"), and call tokenizer.tokenize("I don't speak French."). The result separates words and punctuation into reusable subword tokens.

Loading GPT-2XL parameters

Create GPT2LMHeadModel.from_pretrained('gpt2-xl'), obtain model_hf.state_dict(), and transpose selected Conv1d weights before copying them into a separately implemented linear-layer model.

How Learn Generative AI with PyTorch covers it

Cover of Learn Generative AI with PyTorch

liveBook manning.com

Read Learn Generative AI with PyTorch on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Natural Language Processing in Action, Second EditionHugging Face Transformers is an open-source library providing pretrained transformer models and utilities for natural-language and other machine-learning tasks.
liveBook manning.com

Hugging Face in ActionTransformers is an open-source library for working with pretrained machine-learning models, especially models for natural-language processing and computer vision.
liveBook manning.com

Domain-Specific Small Language ModelsThe Hugging Face library for working with pretrained transformer models, tokenizers, pipelines, and related training and inference utilities.
liveBook manning.com

Data Analysis with LLMsHugging Face Transformers is an open-source library providing pretrained Transformer models and tools for natural-language, vision, and multimodal machine-learning tasks.
liveBook manning.com

Transformers in ActionTransformers is an open-source software library for using pretrained transformer models across machine-learning tasks.

Compare how 6 books explain transformers →

Related concepts

PyTorchuses · PyTorch is an open-source machine-learning framework for Python that provides tensor computation, automatic differentiation, and tools for building and training neural networks.
Hugging Faceused by · Hugging Face is a platform and ecosystem for discovering, hosting, sharing, and using machine-learning models, datasets, applications, and related tools.
pipelineuses · A pipeline is an ordered or coordinated sequence of connected processing stages or tasks that passes data, work, or results toward an outcome.
Pythonuses · Python is a high-level, general-purpose, dynamically typed programming language known for readable syntax and automatic memory management.
relation extractionuses · Relation extraction is a natural-language-processing technique that identifies semantic relationships between entities expressed in text.

This explainer was generated by Manning's knowledge graph from the text of Learn Generative AI with PyTorch. The section links point to the parts of the book it draws on.