Auto class
A Hugging Face Transformers auto class is a factory abstraction that selects and instantiates the appropriate pretrained tokenizer or model class for a specified model and backend.
Deep Learning with JAX
A Transformers abstraction that selects and instantiates the appropriate tokenizer or model class from a pretrained model name and backend.
- How it is used
- Loads pretrained models and tokenizers for downstream tasks without hard-coding architecture-specific classes.
- How deep it goes
- Explains name-based class selection and tokenizer matching, then demonstrates loading; it defers auto-class internals and broader memory optimization.
- Examples
- Demonstrates loading GPT-J-6B with FlaxAutoModelForCausalLM and AutoTokenizer, then tokenizing a prompt for generation.
- What you can do after
- Apply Transformers auto classes to load a compatible Flax model and tokenizer from a pretrained model name.
Hugging Face in Action
AutoTokenizer is a Transformers factory class that selects and loads the tokenizer appropriate for a specified pretrained model.
- How it is used
- Selects and loads the tokenizer needed to prepare text for pretrained Transformers models and datasets.
- How deep it goes
- Explains loading, tokenization, padding, masks, and ID conversion, but does not detail tokenizer-selection internals.
- Examples
- Demonstrates DistilBERT sentiment inputs and BERT IMDb dataset fields, including input IDs, tokens, padding, and attention masks.
- What you can do after
- Apply it via Hugging Face Transformers to prepare single texts or dataset batches for model inference.
Related concepts
- transformersuses · Hugging Face Transformers is an open-source library providing pretrained transformer models, tokenizers, and tools for using and training models across machine-learning tasks.
- foundational modeluses · A base or foundation model is a pretrained model that provides learned parameters or capabilities as a starting point for reuse or adaptation to downstream tasks.
- tokenizerkind of · A tokenizer is a text-processing component or algorithm that segments text into discrete tokens and may map them to identifiers for computational processing.
- FlaxAutoModelForCausalLMincludes · A Hugging Face Flax auto class for loading causal language models from pretrained names.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.