liveBook · Manning

Language model

As explained in Deep Learning with JAX

A large neural language model used for language-modeling tasks such as generation, training, fine-tuning, and serving.

A large language model (LLM) is a large neural language model used to generate and process language. In practice, you can start from a pretrained model or train one for a specialized application, then fine-tune, evaluate, and serve it. section 11.3 section 12.1.2

Common LLM workflows include:

  • Using pretrained models for natural language processing tasks. section 11.3
  • Pretraining models from scratch when substantial cloud GPU or TPU resources are available. section 12.1.2
  • Fine-tuning, evaluating, and serving models with JAX/Flax tooling. section 12.1.2
  • Scaling training across many accelerators through distributed parallelism. section 12.1.2

Representative model families

The ecosystem includes GPT-like families such as Llama, Gemma, GPT-J, GPT-Neo, BLOOM, and GPT-2, alongside transformer families such as T5 and BERT. section 11.3

pip install -q git+https://github.com/huggingface/transformers.git

Install the Hugging Face Transformers development version to begin working with models from its ecosystem. section 11.3

How Deep Learning with JAX covers it

How deep it goes
Surveys model families and JAX tooling but stops short of explaining transformer internals or language-model training algorithms.
Examples
Uses Llama, Gemma, GPT-J, GPT-Neo, BLOOM, GPT-2, T5, BERT, EasyLM, Pax, and MaxText as concrete examples.
What you can do after
Operate pretrained models and evaluate JAX-based options for training, fine-tuning, scaling, or serving LLMs.
How it connects
Connects LLMs with Hugging Face Transformers, JAX, Flax, pjit, distributed training, fine-tuning, evaluation, and serving.
Cover of Deep Learning with JAX
liveBookmanning.com
Read Deep Learning with JAX on liveBook Get the book at manning.com

Also explained in

Compare how 67 books explain language model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with JAX. The section links point to the parts of the book it draws on.