liveBook · Manning

Foundational model

As explained in Deep Learning with JAX

A model that has already been trained on a large corpus and can be reused directly or adapted to a new task.

A pretrained model has already learned from a large corpus, so you can reuse it for a task instead of starting training from scratch. Hugging Face Model Hub provides thousands of open models, including JAX-compatible options for language and image generation. section 11.3

For language-model tasks, begin with prompting. If the results are not good enough, adapt the existing model with fine-tuning rather than paying the much higher cost of pretraining a new one. section 11.3.2

Common adaptation choices include:

  • Transfer learning updates all or selected network layers. section 11.3.2
  • Low-rank adaptation adds a small delta file to the original weights. section 11.3.2
  • Reinforcement learning from human feedback can further tune model behavior. section 11.3.2

Model choices

Examples include Llama and Gemma language models, GPT-J and GPT-Neo, BERT variants such as FinBERT and BioBERT, T5-family models, PEGASUS, and DALL·E Mini for image generation. section 11.3

pip install -q git+https://github.com/huggingface/transformers.git

Install the Hugging Face Transformers development version before exploring compatible pretrained models. section 11.3

How Deep Learning with JAX covers it

How deep it goes
Explains prompting, pretraining, fine-tuning approaches, and cost tradeoffs, while deferring retrieval-augmented generation.
Examples
Names Llama, Gemma, GPT-J, GPT-Neo, T5, BERT, FinBERT, BioBERT, PEGASUS, and DALL·E Mini.
What it assumes
Assumes a task context and prompt engineering, then contrasts adapting an existing model with training one from scratch.
What you can do after
Apply pretrained models through Hugging Face Transformers and choose fine-tuning when prompting is insufficient.
How it connects
Connects pretrained models with prompt engineering, pretraining, fine-tuning, transfer learning, LoRA, RLHF, RAG, and Hugging Face Transformers.
Cover of Deep Learning with JAX
liveBookmanning.com
Read Deep Learning with JAX on liveBook Get the book at manning.com

Also explained in

Compare how 21 books explain foundational model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with JAX. The section links point to the parts of the book it draws on.