liveBook · Manning

Foundational model

As explained in Transformers in Action

A pretrained model is a machine-learning model whose parameters have already been learned from a broad training corpus before adaptation to a specific task.

A foundational model is a broad generative AI system trained on large-scale datasets for many downstream tasks. It may process language, visual, auditory, or time-series data rather than language alone. Pretraining gives the model learned patterns of syntax, semantics, and context before task-specific adaptation. section 3.1 section 1.2

Distinguish foundational models from nearby terms:

  • A pretrained model has already learned parameters from extensive data, reducing the need to train a task model from scratch. section 1.2
  • An LLM is a language-focused member of the broader foundational-model category. section 3.1
  • An instruction-tuned or chat model is an LLM optimized to follow structured prompts and perform goal-directed tasks. section 3.1
  • Zero-shot use applies existing knowledge to an unseen task, while few-shot use adapts that knowledge from a small task-specific dataset. section 1.3

Task adaptation

Use a pretrained transformer through Hugging Face for sentiment analysis: adapt it with labeled sentence pairs marked neutral, negative, or positive instead of training from scratch. Fine-tuning can often take only a few hours, although transformer training and use may require GPU resources. section 1.2 section 1.3

Table 3.1 Transformer architecture variants and their core capabilities and use cases
Architecture type
Primary capabilities
Typical use cases
Encoder–decoder
Sequence-to-sequence modeling
Machine translation, summarization, question answering
Decoder-only
Autoregressive generation, instruction following
Text generation, code synthesis, chat interfaces
Encoder-only
Representation learning, contextual embeddings
Text classification, semantic search, entity recognition
Embedding models
Learning dense or sparse vector representations
Retrieval-augmented generation, similarity search, clustering
MoE
Sparse expert routing, scalable compute efficiency
Efficient large-scale generation, multitask learning

Compare foundational-model implementations by architecture, capability, and use case, including encoder-only, decoder-only, encoder–decoder, embedding, and mixture-of-experts variants. section 3.1

How Transformers in Action covers it

How deep it goes
Defines the category and contrasts related model types, but stops short of explaining training internals or parameter-update procedures.
Examples
Uses Hugging Face sentiment analysis, broad task adaptation, zero-shot use, few-shot use, and multiple transformer architecture variants.
What it assumes
Assumes basic machine-learning awareness while introducing pretrained models, model families, downstream adaptation, and zero-shot and few-shot use.
What you can do after
Apply it via Hugging Face Transformers by selecting a pretrained model and adapting it to a specific labeled or prompt-based task.
How it connects
Connects foundational models with pretrained models, transformers, LLMs, instruction-tuned models, zero-shot learning, and few-shot learning.
Cover of Transformers in Action
liveBookmanning.com
Read Transformers in Action on liveBook Get the book at manning.com

Also explained in

Compare how 21 books explain foundational model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.