What will I learn in chapter 4 of LLMs in Production?

The chapter introduces the model and data assets to prepare before training or finetuning an LLM, then previews how it will evaluate models and build a reusable dataset. After reading it you should be able to: Explain why an LLM project begins by acquiring, cleaning, preparing, and curating data: these processes turn raw data into high-quality, consistent information for downstream use. Explain that a task-specific LLM will generally start from an existing base or foundation model rather than from scratch, and that choosing the right starting model can make or break the project. Explain why collecting a pretrained model’s weights is a useful starting point: a model already trained on general language can be tweaked for a specific task. Describe how knowing a candidate model’s training process and training data helps in choosing one to adapt. Explain why comparing model families and open-source variants matters amid the many available models, and what to consider if training from scratch. Trace how GPT-1, GPT-2, and GPT-3 grew in parameter count and training data while retaining broadly the same original architecture. Explain how the section says OpenAI’s closed-source GPT models can be accessed, and why the book uses GPT-2 instead. Explain why the authors no longer recommend BLOOM as a foundation model: it often gives poor responses and weak benchmark results, possibly because it was undertrained, so users should expect substantial finetuning. Balance that against its transparent, well-understood training and its contribution of petals for distributed training. Explain that Meta initially released LLaMA to researchers under a noncommercial license, but leaked weights helped make it a standard for experimentation and a basis for other models.

What does section 4.1, Models are the foundation, cover?

This section introduces pretrained model weights as a starting point for task-specific LLMs and explains how to approach choosing a model or architecture. Explain why collecting a pretrained model’s weights is a useful starting point: a model already trained on general language can be tweaked for a specific task. Describe how knowing a candidate model’s training process and training data helps in choosing one to adapt. Explain why comparing model families and open-source variants matters amid the many available models, and what to consider if training from scratch.

What does section 4.1.1, GPT, cover?

This section introduces the GPT family, traces its growth, distinguishes GPT models from ChatGPT and unrelated uses of the name, and explains why the book uses GPT-2. Trace how GPT-1, GPT-2, and GPT-3 grew in parameter count and training data while retaining broadly the same original architecture. Distinguish ChatGPT from the GPT-3 model it was based on by describing its conversational finetuning with reinforcement learning with human feedback. Explain why a model called GPT is not necessarily an OpenAI model, while noting that OpenAI’s later commercial models have become less transparent.

What does section 4.1.2, BLOOM, cover?

This section introduces BLOOM’s transparent development, model variants, and licensing, then weighs its research contributions against its limitations as a foundation model. Describe BLOOM as a publicly developed, GPT-3-scale model trained on the 1.6 TB BigScienceCorpus, and explain that its RAIL license permits commercialization but restricts some uses, so it is not technically open source. Distinguish the full 176B-parameter BLOOM from its smaller 560M-, 3B-, and 7B-parameter versions and from BLOOMZ, a multitask-finetuned version intended to improve cross-lingual generalization. Explain why the authors no longer recommend BLOOM as a foundation model: it often gives poor responses and weak benchmark results, possibly because it was undertrained, so users should expect substantial finetuning. Balance that against its transparent, well-understood training and its contribution of petals for distributed training.

What does section 4.1.3, LLaMA, cover?

This section explains how LLaMA became a widely used model family and describes Llama 2’s licensing, sizes, training data, and finetuned variants. Explain that Meta initially released LLaMA to researchers under a noncommercial license, but leaked weights helped make it a standard for experimentation and a basis for other models. Describe Llama 2 as a capable model released with research and commercial licensing, available in 7B, 13B, and 70B parameter sizes, with access requiring a request and acceptance of terms. Describe Llama 2’s training on 2 trillion tokens from curated internet data, including the removal of sites known to contain personal information and upsampling of sources considered factual. Distinguish those stated practices from the section’s speculation about exact source datasets, and explain that RLHF finetuning produced chat and code models.

What does section 4.1.4, Wizard, cover?

This section explains how the Wizard methodology makes instruction datasets more complex and how to use Wizard models for coding, math, and other tasks. Explain that Wizard is a methodology for turning existing datasets into denser, more complex instruction tasks, then using them to finetune models. Describe trying it on a dataset you already use when progress with another model or architecture stalls. Distinguish WizardCoder’s coding focus from WizardMath’s potential use in talk-to-your-data applications, and describe the Wizard models’ praised human-like prose and handling of complex problems.

What does section 4.1.5, Falcon, cover?

This section introduces Falcon as an openly licensed LLM family and explains its competitive performance and training-data foundation. Explain that Falcon is a model family from the Technology Innovation Institute in Abu Dhabi, released under the Apache 2.0 open-source license and readily available to practitioners. Describe Falcon’s progression from 7B and 40B parameter models to a 180B model, and explain that its competitiveness is reflected both in comparison with GPT-3–sized models and in its strong results on LLM benchmark leaderboards. Explain that Falcon was primarily trained on RefinedWeb, a smaller, higher-quality dataset carefully curated and extracted from Common Crawl.

What does section 4.1.6, Vicuna, cover?

Vicuna shows how training on shared ChatGPT conversations can produce a capable, inexpensive model while creating risks when its training data does not cover a task. Explain that Vicuna was trained on user-shared ShareGPT conversations to emulate ChatGPT by learning from its outputs, and that it was praised for strong performance at relatively low training cost. Describe model collapse as less diverse output arising from training on synthetic data, and explain how Vicuna’s good performance on familiar tasks contrasts with its greater hallucination on generative or agent-like tasks. State that Vicuna is not licensed for commercial use, though the section presents it as suitable for personal projects.

What does section 4.1.7, Dolly, cover?

This section explains why Dolly is useful as a starting point for adaptation despite weak comparative performance, and how its training data and open release differ by version. Explain why Databricks’ Dolly is useful for finetuning or creating LoRAs despite performing poorly against other models of the same size. Distinguish the datasets used to train Dolly 1.0 and Dolly 2.0, and describe what was openly released for Dolly 2.0.

What does section 4.1.8, OpenChat, cover?

The section introduces OpenChat’s training approach and task-specific models, then points readers toward comparing and evaluating model families for a project. Explain how OpenChat resembles Vicuna in its use of 80K ShareGPT conversations but differs through conditioning and weighted loss strategies aimed at human-like, human-preferred responses. Describe OpenChat as a collection of task-specific finetunings for coding, agents, and chatting, rather than the open source chatbot console. State that the models are free for commercial use under the Llama 2 Community License. Explain that Table 4.1 summarizes points of comparison among the discussed model families, including commercial availability, but that evaluating and comparing models is the next step toward choosing one for a project.

What does section 4.2, Evaluating LLMs, cover?

Introduces why choosing and improving an LLM requires objective evaluation, and identifies the metric and dataset needed to do it. Explain how evaluation helps compare the many available models, check whether a model works on a task without finetuning, determine whether finetuning helped, and weigh model size against performance. Distinguish the two things needed to evaluate a model: a metric, which compares results to ground truth, and a dataset, which supplies the tasks the model runs.

What does section 4.2.1, Metrics for evaluating text, cover?

The section explains how ROUGE, BLEU, and entropy-based metrics evaluate generated text, and why their scores need careful interpretation. Explain why exact-match evaluation is too rigid for generated text: two answers can express the same meaning with different words or syntax, so a model’s output must be compared with a ground truth more flexibly. Describe how ROUGE compares generated text with a ground-truth text by scoring overlap rather than demanding an exact match. Explain its use for summaries, its primarily recall-oriented character, and the overlap forms the section mentions: N-grams, longest common subsequences, and skip-bigrams. Explain that BLEU evaluates translations through N-gram precision against reference text, in contrast with ROUGE’s recall emphasis. Describe why an overly short prediction can receive misleadingly high ordinary precision and how BLEU addresses this with a brevity penalty and modified N-gram precision that supports multiple references.

What does section 4.2.2, Industry benchmarks, cover?

This section compares GLUE, SuperGLUE, and MMLU, shows how to try two of them, and explains why benchmark results need to be interpreted together and with care. Explain why evaluating an LLM usually requires several benchmarks rather than choosing one winner: each tends to measure a particular quality, while deployed LLMs perform many tasks. Distinguish GLUE’s standardized language-understanding tasks from SuperGLUE’s harder, more diverse successor tasks. Explain why surpassing human baselines makes high scores useful for comparison but insufficient proof of genuine understanding or acceptable real-world output. Describe the section’s SuperGLUE demonstration: load MultiRC paragraphs with comprehension questions and correct answers, run a small model, and compare its generated answers qualitatively with the answers.

What should I know before reading chapter 4?

Transformers (chapter 2): GPT models are based on transformers.; Hallucinations cause confusion (chapter 3): The earlier discussion of hallucinations provides context for Vicuna’s failures on less familiar tasks.; N-gram and corpus-based techniques (chapter 2): Introduces N-grams as sequences of words.; Semiotics (chapter 2): The distinction between literal language and meaning in context supports analysis of speech acts.; Multilingual NLP (chapter 2): The multilingual challenges that motivate attention to tokenization.; Token limits create bottlenecks (chapter 3): The effects of token limits and differing token counts.; Peculiarities of text data (chapter 3): Text data has operational peculiarities.; Embeddings (chapter 2): The earlier introduction to embeddings.

Chapter 4: Data engineering for large language models: Setting up for success

What you will learn in chapter 4 of LLMs in Production

The chapter introduces the model and data assets to prepare before training or finetuning an LLM, then previews how it will evaluate models and build a reusable dataset.

After this chapter you will be able to

Data engineering foundation. Explain why an LLM project begins by acquiring, cleaning, preparing, and curating data: these processes turn raw data into high-quality, consistent information for downstream use.
Foundation model selection. Explain that a task-specific LLM will generally start from an existing base or foundation model rather than from scratch, and that choosing the right starting model can make or break the project.
Pretrained Weights for Adaptation. Explain why collecting a pretrained model’s weights is a useful starting point: a model already trained on general language can be tweaked for a specific task.
Model Choice and Provenance. Describe how knowing a candidate model’s training process and training data helps in choosing one to adapt. Explain why comparing model families and open-source variants matters amid the many available models, and what to consider if training from scratch.
GPT model growth. Trace how GPT-1, GPT-2, and GPT-3 grew in parameter count and training data while retaining broadly the same original architecture.
Model access and book choice. Explain how the section says OpenAI’s closed-source GPT models can be accessed, and why the book uses GPT-2 instead.
Foundation model tradeoffs and contributions. Explain why the authors no longer recommend BLOOM as a foundation model: it often gives poor responses and weak benchmark results, possibly because it was undertrained, so users should expect substantial finetuning. Balance that against its transparent, well-understood training and its contribution of petals for distributed training.
LLaMA release and adoption. Explain that Meta initially released LLaMA to researchers under a noncommercial license, but leaked weights helped make it a standard for experimentation and a basis for other models.
Llama 2 training and variants. Describe Llama 2’s training on 2 trillion tokens from curated internet data, including the removal of sites known to contain personal information and upsampling of sources considered factual. Distinguish those stated practices from the section’s speculation about exact source datasets, and explain that RLHF finetuning produced chat and code models.
Wizard instruction data methodology. Explain that Wizard is a methodology for turning existing datasets into denser, more complex instruction tasks, then using them to finetune models. Describe trying it on a dataset you already use when progress with another model or architecture stalls.
Falcon licensing and access. Explain that Falcon is a model family from the Technology Innovation Institute in Abu Dhabi, released under the Apache 2.0 open-source license and readily available to practitioners.
Falcon scale and benchmark performance. Describe Falcon’s progression from 7B and 40B parameter models to a 180B model, and explain that its competitiveness is reflected both in comparison with GPT-3–sized models and in its strong results on LLM benchmark leaderboards.
Vicuna training data strategy. Explain that Vicuna was trained on user-shared ShareGPT conversations to emulate ChatGPT by learning from its outputs, and that it was praised for strong performance at relatively low training cost.
Synthetic data generalization risks. Describe model collapse as less diverse output arising from training on synthetic data, and explain how Vicuna’s good performance on familiar tasks contrasts with its greater hallucination on generative or agent-like tasks.
Dolly as an adaptation base. Explain why Databricks’ Dolly is useful for finetuning or creating LoRAs despite performing poorly against other models of the same size.
OpenChat training distinction. Explain how OpenChat resembles Vicuna in its use of 80K ShareGPT conversations but differs through conditioning and weighted loss strategies aimed at human-like, human-preferred responses.
OpenChat model options. Describe OpenChat as a collection of task-specific finetunings for coding, agents, and chatting, rather than the open source chatbot console. State that the models are free for commercial use under the Llama 2 Community License.
Model selection questions. Explain how evaluation helps compare the many available models, check whether a model works on a task without finetuning, determine whether finetuning helped, and weigh model size against performance.
Metric and dataset roles. Distinguish the two things needed to evaluate a model: a metric, which compares results to ground truth, and a dataset, which supplies the tasks the model runs.
Limits of exact matching. Explain why exact-match evaluation is too rigid for generated text: two answers can express the same meaning with different words or syntax, so a model’s output must be compared with a ground truth more flexibly.
ROUGE overlap scoring. Describe how ROUGE compares generated text with a ground-truth text by scoring overlap rather than demanding an exact match. Explain its use for summaries, its primarily recall-oriented character, and the overlap forms the section mentions: N-grams, longest common subsequences, and skip-bigrams.
BLEU precision adjustments. Explain that BLEU evaluates translations through N-gram precision against reference text, in contrast with ROUGE’s recall emphasis. Describe why an overly short prediction can receive misleadingly high ordinary precision and how BLEU addresses this with a brevity penalty and modified N-gram precision that supports multiple references.
Entropy metrics and tokenization. Describe bits per character (BPC) as an entropy-based measure of average information per character that a model tries to minimize, and give the section’s intuitive account of perplexity as reflecting how often a model draws sequences from its corpus or vocabulary. Explain why entropy or perplexity cannot fairly compare models with different tokenization strategies, though they remain useful during training of the same model.
Complementary benchmark results. Explain why evaluating an LLM usually requires several benchmarks rather than choosing one winner: each tends to measure a particular quality, while deployed LLMs perform many tasks.
GLUE and SuperGLUE comparison. Distinguish GLUE’s standardized language-understanding tasks from SuperGLUE’s harder, more diverse successor tasks. Explain why surpassing human baselines makes high scores useful for comparison but insufficient proof of genuine understanding or acceptable real-world output.
MMLU knowledge evaluation. Explain how MMLU tests broader, deeper subject knowledge and problem-solving than GLUE-style tasks, using questions across subjects and difficulty levels. Describe how the example compares OpenAI models and lets the reader vary the number of few-shot examples.
Bias in benchmark questions. Explain why deeper-knowledge evaluations require ethical scrutiny: questions and expected answers about subjects such as history or law can embed disputed assumptions or biases, which models may learn to reproduce.
Bias evaluation rationale. Explain why capability benchmarks alone are insufficient and why evaluating a model’s bias is both a responsibility and a practical way to identify blind spots before they cause problems.
Group comparison evaluation pipeline. Describe how to evaluate bias by running prompts through a model, applying relevant evaluators, segmenting results by group, and comparing the groups’ results or distributions. Explain why these checks should be automated alongside performance evaluations.
Task-specific benchmark design. Explain why a general benchmark may not capture a model’s weaknesses and describe a simple custom evaluation that compares prompts for a specialized task with expected responses.
Evaluation library tradeoffs. Distinguish OpenAI Evals as a framework and registry of curated edge-case datasets from Hugging Face Evaluate as a framework whose datasets and metrics can live in separate, private or public Hub spaces. Explain how that difference affects using your own model or private evaluation data.
Generated code evaluation workflow. Describe how to evaluate code-generating LLMs by having each model generate code from docstrings, running the code safely, measuring its behavior and quality, and comparing the results across models.
Parameter based training assessment. Explain how inspecting a model’s parameter distributions can indicate whether its layers are over- or undertrained without running the model, using WeightWatcher’s analysis of GPT-2 as the example.
Parameter evaluation uses and limits. Explain why parameter analysis is useful for comparing base models or monitoring training and updates, and why it must be paired with other evaluations to judge performance on particular tasks.
Training data quality priorities. Explain why the section places high-quality training data and clever modeling ahead of parameter count or training-data size, while acknowledging that data quality is difficult to measure.
Training data and model behavior. Explain how knowing what a model was and was not trained on helps predict its strengths and failures, using the section’s C versus C++ coding example.
Curated corpora and provenance. Distinguish a corpus from a raw text dataset by its searchable, curated text and linguistic metadata, and describe why an organization building one from its own language data should record where and when data originated and its intended purpose.
Task Focused Data Preparation. Explain why self-supervised training reduces the need for traditional labels but does not eliminate data cleaning. Describe the section’s preparation sequence: define a schema, check whether the data distribution fits the task, inspect bias and anomalies, convert data for model ingestion, then train and reassess.
Instruction Dataset Structure. Describe the instruction, input, and response parts of an instruct-schema example and explain why showing a model how to perform the intended task can be more useful than supplying tasks and answers alone. Explain how recurring format landmarks and task-relevant examples help guide responses.
Speech Acts And Task Fit. Explain how speech acts classify what an utterance is doing in a conversation, not just its literal wording. Describe how anticipating likely, unusual, and inappropriate speech acts helps select examples and desired responses for a task such as a fast-food ordering assistant.
Purposeful Sample Annotation. Explain how annotations serve as metadata for checking whether a dataset fits its task, with emphasis on speech acts, language, bias, and anomalies rather than labeling every word. Describe why the section recommends annotating a representative sample with a defined purpose instead of trusting an unchecked dataset or hand-labeling every row.
Text processing roadmap. Describe how text from a training or finetuning dataset must be turned into numbers an LLM can consume, and identify training BPE or SentencePiece tokenizers and obtaining model embeddings as the approaches this section introduces.
Use-case-specific tokenizer benefits. Explain why training tokens for a specific use case can improve performance: there are multiple valid ways to tokenize text, and choosing one for the task can minimize unknown tokens while maximizing encoded semantics.
Tokenization and model performance. Explain why tokenization strategy can affect model performance: the way text or numbers are split determines what units the model receives. Use the GOAT 7B and GPT-4 arithmetic comparison to illustrate the importance of representing individual digits.
Tokenization's two basic steps. Describe tokenization as splitting text into units and mapping those units to numbers so text can be encoded and decoded. Explain why simply splitting on whitespace is unreliable across languages and code.
Word character and subword tradeoffs. Distinguish word-, character-, and subword-based tokenizers by their splitting units and tradeoffs. Explain why the section favors subwords as a compromise between vocabulary size, unknown tokens, and preservation of useful information.
Meaning Beyond Token IDs. Explain how embeddings build on tokenization: token IDs are largely arbitrary numbers, while embeddings provide positional and semantic context the model can use.
Comparable Embeddings and Efficient Search. Explain that distances between embeddings support fast comparison and retrieval when the embeddings come from the same model and dimensional space. Describe why this can be useful without hosting a gigantic model for every search task.
Retrieval Augmented Generation Flow. Trace the section’s RAG example: store task-relevant data, retrieve pertinent portions using embeddings, and provide them to a larger model at prompting time. Use the comparable-house example to illustrate how current retrieved data can inform an appraisal.
Embedding Extraction and Consistency. Describe the section’s extraction advice: use hidden-state representations generally from the final layer before decoding, final linear layers, or softmax; use the intended inference embedding model for documents that may be stored in a VectorDB. State the suggested PyTorch inference settings when not using sentence transformers.
Slack dataset preparation. Describe the example workflow: pull company Slack data, filter it to one user's messages, and save the result as a Parquet file for later LLM training.

liveBook manning.com

Read chapter 4 on liveBook Get the book at manning.com

What it builds on

Transformers (chapter 2) · GPT models are based on transformers.
Hallucinations cause confusion (chapter 3) · The earlier discussion of hallucinations provides context for Vicuna’s failures on less familiar tasks.
N-gram and corpus-based techniques (chapter 2) · Introduces N-grams as sequences of words.
Semiotics (chapter 2) · The distinction between literal language and meaning in context supports analysis of speech acts.
Multilingual NLP (chapter 2) · The multilingual challenges that motivate attention to tokenization.
Token limits create bottlenecks (chapter 3) · The effects of token limits and differing token counts.
Peculiarities of text data (chapter 3) · Text data has operational peculiarities.
Embeddings (chapter 2) · The earlier introduction to embeddings.

Section by section

Introduction

The chapter introduces the model and data assets to prepare before training or finetuning an LLM, then previews how it will evaluate models and build a reusable dataset.

Data engineering foundation: Explain why an LLM project begins by acquiring, cleaning, preparing, and curating data: these processes turn raw data into high-quality, consistent information for downstream use.
Why it matters: The quality of the data foundation affects the model you can build from it.
Foundation model selection: Explain that a task-specific LLM will generally start from an existing base or foundation model rather than from scratch, and that choosing the right starting model can make or break the project.
Why it matters: The starting model is another consequential asset to settle before finetuning.
Chapter asset preparation roadmap: Describe the chapter's planned sequence: compare foundation models, examine data sources and prepare data, consider custom tokenizers or embeddings, and create a Slack dataset for finetuning in the next chapter.
Why it matters: The roadmap connects this chapter's preparation work to the later finetuning exercise.

Key terms: data engineering, data assets, high-quality, consistent information, base model, foundation models, finetuning, datasets, tokenizer, embeddings, Slack dataset

4.1 Models are the foundation

This section introduces pretrained model weights as a starting point for task-specific LLMs and explains how to approach choosing a model or architecture.

Pretrained Weights for Adaptation: Explain why collecting a pretrained model’s weights is a useful starting point: a model already trained on general language can be tweaked for a specific task.
Why it matters: Starting with pretrained weights can spare you from training a language model from scratch.
Model Choice and Provenance: Describe how knowing a candidate model’s training process and training data helps in choosing one to adapt. Explain why comparing model families and open-source variants matters amid the many available models, and what to consider if training from scratch.
Why it matters: A model’s origins and architecture offer a more useful basis for choosing it than a new benchmark record alone.

Key terms: model weights, pretrained model, training data, open source models, benchmark-breaking records, architecture, model family

4.1.1 GPT

This section introduces the GPT family, traces its growth, distinguishes GPT models from ChatGPT and unrelated uses of the name, and explains why the book uses GPT-2.

GPT model growth: Trace how GPT-1, GPT-2, and GPT-3 grew in parameter count and training data while retaining broadly the same original architecture.
Why it matters: The progression shows how GPT scaled from an early transformer model into a much larger language model.
ChatGPT and GPT distinction: Distinguish ChatGPT from the GPT-3 model it was based on by describing its conversational finetuning with reinforcement learning with human feedback.
Why it matters: It prevents a conversational product from being confused with the underlying model family.
GPT name and ownership: Explain why a model called GPT is not necessarily an OpenAI model, while noting that OpenAI’s later commercial models have become less transparent.
Why it matters: The name alone does not tell you who built a model or how much is known about it.
Model access and book choice: Explain how the section says OpenAI’s closed-source GPT models can be accessed, and why the book uses GPT-2 instead.
Why it matters: The choice of GPT-2 sets expectations for the book’s hands-on examples.

Key terms: GPT, GPT-2, GPT-3, parameters, BookCorpus, WebText, ChatGPT, finetuned, reinforcement learning with human feedback, RLHF

4.1.2 BLOOM

This section introduces BLOOM’s transparent development, model variants, and licensing, then weighs its research contributions against its limitations as a foundation model.

Transparent origins and licensing: Describe BLOOM as a publicly developed, GPT-3-scale model trained on the 1.6 TB BigScienceCorpus, and explain that its RAIL license permits commercialization but restricts some uses, so it is not technically open source.
Why it matters: BLOOM’s transparency makes its training data and development unusually accessible, while its license still matters for adoption.
BLOOM variants and language goals: Distinguish the full 176B-parameter BLOOM from its smaller 560M-, 3B-, and 7B-parameter versions and from BLOOMZ, a multitask-finetuned version intended to improve cross-lingual generalization.
Why it matters: The variants serve different purposes; BLOOMZ is not simply another smaller BLOOM.
Foundation model tradeoffs and contributions: Explain why the authors no longer recommend BLOOM as a foundation model: it often gives poor responses and weak benchmark results, possibly because it was undertrained, so users should expect substantial finetuning. Balance that against its transparent, well-understood training and its contribution of petals for distributed training.
Why it matters: BLOOM is important to understand historically without necessarily being the best model to build on today.

Key terms: BLOOM, BigScience, BigScienceCorpus, 176B parameters, RAIL, BLOOMZ, multitask, finetuned, cross-lingual generalization, 46 languages

4.1.3 LLaMA

This section explains how LLaMA became a widely used model family and describes Llama 2’s licensing, sizes, training data, and finetuned variants.

LLaMA release and adoption: Explain that Meta initially released LLaMA to researchers under a noncommercial license, but leaked weights helped make it a standard for experimentation and a basis for other models.
Why it matters: LLaMA’s adoption helps explain why later models build on it.
Llama 2 commercial availability: Describe Llama 2 as a capable model released with research and commercial licensing, available in 7B, 13B, and 70B parameter sizes, with access requiring a request and acceptance of terms.
Why it matters: Its licensing and access conditions affect whether a team can use it in an application.
Llama 2 training and variants: Describe Llama 2’s training on 2 trillion tokens from curated internet data, including the removal of sites known to contain personal information and upsampling of sources considered factual. Distinguish those stated practices from the section’s speculation about exact source datasets, and explain that RLHF finetuning produced chat and code models.
Why it matters: The data preparation and finetuning choices help explain how the base model became specialized for different uses.

Key terms: LLaMA, Meta, noncommercial license, weights, Llama 2, commercial license, 7B, 13B, 70B, curated dataset

4.1.4 Wizard

This section explains how the Wizard methodology makes instruction datasets more complex and how to use Wizard models for coding, math, and other tasks.

Wizard instruction data methodology: Explain that Wizard is a methodology for turning existing datasets into denser, more complex instruction tasks, then using them to finetune models. Describe trying it on a dataset you already use when progress with another model or architecture stalls.
Why it matters: Wizard is presented as a way to improve training data, not just as another model to choose.
Wizard model strengths and variants: Distinguish WizardCoder’s coding focus from WizardMath’s potential use in talk-to-your-data applications, and describe the Wizard models’ praised human-like prose and handling of complex problems.
Why it matters: The variants illustrate how the same methodology can serve different kinds of applications.

Key terms: WizardLM, Wizard methodology, high-complexity tasks, finetune, WizardCoder, WizardMath, human-like prose, talk-to-your-data applications

4.1.5 Falcon

This section introduces Falcon as an openly licensed LLM family and explains its competitive performance and training-data foundation.

Falcon licensing and access: Explain that Falcon is a model family from the Technology Innovation Institute in Abu Dhabi, released under the Apache 2.0 open-source license and readily available to practitioners.
Why it matters: Falcon’s license and availability affect whether teams can use it in their own projects.
Falcon scale and benchmark performance: Describe Falcon’s progression from 7B and 40B parameter models to a 180B model, and explain that its competitiveness is reflected both in comparison with GPT-3–sized models and in its strong results on LLM benchmark leaderboards.
Why it matters: Model size and benchmark results help explain why Falcon is presented as a serious model option.
RefinedWeb training data origin: Explain that Falcon was primarily trained on RefinedWeb, a smaller, higher-quality dataset carefully curated and extracted from Common Crawl.
Why it matters: Falcon’s training data provides another reason to examine the quality of a model, not just its size.

Key terms: Falcon, Technology Innovation Institute, Apache 2.0, 7B, 40B, 180B, GPT-3, LLM leaderboards, RefinedWeb, Common Crawl

4.1.6 Vicuna

Vicuna shows how training on shared ChatGPT conversations can produce a capable, inexpensive model while creating risks when its training data does not cover a task.

Vicuna training data strategy: Explain that Vicuna was trained on user-shared ShareGPT conversations to emulate ChatGPT by learning from its outputs, and that it was praised for strong performance at relatively low training cost.
Why it matters: Vicuna illustrates how the source and quality of training data can shape a model’s performance.
Synthetic data generalization risks: Describe model collapse as less diverse output arising from training on synthetic data, and explain how Vicuna’s good performance on familiar tasks contrasts with its greater hallucination on generative or agent-like tasks.
Why it matters: Success on tasks resembling the training data does not guarantee reliable behavior on different tasks.
Vicuna usage restriction: State that Vicuna is not licensed for commercial use, though the section presents it as suitable for personal projects.
Why it matters: A model’s license matters when deciding whether it can be used in a product.

Key terms: Vicuna, ShareGPT, ChatGPT, data coverage, data quality, model collapse, synthetic data, hallucinate, commercial use, personal projects

4.1.7 Dolly

This section explains why Dolly is useful as a starting point for adaptation despite weak comparative performance, and how its training data and open release differ by version.

Dolly as an adaptation base: Explain why Databricks’ Dolly is useful for finetuning or creating LoRAs despite performing poorly against other models of the same size.
Why it matters: Dolly’s value as a starting point is different from its performance as a finished model.
Dolly version data and release: Distinguish the datasets used to train Dolly 1.0 and Dolly 2.0, and describe what was openly released for Dolly 2.0.
Why it matters: The datasets and released artifacts help explain what someone can inspect or reuse.

Key terms: Dolly, finetuning, low-ranking adaptations, LoRAs, Dolly 1.0, Stanford Alpaca Dataset, Dolly 2.0, instruction-following dataset, commercial use license

4.1.8 OpenChat

The section introduces OpenChat’s training approach and task-specific models, then points readers toward comparing and evaluating model families for a project.

OpenChat training distinction: Explain how OpenChat resembles Vicuna in its use of 80K ShareGPT conversations but differs through conditioning and weighted loss strategies aimed at human-like, human-preferred responses.
Why it matters: The shared dataset alone does not explain what distinguishes OpenChat from Vicuna.
OpenChat model options: Describe OpenChat as a collection of task-specific finetunings for coding, agents, and chatting, rather than the open source chatbot console. State that the models are free for commercial use under the Llama 2 Community License.
Why it matters: The available variants and license affect whether OpenChat is a practical starting point for an application.
Model comparison and evaluation: Explain that Table 4.1 summarizes points of comparison among the discussed model families, including commercial availability, but that evaluating and comparing models is the next step toward choosing one for a project.
Why it matters: A promising model family is a starting point, not by itself a confirmed choice for a project.

Key terms: OpenChat, Vicuna, ShareGPT, conditioning, weighted loss, human-preferred responses, OpenChat models, finetunings, coding, agents

4.2 Evaluating LLMs

Introduces why choosing and improving an LLM requires objective evaluation, and identifies the metric and dataset needed to do it.

Model selection questions: Explain how evaluation helps compare the many available models, check whether a model works on a task without finetuning, determine whether finetuning helped, and weigh model size against performance.
Why it matters: These questions make evaluation useful for choosing a model rather than relying on claims that it is the best.
Metric and dataset roles: Distinguish the two things needed to evaluate a model: a metric, which compares results to ground truth, and a dataset, which supplies the tasks the model runs.
Why it matters: Separating the tasks from the way results are judged makes model comparisons clearer.

Key terms: finetuning, model size, industry standards, metric, dataset, ground truth

4.2.1 Metrics for evaluating text

The section explains how ROUGE, BLEU, and entropy-based metrics evaluate generated text, and why their scores need careful interpretation.

Limits of exact matching: Explain why exact-match evaluation is too rigid for generated text: two answers can express the same meaning with different words or syntax, so a model’s output must be compared with a ground truth more flexibly.
Why it matters: A useful text metric should not treat every valid rewording as a failure.
ROUGE overlap scoring: Describe how ROUGE compares generated text with a ground-truth text by scoring overlap rather than demanding an exact match. Explain its use for summaries, its primarily recall-oriented character, and the overlap forms the section mentions: N-grams, longest common subsequences, and skip-bigrams.
Why it matters: ROUGE offers a quick way to compare short texts when a valid summary may use different wording or word order.
BLEU precision adjustments: Explain that BLEU evaluates translations through N-gram precision against reference text, in contrast with ROUGE’s recall emphasis. Describe why an overly short prediction can receive misleadingly high ordinary precision and how BLEU addresses this with a brevity penalty and modified N-gram precision that supports multiple references.
Why it matters: BLEU’s adjustments make a precision-based score less easily flattered by an incomplete output.
BLEU tokenizer comparability: State why BLEU scores produced with different tokenizers cannot be directly compared, and describe SacreBLEU as a variant intended to improve score comparability.
Why it matters: A reported BLEU score is only meaningful alongside a consistent way of splitting text into tokens.
Entropy metrics and tokenization: Describe bits per character (BPC) as an entropy-based measure of average information per character that a model tries to minimize, and give the section’s intuitive account of perplexity as reflecting how often a model draws sequences from its corpus or vocabulary. Explain why entropy or perplexity cannot fairly compare models with different tokenization strategies, though they remain useful during training of the same model.
Why it matters: Tokenization can change an entropy-based score without indicating that one model performs better overall.

Key terms: exact match, ground truth, semantically, syntactically, ROUGE, Recall-Oriented Understudy for Gisting Evaluation, N-grams, longest common subsequences, skip-bigrams, BLEU

4.2.2 Industry benchmarks

This section compares GLUE, SuperGLUE, and MMLU, shows how to try two of them, and explains why benchmark results need to be interpreted together and with care.

Complementary benchmark results: Explain why evaluating an LLM usually requires several benchmarks rather than choosing one winner: each tends to measure a particular quality, while deployed LLMs perform many tasks.
Why it matters: Using benchmarks together gives a more useful picture of a model’s strengths and weaknesses.
GLUE and SuperGLUE comparison: Distinguish GLUE’s standardized language-understanding tasks from SuperGLUE’s harder, more diverse successor tasks. Explain why surpassing human baselines makes high scores useful for comparison but insufficient proof of genuine understanding or acceptable real-world output.
Why it matters: A leaderboard score is more meaningful when you know what its tasks test and what they cannot establish.
SuperGLUE MultiRC example: Describe the section’s SuperGLUE demonstration: load MultiRC paragraphs with comprehension questions and correct answers, run a small model, and compare its generated answers qualitatively with the answers.
Why it matters: The example turns an abstract benchmark into a concrete model-evaluation task.
MMLU knowledge evaluation: Explain how MMLU tests broader, deeper subject knowledge and problem-solving than GLUE-style tasks, using questions across subjects and difficulty levels. Describe how the example compares OpenAI models and lets the reader vary the number of few-shot examples.
Why it matters: MMLU probes abilities that simpler language-understanding tests may miss.
Bias in benchmark questions: Explain why deeper-knowledge evaluations require ethical scrutiny: questions and expected answers about subjects such as history or law can embed disputed assumptions or biases, which models may learn to reproduce.
Why it matters: A model can score well while reinforcing harmful assumptions built into its evaluation.

Key terms: benchmarks, evaluation metrics, GLUE, SuperGLUE, human parity, human baselines, few-shot, zero-shot, MultiRC, raw generated text

4.2.3 Responsible AI benchmarks

The section shows how to evaluate LLM bias by comparing metric results across demographic groups and argues for automating those checks alongside performance evaluations.

Bias evaluation rationale: Explain why capability benchmarks alone are insufficient and why evaluating a model’s bias is both a responsibility and a practical way to identify blind spots before they cause problems.
Why it matters: A model can do its assigned job while still producing unequal or harmful results for different groups.
Group comparison evaluation pipeline: Describe how to evaluate bias by running prompts through a model, applying relevant evaluators, segmenting results by group, and comparing the groups’ results or distributions. Explain why these checks should be automated alongside performance evaluations.
Why it matters: A repeatable group comparison can expose differences that an overall performance score hides.
Gender bias evaluation examples: Distinguish the section’s examples: HONEST compares the hurtfulness of completions for different genders, while the Regard pipeline scores polarity on WinoBias prompts segmented by gender. State that the example’s positive and negative results were fairly comparable across genders.
Why it matters: The examples show how the choice of metric and prompt dataset determines what kind of bias a comparison can reveal.

Key terms: industry benchmarks, bias, blind spots, segment the data, evaluation pipeline, toxicity, polarity, hurtfulness, HONEST, Regard

4.2.4 Developing your own benchmark

The section explains when to build a task-specific LLM benchmark and how OpenAI Evals and Hugging Face Evaluate support different evaluation needs.

Task-specific benchmark design: Explain why a general benchmark may not capture a model’s weaknesses and describe a simple custom evaluation that compares prompts for a specialized task with expected responses.
Why it matters: A targeted test can reveal failures that a broad benchmark misses.
Evaluation library tradeoffs: Distinguish OpenAI Evals as a framework and registry of curated edge-case datasets from Hugging Face Evaluate as a framework whose datasets and metrics can live in separate, private or public Hub spaces. Explain how that difference affects using your own model or private evaluation data.
Why it matters: The structure of an evaluation tool affects how easily you can adapt it to your model and data.
Metric and dataset quality: Describe SQuAD as an example of a reading-comprehension benchmark available through Evaluate, with exact-match and F1 metrics, and explain why finding a trustworthy metric or dataset on the Hub still takes care.
Why it matters: Easy access to metrics does not guarantee that a community-uploaded evaluation is suitable or clean.

Key terms: benchmark datasets, edge cases, custom list of prompts, expected responses, OpenAI’s Evals, Hugging Face’s Evaluate, registry, Hugging Face Hub, metric space, SQuAD

4.2.5 Evaluating code generators

This section shows how to evaluate code-generating LLMs by testing the code they produce and comparing the results, while recognizing the limits of automated tests.

Generated code evaluation workflow: Describe how to evaluate code-generating LLMs by having each model generate code from docstrings, running the code safely, measuring its behavior and quality, and comparing the results across models.
Why it matters: Established tools for evaluating code provide a practical way to evaluate a model that writes it.
Test quality and human oversight: Explain why this evaluation approach requires prepared coding prompts and tests, and why a human may need to check whether those tests cover important edge cases.
Why it matters: A model can appear successful when the tests are too easy or miss cases where its code fails.

Key terms: docstrings, prebuilt tests, profiler, security scanner, architectural fitness functions, coding challenges, edge cases, human in the loop

4.2.6 Evaluating model parameters

The section introduces evaluation of LLM training quality from model parameters, explains a WeightWatcher spectral plot, and clarifies why parameter analysis must complement task-based evaluation.

Parameter based training assessment: Explain how inspecting a model’s parameter distributions can indicate whether its layers are over- or undertrained without running the model, using WeightWatcher’s analysis of GPT-2 as the example.
Why it matters: Parameter analysis offers a way to assess training quality without generating and scoring outputs.
Spectral plot tail interpretation: Describe what WeightWatcher’s spectral analysis plots show and how the section interprets the tail of each layer’s eigenvalue-frequency distribution.
Why it matters: The plot gives a visible basis for the section’s layer-level training assessment.
Parameter evaluation uses and limits: Explain why parameter analysis is useful for comparing base models or monitoring training and updates, and why it must be paired with other evaluations to judge performance on particular tasks.
Why it matters: A model that appears well trained from its weights is not necessarily the right model for a specific task.

Key terms: model parameters, weightwatcher, GPT-2, overtrained, undertrained, spectral analysis plots, eigenvalues, tail, heavy tail, Empirical Spectral Density (ESD)

4.3 Data for LLMs

This section argues that high-quality data is central to LLM training and previews how to find, assess, and prepare datasets.

Training data quality priorities: Explain why the section places high-quality training data and clever modeling ahead of parameter count or training-data size, while acknowledging that data quality is difficult to measure.
Why it matters: This priority helps you focus on the data rather than assuming a larger model or dataset will produce better results.
Dataset and preparation roadmap: Describe the section's plan to cover common datasets—their contents, uses, and sources—followed by processing and preparation techniques for getting better LLM results.
Why it matters: Knowing both where data comes from and how to prepare it frames the practical work ahead.

Key terms: high-quality training data, clever modeling, number of parameters, size of training data, common datasets, processing and preparation techniques

4.3.1 Datasets you should know

This section surveys commonly used LLM datasets and corpora, showing how their sources, coverage, freshness, and curation affect model behavior and dataset choice.

Training data and model behavior: Explain how knowing what a model was and was not trained on helps predict its strengths and failures, using the section’s C versus C++ coding example.
Why it matters: A model’s training data can explain a failure that otherwise looks surprising.
Small reference and multilingual datasets: Distinguish the uses and limitations of Wikitext, Wiki-40B, and Europarl: quick English prototyping, multilingual Wikipedia-based modeling, and paired-language translation data, respectively.
Why it matters: Small, convenient datasets can be useful for demos without representing the language or information a production application needs.
Web crawl pretraining sources: Contrast Common Crawl and its cleaned C4 variant with OpenWebText as sources for pretraining, including how each is collected and why cleaning or curation does not remove bias.
Why it matters: The collection and filtering method affects the diversity and biases a model inherits.
Large composite and multilingual datasets: Compare The Pile, RedPajama, and OSCAR by their purposes and composition: a mixed-source pretraining collection, a larger open collection modeled on LLaMA’s described data, and an actively updated multilingual collection with uneven language representation.
Why it matters: A dataset’s size alone does not tell you which sources or languages it represents well.
Curated corpora and provenance: Distinguish a corpus from a raw text dataset by its searchable, curated text and linguistic metadata, and describe why an organization building one from its own language data should record where and when data originated and its intended purpose.
Why it matters: Documented, curated data can support data quality and model explainability, while defining everything the model will see.

Key terms: training data, datasets, Wikitext, Wiki-40B, Europarl, parallel corpora, Common Crawl, C4, OpenWebText, self-supervised pretraining

4.3.2 Data cleaning and preparation

This section explains how to curate LLM data around the intended task, using instruction formats, speech-act analysis, and targeted annotation rather than relying on raw text volume alone.

Task Focused Data Preparation: Explain why self-supervised training reduces the need for traditional labels but does not eliminate data cleaning. Describe the section’s preparation sequence: define a schema, check whether the data distribution fits the task, inspect bias and anomalies, convert data for model ingestion, then train and reassess.
Why it matters: A large text dump can produce fluent language without reliably supporting the behavior your application needs.
Instruction Dataset Structure: Describe the instruction, input, and response parts of an instruct-schema example and explain why showing a model how to perform the intended task can be more useful than supplying tasks and answers alone. Explain how recurring format landmarks and task-relevant examples help guide responses.
Why it matters: The structure of examples helps teach the model both what a request means and what kind of answer to produce.
Speech Acts And Task Fit: Explain how speech acts classify what an utterance is doing in a conversation, not just its literal wording. Describe how anticipating likely, unusual, and inappropriate speech acts helps select examples and desired responses for a task such as a fast-food ordering assistant.
Why it matters: Users can make unexpected requests, so examples limited to typical exchanges may leave an assistant responding poorly.
Purposeful Sample Annotation: Explain how annotations serve as metadata for checking whether a dataset fits its task, with emphasis on speech acts, language, bias, and anomalies rather than labeling every word. Describe why the section recommends annotating a representative sample with a defined purpose instead of trusting an unchecked dataset or hand-labeling every row.
Why it matters: A purposeful sample can expose gaps or harmful patterns without requiring exhaustive manual labeling.

Key terms: self-supervised, schema, distribution, normalization, scaling, bias, instruct schema, instruction tuning, ###Instruction, ###Input

4.4 Text processors

Introduces the conversion of training text into numbers and explains why use-case-specific tokenization and embeddings matter for LLM performance.

Text processing roadmap: Describe how text from a training or finetuning dataset must be turned into numbers an LLM can consume, and identify training BPE or SentencePiece tokenizers and obtaining model embeddings as the approaches this section introduces.
Why it matters: This connects a prepared dataset to the representations an LLM can use.
Use-case-specific tokenizer benefits: Explain why training tokens for a specific use case can improve performance: there are multiple valid ways to tokenize text, and choosing one for the task can minimize unknown tokens while maximizing encoded semantics.
Why it matters: Tokenizer choices can affect how well the model represents the text in your application.

Key terms: tokenization, byte-pair encoding (BPE), SentencePiece, embeddings, unknown tokens, <UKN>, encoded semantics, tokenize

4.4.1 Tokenization

This section explains how tokenization choices shape what an LLM can process, compares word, character, and subword approaches, and identifies considerations for training a tokenizer on your own data.

Tokenization and model performance: Explain why tokenization strategy can affect model performance: the way text or numbers are split determines what units the model receives. Use the GOAT 7B and GPT-4 arithmetic comparison to illustrate the importance of representing individual digits.
Why it matters: A model's size alone does not tell you whether its input representation suits a task.
Tokenization's two basic steps: Describe tokenization as splitting text into units and mapping those units to numbers so text can be encoded and decoded. Explain why simply splitting on whitespace is unreliable across languages and code.
Why it matters: Those basic choices determine whether a tokenizer handles the inputs an application actually encounters.
Word character and subword tradeoffs: Distinguish word-, character-, and subword-based tokenizers by their splitting units and tradeoffs. Explain why the section favors subwords as a compromise between vocabulary size, unknown tokens, and preservation of useful information.
Why it matters: The tradeoffs help you choose a tokenizer suited to the data rather than assuming one splitting rule works everywhere.
Custom subword tokenizer choices: Describe the subword methods named in the section and the three parameters highlighted for training a custom BPE or SentencePiece tokenizer: vocab_size, min_frequency, and special_tokens. Explain the stated tradeoff for vocabulary size and minimum frequency.
Why it matters: A custom tokenizer must reflect the dataset and task without needlessly expanding its vocabulary or losing useful rare tokens.
Limits of tokenized representations: Explain that turning text into tokenized numbers does not itself preserve all syntactic or semantic information, and state why the section turns next to embeddings to represent meaning mathematically.
Why it matters: Token IDs alone are not the richer representation a model needs to work with meaning.

Key terms: tokenization strategy, GOAT 7B, GPT-4, Llama, digits, tokenization, encode, decode, whitespace, word-to-integer dictionary

4.4.2 Embeddings

This section explains how embeddings add context to tokenized text, enable similarity search and retrieval-augmented generation, and can be extracted and used in practice.

Meaning Beyond Token IDs: Explain how embeddings build on tokenization: token IDs are largely arbitrary numbers, while embeddings provide positional and semantic context the model can use.
Why it matters: Token IDs and embeddings serve different purposes, even though both represent text as numbers.
Comparable Embeddings and Efficient Search: Explain that distances between embeddings support fast comparison and retrieval when the embeddings come from the same model and dimensional space. Describe why this can be useful without hosting a gigantic model for every search task.
Why it matters: Similarity search is a practical reason businesses use embeddings.
Retrieval Augmented Generation Flow: Trace the section’s RAG example: store task-relevant data, retrieve pertinent portions using embeddings, and provide them to a larger model at prompting time. Use the comparable-house example to illustrate how current retrieved data can inform an appraisal.
Why it matters: RAG connects embedding-based retrieval to a model’s generated answer.
Embedding Extraction and Consistency: Describe the section’s extraction advice: use hidden-state representations generally from the final layer before decoding, final linear layers, or softmax; use the intended inference embedding model for documents that may be stored in a VectorDB. State the suggested PyTorch inference settings when not using sentence transformers.
Why it matters: Consistent extraction makes stored embeddings useful for later similarity searches.
Limits of Embedding Comparisons: Describe the section’s caution that embeddings are imperfect and that comparing different embedding spaces remains unresolved. Explain that an appropriate comparison depends on the modality and the linguistic goal, not mathematical soundness alone.
Why it matters: A numerical comparison is not automatically a meaningful comparison for the task.

Key terms: tokenization, embeddings, positional and semantic encodings, distance, dimensional space, similarity search, retrieval-augmented generation, RAG, prompting time, Boston Housing dataset

4.5 Preparing a Slack dataset

The section introduces a Slack-data preparation exercise for an LLM chatbot and explains why a chatbot could help answer recurring workplace questions.

Slack dataset preparation: Describe the example workflow: pull company Slack data, filter it to one user's messages, and save the result as a Parquet file for later LLM training.
Why it matters: The saved dataset provides material for the model-training work in later chapters.
Workplace chatbot use case: Explain why the section proposes an LLM chatbot for a workplace help channel: people repeatedly ask the same HR questions rather than searching FAQ pages. Describe how filtering Slack data to one user could instead support a bot that talks like that person.
Why it matters: The use case connects dataset preparation to a concrete problem the chatbot is meant to address.

Key terms: Slack data, user's data, parquet file, HR help channel, FAQ pages, chatbot

4.6 Summary

This summary connects model choice and evaluation with the data preparation and text-processing decisions that shape an LLM’s performance.

Where this is used later

Transfer learning (finetuning) (chapter 5) · Builds on an existing model through finetuning.
From scratch (chapter 5) · Training from scratch requires choosing an architecture or model family.
Reinforcement learning with human feedback (chapter 5) · That section examines reinforcement learning with human feedback.
Implementing Meta’s Llama (chapter 9) · The Llama implementation project builds on familiarity with the model family.
LoRA and PEFT (chapter 5) · LoRA builds on the adaptation option mentioned here.
Government and regulation (chapter 12) · The idea of evaluating text with metrics
Few-shot prompting (chapter 7) · Few-shot prompting builds on the idea of supplying examples and observing their effect on results.
Datasets (chapter 12) · Developing benchmarks for model evaluation

Concepts explained in this chapter

self-supervised learningSelf-supervised learning is a machine-learning approach that derives training targets from the input data itself rather than requiring manually supplied labels.
fine-tuningFine-tuning is the process of continuing model training on task-specific data to adapt a pretrained model to a particular behavior or domain.
data cleaningData cleaning is the process of detecting, correcting, removing, or managing inaccurate, inconsistent, incomplete, or unsuitable data.
instruct schemaAn instruct schema is a structured data format that separates an instruction, its relevant input or metadata, and the desired response.
speech actA speech act is an utterance considered in terms of the communicative action it performs, such as requesting, asserting, or instructing.
ambiguitySemantic ambiguity is the condition in which an expression supports multiple possible meanings or interpretations.
data annotationData annotation is the process of attaching labels or metadata to data instances, often with awareness of their positions or spans.
probability distributionA data distribution is the pattern of frequencies or probabilities with which data values or feature combinations occur.
schemaA schema is a formal specification of the structure, fields, types, and constraints of data.
RedPajamaRedPajama is a large open text dataset designed to reproduce the data composition described for the LLaMA model.
tokenizerA tokenizer is a text-processing component that divides input text into discrete tokens according to specified rules.
benchmarkA benchmark dataset is a curated dataset used to evaluate and compare model performance on defined tasks.
GPT modelA GPT model is a generative pretrained transformer language model that produces text from an input context.
SentencePieceSentencePiece is a library and tokenization framework that learns subword vocabularies directly from raw text without requiring pretokenization by whitespace.
byte pair encodingByte pair encoding is a subword tokenization algorithm that iteratively merges frequent adjacent symbol pairs into larger vocabulary units.
tokenizationTokenization is the process of dividing text into discrete units that can be represented and processed by a language model.
embeddingA vector is an ordered numerical representation of values in a mathematical space.
datasetA dataset is an organized collection of data used for analysis, evaluation, or machine-learning development.
subword tokenizationA subword is a tokenization unit corresponding to part of a word rather than necessarily a complete word.
minimum token frequencyMinimum token frequency is the threshold number of dataset occurrences required for a token to enter a tokenizer vocabulary.
special tokenA special token is a reserved token with a syntactic or control function in tokenized model input.
vocabularyA vocabulary is the set of tokens and their associated identifiers recognized by a tokenizer or language model.
GLUE BenchmarkGLUE is a standardized benchmark suite for evaluating language-model performance on diverse language-understanding tasks.
MMLUMMLU is a benchmark suite that evaluates broad language understanding, world knowledge, and problem-solving across subjects and difficulty levels.
SuperGLUESuperGLUE is a benchmark suite with more difficult and diverse language-understanding tasks than GLUE.
BLEU scoreBLEU is an evaluation metric for machine translation that measures modified n-gram precision against one or more reference translations.
BPCBPC is an entropy-based language-model evaluation metric measuring the average number of binary digits needed to represent each character.
ROUGEROUGE is a family of automatic evaluation metrics that measure lexical overlap between generated text and reference text, commonly for summarization.
n-gramAn n-gram is a contiguous sequence of n linguistic units, commonly used to model local patterns in text.
precisionPrecision is the proportion of predicted items that are correct among all predicted items.
recallRecall is the proportion of reference items that are correctly identified by a prediction.
EvaluateEvaluate is a Hugging Face library for building and applying machine-learning evaluation methods and metrics.
similarity searchSemantic search retrieves information according to the meaning represented by queries and documents rather than relying only on exact lexical matches.
vectorizationVectorization is the transformation of input data into numerical vector representations.
foundational modelA foundation model is a broadly pretrained model that can be adapted to multiple downstream tasks or domains.
language modelA large language model is an artificial intelligence model trained on large text collections to predict and generate human language.
biasA systematic pattern in model outputs that reflects skewed data or assumptions and produces inaccurate or unfair representations.
open-source modelA machine-learning model whose weights, code, or other defining artifacts are made available under terms permitting specified use, modification, or redistribution.
parameterA model parameter is a learned numerical value that determines part of a machine-learning model's computation.
conversational aiAn AI assistant is software that interacts with users and performs tasks through artificial-intelligence capabilities.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.