liveBook · Manning

Language model

As explained in AI Model Evaluation

A neural language model trained on large text corpora to understand, generate, or transform natural-language content.

A large language model (LLM) is an AI model trained on large text corpora to generate or transform natural-language content. A chatbot answering questions about a dripping faucet or making tomato sauce is one application. Like other AI models, it learns patterns from historical data and uses them to produce outputs for new inputs. section 1.3

An AI model transforms inputs into outputs using an architecture and learnable parameters such as weights and biases. Training adjusts those parameters; deployment adds an inference pipeline and serving infrastructure. This is a general model lifecycle, not a description of a specific LLM architecture. section 1.3

Figure 1.2 A high level conceptual overview of AI systems in an industry setting. The diagram illustrates the key components typically required to build and deploy an AI model. Starting from left to right, input features and training data are closely linked, as both are fed into the model. The model architecture, which is the core of the system, includes trainable weights and other configuration parameters. Hyperparameters, which are not trainable, are used to define the learning process. The loss function guides model training by measuring error, while the optimizer (e.g., gradient descent) updates the weights based on this feedback. Operational and deployment components include the inference pipeline, model output (such as prediction scores and confidence intervals), version control, and model serving infrastructure.

This overview connects training components with inference and serving infrastructure for AI models generally. section 1.3

Evaluate a language model as part of a product, not just in isolation. Offline evaluations can catch regressions and safety issues before launch, but online data is needed to assess real user impact. A model can perform well against historical labels yet disappoint when latency, user behavior, and downstream experience come into play. section 6.0

When evaluating models in user-facing products, check for:

  • Misaligned metrics: a local improvement such as click-through rate can coincide with lower satisfaction or higher latency. section 6.0
  • Experiment complications: behavior can shift over time, inference costs can fluctuate, and cold starts or training-data bias can make results harder to interpret. section 6.4 section 6.5
  • Feedback loops: model outputs affect user behavior and logged data, which can influence later model iterations. Increased engagement may reflect exposure rather than relevance. section 8.2

A recommendation feedback loop

A recommender that shows one genre more often may collect more clicks on that genre. Those clicks can reinforce the initial preference, making engagement look stronger while narrowing the user experience. This illustrates a general model-evaluation risk, rather than an LLM-specific example. section 8.2

How AI Model Evaluation covers it

How deep it goes
Defines model anatomy and lifecycle at a high level, while assuming familiarity with model basics rather than developing an LLM architecture.
Examples
Illustrates LLM use with chatbot questions about a dripping faucet and making tomato sauce.
What it assumes
Assumes familiarity with basic model anatomy and use, while providing a high-level definition of inputs, parameters, and training.
What you can do after
Evaluate/understand model quality by distinguishing offline promise from online user impact and recognizing feedback-loop risks.
How it connects
Connects AI models to offline evaluation, A/B testing, feedback loops, and exposure bias.

Also explained in

Compare how 89 books explain language model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.