liveBook · Manning

Large language model

As explained in Applied Reinforcement Learning

A machine-learning model trained on large text corpora to model and generate language.

A large language model (LLM) is a neural network trained on vast amounts of text to predict the next token. It generates a response one token at a time. This produces fluent language, but next-token prediction alone does not explicitly optimize for safety, helpfulness, or appropriateness. section 10.1 section 10.2

Before processing text, the model tokenizes it into units such as words, word pieces, or punctuation, then maps those tokens to integer IDs. A transformer processes the resulting sequence and produces probabilities for the next token. During inference, the input includes the prompt and conversation history. section 10.2 section 10.3

Fluency versus a useful response

For a mental-health query, both “Consider talking to a professional” and a longer response acknowledging distress and addressing immediate danger are plausible. Human raters may prefer the latter for its empathy, urgency handling, and clearer escalation advice. Reinforcement learning from human feedback (RLHF) provides a way to express such preferences. section 10.1

Language generation can be viewed as a reinforcement learning problem:

  • State: the prompt plus all tokens generated so far. section 10.3
  • Action: select the next token from the vocabulary. section 10.3
  • Policy: the language model assigns probabilities to possible next tokens. section 10.3
  • Reward: a learned reward model scores responses based on human preferences. section 10.1 section 10.9
Figure 10.2 Language generation mapped as a Markov Decision Process.

Use the diagram to trace how a growing response maps to states and token choices. section 10.3

One RLHF workflow first uses supervised fine-tuning, then trains a reward model on pairwise preferences. Proximal Policy Optimization (PPO) updates the language-model policy against that reward while a KL-divergence penalty keeps it close to the supervised reference. Evaluation still needs multiple measures: safety, empathy, urgency matching, fluency, and correctness may not move together. section 10.9

Turning financial news into an agent’s state

An LLM can also act as a representation layer rather than a trading agent. It converts news headlines into a structured numeric summary that a portfolio-management agent can use as part of its state. A fixed 13-field signal schema lets a rule-based extractor, an off-the-shelf model, or a fine-tuned model feed the same interface without changing the agent or environment. section 12.5 section 12.9.1

Listing 12.6 The signal schema: thirteen named fields
@dataclass
class MarketSignal:
    market_sentiment:   float = 0.0   #A
    risk_on_signal:     float = 0.0
    rates_pressure:     float = 0.0
    inflation_pressure: float = 0.0
    growth_signal:      float = 0.0
    tech_signal:        float = 0.0   #B
    energy_signal:      float = 0.0
    financials_signal:  float = 0.0
    defensive_signal:   float = 0.0
    volatility_risk:    float = 0.0
    liquidity_risk:     float = 0.0
    geopolitical_risk:  float = 0.0
    confidence:         float = 0.0   #C
 
    def to_vector(self) -> np.ndarray:
        return np.asarray([getattr(self, f) for f in self.FIELDS], 
        ➥ dtype=np.float32)  #D

Inspect the named market-signal fields and their conversion to a numeric vector. section 12.5

How Applied Reinforcement Learning covers it

How deep it goes
Explains tokenization, transformer flow, RL mapping, and fine-tuning workflow without the full architectural mathematics.
Examples
Illustrates response preferences with a mental-health query and structured signal extraction with financial news headlines.
What it assumes
Assumes familiarity with neural networks, attention, policy gradients, and sequence models while introducing tokenization and the language-model architecture.
What you can do after
Evaluate language-model alignment and representation choices using preference comparisons, state mappings, and multiple response-quality measures.
How it connects
Connects next-token prediction with Markov decision processes, reinforcement learning from human feedback, supervised fine-tuning, reward models, and PPO.

Also explained in

Compare how 6 books explain large language model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.