A large language model (LLM) is a neural network trained on vast amounts of text to predict the next token. It generates a response one token at a time. This produces fluent language, but next-token prediction alone does not explicitly optimize for safety, helpfulness, or appropriateness. section 10.1 section 10.2
Before processing text, the model tokenizes it into units such as words, word pieces, or punctuation, then maps those tokens to integer IDs. A transformer processes the resulting sequence and produces probabilities for the next token. During inference, the input includes the prompt and conversation history. section 10.2 section 10.3
Fluency versus a useful response
For a mental-health query, both “Consider talking to a professional” and a longer response acknowledging distress and addressing immediate danger are plausible. Human raters may prefer the latter for its empathy, urgency handling, and clearer escalation advice. Reinforcement learning from human feedback (RLHF) provides a way to express such preferences. section 10.1
Language generation can be viewed as a reinforcement learning problem:
- State: the prompt plus all tokens generated so far. section 10.3
- Action: select the next token from the vocabulary. section 10.3
- Policy: the language model assigns probabilities to possible next tokens. section 10.3
- Reward: a learned reward model scores responses based on human preferences. section 10.1 section 10.9
Figure 10.2 Language generation mapped as a Markov Decision Process.

Use the diagram to trace how a growing response maps to states and token choices. section 10.3
One RLHF workflow first uses supervised fine-tuning, then trains a reward model on pairwise preferences. Proximal Policy Optimization (PPO) updates the language-model policy against that reward while a KL-divergence penalty keeps it close to the supervised reference. Evaluation still needs multiple measures: safety, empathy, urgency matching, fluency, and correctness may not move together. section 10.9
Turning financial news into an agent’s state
An LLM can also act as a representation layer rather than a trading agent. It converts news headlines into a structured numeric summary that a portfolio-management agent can use as part of its state. A fixed 13-field signal schema lets a rule-based extractor, an off-the-shelf model, or a fine-tuned model feed the same interface without changing the agent or environment. section 12.5 section 12.9.1
Listing 12.6 The signal schema: thirteen named fields
@dataclass
class MarketSignal:
market_sentiment: float = 0.0 #A
risk_on_signal: float = 0.0
rates_pressure: float = 0.0
inflation_pressure: float = 0.0
growth_signal: float = 0.0
tech_signal: float = 0.0 #B
energy_signal: float = 0.0
financials_signal: float = 0.0
defensive_signal: float = 0.0
volatility_risk: float = 0.0
liquidity_risk: float = 0.0
geopolitical_risk: float = 0.0
confidence: float = 0.0 #C
def to_vector(self) -> np.ndarray:
return np.asarray([getattr(self, f) for f in self.FIELDS],
➥ dtype=np.float32) #DInspect the named market-signal fields and their conversion to a numeric vector. section 12.5





