Policy

As explained in Transformers in Action

A policy is a model or probability distribution that determines the actions or responses produced in a decision-making process.

A policy is the decision rule that maps the agent’s present state to its next action. In sequential environments, it guides choices toward maximizing total rewards under uncertainty. section 5.1.1

Policy operates across two related decision-making settings:

• Within an MDP, the policy connects states, actions, rewards, and the agent’s long-term objective. section 5.1.1
• Within preference alignment, an LLM policy is adjusted to favor responses labeled preferred over undesirable alternatives. section 5.2
• RLHF maximizes policy performance through a reward model, whereas DPO directly optimizes the policy without an explicit reward model. section 5.2

Dog-training obstacle course

The dog acts as the agent, obstacles represent states, jumping or crawling supplies actions, and treats provide rewards. A learned policy selects actions that increase expected treats over the course. section 5.1.1

Figure 5.1 A simplified comparison of RLHF and DPO and how DPO directly optimizes for the policy without an explicit reward function

Compare RLHF’s reward-model pathway with DPO’s direct policy-optimization pathway. section 5.2

How Transformers in Action covers it

How deep it goes

Defines state-based action selection, then contrasts reward-model optimization with DPO’s direct policy adjustment and KL-divergence constraint.

Examples

Uses a dog obstacle course, portfolio construction, preferred-response labels, and direct language-model preference optimization.

What it assumes

Builds from MDP components, while DPO discussion assumes the surrounding RLHF and reward-model framing.

What you can do after

Understand how policies select actions and how DPO favors preferred language-model responses without an explicit reward model.

How it connects

Connects policy with MDPs, reinforcement learning, rewards, RLHF, reward models, DPO, preference labels, and KL divergence.

liveBookmanning.com

Read Transformers in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Authorization in ActionAn authorization policy is a declarative rule or set of rules that determines which subjects may perform which actions on which resources under specified conditions.
• 

  liveBookmanning.com
  ASP.NET Core in Action, 3eAn authorization policy is a named set of requirements that determines whether an authenticated subject may access a protected resource or operation.
• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsA policy is a reinforcement-learning decision rule that maps observed states or observations to actions or action probabilities.
• 

  liveBookmanning.com
  Build a Robo-Advisor with Python (From Scratch)A policy is a rule or mapping that specifies which action an agent should take for each state.
• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackA policy is a function that maps an environment state to a probability distribution over actions.
• 

  liveBookmanning.com
  Causal AIA policy is a rule or mapping that selects an action from observed variables, potentially through a stochastic process.
• 

  liveBookmanning.com
  Google Anthos in ActionA policy is a declarative rule that specifies required behavior, configuration, or access conditions for managed resources.
• 

  liveBookmanning.com
  Microsoft Azure in ActionAn access policy is a defined set of rules that determines whether a user or group may access a service or resource.
• 

  liveBookmanning.com
  Terraform in Depth: Infrastructure as Code with Terraform and OpenTofuA policy is a rule set that governs technical, compliance, financial, or contractual decisions about infrastructure and software.
• 

  liveBookmanning.com
  .NET in Action, Second EditionAn authorization policy is a set of access-control rules composed of requirements that determine whether an authenticated principal may perform an action.
• 

  liveBookmanning.com
  Bayesian Optimization in ActionA policy is a decision rule that determines an action from the information available to an optimization procedure.
• 

  liveBookmanning.com
  Building Web APIs with ASP.NET CoreAn authorization policy is a named set of requirements that determines whether an authenticated principal may perform an operation.

Compare how 13 books explain policy →

Related concepts

• attributeuses · An attribute is a named property, value, or piece of information associated with an entity, object, or program element.
• attribute-based access controlused by · Attribute-based access control is an authorization model that evaluates subject, resource, action, and contextual attributes against policies to make access decisions.
• multi-armed banditincludes · A multi-armed bandit is a sequential decision problem in which an agent chooses among options with uncertain rewards, balancing reward seeking with gathering information about those options.
• reinforcement learningproduced by · Reinforcement learning is a machine-learning paradigm in which an agent learns actions or a policy through interaction with an environment and evaluative feedback, such as rewards.
• actionproduces · An action is a choice or operation selected or performed by an actor that can affect a system or environment or produce an outcome.
• stateuses · State is information or a configuration describing the condition of a system, entity, or computation at a particular time and potentially influencing subsequent behavior or decisions.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.