liveBook · Manning

Sigmoid

As explained in Applied Reinforcement Learning

A differentiable function that maps real numbers to values between zero and one.

A sigmoid smoothly maps any real input to a value between 0 and 1: sigmoid(x) = 1 / (1 + exp(−x)). In a Gaussian policy, it squashes unbounded samples into the range needed for fan-speed actions. Because the sigmoid’s tails compress values near the boundaries, the transformation changes probability density; a squash correction adjusts log-probabilities for this change. section 9.7

Listing 9.7 Action sampling with Gaussian policy and squashing
def get_action_and_value(self, obs: torch.Tensor,
                         deterministic: bool = False):
    action_mean, action_std, values, _ = self.forward(obs) #A
    dist = Normal(action_mean, action_std)  #B
    raw_actions = action_mean if deterministic
        else dist.rsample()  #C
    actions = torch.sigmoid(raw_actions)  #D
    eps = 1e-6
    log_prob_raw = dist.log_prob(raw_actions)  #E
    squash_correction = (
        torch.log(actions.clamp(eps, 1-eps)) +
        torch.log((1 - actions).clamp(eps, 1-eps)))  #F
    log_probs = (
        log_prob_raw + squash_correction).sum(dim=[1, 2])  #G
    entropy = dist.entropy().sum(dim=[1, 2])  #H
    return actions, log_probs, entropy, values

See the Gaussian-policy action sampling code, including sigmoid squashing and the log-probability correction. section 9.7

How Applied Reinforcement Learning covers it

How deep it goes
Defines the sigmoid and explains action squashing, boundary compression, and log-probability correction without deriving them.
Examples
Uses Gaussian samples transformed into fan-speed actions between zero and one.
What it assumes
Assumes real-valued Gaussian samples and bounded actions, while defining sigmoid from its formula.
What you can do after
Apply it in a squashed Gaussian policy to produce bounded actions and corrected log-probabilities.
How it connects
Connects Gaussian policies, reparameterization, and policy gradients through differentiable action sampling.

Also explained in

Compare how 23 books explain sigmoid →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.