liveBook · Manning

Softplus function

As explained in Applied Reinforcement Learning

A smooth function, log(1 + exp(x)), commonly used to provide differentiable positive-valued outputs or penalties.

Softplus, log(1 + exp(x)), is a smooth alternative to a hard threshold. In a penalty, it provides a gradual gradient that can guide optimization before a constraint is violated. section 9.7

Data-center cooling

A cooling reward penalizes temperatures and pressure outside safe ranges using smooth hinge penalties. Unlike a hard penalty, these give gradient-based methods a learning signal as conditions approach the limits. section 9.7

Listing 9.5 Reward computation with energy and violation penalties
energy_cost = (self.cfg.w_energy_fan *
    (fan ** 2).mean(dim=1) +
    self.cfg.w_energy_valve *
    (valve ** 2).mean(dim=1))  #A
 
temp_penalty = soft_hinge_penalty(
    self.temp, self.cfg.temp_min, self.cfg.temp_max,
    self.cfg.softplus_beta).mean(dim=1) *
    self.cfg.w_temp_violation  #B
 
pressure_penalty = soft_hinge_penalty(
    self.dp, self.cfg.dp_min, self.cfg.dp_max,
    self.cfg.softplus_beta).mean(dim=1) *
    self.cfg.w_pressure_violation  #C
 
rewards = -(energy_cost + temp_penalty +
    pressure_penalty)  #D

See how temperature and pressure penalties combine with energy cost in the reward calculation. section 9.7

How Applied Reinforcement Learning covers it

How deep it goes
Contrasts smooth gradients with hard thresholds but does not detail the penalty implementation.
Examples
Applies smooth hinge penalties to temperature and pressure bounds in data-center cooling.
What you can do after
Understand why gradual penalties provide learning signals near safety boundaries.
How it connects
Connects to reward functions, gradient-based optimization, and temperature and pressure constraints.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.