Convexity

As explained in Math and Architectures of Deep Learning

The property of a function or set that, in this section, is characterized by being above its tangent and by a nonnegative second derivative.

Convexity describes a geometric and analytic property of sets and functions. For a convex set, the straight line joining any two points remains entirely inside the set. For a convex function, weighted averages of function values are at least as large as the function evaluated at the corresponding weighted-average input. section 3.7 section 3.7.1 section 3.7.2

Figure 3.15 Convex and nonconvex sets. The points in the left-hand shaded region form a convex set.The line joining any pair of points in that shaded region lies entirely in the shaded region: for example, AB.The points in the right-hand shaded region form a nonconvex set. For instance, the line joining points C and D passes through a nonshaded region even though both end points belong to a shaded region.

Compare convex and nonconvex regions by testing whether every line segment between two points stays inside the region. section 3.7.1

Geometrically, a convex curve lies below every secant line joining two points on it. Equivalently, the weighted-average point on the secant lies above the curve at the same horizontal coordinate. A related tangent characterization says that the entire convex curve lies above each tangent line, touching it at the point of tangency. section 3.7.2 section 3.7.2

For a twice-differentiable one-dimensional function, convexity can be checked through its second derivative:

• If the second derivative is always positive, the curve bends so that it lies above its tangent locally. section 3.7.3
• The first-order Taylor approximation is the tangent line; adding the second-order term produces a quadratic approximation that is at least as large when the second derivative is positive. section 3.7.3
• The criterion connects local curvature with the tangent-line and weighted-average definitions of convexity. section 3.7.2 section 3.7.3

Common convex functions

The function g(x) = x² is convex because its second derivative is 2. More generally, even powers such as x⁴ and x⁶ are convex, as is eˣ. Since log x is concave, −log x is convex. Multiplying a convex function by a positive scalar preserves convexity, and sums of convex functions remain convex. section 3.7.4

How Math and Architectures of Deep Learning covers it

How deep it goes

Defines sets and functions geometrically, connects tangents to Taylor expansions, and verifies several functions with second derivatives.

Examples

Uses convex and nonconvex regions, x², even powers, eˣ, log x, −log x, positive scaling, and sums.

What it assumes

Assumes points, line segments, weighted averages, derivatives, and Taylor expansions, while developing convexity from geometric definitions.

What you can do after

Evaluate/understand convexity by testing line segments, weighted-average inequalities, tangent behavior, or second derivatives.

How it connects

Connects convexity with local minima, weighted averages, secant lines, tangent lines, Taylor series, second derivatives, and Jensen’s inequality.

liveBookmanning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Build a Robo-Advisor with Python (From Scratch)A mathematical property of functions or sets that ensures line-segment combinations remain compatible with global optimization guarantees.

Compare how 2 books explain convexity →

Related concepts

• linear regressionincludes · Linear regression is a statistical modeling method that estimates a continuous response as a linear combination of predictor variables and fitted coefficients.
• second derivativemeasured by · A second derivative is the derivative of a function’s first derivative with respect to the same variable, describing how its rate of change varies.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.