Entropy

As explained in Machine Learning System Design

Entropy is a measure of uncertainty or dispersion in a probability distribution.

How Machine Learning System Design covers it

How deep it goes

Applies entropy as an optional loss component but does not derive its formula or implementation.

Examples

Uses the category distribution of PhotoStock search results as the concrete diversity example.

What you can do after

Evaluate result diversity and consider adding an entropy-based penalty to a search-ranking loss.

How it connects

Connects entropy with diversity, loss functions, search engines, binary cross-entropy, and focal loss.

liveBookmanning.com

Read Machine Learning System Design on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningEntropy is an information-theoretic impurity measure that is higher when class probabilities are more diverse and zero for a pure set.
• 

  liveBookmanning.com
  Bayesian Optimization in ActionEntropy is an information-theoretic measure of the uncertainty or variability of a random variable, defined for a discrete distribution as the negative sum of each event probability times its base-2 logarithm.
• 

  liveBookmanning.com
  Hacking CryptographyEntropy is a measure of uncertainty in a probability distribution, calculated as the negative sum of each outcome probability multiplied by its logarithmic probability.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningA measure of uncertainty or unpredictability in a probability distribution, high when outcomes are nearly equally likely and low when one outcome dominates.
• 

  liveBookmanning.com
  Real-World CryptographyEntropy is a measure of the uncertainty or unpredictability contained in a random source or secret.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIEntropy is a measure of disorder or randomness in a distribution of class labels.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionEntropy is a measure of disorder or uncertainty in a probability distribution or dataset.
• 

  liveBookmanning.com
  Sutskever's ListEntropy is a thermodynamic quantity related to the number of microscopic configurations compatible with a macroscopic state and to the dispersal of energy and matter.
• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningEntropy is an information-theoretic measure of uncertainty or impurity in a probability distribution.
• 

  liveBookmanning.com
  Outlier Detection in PythonEntropy is a measure of uncertainty or information content in a probability distribution.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthEntropy is a measure of uncertainty in a probability distribution, computed as the expected negative log probability.
• 

  liveBookmanning.com
  Build a Reasoning Model (From Scratch)A measure of uncertainty in the model's output distribution during generation, where low entropy indicates collapse and high entropy indicates randomness.

Compare how 13 books explain entropy →

Related concepts

• Gini impurityalternative to · Gini impurity is a measure of class-label heterogeneity in a set or decision-tree node, calculated from the proportions of its classes.
• uncertaintymeasures · Uncertainty is incomplete knowledge or confidence about a value, state, interpretation, behavior, or outcome.
• bitmeasured by · A bit is a binary digit and unit of digital information that represents one of two distinguishable values, conventionally 0 or 1.
• probability distributionmeasures · A probability distribution is a mathematical specification assigning probabilities to the possible values or outcomes of a random variable, with total probability one.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.