liveBook · Manning

Benford's law

As explained in Statistics Every Programmer Needs

A Benford distribution is a probability distribution in which the first significant digits of numerical values occur with logarithmically decreasing frequencies.

Benford’s law, or the first-digit law, describes a logarithmic pattern in some numeric data: smaller leading digits occur more often than larger ones. It is useful for spotting distributions that merit closer inspection, particularly in fraud detection and data-integrity work. A mismatch is a reason to investigate, not proof of manipulation. section 12.1 section 12.6

For a leading digit d from 1 through 9, the expected probability is P(d) = log10(1 + 1/d). This gives about 30.1% for 1 and 4.6% for 9; zero is excluded. These probabilities provide the benchmark against which observed first-digit frequencies are compared. section 12.1

>>> import numpy as np
>>> df1 = df1.assign(benford = round(np.log10(1 + (1 / df1.d)), 3))
>>> print(df1)
   d  benford
0  1    0.301
1  2    0.176
2  3    0.125
3  4    0.097
4  5    0.079
5  6    0.067
6  7    0.058
7  8    0.051
8  9    0.046

The pandas and NumPy listing calculates the expected probability for each digit, producing the benchmark distribution used in comparisons. section 12.1

Check whether the data is a plausible candidate before testing it:

  • Look for values spanning multiple orders of magnitude, without imposed minimums or maximums. section 12.2
  • Examine whether the values are right-skewed, with a mean greater than the median. section 12.2
  • Exclude numbers used merely as identifiers, such as account or invoice numbers; their assigned digits need not follow this pattern. section 12.2

A practical analysis imports the relevant numeric field, checks its structure and missing values, extracts the first digit, and retains digits 1–9. It then groups those digits into observed frequencies and plots them against the expected curve. Extraction needs care: string conversion can return a minus sign for a negative amount or a zero for a zero amount, neither of which belongs in this first-digit comparison. section 12.4.1 section 12.4.3

Street-address check

Among 194,836 imported East Baton Rouge Parish address entries, the first character included letters as well as digits. After retaining leading digits 1–9, digits 1, 2, and 3 accounted for 60.4% of addresses, versus 60.2% in the ideal Benford distribution. The close aggregate match is useful visual evidence, but it does not establish that every digit matches. section 12.4.1

Visual resemblance is a starting point, not validation. Compare observed and expected frequencies using statistical checks such as chi-square goodness of fit and mean absolute deviation. A distortion factor compares actual and expected mean leading digits; values nearer zero indicate a closer match. Mantissa analysis offers another check: for data obeying the law, the fractional parts of their base-10 logarithms should be approximately uniform. Interpret tests together rather than basing a decision on one chart or statistic. section 12.0 section 12.6

How Statistics Every Programmer Needs covers it

How deep it goes
Works through the probability formula, data preparation, plots, and validation while concentrating on first digits rather than first-two-digit analysis.
Examples
Compares street addresses, world population counts, and utility payment amounts with an ideal Benford distribution.
What it assumes
Introduces the probability calculation and pandas data-frame operations while explaining statistical measures used for validation.
What you can do after
Apply it via Python to screen suitable data sets, validate apparent deviations, and decide whether further investigation is warranted.
How it connects
Connects logarithmic distributions and leading digits to skewness, goodness-of-fit testing, and mantissa statistics.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Statistics Every Programmer Needs. The section links point to the parts of the book it draws on.