liveBook · Manning

Frequency analysis

As explained in Natural Language Processing in Action, Second Edition

Frequency analysis is the examination of how often symbols, tokens, or other elements occur in a text or signal.

Frequency analysis counts how often symbols or tokens occur, turning a text into measurable evidence. Comparing character frequencies can reveal language patterns, support authorship attribution, or expose a hidden transformation. section 3.4.1

A practical workflow uses:

  • CountVectorizer configured for words, characters, or character trigrams. section 3.4.1
  • Explicit tokenizer, punctuation, stop-word, and vocabulary choices so downstream bugs are easier to diagnose. section 3.4.1
  • Normalized counts and plots for comparing distributions across texts. section 3.4.1

Revealing a hidden message

Counting characters in Python’s encoded this message produces a distinctive histogram. Comparing its peaks with an ordinary English article suggests a fixed shift; applying ROT-13 reveals “The Zen of Python.” section 3.4.1

Listing 3.14 A secret message from Tim Peters
>>> from this import s as secret
>>> print(secret)

Gur Mra bs Clguba, ol Gvz Crgref
Ornhgvshy vf orggre guna htyl.
Rkcyvpvg vf orggre guna vzcyvpvg.
Fvzcyr vf orggre guna pbzcyrk.
...
Nygubhtu arire vf bsgra orggre guna *evtug* abj.
Vs gur vzcyrzragngvba vf uneq gb rkcynva, vg'f n onq vqrn.
Vs gur vzcyrzragngvba vf rnfl gb rkcynva, vg znl or n tbbq vqrn.
Anzrfcnprf ner bar ubaxvat terng vqrn -- yrg'f qb zber bs gubfr!

Inspect the encoded message before counting its characters and comparing distributions. section 3.4.1

How Natural Language Processing in Action, Second Edition covers it

How deep it goes
Builds reusable CountVectorizer-based counting, normalization, plotting, and peak comparison, while stopping short of general cipher theory.
Examples
Uses Python’s encoded this message, an English machine-learning article, character histograms, and ROT-13 decoding.
What it assumes
Assumes familiarity with CountVectorizer and introduces character tokenization, vectorizer configuration, and frequency comparison through runnable examples.
What you can do after
Apply it via scikit-learn CountVectorizer to count configurable tokens, compare normalized distributions, and inspect suspicious patterns.
How it connects
Connects frequency analysis with term frequency–inverse document frequency vectors, character n-grams, language detection, and authorship attribution.

Also explained in

Compare how 2 books explain frequency analysis →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Natural Language Processing in Action, Second Edition. The section links point to the parts of the book it draws on.