Frequency analysis counts how often symbols or tokens occur, turning a text into measurable evidence. Comparing character frequencies can reveal language patterns, support authorship attribution, or expose a hidden transformation. section 3.4.1
A practical workflow uses:
- CountVectorizer configured for words, characters, or character trigrams. section 3.4.1
- Explicit tokenizer, punctuation, stop-word, and vocabulary choices so downstream bugs are easier to diagnose. section 3.4.1
- Normalized counts and plots for comparing distributions across texts. section 3.4.1
Revealing a hidden message
Counting characters in Python’s encoded this message produces a distinctive histogram. Comparing its peaks with an ordinary English article suggests a fixed shift; applying ROT-13 reveals “The Zen of Python.” section 3.4.1
Listing 3.14 A secret message from Tim Peters
>>> from this import s as secret >>> print(secret) Gur Mra bs Clguba, ol Gvz Crgref Ornhgvshy vf orggre guna htyl. Rkcyvpvg vf orggre guna vzcyvpvg. Fvzcyr vf orggre guna pbzcyrk. ... Nygubhtu arire vf bsgra orggre guna *evtug* abj. Vs gur vzcyrzragngvba vf uneq gb rkcynva, vg'f n onq vqrn. Vs gur vzcyrzragngvba vf rnfl gb rkcynva, vg znl or n tbbq vqrn. Anzrfcnprf ner bar ubaxvat terng vqrn -- yrg'f qb zber bs gubfr!
Inspect the encoded message before counting its characters and comparing distributions. section 3.4.1

