liveBook · Manning

Frequency analysis

As explained in Starting Data Analytics with Generative AI and Python

Frequency-based text analysis infers textual meaning or themes from the occurrence counts of words, phrases, or terms.

What do customers mention most often? Frequency analysis counts how often words or phrases occur across their reviews. It offers a faster first look at recurring topics than reading a few reviews, which may not represent the whole collection. Counts point to subjects worth investigating; they do not, by themselves, establish what customers meant. section 5.3 section 5.3.1

For a collection of reviews, use a repeatable workflow:

  • Ask a generative AI assistant for Python code, naming the DataFrame, review column, and language. Check any filename it supplies against your actual file. section 5.2.2 section 5.3.4
  • Tokenize the text, lowercase it, discard punctuation, remove language-appropriate stopwords, and count the remaining words. Run and inspect the generated code rather than assuming its language settings are correct. section 5.3.4
  • Read the leading words alongside their counts; examine frequent phrases when individual words lack context. Translate unfamiliar terms before summarizing them. section 5.3.6

A small Portuguese review check

A direct request to count words in pasted reviews produced frequent articles and conjunctions such as “a” and “e.” Those counts reflected grammar more than customer concerns. Repeated content words including “prazo” (deadline) and “aparelho” (device) offered more useful leads about delivery and products. A follow-up request can refine an unhelpful first result. section 5.3.2

('chegou', 'antes', 'prazo') - (arrived, before, deadline)
('bem', 'antes', 'prazo') - (well, before, deadline)
('entregue', 'antes', 'prazo') - (delivered, before, deadline)
('produto', 'chegou', 'antes') - (product, arrived, before)
('entrega', 'antes', 'prazo') - (delivery, before, deadline)
('chegou', 'bem', 'antes') - (arrived, well, before)
('produto', 'entregue', 'antes') - (product, delivered, before)
('entrega', 'super', 'rápida') - (delivery, super, fast)
('antes', 'prazo', 'previsto') - (before, deadline, expected)
('produto', 'ótima', 'qualidade') - (product, great, quality)

The translated three-word combinations give a more specific lead than isolated counts: several mention arrival or delivery before the deadline. Check the underlying reviews before treating that pattern as a conclusion about all deliveries. section 5.3.6

For reporting, a word cloud can make common terms easy to scan, but keep the counts and translations available. Spelling, punctuation, capitalization, and other noise can distort text analysis; a generated tokenizer may also ignore a stated language. Validate promising themes against raw reviews. Where review scores are available, they offer a separate check on claims about satisfaction. section 5.3.6 section 5.0 section 5.3.4 section 5.2.2

How Starting Data Analytics with Generative AI and Python covers it

How deep it goes
Works through direct prompting and a Python counting workflow, including language-sensitive preprocessing and interpretation; it does not develop advanced text-mining methods.
Examples
Uses Portuguese customer reviews, repeated delivery terms, translated three-word phrases, and a word-cloud visualization.
What it assumes
Assumes access to review text and a Python DataFrame while showing how to request code for a named column and language.
What you can do after
Apply it via Python, pandas, and NLTK to count review terms, then inspect and validate the results.
How it connects
Connects frequency analysis to tokenization, stopwords, phrase analysis, co-occurrence analysis, and sentiment analysis.

Also explained in

Compare how 2 books explain frequency analysis →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Starting Data Analytics with Generative AI and Python. The section links point to the parts of the book it draws on.