Co-occurrence analysis

As explained in Starting Data Analytics with Generative AI and Python

Co-occurrence analysis is a text-analysis technique that measures how often terms appear near one another within a defined context window.

Co-occurrence analysis finds words that appear near one another in a text. Unlike n-gram analysis, which focuses on adjacent sequences, it can connect terms separated by other words. Choose a context window—the number of nearby words to consider—then count word pairings in a co-occurrence matrix. section 5.4 section 5.4.1

A basic workflow is:

• Clean and tokenize the text, then select a context window. section 5.4
• Count which words fall within each other’s windows and record the counts in a matrix. section 5.4
• Inspect frequent pairs as clues to themes, preferences, or issues; check them against the underlying text before drawing conclusions. section 5.4.1

From software updates to performance concerns

Reviews might discuss a software update that “slows down” a device “over time.” These terms need not appear together in one three-word sequence to be related by a co-occurrence analysis. section 5.4

Figure 5.2 Heatmap of the co-occurrence matrix for the top 30 words in customer reviews with English translations—visualizing word associations and relationships to uncover patterns and themes in customer feedback

A heatmap visualizes frequent word associations in customer reviews, with English translations. section 5.4.3

How Starting Data Analytics with Generative AI and Python covers it

How deep it goes

Defines cleaning, tokenization, window selection, pair counting, and matrix construction without a detailed mathematical treatment.

Examples

Uses software-update performance complaints and customer-review word pairs about delivery, product quality, and receipt.

What it assumes

Requires cleaned, tokenized text and a chosen context window; needs no deep linguistic knowledge.

What you can do after

Apply it via Python to customer-review text by preprocessing tokens, choosing a window, and constructing a co-occurrence matrix.

How it connects

Compares co-occurrence analysis with n-gram analysis and connects it to sentiment lexicon bootstrapping.

liveBookmanning.com

Read Starting Data Analytics with Generative AI and Python on liveBook Get the book at manning.com

Related concepts

• sparse representationuses · A sparse representation stores non-default or present values and information locating them, while omitting or compressing default or absent values.
• co-occurrence matrixproduces · A co-occurrence matrix is a table whose entries record the frequency or weight with which pairs of items occur together within specified contexts.
• text miningpart of · Text mining is the computational extraction of patterns, relationships, facts, information, or meaning from textual data.
• reviewuses · A review is a user-authored evaluative statement or assessment of a product, service, business, experience, or work.

This explainer was generated by Manning's knowledge graph from the text of Starting Data Analytics with Generative AI and Python. The section links point to the parts of the book it draws on.