liveBook · Manning

Vocabulary

As explained in Starting Data Analytics with Generative AI and Python

A vocabulary is the fixed collection of tokens that a language model can represent.

A language model’s vocabulary is the fixed collection of tokens it can represent. A text-mining lexicon is a different kind of vocabulary: a collection of terms used to find patterns in text, sometimes with sentiment labels or scores. section 5.6.1 section 5.6.4

A lexicon can give an analysis a stronger starting point than a quickly assembled keyword list. Sentiment lexicons help identify positive or negative language, while other dictionaries cover emotions, specialist terminology, slang, or opinions. section 5.6 section 5.6.1

Before using a lexicon, check whether it fits the task:

  • Compare its domain and coverage with the language in your records. A medical lexicon may not suit finance. section 5.6.2
  • Check its source, publication date, accuracy, and potential cultural, linguistic, or domain bias. Compare resources with pilot tests rather than choosing the first search result. section 5.6.2
  • Inspect the file format, encoding, term representation, and sentiment labels before adapting code to load and score it. section 5.6.3 section 5.6.4

Portuguese product reviews

An AI-generated Python workflow assumes that the SentiLex sentiment lexicon is a CSV file with word and polarity columns. The actual file has a different format, so the loading code needs adjustment. Ask the assistant to revise the code, but inspect the actual entries and test the result rather than trusting the initial assumption. section 5.6.4

def load_sentilex(filepath):
    sentilex_dict = {}
    with open(filepath, 'r', encoding='utf-8') as file:
        for line in file:
            parts = line.strip().split(';')
            word = parts[0].split('.')[0]  # Get the word before '.PoS='
            pol_entry = [part for part in parts if 
            part.startswith('POL:N0=')]
            if pol_entry:
                # Assuming there's exactly one 'POL:N0=' entry per line
                polarity = int(pol_entry[0].split('=')[1])
                sentilex_dict[word] = polarity
    return sentilex_dict

# Example usage
filepath = 'SentiLex-lem-PT02.txt'
sentilex_dict = load_sentilex(filepath)

# Now sentilex_dict is ready to be used as in the previous example.

This loader extracts words and polarity values from a SentiLex lemma file; its assumptions should be checked against the actual entries. section 5.6.4

Matching also depends on how terms are represented. If the lexicon contains lemmas, reviews may need lemmatization before matching; otherwise, word forms in reviews can be missed. Even a match is not enough to establish sentiment: a simple search for “good” can put “so good” and “not so good” in the same bucket. Inspect sample reviews and compare results after preprocessing changes. section 5.6.5 section 5.6.1

How Starting Data Analytics with Generative AI and Python covers it

How deep it goes
Explains lexicon selection, file loading, scoring assumptions, and preprocessing adjustments without detailing language-model vocabulary internals.
Examples
Uses Portuguese product reviews and SentiLex files to illustrate format mismatches and lemma-based matching.
What you can do after
Evaluate and apply lexicons by checking their suitability, adapting loading code, and aligning review preprocessing with term forms.
How it connects
Connects lexicons with sentiment analysis, keyword matching, and lemmatization.

Also explained in

Compare how 17 books explain vocabulary →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Starting Data Analytics with Generative AI and Python. The section links point to the parts of the book it draws on.