A language model’s vocabulary is the fixed collection of tokens it can represent. A text-mining lexicon is a different kind of vocabulary: a collection of terms used to find patterns in text, sometimes with sentiment labels or scores. section 5.6.1 section 5.6.4
A lexicon can give an analysis a stronger starting point than a quickly assembled keyword list. Sentiment lexicons help identify positive or negative language, while other dictionaries cover emotions, specialist terminology, slang, or opinions. section 5.6 section 5.6.1
Before using a lexicon, check whether it fits the task:
- Compare its domain and coverage with the language in your records. A medical lexicon may not suit finance. section 5.6.2
- Check its source, publication date, accuracy, and potential cultural, linguistic, or domain bias. Compare resources with pilot tests rather than choosing the first search result. section 5.6.2
- Inspect the file format, encoding, term representation, and sentiment labels before adapting code to load and score it. section 5.6.3 section 5.6.4
Portuguese product reviews
An AI-generated Python workflow assumes that the SentiLex sentiment lexicon is a CSV file with word and polarity columns. The actual file has a different format, so the loading code needs adjustment. Ask the assistant to revise the code, but inspect the actual entries and test the result rather than trusting the initial assumption. section 5.6.4
def load_sentilex(filepath):
sentilex_dict = {}
with open(filepath, 'r', encoding='utf-8') as file:
for line in file:
parts = line.strip().split(';')
word = parts[0].split('.')[0] # Get the word before '.PoS='
pol_entry = [part for part in parts if
part.startswith('POL:N0=')]
if pol_entry:
# Assuming there's exactly one 'POL:N0=' entry per line
polarity = int(pol_entry[0].split('=')[1])
sentilex_dict[word] = polarity
return sentilex_dict
# Example usage
filepath = 'SentiLex-lem-PT02.txt'
sentilex_dict = load_sentilex(filepath)
# Now sentilex_dict is ready to be used as in the previous example.
This loader extracts words and polarity values from a SentiLex lemma file; its assumptions should be checked against the actual entries. section 5.6.4
Matching also depends on how terms are represented. If the lexicon contains lemmas, reviews may need lemmatization before matching; otherwise, word forms in reviews can be missed. Even a match is not enough to establish sentiment: a simple search for “good” can put “so good” and “not so good” in the same bucket. Inspect sample reviews and compare results after preprocessing changes. section 5.6.5 section 5.6.1
















