A vocabulary is a collection of terms available for a task. In dictionary-based text mining, it can be a lexicon of words associated with sentiments, emotions, or a subject area. Such a lexicon gives an analysis a starting point beyond keywords chosen by hand. This is a task-specific word list, not an explanation of the fixed token vocabulary a language model can represent. section 5.6 section 5.6.1
Suppose you need to assess sentiment in Portuguese product reviews. A sentiment lexicon supplies terms and polarity information, but choosing one is not enough. Check whether its language and domain fit the reviews, whether it is current, and whose language it may miss. Compare candidates on sample reviews rather than trusting the first resource you find. section 5.6.2 section 5.6.4
Inspect the actual file before asking an AI assistant to write a loader. An initial Python suggestion assumed SentiLex was a CSV with word and polarity columns; the chosen file instead needed parsing as text. Check the resulting entries and scores. If matches remain sparse, check word forms too: a lexicon of lemmas may not match inflected words in reviews until preprocessing produces lemmas. section 5.6.4 section 5.6.5
def load_sentilex(filepath):
sentilex_dict = {}
with open(filepath, 'r', encoding='utf-8') as file:
for line in file:
parts = line.strip().split(';')
word = parts[0].split('.')[0] # Get the word before '.PoS='
pol_entry = [part for part in parts if
part.startswith('POL:N0=')]
if pol_entry:
# Assuming there's exactly one 'POL:N0=' entry per line
polarity = int(pol_entry[0].split('=')[1])
sentilex_dict[word] = polarity
return sentilex_dict
# Example usage
filepath = 'SentiLex-lem-PT02.txt'
sentilex_dict = load_sentilex(filepath)
# Now sentilex_dict is ready to be used as in the previous example.
A Python loader for the selected text file shows how entries and polarity values are extracted instead of assuming a CSV layout. section 5.6.4
