A DataFrame organizes tabular data into labeled rows and columns. Each column represents a dimension, such as a vocabulary term, while cells hold values such as token counts. Columns may contain different kinds of values, and pandas aligns labels while handling missing entries. section 3.2
A DataFrame is useful when you need to:
- Convert a list of dictionaries into columns automatically. section 3.2
- Represent documents as rows and vocabulary terms as dimensions. section 3.2
- Replace missing dictionary keys with NaN, then clean values with fillna(0).astype(int). section 3.2
Document-term counts
Counting tokens for 16 short documents produces a DataFrame with 16 rows and 246 vocabulary columns. A row such as df.iloc[10] exposes the counts for one document, while df.columns contains the vocabulary. section 3.2
Listing 3.1 Short documents about bias
>>> docs = [nlp(s) for s in bias_intro.split('\n')
... if s.strip()] #1
>>> counts = []
>>> for doc in docs:
... counts.append(Counter([
... t.text.lower() for t in doc])) #2
>>> df = pd.DataFrame(counts)
>>> df = df.fillna(0).astype(int) #3
>>> len(df)
16
>>> df.head()
algorithmic bias describes systematic ... between same service
0 1 1 1 1 ... 0 0 0
1 0 1 0 0 ... 0 0 0
2 1 1 0 0 ... 0 0 0
3 1 1 0 1 ... 0 0 0
4 0 1 0 0 ... 0 0 0
Inspect a runnable construction that counts tokens, creates a DataFrame, fills missing values, and displays its shape and first rows. section 3.2
