spaCy is an open-source Python library for efficient natural language processing and linguistic feature extraction. It processes text into document objects that expose word-level information, including part-of-speech tags and sentence structure. section 9.3.1
A practical workflow uses spaCy to turn text into interpretable features:
- Install spaCy and download the en_core_web_lg model, or another suitable model. section 9.3.1
- Process each message into a Doc, then count words, unique words, stop words, and sentences. section 9.3.1
- Use these simple features to identify obvious or potentially anomalous records, while recognizing that they may be insufficient for subtle cases. section 9.3.1
- Consider additional measures such as characters per word or words per sentence; sentiment estimation is less suitable for outlier detection. section 9.3.1
Listing 9.8 Extracting features from text using spaCy
import pandas as pd
import spacy
from spacy.lang.en.stop_words import STOP_WORDS
from spacy.lang.en import English
from sklearn.datasets import fetch_20newsgroups
data = fetch_20newsgroups(
remove=('headers', 'footers', 'quotes')) #1
df = pd.DataFrame({"Content": data['data']})
df['Num Chars'] = df['Content'].str.len() #2
df['Num Unique Chars'] = df['Content'].apply(lambda x: len(set(x)))
nlp = spacy.load('en_core_web_lg') #3
stopwords = list(STOP_WORDS)
num_words_arr = [] #4
num_unique_words_arr = []
num_stop_words = []
num_sentences_arr = []
for v in df['Content']:
doc = nlp(v)
num_words_arr.append(len(doc))
num_unique_words_arr.append(len(set([x.text for x in doc])))
num_stop_words.append(len([x for x in doc if x.text in stopwords]))
num_sentences_arr.append(len(list(doc.sents)))
df['Num Words'] = num_words_arr
df['Num Unique Words'] = num_unique_words_arr
df['Num Non-Stop Words'] = df['Num Words'] - num_stop_words
df['Num Sentences'] = num_sentences_arr
The listing demonstrates spaCy-based feature extraction alongside pandas and the 20 Newsgroups dataset. section 9.3.1
