spaCy is an open-source natural-language-processing library that turns text into structured documents. Its tokenizer splits words and punctuation while a language model adds sentence boundaries, part-of-speech tags, lemmas, and syntax-tree relationships. section 2.1.4
A typical workflow loads an English model, processes text into a Doc object, and inspects its tokens. The same document exposes sentences, entities, token attributes, dependency roles, and fine-grained grammatical tags. section 2.1.4 section 11.4.2
>>> import spacy #1
>>> spacy.cli.download('en_core_web_sm') #2
>>> nlp = spacy.load('en_core_web_sm') #3
>>> doc = nlp(texts[-1])
>>> type(doc)
spacy.tokens.doc.Doc
>>> tokens = [tok.text for tok in doc]
>>> tokens[:9]
['There', "'s", 'no', 'such', 'thing', 'as', 'survival', 'of', 'the']
>>> tokens[9:17]
['fittest', '.', 'Survival', 'of', 'the', 'most', 'adequate', ',']
Load an English model, process text, and inspect the resulting Doc and token sequence. section 2.1.4
Inspecting entities and syntax
After processing text, doc.ents returns detected named entities. Token attributes can be collected into a table containing the original token, POS tag, entity type, dependency role, and children. For example, a token can be identified as a proper noun, a PERSON entity, and a passive nominal subject. displaCy can render dependency relationships as an SVG file, a browser view, or inline notebook output. section 11.4.2 section 2.1.4
spaCy’s breadth has a cost. Processing roughly 160,000 characters took about 4.67 seconds, or 7,000 words per second, while disabling pipeline components reduced the measured time to about 199 milliseconds. When linguistic annotations are unnecessary, a regular-expression tokenizer can be substantially faster; when accuracy and annotations matter, spaCy provides the fuller pipeline. section 2.1.5 section 2.1.5
