Corpus

Q: How deeply does Natural Language Processing in Action, Second Edition cover corpus?

Defines document collection, indexing, counting, and vectorization, while deferring deeper language-model mechanics to Word2Vec.

As explained in Natural Language Processing in Action, Second Edition

A corpus is a structured collection of documents or language samples used for linguistic analysis or model development.

A corpus is a structured collection of documents or language samples. Counting words across its documents reveals patterns that support linguistic analysis and model development. section 1.5.3

A corpus can support a simple statistical workflow:

Decode character groups with a Morse or ASCII table. section 1.5.3
Count words or short sequences and record the documents where they occur. section 1.5.3
Use the resulting counts as statistical vectors, while recognizing that binning loses information about language. section 1.5.3

Counting words

Counter("Guten Morgen Rosa".split()) produces counts for Guten, Morgen, and Rosa; punctuation and capitalization affect the separate counts in "Good morning Rosa!". section 1.5.3

How Natural Language Processing in Action, Second Edition covers it

Cover of Natural Language Processing in Action, Second Edition

liveBook manning.com

Read Natural Language Processing in Action, Second Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Software Testing with Generative AIA corpus is a curated collection of documents or other information assembled for retrieval and analysis.
liveBook manning.com

Generative AI in ActionA corpus is a collection of documents or passages assembled for information retrieval or language processing.
liveBook manning.com

AI-Powered SearchA document corpus is a collection of documents assembled for linguistic analysis or information retrieval.

Compare how 4 books explain corpus →

Related concepts

JSONhas part · JSON is a standardized, language-independent text format for representing structured data using objects, arrays, strings, numbers, Boolean values, and null.
language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
retrieval-augmented generationused by · Retrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.
evaluation datasetpart of · An evaluation dataset is a collection of examples or test cases, often with expected or judged outcomes, designated to assess a model or system against defined criteria.

This explainer was generated by Manning's knowledge graph from the text of Natural Language Processing in Action, Second Edition. The section links point to the parts of the book it draws on.