Singular value decomposition

As explained in Natural Language Processing in Action, Second Edition

Singular value decomposition is a matrix factorization that expresses a matrix as the product of orthogonal singular-vector matrices and a diagonal singular-value matrix.

Singular value decomposition (SVD) factorizes a matrix W into U, S, and V: orthogonal matrices U and V plus a diagonal matrix S. Multiplying the factors reconstructs W exactly, while the largest singular-value dimensions capture the directions selected for reduction. section 4.3.2

For text analysis, SVD transforms bag-of-words or TF–IDF document vectors into lower-dimensional topic vectors. Truncated SVD keeps only p dimensions, making large sparse matrices more manageable and supporting latent semantic analysis and semantic search. section 4.11 section 4.3.2

A practical SVD workflow for semantic analysis is:

• Represent documents as a bag-of-words or TF–IDF matrix. section 4.11
• Apply truncated SVD and choose the number p of topic dimensions to retain. section 4.3.2
• Compare document distances or similarities in the reduced topic-vector space. section 4.6
• Use a hyperparameter table to compare pipelines and models before deploying semantic search. section 4.11

TruncatedSVD or PCA?

Both scikit-learn methods use SVD-based matrix decomposition, but PCA centers the input while TruncatedSVD does not. Centering a sparse TF–IDF matrix can make it dense, increasing memory use and slowing computation. To compare equivalent results, center the data explicitly before TruncatedSVD: `tfidf_docs = tfidf_docs - tfidf_docs.mean()`. section 4.4.2

SVD-based reduction preserves large distances better than fine local structure because it maximizes overall variance. Check whether documents with similar subject matter remain close, and choose a distance measure suited to the NLP pipeline. section 4.6

Equation 4.2 Truncated SVD

Use the truncated-SVD equation to relate the original matrix dimensions m and n to the retained topic dimensions p. section 4.3.2

How Natural Language Processing in Action, Second Edition covers it

How deep it goes

Defines full and truncated factorization, explains sparsity and centering, and evaluates distance preservation without deriving the algorithm numerically.

Examples

Uses TF–IDF document matrices, a comment dataset, PCA comparisons, topic vectors, and pairwise distance measures.

What it assumes

Assumes matrix-based document representations and basic distance concepts while motivating their use through TF–IDF and sparse text data.

What you can do after

Apply it via scikit-learn to reduce sparse text matrices and assess whether topic-vector distances retain document meaning.

How it connects

Connects SVD to PCA, latent semantic analysis, TF–IDF, bag-of-words, topic modeling, semantic search, sparse matrices, NMF, and random projection.

liveBookmanning.com

Read Natural Language Processing in Action, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Data Without LabelsSingular value decomposition is a matrix factorization that expresses a matrix as the product of two orthogonal matrices and a diagonal matrix of singular values.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningThe statement that any matrix can be decomposed as a product of orthogonal matrices and a diagonal matrix in singular value decomposition form.
• 

  liveBookmanning.com
  LLMs in ProductionSingular value decomposition is a matrix factorization that expresses a matrix as a product of orthogonal and diagonal matrices.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA matrix factorization that expresses a matrix as the product of orthogonal matrices and a diagonal matrix of singular values.
• 

  liveBookmanning.com
  Grokking Machine LearningA matrix decomposition that expresses a matrix as the product of orthogonal matrices and a diagonal matrix of singular values.

Compare how 6 books explain singular value decomposition →

Related concepts

• principal component analysisused by · Principal component analysis is a linear dimensionality-reduction technique that transforms data into orthogonal components ordered by the variance they capture.
• singular valueproduces · A singular value is a nonnegative scalar in a matrix’s singular value decomposition that measures stretching along a corresponding singular direction.
• co-occurrence matrixuses · A co-occurrence matrix is a table whose entries record the frequency or weight with which pairs of items occur together within specified contexts.
• dimensionality reductionuses · Dimensionality reduction is the transformation of data into a representation with fewer dimensions while retaining selected information, structure, or relationships.
• truncated singular value decompositionincludes · Truncated singular value decomposition is a matrix factorization technique that approximates a matrix using a selected number of singular components.
• compressionuses · Compression is the transformation of information into a more compact representation to reduce storage or transmission requirements while retaining information needed for recovery or use.

This explainer was generated by Manning's knowledge graph from the text of Natural Language Processing in Action, Second Edition. The section links point to the parts of the book it draws on.