liveBook · Manning

Chapter 2: Representing language

What you will learn in chapter 2 of Machines that Think

This chapter introduction frames language as a difficult computational problem and previews the progression from rules and statistical units to numerical representations of meaning.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This chapter introduction frames language as a difficult computational problem and previews the progression from rules and statistical units to numerical representations of meaning.

Key terms: context, intent, ambiguity, long-range dependencies, pragmatic inference, rules, statistics, machine-readable units, tokens, embeddings

2.1 Language equals intelligence?

This section introduces the historical link between language and thought and asks whether humanlike language generation demonstrates machine thinking.

Key terms: speech, thought, language, intelligence, logos, primary interface, evaluate artificial intelligence, generating language, thinking, human

2.1.1 When words stand in for minds

This section explains how language structures thought, guides attention and perception, and becomes a common proxy for judging intelligence.

Key terms: scaffolding of culture, naming, syntax, symbols, reasoning, Guugu Yimithirr, cardinal directions, goluboy, siniy, linguistic categories

2.1.2 What machine language does reveal

This section explains why fluent machine language can both mislead people into attributing understanding and still deliver practically meaningful, useful results.

Key terms: anthropomorphism, fluency, comprehension, human cognition, statistical patterns, regularities, human text, grounded model of the world, collective expression, performance

2.1.3 Thought beyond language

This section distinguishes language from the broader range of human thought and explains why language-model fluency is not a complete measure of intelligence.

Key terms: spatial reasoning, visual imagination, motor planning, emotional inference, intuition, expression of intelligence, substance, linguistic ability, broader intelligence, language models

2.1.4 Why language still matters

This section justifies using language and large language models as a practical starting point for examining intelligence without equating language with intelligence.

Key terms: cognitive abilities, common form, intelligent behavior, language, artificial intelligence systems that operate through languag, large language models, integrated form of machine behavior, intelligence reducible to language, broader perspectives on artificial intelligence, underlying mechanisms

2.2 Machines’ first words

This section introduces how language became data for NLP and why early hand-crafted grammar rules gave way to learning patterns from language data.

Key terms: stable artifacts, language as data, writing, recording, digitization, natural language processing, NLP, artificial intelligence, human language, hand-crafted rules

2.2.1 When language was logic

This section explains how early language systems used handcrafted grammatical and semantic rules, and why that approach broke down on real, expanding language use.

Key terms: grammar rules, logical constraints, semantic templates, formal structure, machine translation, subject, verb, object, inflect, idiom

2.2.2 Patterns without understanding

This section shows how early template-based language programs such as ELIZA could produce convincing conversational patterns without interpreting meaning or context.

Key terms: patterns, templates, pattern matching, scripted transformations, ELIZA, psychotherapist, reflection, conversational patterns, ambiguity, context

2.2.3 From rules to evidence

This section explains the shift in NLP from handcrafted grammatical rules to data-driven statistical methods that infer language patterns from real usage.

Key terms: handcrafted rules, real text, patterns, observation, usage, linguistic behavior, probabilistic methods, counts, probabilities, sequences

2.3 Counting words

This section introduces statistical NLP as an early data-driven approach that represents language through word counts and predicts likely next words.

Key terms: statistical NLP, rule-based NLP, words, numbers, sentences, sequences of numbers, most likely next word, given a few words

2.3.1 From intuition to probability

This section explains how probabilistic language models learn likely word sequences from text counts and use a limited recent context to make prediction tractable.

Key terms: probabilistic language modeling, probabilities, training data, word sequences, Markov property, n-gram model, limited number of previous words, local dependencies, long-range relationships, probability of a sequence

2.3.2 What is an n-gram?

This section explains how n-grams use counts of consecutive word sequences in a corpus to estimate the probability of a next word from its preceding context.

Key terms: n-gram, unigram, bigram, trigram, four-gram, conditional probability, context, count, trigram model, statistical language modeling

2.3.3 Building language from statistics

This section shows how probabilistic word-transition estimates generate language-like text and power early practical language technologies without giving machines linguistic understanding.

Key terms: probability, word transition, continuation, context, probabilistic language model, grammar rules, syntax, meaning, statistics, frequency counts

2.3.4 The problem of sparsity

This section explains why n-gram models fail on unseen word combinations and how smoothing lets them assign such combinations a plausible nonzero probability.

Key terms: sparsity, n-gram models, training data, probabilities, zero probability, unseen sequences, n-gram model, smoothing, small probability, generalize

2.3.5 Balancing context and complexity

This section explains the trade-off in n-gram models between remembering longer context and keeping statistical estimation, storage, and computation feasible.

Key terms: n-gram, bigram, trigram, context, higher-order n-gram, sparsity, expressive, feasibility, storage, computation

2.4 A token full of meaning

This section introduces tokenization as the necessary transformation of byte-level text into manipulable units, and explains why choosing those units is difficult and consequential.

Key terms: bytes, raw text, units, model, words, compound terms, inflections, contractions, languages without spaces, segmenting text

2.4.1 From raw text to tokens

This section explains tokenization as the modeling choice that turns raw character strings into tokens while trading vocabulary simplicity against preservation of meaning-bearing distinctions.

Key terms: tokenization, tokens, string of characters, word as a unit, fixed boundaries, punctuation, inflections, contractions, capitalization, tokenization scheme

2.4.2 Words, subwords, and characters

This section compares word-, character-, and subword-level tokenization and explains why subwords became the practical standard for handling diverse and unfamiliar language.

Key terms: tokenization, linguistic structure, morphologically rich languages, word-level tokenization, character-level tokenization, subword-level tokenization, vocabulary, granularity, out-of-vocabulary words, Byte Pair Encoding

2.4.3 Compressing language

This section explains how byte-pair encoding compresses text into reusable subword tokens by repeatedly merging frequent adjacent symbols.

Key terms: byte-pair encoding, BPE, adjacent symbols, merges, vocabulary size, subword tokens, vocabulary, low, est, lowest

2.4.4 From BPE to modern tokenizers

This section explains how WordPiece and SentencePiece extend BPE to learn more effective, language-agnostic tokenizations for modern multilingual neural models.

Key terms: WordPiece, BPE, merge, vocabulary, training corpus, language-modeling objective, reusable units, rare fragments, segmentation, rich morphology

2.4.5 Handling the unknown

This section explains how subword tokenization handles unfamiliar language more informatively than mapping every unseen word to one unknown token.

Key terms: unknown token, unseen word, early NLP systems, subword tokenization, existing vocabulary, known components, cybergenomics, cyber, geno, novelty

2.4.6 Tokens as information boundaries

This section explains how token boundaries determine the resolution, context capacity, efficiency, and cross-language behavior of language-model input.

Key terms: tokenization, tokens, resolution, context window, sequence, token, memory, limited context window, writing systems, scripts

2.4.7 Why tokens matter

This section explains how token choices set the resolution and limits of a model’s language input, motivating representations that capture relationships among tokens.

Key terms: tokenization, tokens, basic units, initial resolution, discrete symbol, identity, similarity, standalone units, relationships between words, representation

2.5 Embedding reality

This introduction explains why tokenized language must move beyond isolated symbols to a representation that captures relationships among tokens.

Key terms: tokenization, tokens, discrete symbols, similarity, contrast, association

2.5.1 From indexes to vectors

This section explains why token IDs and one-hot vectors are inadequate for language meaning and how dense, learned embeddings represent relationships through patterns of word use.

Key terms: token, token ID, one-hot encoding, vocabulary, sparse vectors, dense vectors, embeddings, vector space, distributional hypothesis, statistical patterns

2.5.2 Learning embeddings

This section explains how word2vec learns efficient, static word embeddings by predicting words from their nearby context.

Key terms: word2vec, word embeddings, static embeddings, shallow architecture, Continuous Bag of Words, CBOW, Skip-gram, context, similar internal representations, training

2.5.3 Embeddings go global

This section explains how GloVe builds word embeddings from corpus-wide co-occurrence patterns rather than only local context.

Key terms: GloVe, Global Vectors for Word Representation, co-occurrence statistics, corpus, embedding space, global structure, Word2vec, local context, local word sequences

2.5.4 The geometry of meaning

This section explains how learned embedding spaces turn contextual similarity and linguistic relationships into geometric positions and directions that models can compute with.

Key terms: embedding space, vectors, contexts, clusters, neighborhoods, transformations, directions, singular to plural, present to past, masculine to feminine

2.5.5 Meaning in context

This section explains how contextual embeddings replace one fixed vector per word with representations that change according to the word’s surrounding context.

Key terms: static embeddings, embedding space, vector, contextualized embeddings, contextual embeddings, ELMo, Embeddings from Language Models, layered language models, fully contextualized representations, dynamic pattern

2.5.6 Why embeddings matter

This section explains how embeddings made language tractable for machine learning, what capabilities they enabled, and the costs of their scale and opacity.

Key terms: embeddings, continuous vector space, geometry of meaning, learned numerical patterns, handcrafted rules, semantic relationships, co-occur, transfer learning, vector spaces, grounding

2.6 The last frontier

This section frames language as a uniquely difficult frontier for computational models because its forms, meanings, distributions, and structures resist simple statistical treatment.

Key terms: effectively infinite space of forms, unbounded combinations, uneven distributions, sparsity, rare terms, relationships between words, long stretches of text, implications, ambiguity, pragmatic reasoning

2.6.1 One reality, infinite expressions

This section explains why language models must generalize across infinitely varied expressions while recognizing that grammatical form alone does not ensure meaningful content.

Key terms: vast space of possible expressions, memorization, generalize, variations, nuance, syntax, semantics, syntactically correct, semantically nonsensical, Colorless green ideas sleep furiously

2.6.2 Meaning stretched across distance

This subsection explains why language models must preserve relationships between distant parts of text, not just nearby words or similarity patterns.

Key terms: dependencies, agreement, pronouns, conditionals, nearby context, local patterns, narrow windows, brittle interpretations, drift, embeddings

2.6.3 What language leaves unsaid

This section explains why language leaves meaning implicit and ambiguous, requiring people and models to infer intentions from incomplete contextual signals.

Key terms: ambiguity, intention, expression, linear string of words, misunderstanding, multiple meanings, interpretations, context, world knowledge, pragmatic inference

2.6.4 The gaps between the words

This section explains how language variability and Zipf’s long-tailed word-frequency distribution leave models with sparse evidence for many meaningful expressions.

Key terms: static datasets, variation, sparsely observed, statistical learning, long tail, long-tailed vocabulary, rare forms, technical terms, named entities, emerging slang

2.6.5 Diversity across languages

This section explains why language models must handle fundamentally different ways that human languages encode information rather than treating English-like structure as universal.

Key terms: word order, morphology, tone, script, multilingual representation

2.6.6 Measuring success

This section explains why language-model evaluation needs multiple task-specific measures rather than a single score.

Key terms: metric, fluently, factually wrong, nuance, coherence, single score, translation, faithfulness, natural phrasing, summarization

2.7 Summary

This summary traces how language became computable through statistics, tokens, and embeddings while emphasizing that fluent language use is not proof of understanding and remains hard to model or evaluate.

Where this is used later

Concepts explained in this chapter

← Chapter 1: Old questions, new machines Chapter 3: Artificial neural networks →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.