Natural language

As explained in How Large Language Models Work

Human language is a naturally evolved system of symbolic communication used by people.

Natural language is a naturally evolved system of symbolic communication used by people. LLMs must convert it into an internal representation and convert generated representations back into language; that representation affects what they learn and how solutions can be built. section 1.9

Language alongside code

Natural language and programming languages can share tokenization patterns. Meaningful variable names and comments connect code with human descriptions of intent, helping an LLM correlate the two. section 6.1

Figure 6.2 Two similar samples of code written in the programming languages Python (left) and Java (right). These show how byte-pair encoding can identify similar tokens across different languages. The boxes show individual tokens. Standard tokenization methods for human languages do a reasonable job on code since it has many similarities to natural language.

Compare Python and Java tokens to see why methods designed for human language can also process code. section 6.1

How How Large Language Models Work covers it

How deep it goes

Explains conversion to internal representations and links tokenization to code, but stops short of detailing representation formation.

Examples

Uses Python, Java, Modula-3, variable names, comments, and tokenized code to illustrate language relationships.

What you can do after

Evaluate how language representation influences what an LLM learns and how it can support software solutions.

How it connects

Connects natural language with LLMs, generative AI, tokenization, programming languages, and internal representations.

liveBookmanning.com

Read How Large Language Models Work on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionA natural language is a human language that develops and is used for communication rather than being defined solely by formal syntactic rules.
• 

  liveBookmanning.com
  LLMs in ProductionA human communication system expressed through speech or writing and governed by linguistic conventions.
• 

  liveBookmanning.com
  Learn AI-Assisted Python Programming, Second EditionA natural language is a human language that arises through ordinary human communication rather than being designed primarily for programming computers.

Compare how 4 books explain natural language →

Related concepts

• morphologyhas part · Morphology is the study of word structure and the processes by which words are formed and grammatical information is expressed.
• syntaxhas part · Syntax is the set of rules governing how elements of a language or notation may be arranged to form valid expressions or constructs.
• phoneticshas part · Phonetics is the study of speech sounds, including their physical production, transmission, and perception.
• semanticshas part · Semantics is the meaning and behavior conveyed or assigned to expressions, constructs, representations, or their combinations in a language or symbolic system.
• pragmaticshas part · Pragmatics is the study of how context, speaker intentions, shared knowledge, and circumstances affect language meaning and use.
• language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.

This explainer was generated by Manning's knowledge graph from the text of How Large Language Models Work. The section links point to the parts of the book it draws on.