Lexical analysis

As explained in Kotlin in Action, 2E

A lexical analyzer converts an input character sequence into a list of tokens according to the input language's lexical rules.

Lexical analysis converts an input character sequence into an ordered list of tokens. In JSON deserialization, it is the first stage: the lexer prepares input for the parser, which recognizes higher-level structures before deserialization creates Kotlin objects. section 12.2.4

JSON lexical analysis produces two token categories:

• Character tokens represent syntax characters such as commas, colons, braces, and brackets. section 12.2.4
• Value tokens represent string, number, Boolean, and null constants; examples include "Catch-22" and 42. section 12.2.4

Figure 12.8 The process of parsing JSON: first, the lexer takes the input text and divides it into tokens. Then, the parser processes the different semantic elements. Finally, the deserializer turns them into the final Kotlin objects.

Follow the parsing pipeline from source text to tokens, semantic JSON elements, and final Kotlin objects. section 12.2.4

How Kotlin in Action, 2E covers it

How deep it goes

Defines token categories and pipeline position but does not detail lexer implementation.

Examples

Uses braces, strings such as "Catch-22", integers such as 42, commas, colons, Booleans, and nulls.

What you can do after

Evaluate how JSON text is divided into tokens before parsing and object construction.

How it connects

Connects lexical analysis to JSON, parsing, syntax analysis, deserialization, and reflection.

liveBookmanning.com

Read Kotlin in Action, 2E on liveBook Get the book at manning.com

Related concepts

• tokenproduces · A token is a discrete unit of text, such as a word, word fragment, character, or symbol, represented and processed as an element by a language model.

This explainer was generated by Manning's knowledge graph from the text of Kotlin in Action, 2E. The section links point to the parts of the book it draws on.