A token is a lexical unit representing either a syntactic symbol or a value in an input language. During JSON deserialization, a lexer splits character input into tokens before a parser recognizes objects, arrays, and key-value pairs. section 12.2.4
JSON token categories include:
- Character tokens represent syntax with special meaning, including commas, colons, braces, and brackets. section 12.2.4
- Value tokens represent string, number, Boolean, and null constants. section 12.2.4
Tokenizing a JSON fragment
For {"title": "Catch-22", "count": 42}, the left brace is a character token, "title" and "Catch-22" are string value tokens, the colon is a character token, and 42 is an integer value token. The parser then uses these tokens to build higher-level JSON structure. section 12.2.4
Figure 12.8 The process of parsing JSON: first, the lexer takes the input text and divides it into tokens. Then, the parser processes the different semantic elements. Finally, the deserializer turns them into the final Kotlin objects.
Trace the JSON pipeline from input characters through lexical tokens and parsed semantic elements to final Kotlin objects. section 12.2.4
