Tokenization, or lexing, converts raw source text into a stream of lexical tokens. In Rust, procedural macros receive code as a TokenStream and return another TokenStream for generated code. Parsing follows tokenization by organizing tokens into an abstract syntax tree (AST); spans connect tokens to their original source for diagnostics. section 3.2
From expression to tokens
Tokenizing `1 + 11` produces three tokens: an integer literal, the `+` punctuation token, and another integer literal. Parsing can then interpret `+` as the AST root with the two literals as branches. section 3.2
TokenStream [Literal { kind: Integer, symbol: "1" }, Punct { ch: '+', spacing: Alone }, Literal { kind: Integer, symbol: "11" }]
Inspect the concrete TokenStream representation produced from `1 + 11`, including literal kinds and punctuation spacing. section 3.2
