TokenStream is the input and output format of Rust procedural macros: a sequence of tokens representing code rather than raw source text. A macro receives every relevant detail of an item—such as a struct—and returns tokens that Rust compiles as generated code. Lexing first turns source text into tokens; parsing then interprets them as an abstract syntax tree (AST). section 3.2 section 1.3.1
TokenStream [Literal { kind: Integer, symbol: "1" }, Punct { ch: '+', spacing: Alone }, Literal { kind: Integer, symbol: "11" }]
A simple expression becomes separate literal and punctuation tokens, illustrating the lower-level form a macro receives. section 3.2
A TokenStream is iterable because it is a sequence of TokenTrees. TokenTrees can recursively contain further trees or tokens, letting a macro inspect individual identifiers, punctuation, literals, and grouped input. Direct iteration can work for narrowly controlled input, but extracting even a type name requires manual assumptions about token order and shape. section 3.6
Procedural macro forms use TokenStream differently:
- Derive macros accept an item stream and return additional generated code; the annotated type remains unchanged. section 1.3.1 section 3.9
- Attribute macros receive an attribute-argument stream plus an item stream; their output replaces the annotated input. section 4.0 section 8.7
- Function-like macros receive one stream and replace it with generated output, so they must recreate any input code that should remain. section 5.1.1 section 5.1.2
Preserving input while adding code
A function-like macro that parses a struct must emit that struct again before adding an implementation. One approach clones the incoming TokenStream before parsing, converts the clone for quote!, and interpolates it into the output. Parsing into an AST and quoting that AST is simpler when the parsed type implements ToTokens and can still be borrowed for its name. section 5.1.2
Use syn to parse a TokenStream into structured Rust syntax and quote! to assemble output with interpolated values. For example, field identifiers and types can be mapped into small generated TokenStream fragments, then repeated with commas inside a reconstructed struct. Small fragments need not be complete programs: quote! can stitch them into final output. section 4.4 section 6.4.4
Manual parsing avoids syn and may compile faster for a small, constrained macro, but it makes both extraction and code generation more burdensome. A lightweight parser is another option. Whatever approach is used, test generated tokens: compare string forms when exact output matters, or parse output back into an AST and inspect it. cargo expand helps inspect expanded code. section 3.6 section 6.3 section 3.9
