liveBook · Manning

Token tree

As explained in Idiomatic Rust

A token tree is a recursive syntactic structure containing tokens, sequences of tokens, or nested token trees for macro matching.

A token tree is a recursive structure used by Rust macros. It can contain one identifier, a sequence of identifiers, or nested token trees, allowing a macro to match complete argument sequences. section 5.1.2

Matching variadic macro input

In `macro_rules! special_println`, `($($arg:tt)*)` names each matched token tree as `$arg`. Parentheses group the repeated rule, while `*` permits any number of matches. The expansion `$($arg)*` forwards the captured tokens to `println!`, preserving arguments without adding punctuation. section 5.1.2

Token-tree repetition uses familiar quantifiers:

  • `*` matches zero or more occurrences, `+` matches one or more, and `?` matches zero or one. section 5.1.2
  • Because a token tree may itself be a sequence, a matcher can pass the complete captured sequence through without inserting separators. section 5.1.2
macro_rules! special_println {
    ($($arg:tt)*) => {           #1
        println!($($arg)*)       #2
    };
}

Inspect the compact `special_println!` definition to see token-tree capture and forwarding together. section 5.1.2

How Idiomatic Rust covers it

How deep it goes
Defines recursive token trees, repetition quantifiers, and transcription, but does not develop broader macro parsing internals.
Examples
Uses special_println!, println!, format!, format_args!, and var_print! to demonstrate capture, forwarding, and repeated identifiers.
What you can do after
Build macro_rules! matchers that capture repeated token trees and replay them in an expansion.
How it connects
Connects token trees to macro_rules!, macro expansion, repetition syntax, and regular-expression quantifiers.

Also explained in

Compare how 3 books explain token tree →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Idiomatic Rust. The section links point to the parts of the book it draws on.