liveBook · Manning

Token

As explained in Learn Rust in a Month of Lunches

A token is a lexical unit of source input, such as a number, identifier, punctuation mark, or other syntactic element.

A token is an input unit a macro can match, such as the number `6`. It is not yet a typed Rust value: the compiler checks the macro’s output afterward. section 22.2

A macro can require an exact token sequence. For example, a pattern for `6` does not match `66`; other patterns can capture input as an expression, statement, or token tree. section 22.2

macro_rules! might_print {
    (THis is strange input 하하はは哈哈 but it still works) => {
        println!("You guessed the secret message!")
    };
    () => {
        println!("You didn't guess it");
    };
}
 
fn main() {
    might_print!(THis is strange input 하하はは哈哈 but it still works);
    might_print!();
}

This macro matches an unusual input containing characters from several writing systems, showing that token input need not be ordinary numbers or ASCII text. section 22.2

How Learn Rust in a Month of Lunches covers it

How deep it goes
Explains literal matching and token-tree, expression, and statement captures, without detailing how Rust tokenizes source.
Examples
Uses the number `6`, a rejected `66`, and an unusual multilingual input to show macro matching.
What you can do after
Understand why macro inputs can fail to match before generated code is checked by the compiler.
How it connects
Connects tokens to Rust macros, `macro_rules!`, and compiler errors for unmatched input.

Also explained in

Compare how 31 books explain token →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Learn Rust in a Month of Lunches. The section links point to the parts of the book it draws on.