liveBook · Manning

Std.mem.tokenizeScalar

As explained in Systems Programming with Zig

std.mem.tokenizeScalar is a Zig standard-library generic function that returns an iterator splitting text on a scalar delimiter.

To count words in a newline-separated dictionary, use std.mem.tokenizeScalar to create an iterator over the text. Call next() until it returns no more tokens, counting each word along the way. section 4.1.3

Listing 4.1 Counting the dictionary words
fn countWordlist(text: []const u8) usize {
    var count: usize = 0;
    var word_iter = std.mem.tokenizeScalar(u8, text, '\n'); #1
    while (word_iter.next()) |_| { #2
        count += 1;
    }

    return count;
}

This function splits a byte slice on '\n' and counts the resulting words. section 4.1.3

The function is generic: its comptime type parameter T determines the buffer element and delimiter types. It returns a TokenIterator(T, .scalar). The function body must be valid for the supplied type. section 4.1.3

Listing 4.2 Comptime type parameters for fun and profit
pub fn tokenizeScalar(
    comptime T: type, #1
    buffer: []const T, #2
    delimiter: T, #3
) TokenIterator(T, .scalar) #4

The declaration ties the buffer and scalar delimiter to the same element type. section 4.1.3

The same iterator can parse a compile-time wordlist into an array. The parser checks that each word has four characters and reports a compile error otherwise. section 4.1.3

Listing 4.4 Parsing the wordlist into an array
fn parseWordlist(
    comptime wordlist: []const u8, #1
) [countWordlist(wordlist)][4]u8 { #2
    var list: [countWordlist(wordlist)][4]u8 = undefined;
    var i: usize = 0;

    var word_iter = std.mem.tokenizeScalar(u8, wordlist, '\n');
    while (word_iter.next()) |word| : (i += 1) { #3
        if (word.len != 4) {
            @compileError(std.fmt.comptimePrint( #4
                "the word '{s}' is not four characters long.",
                .{word},
            ));
        }
        list[i] = word[0..4].*; #5
    }

    return list;
}

The parser counts entries for its array size, then tokenizes again to populate it. section 4.1.3

How Systems Programming with Zig covers it

How deep it goes
Explains the generic signature and iterator use in wordlist routines without examining iterator internals.
Examples
Illustrates dictionary counting and a parser that rejects words not four characters long.
What you can do after
Apply it via Zig’s std.mem.tokenizeScalar to iterate over delimiter-separated text.
How it connects
Connects iterators with compile-time evaluation, generic functions, and duck typing.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Systems Programming with Zig. The section links point to the parts of the book it draws on.