liveBook · Manning

Tokenization

As explained in Server-Side WebAssembly

Tokenization is the process of dividing text into discrete tokens suitable for processing by a machine-learning model.

Tokenization divides text into discrete tokens that a machine-learning model can process. In a WebAssembly guest, the Rust tokenizers library supplies this capability. section 5.4.1

Enable tokenization by disabling tokenizers' default features and selecting its unstable_wasm feature in the guest project's Cargo.toml. The same project can combine tokenizers with ndarray and a WASI-NN interface for model execution. section 5.4.1

Configure tokenizers for Wasm

Add tokenizers = { version = "0.20.1", default-features = false, features = ["unstable_wasm"] } to the smartcms_ml_guest crate, alongside ndarray and the wasi:nn target dependency.

How Server-Side WebAssembly covers it

How deep it goes
Configures a tokenization library for Wasm but stops short of explaining tokenization algorithms or token-processing internals.
Examples
Demonstrates tokenization through a Cargo.toml dependency configured with tokenizers' unstable_wasm feature.
What it assumes
Assumes a Rust Wasm component project and introduces the dependency configuration required to use tokenizers with Wasm.
What you can do after
Apply tokenization via the Rust tokenizers library in a Wasm guest configured for machine-learning workloads.
How it connects
Connects tokenization with Rust, WebAssembly, ndarray, WASI-NN, and ONNX-backed machine-learning execution.
Cover of Server-Side WebAssembly
liveBookmanning.com
Read Server-Side WebAssembly on liveBook Get the book at manning.com

Also explained in

Compare how 25 books explain tokenization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Server-Side WebAssembly. The section links point to the parts of the book it draws on.