liveBook · Manning

Token

A token is a value used to determine a data partition's placement and ownership in a distributed database cluster.

How 2 Manning books explain it

Timeless Algorithms, The Foundational Ideas

A token is a discrete unit of text represented in a model's vocabulary, such as a word, word fragment, or punctuation mark.

How it is used
Uses tokens as vocabulary units for contextual representation and next-token prediction.
How deep it goes
Explains identifiers, embeddings, and contextual attention with an illustrative calculation, while simplifying the larger representations and layered operations of real transformers.
Examples
Works through a trophy-and-box pronoun example using two-dimensional vectors and illustrative attention weights.
What you can do after
Evaluate how contextual relationships among tokens can influence a next-token prediction without explicitly labeling a pronoun’s referent.

Read how Timeless Algorithms, The Foundational Ideas explains token →

ScyllaDB in Action

A token is a value used to determine a data partition's placement and ownership in a distributed database cluster.

How it is used
Uses tokens to determine partition placement, node ownership, request routing, replication, and cluster scaling.
How deep it goes
Explains hashing, ring positions, vnode ownership, node joining, request routing, and TOKEN inspection without formal mathematics.
Examples
Uses hash-ring diagrams, vnode ownership illustrations, cluster status output, and a cqlsh TOKEN query.
What you can do after
Apply the TOKEN function to inspect a partition key's ring position and trace which node owns its slice.

Read how ScyllaDB in Action explains token →

Also covered in

Related concepts

Questions about token

What is token?

A token is a value used to determine a data partition's placement and ownership in a distributed database cluster.

Which Manning books explain token?

Timeless Algorithms, The Foundational Ideas and ScyllaDB in Action.

What is token part of?

Token is part of sequence. A sequence is an ordered collection of elements whose positions determine the order in which they are arranged or traversed.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.