A tokenizer converts text into discrete tokens that language models process. Token counts affect context quality, response coherence, cost, and performance, making tokenization an operational concern for enterprise applications. section 2.8.3
The tiktoken library implements byte-pair encoding (BPE), a reversible, lossless method that produces compact token sequences and generalizes recurring language patterns. Use the encoding associated with the target model; encoding_for_model() can select it when the encoding name is uncertain. section 2.8.3
Listing 2.1 Counting tokens for GPT
import tiktoken as tk
def count_tokens(string: str, encoding_name: str) -> int:
# Get the encoding
encoding = tk.get_encoding(encoding_name) #1
# Encode the string
encoded_string = encoding.encode(string)
# Count the number of tokens
num_tokens = len(encoded_string)
return num_tokens
# Define the input string
prompt = “I have a white dog named Champ”
# Display the number of tokens in the String
print(“Number of tokens:” , count_tokens(prompt, “cl100k_base”))
Count tokens by selecting an encoding, encoding the input string, and measuring the resulting sequence. The example uses cl100k_base with a short prompt. section 2.8.3
