tiktoken is an OpenAI tokenizer library for converting text into the token sequences used by language models. Token counts help estimate cost, context requirements, and output quality: overly short inputs may lack context, while overly long inputs may lose coherence. section 2.8.3
Use tiktoken to:
- Count tokens before sending prompts to an OpenAI model. section 2.8.3
- Select an encoding directly with get_encoding() or derive one from a model name with encoding_for_model(). section 2.8.3
- Encode text into token IDs and decode token IDs back into the original text. section 2.8.3
Count tokens in a prompt
A Python helper can obtain the cl100k_base encoding, encode “I have a white dog named Champ,” and return len(encoded_string). The demonstrated result is seven tokens. section 2.8.3
Listing 2.1 Counting tokens for GPT
import tiktoken as tk
def count_tokens(string: str, encoding_name: str) -> int:
# Get the encoding
encoding = tk.get_encoding(encoding_name) #1
# Encode the string
encoded_string = encoding.encode(string)
# Count the number of tokens
num_tokens = len(encoded_string)
return num_tokens
# Define the input string
prompt = “I have a white dog named Champ”
# Display the number of tokens in the String
print(“Number of tokens:” , count_tokens(prompt, “cl100k_base”))
Browse the complete Python listing for obtaining an encoding, encoding a string, and counting its token IDs. section 2.8.3
