tiktoken is a Python package for counting tokens in text. It selects a model encoding and measures the length of encoded strings, helping identify sections that may exceed an LLM context limit. section 3.2
Count tokens in a section
Define num_tokens_from_string(string, model="gpt-4"), obtain the model encoding with tiktoken.encoding_for_model(model), encode the string, and return len(encoding.encode(string)). Applying it to sections produces counts such as 154, 254, 4,186, and 2,703.
