A token is a unit produced when text is tokenized for an LLM. Tokenization turns input into tokens and numerical IDs; the model processes numbers and translates its output back into human-readable form. Because services may bill for input and generated tokens, token counts matter for cost management. section 5.1.1
Figure 5.2 The end-to-end process of how human input is transformed, processed by an LLM and converted back into human-readable output
Trace how human input is tokenized, processed by an LLM, and converted into readable output. section 5.1.1
Listing 5.1 Function that counts prompt tokens
import { encoding_for_model } from '@dqbd/tiktoken'; #1
function countTokens(text, model = "gpt-3.5-turbo") {
const enc = encoding_for_model(model);
const tokens = enc.encode(text); #2
enc.free();
return tokens.length;
}
const prompt =
"Translate the following English text to French: 'Hello, how are you?'";
const systemMessage
"You are a helpful assistant that translates English to French.";
const promptTokens = countTokens(prompt); #3
const systemTokens = countTokens(systemMessage); #4
const totalTokens = promptTokens + systemTokens;
console.log(`Prompt tokens: ${promptTokens}`);
console.log(`System message tokens: ${systemTokens}`);
console.log(`Total tokens: ${totalTokens}`);
See a JavaScript function that counts tokens in a prompt and system message using @dqbd/tiktoken. section 5.1.1
