An LLM wrapper is software that sits between the user and the model. It manages prompts, repeats generation, and can add missing capabilities like chat formatting, tool calls, or current information. section 1.1 section 1.34
A bare LLM only predicts the next token from its input. The wrapper turns that into a fuller product by feeding the model’s output back into the next prompt until a stop token appears. section 1.2 section 1.3
Common tricks used by wrappers here include:
- Generating full sentences one token at a time by reattaching each output to the prompt. section 1.2
- Inserting special delimiters such as end-of-prompt and end-of-text markers to manage chat history. section 1.4
- Calling external functions, then inserting the returned result back into the prompt. section 1.6
- Retrieving documents first and augmenting the prompt with their content. section 1.7
ChatGPT-style chat
A chat wrapper can send the model the whole conversation each turn, mark boundaries with special code words, and even route tool calls behind the scenes so the model can answer with up-to-date facts. section 1.4 section 1.6
Figure 1.1 To generate full sentences, the LLM wrapper used the LLM to generate one word, then attached that word to the initial prompt, then used the LLM again to generate one more word, and so on.
Shows the one-token-at-a-time loop that wrappers use to turn next-token guesses into complete text. section 1.2
