A coding assistant may find five useful examples, then discover that its model cannot fit them alongside the user’s request. The context window is the model’s token limit for the context it can process in a request, including generated context. DeciCoder’s 2,048-token window makes long code examples a practical constraint for retrieval-augmented generation (RAG). section 10.1
The term also appears in an earlier language-modeling sense: a continuous bag-of-words model slides a short window across an utterance and predicts a word from its neighbors. That local training window is distinct from the token budget that constrains an LLM request. section 2.2.4 section 10.1
Fit retrieved examples before sending the prompt
In a RAG coding assistant, retrieve candidate examples, then count their tokens with the model’s tokenizer. Add examples only while they fit alongside the instruction and user prompt. A simple counter tokenizes text and reads the length of input_ids; retrieval can return five candidates without requiring all five to enter the final prompt. section 10.2.3
Treat a larger window as capacity, not a reason to send everything:
- Few-shot code examples can consume thousands of tokens; another example competes with space for the user’s request. One-shot prompting can reduce that pressure and cost. section 7.1.1
- Longer windows have grown dramatically, but many smaller deployable models remain constrained, and per-token pricing still makes large requests costly. section 12.2.2
