Test-time compute (TTC) allocates additional computation during inference so a model can explore alternatives, evaluate candidates, and spend more time reasoning without changing its base model. It targets difficult tasks such as mathematics, logic, and coding, trading higher accuracy for greater latency and compute cost. section 15.1
TTC can improve an otherwise static inference pass through:
- Search over candidate outputs with beam search or tree-based exploration, including Monte Carlo tree search, within a compute budget. section 15.1
- Candidate scoring, self-consistency, reflection, or executing generated code and checking it for errors before returning an answer. section 15.1 section 15.5
- Dynamic allocation of reasoning time, which can make inference more reliable but increases hardware use, response time, and operating cost. section 15.1
OptiLLM makes TTC available to OpenAI API–compatible services through an inference proxy. It forwards requests to a model, performs repeated calls and refinement with a selected algorithm, and returns the result after the refinement cycle. It also supports local inference with Hugging Face models, LoRA weights, llama.cpp, or Ollama. section 15.2
Figure 15.1 The flow of requests and responses through the OptiLLM proxy
Use the request-flow diagram to trace how a client, OptiLLM proxy, and OpenAI API–compatible service participate in TTC refinement. section 15.2
Reasoning over a math problem
A small Qwen2.5-0.5B-Instruct model can receive a student-enrollment word problem and produce an answer through ThinkDeeper or chain-of-thought decoding. The OptiLLM benchmark evaluates such decoding approaches on GSM8K samples. section 15.2
