OptiLLM is an open-source inference proxy that adds test-time reasoning and answer refinement to OpenAI API–compatible model services. It targets accuracy and performance improvements for coding, logical, and mathematical queries without requiring a new client interface. section 15.2 section 15.5
OptiLLM operates as a refinement layer:
- Clients send requests through the OpenAI SDK, optionally selecting a reasoning algorithm. section 15.2
- The proxy forwards requests to an OpenAI API–compatible service, including OpenAI, Azure OpenAI, Cerebras, or LiteLLM. section 15.2
- The proxy makes multiple inference calls, applies the selected refinement method, and returns the final response after the cycle completes. section 15.2
- Available approaches include Chain-of-Thought with Reflection, PlanSearch, ReRead, Self-Consistency, the Z3 Solver, and R*. section 15.2
The proxy also supports local inference: it can load Hugging Face models and LoRA weights directly or connect to servers such as llama.cpp and Ollama. This extends test-time compute to open-source large and small language models, although external local servers currently lack multiple-response sampling. section 15.2 section 15.5
Refining a small model's mathematical answer
A Qwen/Qwen2.5-0.5B-Instruct model can receive a word problem through a messages list, then use OptiLLM's ThinkDeeper or CoT decoding functions. CoT decoding can aggregate reasoning paths and return a confidence value alongside the answer. section 15.2
Figure 15.1 The flow of requests and responses through the OptiLLM proxy
Trace the client, OptiLLM proxy, downstream inference service, iterative refinement, and final response. section 15.2
