llama.cpp is an open-source C/C++ library for efficient large language model inference. Ollama wraps it to run models locally, including an instruction-fine-tuned Llama 3 model used for automated response evaluation. section 7.8
Its role in the evaluation workflow is operational:
- Runs a local LLM through Ollama rather than a remote web API. section 7.8
- Supports inference and text generation, but not model training or fine-tuning. section 7.8
- Provides the execution layer behind commands such as `ollama run llama3` and HTTP requests to Ollama’s local chat API. section 7.8
Local evaluation engine
A Python evaluator sends prompts to Ollama at `http://localhost:11434/api/chat`, receives generated responses, and asks the local Llama 3 model to score them from 0 to 100. llama.cpp supplies the efficient C/C++ inference implementation beneath this workflow. section 7.8
