liveBook · Manning

Local-model inference

As explained in Vibe Engineering

Developer-local inference is the execution of an AI model on a developer's own workstation rather than on a remote serving system.

Local-model inference runs an AI model on a developer’s workstation rather than a remote serving system. It removes per-token charges for local requests, but does not eliminate hardware costs or the need to check generated code. section 9.2.8 section 9.6

The strongest reasons to run locally are capability, not automatically savings: sensitive code can stay on the machine, assistance can work offline, and frequent completions avoid cloud quotas and network round trips. Keep a fully local workflow when code cannot leave the machine; a hybrid setup sends selected work to the cloud. section 9.3.1 section 9.3.3

Choose the workload and model against the machine’s limits:

  • Use small local models for tab completion, boilerplate, imports, and simple edits. Complex autonomous and multi-file work can exceed their reasoning capacity. section 9.3.4 section 9.6
  • Check model size and quantization against available unified memory or GPU VRAM, leaving room for the IDE and development tools. CPU-only inference can be usable for small models but is slower. section 9.3.2
  • Account for context memory as well as model weights. Mixture-of-experts models activate only some parameters per token, but all their parameters must fit in memory; the KV cache adds to that requirement. section 9.3.4

A practical setup separates the runtime from the client: Ollama serves the model, an editor extension such as Continue supplies inline completions, and OpenCode provides a terminal agent for tasks involving files and tools. A hybrid configuration can reserve cloud models for harder work when policy permits. section 9.3.3

Listing 9.4 Ollama setup - install to inference in under five minutes
# 1. Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh     # Linux/Mac
# 2. Pull a coding model
ollama pull qwen2.5-coder:14b-instruct-q4_K_M    # 8.5GB download
# 3. Verify it works
ollama run qwen2.5-coder:14b-instruct-q4_K_M \
"Write a TypeScript function that debounces an input handler"

The setup listing installs Ollama, downloads a quantized coding model, and verifies inference with a TypeScript prompt. Check that the model fits the workstation before downloading it. section 9.3.3 section 9.3.2

A subscription may still be cheaper

Replacing an editor subscription with local completions may save the seat fee, yet complex agent work can then incur separate cloud API charges. For a twelve-person team with controlled overages, the local switch was a lateral move rather than a saving. Compare the full usage pattern before removing a subscription. section 9.3.6

Evaluate each workflow on latency, cost, and quality rather than token price alone. Track retries and first-attempt success as well as spending; faster or cheaper inference can still increase total cost if quality declines. section 9.6

How Vibe Engineering covers it

How deep it goes
Explains hardware and model selection, runtime setup, task limits, and subscription economics, with practical memory and throughput examples.
Examples
Uses an Ollama coding-model setup, a TypeScript debounce prompt, and a twelve-person team’s subscription-cost comparison.
What you can do after
Operate local coding assistance with Ollama and a client, choosing tasks and models against hardware, privacy, and cost constraints.
How it connects
Connects inference to quantization, mixture-of-experts models, KV cache, model selection, and AI coding agents.

Also explained in

Compare how 2 books explain local-model inference →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.