liveBook · Manning

Local-model inference

As explained in Building Specialized AI Systems

Local inference is the execution of a trained model on hardware controlled by the user or deploying organization rather than on a remote service.

Local-model inference runs a trained model on hardware controlled by the user or deploying organization. Reload a base model with an adapter, optionally merge and export it, then serve requests without sending sensitive data to a remote service. This preserves confidentiality, avoids network latency, and keeps control over model outputs. section 2.2

A practical local deployment workflow is:

  • Save adapter weights and reload the base model with the selected adapter. section 2.2
  • Create an Ollama Modelfile, register the model locally, and run it offline; use llama.cpp directly for a more transparent toolchain. section 2.2
  • Use LM Studio when graphical experimentation or CPU/GPU execution is preferred. section 2.2

Offline financial assistant

Declare a finance adapter with FROM llama2 and ADAPTER ./finance_adapter, then run ollama create finance-model -f Modelfile followed by ollama run finance-model. The resulting assistant performs financial analysis locally without external calls. section 2.2

How Building Specialized AI Systems covers it

How deep it goes
Explains adapter loading, optional merging, Modelfile creation, registration, and offline execution with named tools.
Examples
Uses finance and casual adapters, financial-analysis prompts, Ollama Modelfiles, and command-line model execution.
What it assumes
Assumes a trained base model and saved adapter weights, then demonstrates how to attach them for inference.
What you can do after
Operate/configure a fine-tuned model locally with Ollama, llama.cpp, or LM Studio.
How it connects
Connects local inference with parameter-efficient fine-tuning, LoRA adapters, model serving, privacy, and data security.

Also explained in

Compare how 2 books explain local-model inference →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.