Chapter 12: Running on your laptop
Introduces the chapter’s focus on running personal SLM-based assistants directly on a laptop, including offline use.
After this chapter you will be able to
- Privacy risks of online LLMs. Explain why using commercial LLM chat interfaces or APIs can raise privacy concerns, even when users can opt out of data collection or model tuning.
- Ollama local model workflow. Describe how Ollama makes models available on a local computer and what happens when a user runs one for the first time versus later runs.
- Local model access interfaces. Describe how to inspect or remove locally installed Ollama models and how a client application can access Ollama through its REST API.
- Custom model import workflow. Trace the steps for making a custom model available in Ollama: build llama.cpp, obtain the model files, convert them to GGUF, create a Modelfile, and use Ollama’s CLI to create and chat with the model.
- Local inference privacy. Explain why local Ollama chats appear to preserve privacy both offline and while the laptop is online, while qualifying that conclusion as specific to the release examined.
- Local history and log exposure. Distinguish the contents of Ollama’s history and server log files and explain why both warrant protection from unauthorized local access.
- Local chat and documents. Describe how to chat with a locally downloaded model in LM Studio and what happens when a document is attached.
- Local API server workflow. Explain what the lms CLI and developer view provide for applications that call a local model.
- Speculative decoding mechanism. Trace how LM Studio’s draft and target models produce an output during speculative decoding.
- SDK access and resource management. Explain how Python clients use the LM Studio SDK to interact with models served by a running LM Studio server, and distinguish its convenience API from its scoped resource API.
- Chat responses and context. Describe how the SDK obtains a chat response, streams it, and supplies a multiturn conversation as model input.
- Agentic tool execution rounds. Trace how act enables multi-round tool calling, and explain the practical cautions the section gives for tool and model selection.
- Local chat in Jan. Describe how Jan’s desktop app supports chatting with local models, including when an internet connection is needed and what the chat view provides.
- Model selection and import. Explain where Jan gets local models, its format requirement, and how it helps users select or import a model.
- Local API server modes. Describe how Jan exposes a local model over HTTP and explain the constraint between API serving and chatting in the desktop UI.
- Local Cortex model workflow. Trace the steps from starting a Cortex server to chatting with a model, including why a model must first be pulled.
What it builds on
- Risks and challenges with generalist LLMs (chapter 1) · Draws on the earlier discussion of risks and challenges with generalist LLMs.
- 4-bit quantization with ggml (chapter 6) · Earlier discussion of quantized models provides context for their reduced resource requirements.
- Retrieval-augmented generation (chapter 2) · The earlier introduction to retrieval-augmented generation.
- Text completion (chapter 4) · The notion of text completion.
Section by section
Introduction
Introduces the chapter’s focus on running personal SLM-based assistants directly on a laptop, including offline use.
12.1 Why use a personal local assistant
Explains why privacy and safety concerns motivate using a personal assistant that runs locally rather than relying on commercial online LLM services.
- Privacy risks of online LLMs: Explain why using commercial LLM chat interfaces or APIs can raise privacy concerns, even when users can opt out of data collection or model tuning.
Why it matters: It explains why an LLM’s productivity benefits do not remove the need to consider privacy and safety. - Local assistant alternatives: Describe the local alternative presented for individuals seeking private assistants and companies seeking secure employee tools.
Why it matters: It frames laptop-based models and tools as an alternative to sending every task to an online service.
Key terms: commercial LLMs, chat interface, API, opt out, privacy and safety concerns, open source small language models, personal local assistant, user’s laptop, RAG applications
12.2 Running an LLM locally with Ollama
This section shows how to run quantized models locally with Ollama, access them through its CLI or API, and chat through a LocalLLM web frontend.
- Ollama local model workflow: Describe how Ollama makes models available on a local computer and what happens when a user runs one for the first time versus later runs.
Why it matters: This is the basic workflow for getting from a model name to a local chat. - Quantized laptop resource needs: Explain why many Ollama models can run on a laptop without hardware acceleration and how RAM needs vary with model size.
Why it matters: Quantization helps make local inference practical, but the laptop still needs enough memory for the chosen model. - Local model access interfaces: Describe how to inspect or remove locally installed Ollama models and how a client application can access Ollama through its REST API.
Why it matters: The CLI manages local models, while the API lets other applications use the locally served models. - LocalLLM frontend and limits: Describe what LocalLLM adds to an Ollama installation, how it is started, and why the section restricts it to local use.
Why it matters: A visual chat interface can make local models easier to use, but it does not make the setup production-ready.
Key terms: Ollama, Modelfile, run, CLI chat, quantized, CPU, RAM, list, rm, REST API
12.2.1 Importing a custom model into Ollama
This section shows how to convert a custom model to GGUF, configure it with a Modelfile, and import it into a running Ollama server for chat.
- Custom model import workflow: Trace the steps for making a custom model available in Ollama: build llama.cpp, obtain the model files, convert them to GGUF, create a Modelfile, and use Ollama’s CLI to create and chat with the model.
Why it matters: This workflow lets you chat with a custom model through Ollama rather than relying only on models already in its catalog. - Modelfile deployment settings: Describe what the Modelfile specifies for a converted custom model and where the section says to put it.
Why it matters: The Modelfile connects the converted model file to the settings Ollama uses when serving it.
Key terms: llama.cpp, GGUF, convert_hf_to_gguf.py, Modelfile, ollama CLI, FROM, TEMPLATE, PARAMETER, context length, stop words
12.2.2 User privacy in Ollama
Explains what Ollama’s local operation does and does not imply for user privacy, including update traffic and unencrypted local files.
- Local inference privacy: Explain why local Ollama chats appear to preserve privacy both offline and while the laptop is online, while qualifying that conclusion as specific to the release examined.
Why it matters: Local operation is a strong privacy advantage, but it should not be treated as a permanent guarantee. - Automatic update-check traffic: Describe the outbound information sent by Ollama’s regular update check and the source-code change the section gives for stopping it.
Why it matters: An online local server can make an update request even when model inference stays local. - Local history and log exposure: Distinguish the contents of Ollama’s history and server log files and explain why both warrant protection from unauthorized local access.
Why it matters: Keeping chats local does not protect sensitive prompts if someone or something gains access to the laptop’s files.
Key terms: local-model inference, packet sniffing, Ollama, update check, IsNewReleaseAvailable, updater.go, lifecycle.go, history file, server log file, prompt text
12.3 Running an LLM locally with LM Studio
This section shows how LM Studio runs local models for chat and API use, and explains how its speculative decoding feature can speed up generation.
- Local chat and documents: Describe how to chat with a locally downloaded model in LM Studio and what happens when a document is attached.
Why it matters: It distinguishes using a local model from needing a network service, and explains how document length changes the chat workflow. - Local API server workflow: Explain what the lms CLI and developer view provide for applications that call a local model.
Why it matters: The local API allows client applications to use the model, while the developer view helps inspect their calls. - Speculative decoding mechanism: Trace how LM Studio’s draft and target models produce an output during speculative decoding.
Why it matters: This explains why a faster model can help generate text without deciding the final accepted tokens. - Speculative decoding tradeoffs: Explain when speculative decoding may speed up generation and why it may instead slow it down.
Why it matters: It prevents treating speculative decoding as an automatic speedup for every model pair or response.
Key terms: GGUF, Apple MLX, chat view, context window, retrieval-augmented generation (RAG), lms CLI, API server, REST API, developer view, server log stream
12.4 The LM Studio Python SDK
This section shows how to use the LM Studio Python SDK for local model interactions, including chat, text completion, embeddings, and multi-round tool calling.
- SDK access and resource management: Explain how Python clients use the LM Studio SDK to interact with models served by a running LM Studio server, and distinguish its convenience API from its scoped resource API.
Why it matters: Choosing the appropriate API helps you use local models without leaving application resources open. - Chat responses and context: Describe how the SDK obtains a chat response, streams it, and supplies a multiturn conversation as model input.
Why it matters: These calls let a Python application get a single reply or conduct a continuing chat. - Completion and embedding calls: Distinguish the SDK’s text-completion call from its embedding-generation call, including the requirement for an embedding model.
Why it matters: The distinction helps you select the SDK call and model for generated text versus a text vector. - Agentic tool execution rounds: Trace how act enables multi-round tool calling, and explain the practical cautions the section gives for tool and model selection.
Why it matters: Tool calling extends a local model beyond text responses, while defensive tools and model evaluation help make the workflow dependable.
Key terms: LM Studio Python SDK, convenience API, scoped resource API, respond, PredictionResult, content, respond_stream, Chat helper class, system prompt, complete
12.5 Running an LLM locally with Jan
This section explains how to chat with local models in Jan, import compatible models, and serve them to local applications through an API.
- Local chat in Jan: Describe how Jan’s desktop app supports chatting with local models, including when an internet connection is needed and what the chat view provides.
Why it matters: You can distinguish using a model locally from downloading it or connecting to a hosted model. - Model selection and import: Explain where Jan gets local models, its format requirement, and how it helps users select or import a model.
Why it matters: The model’s format and hardware demands determine whether it is practical to run in Jan on a particular laptop. - Document chat limitations: State the section’s current restrictions on retrieval and chatting with uploaded documents in Jan.
Why it matters: An upload area in the chat view does not mean every model or file type supports document chat. - Local API server modes: Describe how Jan exposes a local model over HTTP and explain the constraint between API serving and chatting in the desktop UI.
Why it matters: A local client application can call Jan’s model, but the reader must start the server and switch away from UI chat.
Key terms: Jan, desktop app, local models, chat view, Jan Hub, Hugging Face Hub, GGUF, Import Model, My Models, multimodal models
12.6 The Cortex local LLM engine
This section describes Cortex as Jan’s local LLM engine, shows the basic steps for running a model with it, and explains the privacy appeal of local assistant tools.
- Cortex engine capabilities: Describe how Cortex powers Jan and what choices it offers for running local models, including inference backends, GPU use, and quantized model versions.
Why it matters: These options explain what Cortex can run on a laptop and how it relates to Jan. - Local Cortex model workflow: Trace the steps from starting a Cortex server to chatting with a model, including why a model must first be pulled.
Why it matters: Starting the server alone is not enough: a model must be downloaded before you can chat. - Local assistant privacy benefit: Explain the privacy benefit the section attributes to local assistant tools and the basis it gives for choosing among them.
Why it matters: The reason to consider these laptop tools is not just convenience, but access to useful data while keeping it local or on a private network.
Key terms: Cortex, Jan, llama.cpp, ONNX Runtime, quantization modes, cortex CLI, server, Jan catalog, pull, chat session
12.7 Summary
Where this is used later
- Building an agent (chapter 13) · Running an LLM locally, an option this section also uses for agent model serving.
- SLMs with embedded test-time compute (chapter 15) · Earlier coverage of running an LLM locally with Ollama provides context for the Ollama catalog reference.
- Combining a domain-specific SLM with RAG (chapter 13) · The document example provides a local-use context for combining a model with RAG.
- Using a vector database (chapter 13) · Embedding generation supplies vectors relevant to using a vector database.
- RAG + Agentic AI (chapter 14) · The agentic workflow concept provides context for combining RAG with agentic AI.
Concepts explained in this chapter
- OllamaA local model runtime and serving tool that downloads, runs, and exposes quantized language models through an API and chat interface.
- JanA desktop application for chatting with local language models and exposing a local API server compatible with the OpenAI API specification.
- ModelfileAn Ollama configuration file that specifies how to package and serve a converted model, including template and generation parameters.
- LM StudioA local model-serving application and desktop environment that lets users run, chat with, and inspect models offline on their own machine.
- draft modelThe smaller, faster model in speculative decoding that proposes candidate tokens for verification by a target model.
- speculative decodingAn inference-time optimization that predicts multiple future tokens in parallel using draft proposals instead of generating strictly one token at a time.
- LocalLLMA Flask-based local web frontend for Ollama that provides streaming responses and model switching.
- lms CLILM Studio’s command-line tool used to manage local models and its local API server.
- LM Studio Python SDKA Python client library for interacting programmatically with models served by LM Studio, including chat, completion, embeddings, and tool-calling workflows.
- agent workflowA multi-step model-and-tool interaction pattern where the model chooses tools, receives results, and iterates toward a final answer.
- CortexA local AI engine for running language models on a user's machine with support for multiple backends and quantized models.
- GGUFA model packaged in the GGUF format for local inference with llama.cpp-style runtimes.
- retrieval-augmented generationAn application system that combines retrieval from external sources with text generation to ground outputs in outside knowledge.
- ONNX RuntimeA runtime library that efficiently executes ONNX models across hardware targets such as CPUs, GPUs, and edge devices.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.