11 Running on Your Laptop

 

This chapter covers:

  • The reasons for using a personal SLM-based local assistant.
  • How to use SLMs locally, also offline, with the Ollama serve engine.
  • How to use SLMs locally, also offline, with the LM Studio desktop application.
  • How to use SLMs locally, also offline, with the Jan local assistant application.

Chapter 10 presented diverse frameworks to serve private and domain specific SLMs behind endpoints, to be deployed at scale and be used by client applications, and also alternatives to deploy SLMs on edge devices, such as Android phones (typically having limited computational resources, if compared to a backend deployment in a privately or cloud hosted cluster). This chapter explores alternatives to run (completely offline too) personal SLM-based assistants or productivity tools directly on a laptop.

11.1 Why a personal local assistant

11.2 Running an LLM locally with Ollama

11.2.1 Importing a custom model into Ollama

11.2.2 User privacy in Ollama

11.3 Running an LLM locally with LM Studio

11.3.1 The LM Studio Python SDK

11.4 Running an LLM locally with Jan

11.4.1 The Cortex local LLM engine

11.5 Summary