chapter twelve

12 Running on your laptop

 

This chapter covers:

  • Reasons for using a personal, local SLM-based assistant
  • Using Ollama’s server to run SLMs locally and offline
  • Using the LM Studio desktop app to run SLMs locally and offline
  • Using the Jan assistant app to run SLMs locally and offline

In chapter 11, we looked at frameworks for serving private, domain-specific SLMs behind endpoints at scale for client applications, as well as options for running SLMs on edge devices, such as resource-constrained Android phones. This chapter explores ways to run personal SLM-based assistants or productivity tools directly on a laptop, including completely offline.

12.1 Why use a personal local assistant

12.2 Running an LLM locally with Ollama

12.2.1 Importing a custom model into Ollama

12.2.2 User privacy in Ollama

12.3 Running an LLM locally with LM Studio

12.4 The LM Studio Python SDK

12.5 Running an LLM locally with Jan

12.6 The Cortex local LLM engine

Summary