Jan

As explained in Domain-Specific Small Language Models

A desktop application for chatting with local language models and exposing a local API server compatible with the OpenAI API specification.

Jan is an open-source desktop application for chatting with local language models on macOS, Linux, and Windows. It can also connect to commercial API providers, while local chats work offline after model download. section 12.5 section 12.0

Jan’s practical boundaries and capabilities include:

• Uses a simple chat interface with message history, formatted responses, syntax-highlighted code, copy-and-paste, and optional retrieval uploads. section 12.5
• Supports GGUF models from Jan Hub, Hugging Face Hub, or local files; its hardware warnings identify models likely to run slowly or exceed available memory. section 12.5
• Runs through Cortex, a C++ local AI engine using llama.cpp by default or ONNX Runtime, with GPU support and multiple quantization modes. section 12.6
• Includes a local API server, while Cortex separately provides commands for starting the server, pulling models, running chats, and stopping services. section 12.6

Running a local model with Cortex

Start Cortex with `cortex start`, pull a model with `cortex pull tinyllama:1b`, and begin a session using `cortex run tinyllama`. The default server port is 39281. section 12.6

Figure 12.9 The chat view of the Jan desktop application

Use the chat-view figure to inspect Jan’s conversation layout, prompt area, message history, formatting, and document-upload placement. section 12.5

How Domain-Specific Small Language Models covers it

How deep it goes

Explains desktop workflows, model import, hardware warnings, and Cortex operations, but stops short of client-library implementation.

Examples

Uses Jan’s chat interface, TinyLlama conversations, GGUF imports, Cortex CLI commands, and HTTP model-management requests.

What you can do after

Operate local models through Jan’s desktop interface or Cortex commands, including downloading models and starting chat sessions.

How it connects

Connects local language models with GGUF, quantization, llama.cpp, ONNX Runtime, GPU inference, and OpenAI-compatible APIs.

liveBookmanning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Related concepts

• Cortexuses · A local AI engine for running language models on a user's machine with support for multiple backends and quantized models.
• small language modeluses · A small language model is a comparatively compact language model with limited parameters or capacity and typically lower computational and storage requirements than larger models.
• local aiuses · Local AI is artificial-intelligence computation performed by a model on a user's device or other locally controlled infrastructure rather than through a remote service.
• language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• GGUFuses · GGUF is a binary file format for storing machine-learning model tensors and associated metadata, designed to support efficient model loading and inference.
• OpenAI APIimplements · The OpenAI API is a programming interface that allows software applications to send requests to OpenAI models and services and receive responses.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.