How deeply does Domain-Specific Small Language Models cover Cortex?

Explains backend, GPU, quantization, server, and CLI workflows, while deferring released Python and TypeScript clients.

Cortex

As explained in Domain-Specific Small Language Models

A local AI engine for running language models on a user's machine with support for multiple backends and quantized models.

Cortex is a C++ local AI engine for running language models on a user's machine. It supports multiple inference backends, GPU acceleration, and quantized model variants while operating offline. section 12.6 section 12.7

Cortex provides practical controls for local model deployment:

Switch between llama.cpp and ONNX Runtime backends. section 12.6
Use NVIDIA, AMD, or Intel GPUs when available, with GPU use attempted by default. section 12.6
Pull quantized models from the Jan catalog or use bundled models; multiple quantization modes may exist for one model. section 12.6
Expose local management and inference through the cortex CLI and an HTTP API. section 12.6

Run a local model

Start the server, pull TinyLlama, and launch a chat session with `cortex start`, `cortex pull tinyllama:1b`, and `cortex run tinyllama`. The server defaults to port 39281. section 12.6

cortex start

Inspect the command that starts the local Cortex server. section 12.6

How Domain-Specific Small Language Models covers it

Cover of Domain-Specific Small Language Models

liveBook manning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Related concepts

Janused by · A desktop application for chatting with local language models and exposing a local API server compatible with the OpenAI API specification.
llama.cppuses · llama.cpp is an open-source C/C++ software project and runtime for running large language models efficiently on local hardware.
ONNX Runtimeuses · ONNX Runtime is an open-source, cross-platform inference engine for executing and optimizing machine-learning models represented in the ONNX format.
GPUuses · A graphics processing unit (GPU) is a processor designed for highly parallel computation, especially graphics and data-parallel workloads.
quantizationuses · Quantization maps numerical values to a finite set of discrete representations, often using reduced precision to lower model storage or computation requirements.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.