liveBookmanning.com

What you will learn in Domain-Specific Small Language Models

Efficient AI for local deployment

by Guglielmo Iozzia
Foreword by Matthew R. Versaggi

Read it on liveBook Get the book at manning.com

Chapter by chapter

Concepts explained in this book

• retrieval-augmented generationAn application system that combines retrieval from external sources with text generation to ground outputs in outside knowledge.
• quantizationMethods that reduce numeric precision or representational size of model values to improve efficiency, especially for inference; here the absmax method is a specific quantization procedure.
• AI agentAn autonomous software entity that can invoke models or tools as part of a larger system.
• ONNX RuntimeA runtime library that efficiently executes ONNX models across hardware targets such as CPUs, GPUs, and edge devices.
• ONNXA serialized machine-learning model represented in the ONNX format for portable execution outside the training framework.
• CodeAgentA smolagents agent pattern that generates Python code to invoke tools and solve a task step by step.
• int4 quantizationA quantization method that represents model weights or activations with 8-bit numbers to reduce model size and improve efficiency.
• knowledge graphA structured graph representation of entities and relationships used here to enrich RAG with explicit connections among concepts.
• community summaryA high-level summary of a group of related graph entities used as an intermediate unit for search and response generation.
• computeThe practice of allocating more computation at inference time to improve a model’s output quality or reasoning behavior without changing the base model.
• Leiden algorithmA community-detection algorithm that improves graph clustering by optimizing modularity through iterative node movement and aggregation.
• smolagentsA Hugging Face library for building AI agents with code-based or tool-calling workflows.
• ReActAn agent framework that interleaves reasoning and tool use to solve tasks by iterating toward a solution.
• OptiLLMAn open source inference proxy that applies reasoning and answer-refinement techniques to improve model outputs via an OpenAI API–compatible interface.
• AutoThinkAn inference-time reasoning method that chooses a reasoning budget and steers a model toward better reasoning paths using classification and control vectors.
• fine-tuningTraining a pretrained model further on task- or domain-specific data to adapt it to a narrower use case.
• sentence-transformersA library for producing dense text vectors with transformer encoders for embedding and similarity search.
• static inferenceSingle-pass model inference without iterative refinement or adaptive postprocessing of the generated response.
• language modelA very large neural language model trained on broad text corpora to predict or generate language, used here for code generation and assistance.
• OptimumA Hugging Face extension library for optimizing transformer models for training and inference on target hardware, including ONNX workflows.
• Group Relative Policy OptimizationA reinforcement learning optimization method that updates a policy by comparing groups of sampled actions, avoiding the need for a separate value model.
• vector databaseA database designed to store vectors and support similarity search, typically using approximate nearest neighbor methods for retrieval.
• knowledge baseRetrieved outside information supplied to a model at inference time to ground its output.
• vLLMA high-throughput model serving library for running and benchmarking language models with efficient attention and API compatibility.
• agentic RAGA retrieval-augmented generation setup that uses an AI agent to plan, select, and combine retrieval actions across one or more sources before generation.
• BM25A lexical information-retrieval ranking algorithm that scores documents using term frequency, inverse document frequency, and document length.
• hybrid searchA retrieval method that combines keyword-based and vector-based search to improve retrieval quality.
• Additive Manufacturing Reasoning datasetA dataset of additive-manufacturing questions, answers, and reasoning traces used to train and evaluate the example model.
• attentionA sequence-modeling technique that weights relevant parts of an input more heavily than less relevant parts.
• FlexGenA high-throughput inference engine for running large language models under limited GPU memory by offloading and compressing model state across devices.
• SmoothQuantA post-training quantization technique that reduces memory footprint and speeds up inference by smoothing activation and weight ranges.
• vector indexA data structure that accelerates retrieval by organizing vectors for efficient similarity search.
• CrystaLLMA decoder-only Transformer language model in this section that is trained on CIF text to generate crystal structures.
• transformersThe Hugging Face library for working with pretrained transformer models, tokenizers, pipelines, and related training and inference utilities.
• MLC LLMAn open-source ML compiler and deployment engine for serving large language models across multiple platforms.
• OllamaA local model runtime and serving tool that downloads, runs, and exposes quantized language models through an API and chat interface.
• CodeGenA pretrained code-generation transformer model family used for Python code generation and inference optimization.
• kernel fusionAn optimization that combines adjacent operators into one to reduce overhead and improve inference latency.
• execution providerAn ONNX Runtime backend that assigns graph nodes to a specific hardware/runtime implementation such as CPU, CUDA, or ROCm.
• BitNetA 1-bit Transformer model architecture that aggressively quantizes weights to reduce memory and compute cost while preserving competitive language-modeling performance.
• InferenceSessionAn ONNX Runtime class and execution context for loading models, running inference, and applying runtime optimizations.
• mixed precisionUsing reduced-precision numeric formats for some computations or weights to reduce memory use, often with limited loss in performance.
• query-complexity classificationA classifier that estimates whether a query needs high- or low-complexity reasoning so the inference budget can be adjusted adaptively.
• steering vectorA control method that uses learned vectors to steer model behavior during inference.
• communityA group of graph nodes that are more densely connected to each other than to the rest of the graph.
• hierarchical clusteringA clustering approach that builds communities through repeated refinement and aggregation of grouped items.
• Louvain algorithmA community-detection algorithm that iteratively moves nodes and aggregates communities to improve a clustering objective.
• generationThe task of producing output sequences from a model given an input context or prompt, here applied to proteins rather than natural language.
• mlc_llm_compileMLC LLM’s command-line utility for compiling a model library according to a generated chat configuration and target device.
• mlc_llm_convert_weightsAn MLC LLM command that converts model weights into MLC format using specified quantization settings.
• mlc_llm_gen_configMLC LLM’s command-line utility for generating a chat configuration for a selected model.
• MLCEngineMLC LLM’s unified inference engine object used to run compiled models and provide OpenAI-compatible chat completions.
• CodeGenForCausalLMThe Hugging Face auto class specialized for loading CodeGen causal language models.
• CPUExecutionProviderThe ONNX Runtime execution provider that runs model computation on CPU-backed execution.
• ORTModelForCausalLMThe Optimum ONNX Runtime class for loading a causal language model in ONNX Runtime format.
• post-training quantizationA quantization method applied after training to reduce model numeric precision, typically to improve deployment efficiency.
• offloadingA memory-management technique that moves model tensors or state from faster, limited memory to slower or more abundant storage to fit or run a model.
• transformer architectureA neural network architecture for sequence modeling built around self-attention and parallel processing, underlying both large and small language models.
• LM StudioA local model-serving application and desktop environment that lets users run, chat with, and inspect models offline on their own machine.
• LLM.int8()A quantization approach for large language models that uses mixed-precision INT8 computation with special handling for outlier features to preserve accuracy.
• tokenizerA model-specific component that converts concatenated instruction/output text into token sequences for fine-tuning.
• ReferenceEvaluatorThe ONNX Runtime reference evaluator class used to execute ONNX graphs or individual nodes in a Python environment.
• prefillThe inference phase in which prompt tokens are processed in parallel and the KV cache is populated before generation begins.
• speculative decodingAn inference-time optimization that predicts multiple future tokens in parallel using draft proposals instead of generating strictly one token at a time.
• operatorA computation node in an ONNX graph that performs a specific operation during model execution.
• JanA desktop application for chatting with local language models and exposing a local API server compatible with the OpenAI API specification.
• spreading activationA retrieval method that propagates activation from labeled source nodes through linked nodes in an associative network.
• profilingThe practice of collecting runtime measurements for model execution to identify bottlenecks and optimization opportunities.
• similarity searchThe process of searching external sources to find information relevant to a query or prompt.
• parameter-efficient fine-tuningA family of fine-tuning methods that adapt a pretrained model using only a small subset of parameters or added modules to reduce compute and cost.
• arithmetic intensityThe number of compute operations divided by the number of bytes moved to and from memory for an algorithm or model layer.
• FastAPIA Python web framework for building APIs with automatic documentation, asynchronous request handling, and data validation support.
• GPT-2A decoder-only Transformer language model used for text generation, here serving as the architecture underlying ProtGPT2.
• draft modelThe smaller, faster model in speculative decoding that proposes candidate tokens for verification by a target model.
• GPTQA one-shot post-training weight quantization method that uses approximate second-order information to compress generative pretrained transformers with minimal accuracy loss.
• LAMBADAA narrative text dataset used here to evaluate word-prediction and model accuracy after quantization.
• ReCodeAn evaluation approach for code generation models that measures robustness under prompt perturbations.
• ModelfileAn Ollama configuration file that specifies how to package and serve a converted model, including template and generation parameters.
• working memoryConversation memory that keeps recent interactions within a session so they can be reused immediately.
• ORT for PythonThe Python interface to ONNX Runtime used here to evaluate ONNX models, nodes, and intermediate results.
• BERTA transformer-based language model that uses the encoder stack for understanding tasks such as classification.
• GGUFA model packaged in the GGUF format for local inference with llama.cpp-style runtimes.
• coding assistantA model-based assistant intended to help write or suggest program code in an editor or chat workflow.
• lms CLILM Studio’s command-line tool used to manage local models and its local API server.
• LocalLLMA Flask-based local web frontend for Ollama that provides streaming responses and model switching.
• GraphRAGA retrieval-augmented generation approach that uses knowledge graphs and community summaries to support multihop reasoning over document collections.
• DeepSpeedAn open-source deep learning optimization library that supports distributed training and inference and is used here to speed up and memory-optimize Transformer models.
• BitLinearA component that replaces traditional linear layers in BitNet and enables training and inference with 1-bit weights.
• ANARCIAn antibody-numbering and receptor-classification tool used here to validate whether generated antibody sequences are biologically plausible.
• AntibodyGPTA specialized protein generation model in this chapter for generating therapeutic antibody sequences from an antigen context.
• domain-specific large language modelA large language model specialized for a particular domain or task family.
• grouped-query attentionAn attention variant that speeds inference by sharing attention computation across groups of queries.
• StarCoder2A code-generation language model family used here as a custom model example, with the 3B variant specifically mentioned.
• compute-boundA performance regime where execution is limited by available compute units rather than memory transfer rate.
• hardware accelerationThe use of specialized hardware such as GPUs to speed up computation by offloading work from the CPU.
• inference samplingThe process of generating tokens from a Transformer during inference, including prefill and autoregressive decoding.
• ops/byte ratioThe number of floating-point operations available per byte of memory accessed on a given device.
• transfer learningThe process of taking a model trained on one task or domain and applying it to a related task or domain, often with further tuning.
• agent workflowA multi-step model-and-tool interaction pattern where the model chooses tools, receives results, and iterates toward a final answer.
• LM Studio Python SDKA Python client library for interacting programmatically with models served by LM Studio, including chat, completion, embeddings, and tool-calling workflows.
• bfloat16A 16-bit floating-point format with a wider exponent range and reduced mantissa precision than FP16, used in machine learning.
• time per output tokenThe time required to generate each additional output token during response generation.
• time to first tokenThe delay between a request being submitted and the model producing its first output token.
• PagedAttentionAn attention algorithm that stores KV-cache blocks noncontiguously to reduce fragmentation and improve memory flexibility during LLM serving.
• CortexA local AI engine for running language models on a user's machine with support for multiple backends and quantized models.
• perplexityA measure of how well a language model predicts tokens in sequence, with lower values generally indicating better predictive fit.
• precisionA numeric representation setting that determines how many bits are used to store or compute model values during serving.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.