Part 3 Specialized models
Now that we understand how LLMs generate, align, and ground language, we turn our attention to specialization. Real-world deployments often require models that are not only powerful but also efficient, domain-aware, and responsible.
We begin with multimodal models, which combine text with images, audio, video, and structured data. These models enable use cases such as captioning, transcription, and cross-modal retrieval, but they also introduce challenges. Unlike text—which maps cleanly into token embeddings—images, audio, and video must first be processed by modality-specific encoders that turn raw data into patches, frames, or spectrograms before aligning with a language model. Handling these differences is essential for systems that integrate multiple modalities effectively.
Next, we examine efficient and specialized small language models (SLMs). While large LLMs dominate the headlines, smaller models often deliver better performance in constrained environments or as specialists within a larger agentic system. We will see how SLMs can be fine-tuned for classification, empathy, translation, or domain-specific reasoning, and why their efficiency makes them powerful complements to larger models.
We then turn to training and evaluating large-scale models, focusing on hyperparameter tuning, parameter-efficient fine-tuning, and systematic evaluation. These techniques allow you to adapt foundation models to your needs without prohibitive compute costs.