Chapter 4: Downloading an LLM and having your first conversation

Introduction

The introduction establishes that Ollama is running but still needs a model, and tells the reader to keep the service running while using a second terminal for the chapter’s commands.

• Ollama Service and Model Readiness: Explain that having Ollama installed and `ollama serve` running does not mean an AI model is loaded. Describe why the service terminal should stay open while a second terminal is used for commands.
  Why it matters: This distinction helps you keep Ollama running while you download and use a model.

Key terms: Ollama, ollama serve, AI model, second terminal

4.1 Downloading your first model

The section explains why the first download uses a small, reliable model before exploring alternatives.

• First model selection rationale: Explain why the book starts with one small, reliable model before comparing alternatives.
  Why it matters: A manageable first download gives you a useful reference point when you try other models.

Key terms: small, reliable model, baseline

4.1.1 Choosing a starting model

This section chooses Gemma 3 with 4 billion parameters as a practical first Ollama model and explains the basic tradeoff behind that size.

• Parameter meaning and tradeoff: Explain that parameters are numerical values learned during training and that models with more parameters generally give smarter responses but use more RAM and run more slowly.
  Why it matters: This tradeoff helps you understand why a smaller model is a sensible first choice.
• Gemma 3 starting choice: Justify using gemma3:4b for the first download: its 4-billion-parameter size runs smoothly on almost any modern laptop, and keeping this tested model makes the chapter’s commands and results consistent on modest Mac hardware despite the availability of Gemma 4.
  Why it matters: Starting with the tested model makes the first download and conversation easier to follow.

Key terms: parameters, RAM, Gemma 3, gemma3:4b, Gemma 4

4.1.2 Running the pull command

This section explains how to download an AI model with Ollama, what the downloaded file contains, and how to handle an interrupted download.

• Model download and storage: Describe how to download a model with Ollama and what happens after the download finishes.
  Why it matters: This lets you download a model once and use the local copy afterward.
• Downloaded parameters and learning: Explain what the section means by the downloaded model's parameters and distinguish its training from a conversation.
  Why it matters: It clarifies what you downloaded without mistaking a chat for further training.
• Interrupted download recovery: Describe what to do if a large model download appears stalled or ends with one of the stated errors.
  Why it matters: A pause or retryable error need not mean starting the download from scratch.

Key terms: ollama pull, ollama serve, ollama rm, parameters, training, pulling manifest, max retries exceeded, unexpected EOF

4.1.3 Understanding the model name format

Explains how to read a model name and use its size tag to choose a model that fits a Mac’s RAM.

• Model name and tag: Explain the model_name:tag format using gemma3:4b: gemma3 names the model family, and 4b identifies the 4-billion-parameter variant.
  Why it matters: Reading the name tells you which family and variant you are choosing.
• Size tags and RAM fit: Describe how tags such as 7b, 13b, and 70b denote different model sizes and why the choice should fit the machine’s RAM. Give the section’s 8 GB RAM example.
  Why it matters: A model’s size tag helps you avoid choosing one too large for your Mac.

Key terms: model_name:tag, model family, parameter variant, tag, model size, RAM

4.2 Interactive chat

Introduces an upcoming interactive chat with a downloaded model running locally without an internet connection.

4.2.1 Starting interactive chat mode

Shows how to recognize interactive chat mode and explains that messages go to the AI model on the laptop without using the internet.

• Interactive chat prompt and locality: Explain what the triple-arrow terminal prompt indicates and where text entered at that prompt goes.
  Why it matters: You can recognize when you are chatting with the local model and understand where your messages go.

Key terms: triple arrow, interactive chat mode, AI model

4.2.2 A sample conversation

The section demonstrates a first exchange with Gemma and explains why a reader’s response may differ from the example.

• First chat exchange: Describe sending an initial message and then a follow-up asking Gemma for a simple explanation. Explain that the example illustrates how clearly even a small model can answer a straightforward question.
  Why it matters: This exchange gives you a concrete sense of what to try in your first local chat.
• Variation in model replies: Explain why a reply to the same prompt might not match the book’s example word for word.
  Why it matters: A differently worded reply does not mean your chat went wrong.

Key terms: follow-up, 4-billion-parameter model, nondeterministic, prompt

4.2.3 Exiting chat mode

This section shows how to leave an interactive Ollama chat and how to ask a single question without starting one.

• Leaving interactive chat: Describe how to exit an interactive Ollama chat and return to the regular terminal.
  Why it matters: You can finish a chat without wondering how to get back to the terminal.
• One-shot question command: Explain how to ask Ollama one question without entering interactive chat mode.
  Why it matters: A one-shot command is convenient when you want a quick answer rather than a full chat session.

Key terms: /bye, Ctrl-D, regular terminal, one-shot question, ollama run, interactive mode

4.3 Understanding model sizes and parameters

Introduces the upcoming discussion of what model-size labels such as “4B” and “7B” mean and how they relate to performance and memory use.

4.3.1 What are parameters?

Defines model parameters and illustrates how different parameter counts relate to task capability and hardware needs.

• Meaning of model parameters: Explain that a model’s parameter count is the number of adjustable numerical values in its neural network, which are adjusted during training on text to learn language patterns.
  Why it matters: This makes labels such as 4B meaningful rather than just names for models.
• Capabilities across model sizes: Describe the section’s examples of how 4B, 7B, and 70B models differ in the tasks they can handle, including the serious hardware needed to run a 70B model.
  Why it matters: The examples give a practical sense of what different model sizes may offer and demand.

Key terms: parameters, neural network, training, 4B model, 7B model, 70B model

4.3.2 The size, capability, and memory tradeoff

This section explains how to balance model capability and speed against RAM needs, then check whether a Mac has enough memory before downloading a large model.

• Model size tradeoff: Explain why a smaller or larger model might suit a particular computer and task, and why listed RAM requirements are only estimates.
  Why it matters: This tradeoff helps you choose a model your computer can run for the work you want to do.
• Memory pressure check: Describe how to check memory headroom on a Mac before downloading a large model and what to do if memory is tight.
  Why it matters: Checking memory pressure can help you avoid choosing a model that strains your Mac.

Key terms: model size, RAM, quantization, model format, Activity Monitor, Memory tab, Memory Pressure

4.3.3 What “quantization” means

This section explains how quantization reduces a model’s memory needs, what it costs in answer quality, and when that tradeoff is useful.

• Quantization and model tags: Explain what tags such as q4_0 and q8_0 indicate and how quantization affects a model’s memory use and answer quality.
  Why it matters: Those tags help you understand why a version of a model may fit your Mac more easily.
• Choosing a quantized model: Justify using a quantized model when a full-precision model would not fit in RAM, and describe when avoiding heavy quantization makes sense.
  Why it matters: You can choose a model that runs on your available RAM without assuming that a quantized version is unsuitable for everyday use.

Key terms: quantization, q4_0, q8_0, q4, q8, quantized model, full-precision, RAM, heavy quantization, Ollama

4.4 Model comparison: Choosing the right model

This section gives a simple basis for choosing a local model: match its family and size to the task and the Mac’s available RAM.

• Task and RAM model fit: Explain how to choose a model family and size by considering both the intended task and the Mac’s available RAM.
  Why it matters: A model needs to suit both your task and your computer.

Key terms: model family, model size, RAM

4.4.1 The major model families

This subsection explains that model families have different makers and strengths, then gives beginners two models to try in sequence.

• Beginner model comparison choice: Explain why a beginner might try more than one model family, and state the recommended starting and comparison models.
  Why it matters: Trying a second family gives you a simple way to compare models rather than assuming they are interchangeable.

Key terms: model family, Gemma 3 (4B), Qwen 3.5 (4B)

4.4.2 Downloading a second model

This subsection adds Qwen 3.5 as a second local model and explains how to switch between downloaded models.

• Second model download and switching: Describe downloading Qwen 3.5 with 4 billion parameters as a second model for comparison, and explain that you switch between the two downloaded models by giving ollama run a different model name.
  Why it matters: Having two models locally lets you try each one for a comparison.

Key terms: Qwen 3.5, 4 billion parameters, pull command, ollama run

4.4.3 A quick model comparison test

This section compares two models’ answers to the same question to show how their response styles differ.

• Same-question model comparison: Describe how asking Gemma and Qwen the same question reveals differences in their response styles, and explain why trying several models can help you choose one that fits your needs.
  Why it matters: Comparing answers to the same question helps you choose a model whose style suits you.

Key terms: Gemma, Qwen, personalities

4.5 Model management commands

This introduction points readers to a table of Ollama model-management commands and the subsections that explain them.

4.5.1 Listing downloaded models

4.5.2 Checking running models

This subsection explains how to check which Ollama models are loaded in memory and why they remain loaded briefly after use.

• Loaded model status and lifecycle: Explain that Ollama provides a command to check which models are currently loaded in memory, and describe what happens to a model after its last interaction.
  Why it matters: A model can still appear loaded shortly after you finish using it without remaining in memory indefinitely.

Key terms: loaded in memory, fast follow-up queries, automatically unloads

4.5.3 Viewing model details

This section introduces viewing a model’s details and explains how to interpret the context length shown there.

• Model details display: Describe what viewing a model’s details reveals, including its parameters and license.
  Why it matters: Model details help you inspect a downloaded model rather than relying on its name alone.
• Context length in tokens: Explain what a model’s context length measures and interpret the section’s 128,000-token example.
  Why it matters: Reading context length correctly helps you understand how much conversation history a model can consider at once.

Key terms: model, parameters, license, context length, tokens, conversation history

4.5.4 Removing a model

This subsection explains that deleting an unneeded model frees disk space and that it can be downloaded again later.

• Reversible model removal: Explain why you might delete a model you no longer need and how to get it back later.
  Why it matters: You can reclaim storage without giving up the option to use the model again.

Key terms: model, disk space, ollama pull

4.6 Exercises

4.7 Summary