Chapter 9: Recording audio and transcribing speech with MLX Whisper

Introduction

This introduction previews adding microphone recording and speech transcription to the existing text-based AI application as a foundation for a voice-enabled chatbot.

9.1 What is speech recognition?

This section defines speech recognition, explains why it is difficult, and introduces how Whisper transcribes audio locally.

• Speech recognition and its challenges: Explain that automatic speech recognition converts spoken audio into text, and describe why variation in speech, noise, and context make that difficult.
  Why it matters: It explains why turning a microphone recording into reliable text takes more than simply detecting sounds.
• Whisper training and language patterns: Distinguish older rule-based and small-dataset approaches from Whisper's large-scale multilingual training, and explain how learned language patterns help it interpret speech.
  Why it matters: Language patterns help explain how Whisper can make sense of speech when sounds alone are ambiguous.
• Whisper transcription processing flow: Trace the section's high-level path from audio through a spectrogram and encoder to a decoder that uses task and language tokens to generate transcription.
  Why it matters: This gives you a high-level picture of how a recording becomes text without requiring the internal mathematics of the model.
• Local Whisper accuracy and privacy: State the section's claims about Whisper's transcription accuracy and what running it offline means for audio privacy.
  Why it matters: It connects speech transcription to the book's aim of building a private, local AI application.

Key terms: speech recognition, automatic speech recognition (ASR), Whisper, phonemes, multilingual audio, statistical patterns of language, encoder-decoder transformer, spectrogram, Mel filterbank, encoder

9.2 Why use MLX Whisper on Apple silicon?

This section explains why MLX Whisper is a faster, more memory-efficient choice for speech transcription on Apple silicon and what that speed can mean for voice chat.

• Apple silicon hardware advantage: Explain why MLX Whisper makes better use of an Apple silicon Mac than the original PyTorch Whisper implementation.
  Why it matters: It explains why the chapter uses MLX Whisper rather than the original implementation on an Apple silicon Mac.
• Transcription speed expectations: Describe what the reported transcription speed suggests for voice chat and why it is not guaranteed on every Mac.
  Why it matters: The example gives a practical sense of voice-chat responsiveness without promising a fixed wait time.
• Machine and deep learning terms: Distinguish machine learning from deep learning and identify how Whisper and LLMs fit those terms.
  Why it matters: These terms help place speech recognition alongside the text-generation models used earlier in the book.

Key terms: PyTorch Whisper, MLX, MLX Whisper, Apple silicon, unified memory, MLX Whisper large-v3, transcription, system load, machine learning, deep learning

9.3 Installing MLX Whisper

This section identifies the packages needed for local speech transcription, how to install and check them, and the alternative for Intel Macs.

• Installation and verification workflow: Describe activating the virtual environment, installing the four Python packages, adding them to the requirements file, and running check_versions.py to check the installation.
  Why it matters: Checking the environment before recording helps catch missing dependencies early.
• Speech package responsibilities: Explain what each of the four packages contributes to recording and transcription.
  Why it matters: The package roles show how microphone input becomes audio data that can be saved and transcribed.
• Intel Mac Whisper alternative: Explain why the MLX route does not support Intel Macs and identify the alternative named in the section.
  Why it matters: The hardware distinction prevents Intel Mac users from trying to install an unsupported MLX workflow.

Key terms: virtual environment, requirements file, check_versions.py, mlx-whisper, sounddevice, numpy, scipy, audio waveforms, WAV files, Intel Macs

9.4 Understanding digital audio

Introduces the need to understand three numbers that describe digital audio before writing recording scripts.

9.4.1 Audio sample rate

Defines audio sample rate and explains why recordings for Whisper should use 16 kHz or be resampled before transcription.

• Samples and sample rate: Explain that a microphone takes measurements of air-pressure changes, each measurement is a sample, and sample rate is the number of samples taken per second.
  Why it matters: This gives you the vocabulary to describe the audio rate Whisper expects.
• Whisper’s 16 kHz requirement: Explain why this chapter records directly at 16 kHz and what must happen before transcription if audio was recorded at a different rate.
  Why it matters: Choosing 16 kHz when recording avoids an extra conversion before transcription.

Key terms: sample, sample rate, Whisper, 16 kHz, resample

9.4.2 Audio channels

9.4.3 Sample bit depth and data type

This subsection compares two audio sample formats and explains why recordings use float32 before conversion to int16 for WAV files.

• Audio sample formats: Distinguish float32 from int16 audio samples and explain why the chapter records in float32 but converts to int16 when writing WAV files.
  Why it matters: The distinction explains why the recording and the WAV file use different sample formats.

Key terms: sample, bit depth, float32, int16, WAV

9.5 Recording from the microphone

Introduces a beginner microphone-recording script that saves a WAV file for inspection and later transcription.

• Microphone recording output: Describe what the beginner script records, where it saves the result, and why it keeps the WAV file on disk.
  Why it matters: Keeping the recording makes it easier to inspect the audio before trying to transcribe it.
• Reusable function basics: Explain what the section means by a Python function and how to recognize one in the script.
  Why it matters: Recognizing functions helps you follow the recording script without needing to write it from scratch.

Key terms: record.py, recording.wav, WAV file, intermediate audio file, functions, def

9.5.1 sd.rec(): The recording call

Explains how sd.rec() captures a specified number of audio samples and how sd.wait() ensures the recording is complete.

• Recording call and completion: Explain what sd.rec() returns, how its first argument sets the number of samples, and why sd.wait() is called before using the recording.
  Why it matters: Waiting for capture to finish ensures the array holds the complete recording before you use it.

Key terms: sd.rec(), sd.wait(), NumPy array, default input device, sample rate

9.5.2 Squeezing the channel dimension

This subsection explains why a mono recording is squeezed into a flat array for MLX Whisper.

• Mono audio array shape: Explain that sd.rec() with channels=1 returns an array shaped (80000, 1), and that .squeeze() removes the one-column dimension to produce the flat shape (80000,) expected by MLX Whisper.
  Why it matters: The shape change prepares recorded audio for transcription.

Key terms: sd.rec(), channels=1, .squeeze(), channel dimension, MLX Whisper

9.5.3 Converting to int16 for WAV

Explains why a float32 microphone recording is scaled when converting it to int16 samples for a WAV file.

• Float32 to int16 scaling: Explain that the recording contains float32 decimal samples from −1.0 to 1.0, while the WAV samples described here use int16 whole numbers from −32768 to 32767, so the decimal values are multiplied by 32,767 for conversion.
  Why it matters: The scaling step makes the recorded sample values suitable for the WAV representation described here.

Key terms: WAV, int16, float32, samples

9.6 Your first transcription

Shows how to transcribe a saved microphone recording with MLX Whisper, read the resulting text, and handle first-run downloads and common errors.

• Recording path for transcription: Describe how to provide MLX Whisper with the recording created earlier: run record.py first if necessary, then pass the path to recording.wav constructed from the script’s location.
  Why it matters: Using the intended file path lets the transcription script find the audio reliably.
• Transcription text result: Explain what mlx_whisper.transcribe() returns and how this book obtains the recognized words from it.
  Why it matters: The returned dictionary is how the application gets usable text from the transcription call.
• Model location and caching: Explain how path_or_hf_repo identifies the Whisper model and what happens when a Hugging Face repository name is used on the first and later runs.
  Why it matters: This explains why the first transcription may involve a download while later runs usually do not.
• First transcription error checks: Describe the remedies given for an ffmpeg file-not-found error and for a model-download SSL certificate error, and distinguish an unauthenticated-request warning from a failed download.
  Why it matters: These checks help separate a missing dependency or certificate failure from a warning that may not stop transcription.

Key terms: recording.wav, record.py, MLX Whisper, mlx_whisper.transcribe(), Python dictionary, result["text"], path_or_hf_repo, Hugging Face, model weights, cache

9.7 Choosing the right Whisper model

This section introduces the accuracy, speed, and memory tradeoff among MLX Whisper model sizes and shows how their Hugging Face repository names are formed.

• Whisper model size tradeoff: Explain that MLX offers five Whisper model sizes and that larger models tend to be more accurate but slower and more memory-intensive.
  Why it matters: This tradeoff helps you choose a model that fits your accuracy needs and your Mac’s resources.
• Whisper repository naming pattern: Describe the Hugging Face repository naming pattern for MLX Whisper models and give the large-v3 example.
  Why it matters: The naming pattern lets you identify the repository for a chosen model size.

Key terms: Whisper model sizes, accuracy, memory, Hugging Face repository name, mlx-community/whisper-{size}-mlx, large-v3

9.7.1 Which model should you use?

This section recommends MLX Whisper models for short voice input based on unified memory shared with Ollama and the need for accuracy.

• Whisper model selection tradeoffs: Explain how to choose an MLX Whisper model for 5–10 seconds of conversational voice input while leaving memory for an Ollama model: start with small on an 8 GB Mac, try medium first with 16 GB or more, and use large-v3 when accuracy matters and enough memory remains.
  Why it matters: Choosing Whisper with the other running models in mind helps avoid exhausting your Mac’s unified memory.

Key terms: unified memory, whisper-small-mlx, whisper-medium-mlx, whisper-large-v3-mlx, Ollama

9.7.2 Trying a larger model

9.8 Building a reusable transcription script

This section turns recording and transcription into a reusable Python module that can be tested now and imported into the voice chatbot later.

• Reusable voice input module: Explain how voice_input.py combines recording and transcription, why you run it once to check that they work together, and how the resulting module can be reused.
  Why it matters: Keeping a tested voice-input module lets you reuse the same recording-and-transcription capability in another application.

Key terms: voice_input.py, record_and_transcribe, self-contained module

9.9 Troubleshooting

9.9.1 “No default input device”

Explains how to resolve a microphone input error caused by macOS blocking access.

• Microphone access error resolution: Explain that a “No default input device” error can mean macOS has blocked microphone access, and describe how to grant access to the app being used.
  Why it matters: Checking macOS permissions can get microphone recording working again.

Key terms: No default input device, Privacy & Security, Microphone

9.9.2 Audio is captured but Whisper returns empty text

This subsection explains three common reasons a captured recording produces an empty Whisper transcript and how to address each one.

• Empty transcript troubleshooting: Explain how low input volume, background noise, or speaking before recording starts can lead to an empty transcript, and describe the suggested fix for each.
  Why it matters: These checks can help you fix an empty transcript even when audio capture appears to work.

Key terms: empty transcript, input volume, background noise, print("Recording...")

9.9.3 “Cannot import mlx_whisper”

9.9.4 The first download is slow

Explains why MLX Whisper’s first run can be slow and how to let the model download complete.

• First-run model download: Explain why MLX Whisper may be slow on first use and what to do while its model weights download.
  Why it matters: A slow first run may simply mean the model is downloading, not that transcription has failed.

Key terms: MLX Whisper, model weights, Hugging Face, whisper-large-v3-mlx

9.9.5 “File not found” for ffmpeg

9.10 Exercises

9.11 Summary