Whisper

As explained in Build Applications with Local AI Models on a Mac

Whisper is an automatic speech-recognition model developed by OpenAI for transcribing multilingual audio.

Whisper turns speech in an audio recording into text. OpenAI trained it on 680,000 hours of multilingual audio, helping it handle varied accents, languages, and recording conditions. It can run offline, so audio need not leave your machine. section 9.1

Whisper first converts audio into a spectrogram, a representation of sound frequencies over time. An encoder processes that representation; a decoder uses its output and tokens specifying the language and task to generate a transcript. section 9.1

Figure 9.1 Whisper’s encoder-decoder architecture. Audio is converted into a spectrogram, which the encoder processes. The decoder processes the encoder’s output together with control tokens to generate a transcript.

Follow the path from audio through the spectrogram and encoder to the decoder’s transcript. section 9.1

For a local Python voice-input application:

• Prepare mono, 16 kHz audio. Processing uses float32; WAV files can use int16. section 9.11
• Record with sounddevice.rec() and wait for completion with sd.wait(). section 9.11
• Transcribe with mlx_whisper.transcribe(path, path_or_hf_repo=model), then read result["text"]. MLX Whisper is optimized for Apple silicon. section 9.11

Whisper models range from tiny to large-v3. Start with small if memory is limited; try medium with more unified memory, and choose large-v3 only when its accuracy benefit warrants the extra memory and latency. Reusable record_and_transcribe() and transcribe_file() functions can provide voice input for a chatbot. section 9.11 section 9.0

How Build Applications with Local AI Models on a Mac covers it

How deep it goes

Explains the spectrogram-to-encoder-to-decoder flow and gives recording, model-selection, and transcription guidance without detailing model training.

Examples

Uses microphone recording, a transcribe() call, and a reusable voice_input.py module.

What it assumes

Starts with speech-recognition basics before introducing Python microphone recording and transcription.

What you can do after

Apply it via MLX Whisper to transcribe recordings and create reusable microphone voice-input functions.

How it connects

Connects automatic speech recognition with spectrograms, encoder-decoder transformers, MLX, and voice-enabled chatbots.

liveBookmanning.com

Read Build Applications with Local AI Models on a Mac on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Data Analysis with LLMswhisper-1 is the model identifier for the Whisper speech-transcription model in the OpenAI API.

Compare how 2 books explain Whisper →

Related concepts

• encoder-decoder transformerkind of · An encoder-decoder transformer is a neural-network architecture whose encoder processes an input sequence and whose decoder generates an output sequence conditioned on the encoded representation.
• neural networkkind of · A neural network is a computational model of interconnected, parameterized units that learns transformations from inputs to outputs.
• speech recognitionimplements · Speech recognition is the computational conversion of spoken language from audio into a machine-readable representation, typically text.
• MLX Whisperimplemented by · MLX Whisper is a speech-recognition implementation that packages the Whisper model in a format optimized for Apple's MLX framework.

This explainer was generated by Manning's knowledge graph from the text of Build Applications with Local AI Models on a Mac. The section links point to the parts of the book it draws on.