Word error rate

Q: How deeply does Natural Language Processing in Action, Second Edition cover word error rate?

Defines word-level transcription comparison and reports benchmark results, but does not detail the calculation procedure.

As explained in Natural Language Processing in Action, Second Edition

Word error rate is the proportion of word-level transcription errors relative to the reference transcription.

Word error rate (WER) measures the proportion of words that an automatic speech-to-text system transcribes incorrectly against a reference transcription. It turns transcription quality into a comparable metric for choosing an STT system. section 12.7.4

Comparing speech-to-text systems

A benchmark comparison reports mean WERs of 60% for Kaldi, 31% for wav2vec 2.0, and 12% for Whisper across AI, phone, meeting, video, and finance datasets. The domain-specific scores show why a single average may not represent every deployment. section 12.7.4

Listing 12.3 TTS word error rate

|             | AI   | Phone   | Meeting   | Video   | Finance   | Mean   |
|:------------|:-----|:--------|:----------|:--------|:----------|:-------|
| Kaldi       | 66%  | 78%     | 54%       | 69%     | 35%       | 60%    |
| wav2vec 2.0 | 33%  | 41%     | 39%       | 26%     | 17%       | 31%    |
| Whisper     | 6%   | 20%     | 19%       | 9%      | 5%        | 12%    |

Inspect the benchmark table to compare WER across models and speech domains. section 12.7.4

How Natural Language Processing in Action, Second Edition covers it

Cover of Natural Language Processing in Action, Second Edition

liveBook manning.com

Read Natural Language Processing in Action, Second Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Sutskever's ListWord error rate is the proportion of word-level recognition errors, typically computed from substitutions, deletions, and insertions relative to a reference transcript.

Compare how 2 books explain word error rate →

Related concepts

speech recognitionmeasures · Speech recognition is the computational conversion of spoken language from audio into a machine-readable representation, typically text.
Deep Speech 2measures · Deep Speech 2 is a large-scale neural speech-recognition system designed for end-to-end automatic speech recognition.

This explainer was generated by Manning's knowledge graph from the text of Natural Language Processing in Action, Second Edition. The section links point to the parts of the book it draws on.