liveBook · Manning

Curriculum learning

As explained in Sutskever's List

Curriculum learning is a training strategy that presents examples in an organized progression, typically from easier to more difficult cases.

A deep speech-recognition network must learn from utterances of varying length. Starting with the harder cases can make early training unstable. Curriculum learning changes the order of examples so the network encounters easier cases before more challenging ones. section 4.4.2

SortaGrad in Deep Speech 2

SortaGrad sorted training sequences by length during the first epoch. Shorter utterances, treated as typically easier, came first. This helped reduce numerical instability and supported reliable early convergence. Alongside sequence-wise batch normalization, it helped Deep Speech 2 train deeper networks and improve transcription accuracy and stability. section 4.4.2

This was one part of a larger engineering effort, not a stand-alone explanation for the system’s gains. Deep Speech 2 combined curriculum learning with architectural refinements, computational optimizations, and normalization—a pragmatic approach to improving end-to-end learning. section 4.4.5

How Sutskever's List covers it

How deep it goes
Explains first-epoch length sorting and its rationale, but does not specify an ordering procedure for later epochs.
Examples
Uses SortaGrad’s shorter-first ordering of Deep Speech 2 utterances as the concrete example.
What it assumes
Assumes familiarity with training sequences and epochs while explaining the short-before-long ordering in plain terms.
What you can do after
Practice/perform it by sorting speech sequences by length during the first training epoch.
How it connects
Connects curriculum learning to end-to-end speech recognition, sequence-wise batch normalization, and training stability.
Cover of Sutskever's List
liveBookmanning.com
Read Sutskever's List on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.