Guided attention adds an auxiliary loss that penalizes attention matrices deviating from an expected alignment pattern. In text-to-speech, it encodes the assumption that text positions align closely with audio over time. section 9.2.2
Use the technique as a practical training and diagnosis loop:
- Plot attention activations between audio frames and characters, then check whether the pattern approaches a diagonal. section 9.2.2
- Penalize substantial deviations from the diagonal to accelerate training and steer the model toward meaningful alignments. section 9.2.2
- Interpret moderate deviations as timing variation and large residuals as sounds or character combinations the model struggles to learn. section 9.2.2
Text-to-speech alignment
Compare attention plots during training with and without guided-attention loss. The guided version should move more directly toward a near-diagonal alignment, exposing remaining difficult sounds for later data collection. section 9.2.2
Figure 9.12 Attention plot evolution through training without (left) and with (right) guided attention loss
Compare attention-plot evolution without and with guided-attention loss. section 9.2.2
