A model can fit its training data and still perform poorly on new observations. Statistical learning theory studies why this happens and how model capacity affects generalization. Developed over decades by Vladimir Vapnik and Alexey Chervonenkis, it puts performance beyond the training sample at the center of learning. section 8.0
Structural risk minimization considers empirical risk together with model capacity; VC dimension provides a formal measure of that capacity. Margin maximization helps control a classifier’s flexibility. These ideas became operational in support vector machines (SVMs), where a maximum-margin boundary favors separation that is less tied to individual training observations. section 8.9 section 8.0
Soft-margin SVMs allow observations inside the margin or even misclassified. Slack variables quantify these violations, while a penalty parameter balances them against margin width. This trade-off can reduce the influence of unusual observations; cross-validation helps select a penalty value likely to generalize. section 8.0 section 8.9
The same generalization problem applies to transformers, although the connection is interpretive rather than architectural: they do not ordinarily optimize a maximum-margin objective or directly apply Vapnik’s capacity bounds. Training loss alone cannot establish reliable performance on unseen sequences or unfamiliar domains, so regularization and held-out evaluation remain relevant. section 12.3.1
