liveBook · Manning

Semantic association

As explained in Timeless Algorithms, The Foundational Ideas

A semantic association is a learned relationship between concepts that makes one concept more likely to be recalled or predicted in the presence of another.

A semantic association is a learned relationship that makes one concept more likely to be recalled or predicted given another. In a transformer, repeated exposure to a title and its author can strengthen that relationship through prediction-error updates across the model’s parameters. section 12.2.2

The association is distributed rather than stored as a complete fact in one embedding or attention head. Training updates embeddings, attention projections, feed-forward layers, and output parameters. At prediction time, the prompt representation contributes to logits for possible answers; attention helps construct that representation, while the candidate answers are evaluated as possible outputs. section 12.2.2

A title–author association

After encountering The Count of Monte Cristo and Alexandre Dumas across many contexts, a model may assign Dumas a higher output score than Hugo, Tolstoy, or Dickens when prompted for the author. A simplified dot product illustrates how a candidate’s output weights can score against the prompt representation; it does not describe the full transformer computation. section 12.2.2 section 12.2

oDumas=[0.82,0.44]⋅[0.9,0.4]=(0.82)(0.9)+(0.44)(0.4)=0.91

This simplified dot product gives Dumas an output score of 0.91 against the example final representation. It illustrates candidate scoring, not a complete or literal account of how a trained transformer stores the association. section 12.2.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Traces repeated exposure through prediction loss and updates across model parameters, while rejecting a single-location account of the learned fact.
Examples
Uses The Count of Monte Cristo and Alexandre Dumas, with simplified output scores compared against other candidate authors.
What you can do after
Understand how repeated training can make a factual answer more likely without storing the complete fact in one identifiable parameter.
How it connects
Connects factual recall to transformer representations, self-attention, backpropagation, and next-token prediction.

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.