Universal Transformer

As explained in Sutskever's List

The Universal Transformer is a Transformer variant that repeatedly applies shared self-attention and feed-forward layers across depth to iteratively refine token representations.

The Universal Transformer repeatedly applies shared self-attention and feed-forward layers to refine token representations. It brings recurrence into Transformer processing, where self-attention lets tokens interact across a sequence. section 5.1.2

Figure 5.2 The Universal Transformer. Used with permission granted by the lead author (Mostafa Dehghani).

A schematic of the Universal Transformer’s shared-layer design. section 5.1.2

How Sutskever's List covers it

How deep it goes

Defines repeated shared-layer application but gives no supported account of its computation schedule.

What you can do after

Understand how shared attention and feed-forward layers can iteratively refine token representations.

How it connects

Connects Transformer self-attention with recurrence.

liveBookmanning.com

Read Sutskever's List on liveBook Get the book at manning.com

Related concepts

• attentionuses · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• recurrenceuses · A recurrence is a relation or computational process in which each successive value or state is defined using preceding values or states.
• transformer architecturecontrasts with · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.