Output gate

As explained in Sutskever's List

An output gate is an LSTM gating component that controls how much of the processed cell state is exposed as the hidden state.

An output gate is the LSTM component that controls how much of the processed cell state becomes the hidden state. After the cell state is updated, tanh produces a candidate short-term memory; a sigmoid layer scales each element before exposing it at the current time step. section 4.2

Selective exposure

An LSTM can preserve information in its long-term cell state while suppressing it from the current hidden state. The output gate therefore separates retaining memory from exposing memory, supporting different long- and short-term responses. section 4.2

Figure 4.2 A single LSTM cell. The inputs are the current vector X_t, the previous hidden state h_t–1, and the previous cell state C_t–1. There are three learned gates: the forget f_t, input i_t, and output O_t. They are all produced by sigmoid layers (“σ” blocks). A tanh layer forms a candidate update \(\tilde{C}\)_t. Pointwise multiplications (“×”) and an addition (“+”) update the cell state on the top “conveyor” pathway. Arrows indicate vector flow; the legend marks pointwise operations, concatenation, and copies. This is the standard LSTM, emphasizing how gating enables selective memory retention and exposure. Used with permission granted by the author (Chris Olah), though it is slightly modified from its original form.

Use the standard LSTM cell diagram to trace the sigmoid output gate, tanh-processed cell state, and pointwise multiplication that forms the hidden output. section 4.2

How Sutskever's List covers it

How deep it goes

Explains sigmoid scaling after tanh processing and pointwise multiplication, without deriving the gate's learned parameters.

Examples

Uses the conveyor-belt metaphor and contrasts retained cell-state information with selectively exposed hidden-state information.

What it assumes

Assumes basic awareness of LSTM inputs, hidden states, cell states, and the other two gates.

What you can do after

Evaluate how an LSTM selectively exposes memory at each time step.

How it connects

Connects the output gate to LSTM, cell state, hidden state, sigmoid, tanh, forget gate, and input gate.

liveBookmanning.com

Read Sutskever's List on liveBook Get the book at manning.com

Related concepts

• gatingcontrasts with · Gating is the selective regulation of information flow within a computational system, controlling what information is passed, retained, combined, or suppressed.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.