Pointer Network

As explained in Sutskever's List

A Pointer Network is an attention-based sequence model that selects positions in an input sequence as its outputs.

A fixed output vocabulary works poorly when a task’s answers must come from its input. A Pointer Network (Ptr-Net) adapts attention to select input positions: its scores form a distribution over those positions, allowing it to copy or reorder input elements instead of generating new symbols. section 5.4

Figure 5.7 A seq2seq model (left). The input sequence (A) creates a vector used to generate the output sequence (B). The output dimensionality is fixed by the problem’s dimensionality and remains the same during training and inference [51]. Ptr-Net (right) is also an encoder-decoder. The encoder and decoder hidden states are denoted as (h₁,…,h_n) and (s₁,…, s_n), respectively. Ptr-Net encodes the input sequence into a code (A) that is fed to the generating network (B). At each step, the decoder produces a vector that modulates a content-based attention mechanism over the inputs. The attention mechanism outputs a softmax distribution with a dictionary size equal to the input length. Used with permission granted by the lead author (Oriol Vinyals).

The encoder-decoder diagram shows the decoder applying attention to the inputs at each step; the output distribution’s size matches the input length. section 5.4

The architectural shift is small but consequential: attention becomes a way to point, not merely to build a context vector. This makes the model suitable for combinatorial tasks whose outputs are positions or items drawn from a variable-length input. section 5.4 section 5.4.2

Convex hulls and generalization

Given 50 two-dimensional points, a model must identify those forming the convex hull. Ptr-Net reached 72.6% accuracy, compared with 38.9% for LSTM with attention. When trained on lengths 5–50 and tested on 500 points, exact-order accuracy fell to 1.3%, while the area score remained 99.2%. section 5.4.1

Other tasks include Delaunay triangulation, traveling-salesperson tours, sorting, and scheduling. The pointer approach also foreshadowed copy mechanisms that select rare or out-of-vocabulary tokens from source text. section 5.4 section 5.4.1 section 5.4.2

How Sutskever's List covers it

How deep it goes

Defines attention as a distribution over input positions and outlines encoder-decoder operation, but gives no implementation derivation.

Examples

Uses convex-hull detection, Delaunay triangulation, traveling-salesperson tours, sorting, and scheduling as selection tasks.

What you can do after

Understand how attention can select input elements for combinatorial outputs that fixed-vocabulary decoding cannot directly represent.

How it connects

Links attention to Bahdanau attention, sequence-to-sequence models, combinatorial optimization, and copy mechanisms.

liveBookmanning.com

Read Sutskever's List on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsA pointer network is a sequence-to-sequence neural architecture that outputs positions or elements from its input sequence through attention-based pointers.

Compare how 2 books explain Pointer Network →

Related concepts

• attentionuses · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• decodehas part · Decoding is the process of transforming an encoded or internal representation into an interpretable or target-form output.
• encoderhas part · An encoder is a component that transforms input data into an encoded representation for subsequent processing, reconstruction, comparison, or transmission.
• categorical distributionproduces · A categorical distribution is a discrete probability distribution over a finite set of mutually exclusive categories or outcomes.
• Bahdanau attentionuses · Bahdanau attention is an attention mechanism that computes alignment weights between decoder states and encoder outputs in sequence-to-sequence translation.
• encoder-decoder architecturealternative to · An encoder-decoder architecture is a model design in which an encoder transforms input into an intermediate representation and a decoder uses that representation to produce output.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.