Order property

As explained in Build a Large Language Model (From Scratch)

The location of a token or element within a sequence, used here as the basis for relating elements in self-attention.

Positional information identifies where each token occurs in a sequence. Because self-attention and token embeddings are position-independent, adding it helps the model represent order and token relationships. section 2.8

Use two broad positional-embedding strategies:

• Absolute positional embeddings assign a distinct learned vector to each sequence position and add it to the token embedding. section 2.8
• Relative positional embeddings represent distances between tokens, supporting generalization to sequence lengths not seen during training. section 2.8

Combining token and positional embeddings

For inputs shaped [8, 4], a token embedding layer produces [8, 4, 256]. A positional embedding layer produces [4, 256]; broadcasting and addition yield [8, 4, 256] input embeddings for the LLM layers. section 2.8

Figure 2.18 Positional embeddings are added to the token embedding vector to create the input embeddings for an LLM. The positional vectors have the same dimension as the original token embeddings. The token embeddings are shown with value 1 for simplicity.

Visualizes positional vectors matching token-embedding dimensions and being added to form input embeddings. section 2.8

How Build a Large Language Model (From Scratch) covers it

How deep it goes

Explains absolute and relative strategies and demonstrates their tensor-shape implementation, without deriving attention internals.

Examples

Demonstrates PyTorch embeddings on batches of eight four-token sequences with 256-dimensional token and position vectors.

What it assumes

Assumes tokenization and token IDs, while demonstrating token-embedding lookup before adding positional information.

What you can do after

Apply positional embeddings in PyTorch by creating position indices, looking up vectors, and adding them to token embeddings.

How it connects

Connects token embeddings, self-attention, absolute positional embeddings, relative positional embeddings, and input embeddings.

liveBookmanning.com

Read Build a Large Language Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Kafka for ArchitectsMessage ordering is the preservation or enforcement of a sequence among messages, in Kafka guaranteed only within an individual partition.
• 

  liveBookmanning.com
  Apache Kafka in ActionMessage order is the sequence relationship in which messages are stored or processed.
• 

  liveBookmanning.com
  Pro ASP.NET Core 7, Tenth EditionA tag helper execution-order setting that determines the sequence in which tag helpers run for an element.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsInformation that depends on the order or position of elements in a sequence.
• 

  liveBookmanning.com
  MongoDB 8.0 in Action, Third EditionEvent ordering is the determination of the sequence in which events occurred or should be processed.
• 

  liveBookmanning.com
  Clojure: The Essential ReferenceA relation that determines the relative ordering of values, such as less than, greater than, or equality.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionInformation carried by the order of tokens in text, as opposed to only their presence or count.

Compare how 8 books explain Order property →

Related concepts

• Tag Helpersconfigures · Tag Helpers are ASP.NET Core components that process HTML-like elements during Razor view or page rendering.
• partition countcontrasts with · Partition count is the configured number of partitions into which a partitioned data set is divided.
• schedulingused by · Scheduling is the planning and coordination of when and in what order activities or computational work occur, potentially assigning them to available resources subject to constraints.
• AI agentcontrasts with · An AI agent is a computational system that uses artificial intelligence to interpret information, select actions, and pursue goals through interaction with an environment.
• implicit time modelpart of · The representation of time and execution behavior provided implicitly by a programming language or runtime.
• maximumused by · A maximum is the greatest value in a set under a specified ordering, or the greatest value permitted by a specified constraint.

This explainer was generated by Manning's knowledge graph from the text of Build a Large Language Model (From Scratch). The section links point to the parts of the book it draws on.