A Transformer encoder converts a sequence of vectors into context-aware representations: self-attention lets each element use information from the others, while positional encoding preserves word order. Used alone, it supports sequence or set tasks such as text classification; paired with a decoder, it supplies representations for sequence-to-sequence prediction. section 20.1.6
Binary classification from integer sequences
A compact Keras model embeds tokens and positions, applies an encoder, pools the sequence, and predicts a binary label: inputs <- x <- keras_input(c(seq_length), dtype = "int64") x <- TokenAndPositionEmbedding(vocab_size, seq_length, embed_dim)(x) x <- TransformerEncoder(intermediate_dim = 256L, num_heads = 8L)(x) x <- layer_global_max_pooling_1d()(x) outputs <- layer_dense(units = 1, activation = "sigmoid")(x) model <- keras_model(inputs, outputs) model |> compile(optimizer = "adamw", loss = "binary_crossentropy")
A Transformer encoder fits into larger architectures in two common ways:
- Use the encoder alone when one input sequence must produce a classification or other sequence-level result. section 20.1.6
- Feed encoder outputs to a Transformer decoder for sequence-to-sequence tasks such as translation or question answering. section 20.1.6
- Provide positional information explicitly: attention relates sequence elements as a set and otherwise cannot distinguish their order. section 15.3.4
