A decoder-only transformer generates a sequence without a separate input encoder. Compared with the original encoder–decoder transformer, it removes both the encoder and the decoder’s cross-attention to encoder outputs. This makes it a common architecture for language generation rather than an input-to-output task such as translation. section 3.2 section 3.1
Three mechanisms define how it generates:
- Causal attention prevents a position from attending to future tokens, so predictions use preceding context. section 3.2
- A next-token prediction objective trains the model to predict the next token from earlier tokens. section 3.2
- Autoregressive generation feeds each generated token back into the input stream to predict the next; generation remains bounded by the context window. section 3.2
Do not confuse the absent transformer encoder with an input tokenizer named `Encoder`. The GPT-2 example converts raw text into token IDs using Unicode-safe byte encoding and byte pair encoding (BPE); those IDs are then passed to the decoder-only model. section 3.2
Listing 3.1 Input encoder used in decoder-only transformers (BPE abstracted)
class Encoder:
def __init__(self, encoder, bpe_merges, errors='replace'):
self.encoder = encoder
self.decoder = {v: k for k, v in encoder.items()}
self.byte_encoder = bytes_to_unicode()
self.byte_decoder = {v: k for k, v in self.byte_encoder.items()}
self.bpe_ranks = dict(zip(bpe_merges, range(len(bpe_merges))))
self.cache = {}
self.errors = errors
self.pat = re.compile(r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+|
?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+""")
def bpe(self, token):
... #1
def encode(self, text):
bpe_tokens = [] #2
for token in re.findall(self.pat, text): #3
byte_seq = ''.join(self.byte_encoder[b] for b in
token.encode('utf-8')) #4
token_encode = byte_seq
bpe_tokens.extend(
self.encoder[bpe_token] for bpe_token in
self.bpe(byte_seq).split(' ') #5
)
return bpe_tokens
def decode(self, tokens):
text = ''.join([self.decoder[token] for token in tokens]) #6
return text
def __convert_back_to_original_UTF_8_string_using_byte_decoder
(self, bytearray):
text = bytearray([self.byte_decoder[c]
for c in text]).decode('utf-8', errors=self.errors) #7
return text
def get_encoder(model_name, models_dir):
with open(os.path.join(models_dir, model_name,
'encoder.json'), 'r') as f: #8
encoder = json.load(f)
with open(os.path.join(
models_dir, model_name,
'vocab.bpe'), 'r',
encoding='utf-8') as f:
bpe_data = f.read()
bpe_merges = [tuple(merge.split())
for merge in bpe_data.split('\n')[1:-1]] #9
return Encoder(encoder=encoder, bpe_merges=bpe_merges)
The abstracted GPT-2 listing shows how input text becomes vocabulary IDs before it reaches the transformer; this preprocessing is not a separate transformer encoder. section 3.2
Listing 3.2 Masked multihead self-attention in decoder-only models
def attention_mask(nd, ns, *, dtype):
i = tf.range(nd)[:, None] #1
j = tf.range(ns)
m = i >= j - ns + nd
return tf.cast(m, dtype)
def mask_attn_weights(w):
_, nd, ns = shape_list(w) #2
b = attention_mask(nd, ns, dtype=w.dtype) #3
b = tf.reshape(b, [1, 1, nd, ns]) #4
w = w + b * -tf.cast(1e10, w.dtype) * (1 - b) #5
return w
The masked multihead self-attention listing provides a concrete implementation reference for the causal-attention constraint. section 3.2
Decoder-only models underpin many LLMs used for text generation, code synthesis, and chat. An instruction-tuned model is a subset of LLMs optimized to follow prompts, not a separate transformer architecture. section 3.1 section 3.6
