A greedy algorithm chooses the best next action without planning beyond one step. In text generation, it selects the highest-probability next token, appends it to the sequence, and repeats. This can trap generation in repetition loops. section 10.2.1
A GPT-2 continuation
Given “NLP is a,” GPT-2 ranks “ new” above “ non.” Repeated choices can produce circular text, such as repeating claims about data structures. Temperature and repetition penalties can help alter this behavior. section 10.2.1
Listing 10.6 Finding the token with maximum probability
>>> encoded_prompt = tokenizer('NLP is a', return_tensors="pt") #1
>>> encoded_prompt = encoded_prompt["input_ids"]
>>> encoded_prompt = encoded_prompt.to(DEVICE)
>>> output = vanilla_gpt2(input_ids=encoded_prompt)
>>> next_token_logits = output.logits[0, -1, :]
>>> next_token_probs = torch.softmax(next_token_logits, dim=-1)
>>> sorted_ids = torch.argsort(next_token_probs, dim=-1, descending=True)
>>> tokenizer.decode(sorted_ids[0]) #2
' new'
>>> tokenizer.decode(sorted_ids[1]) #3
' non'
Inspect how GPT-2 converts next-token logits into probabilities and ranks candidate tokens. section 10.2.1
