Greedy decoding builds a generated sequence one token at a time, selecting the most probable next token given the input and tokens already produced. It moves forward without reconsidering earlier choices, so a locally likely token may lead to a less coherent continuation. section 4.1.1
The method trades search breadth for a simple, efficient choice at each step:
- Choose the highest-probability next token, conditioned on the input and previous tokens. section 4.1.1
- Continue from that choice without retaining alternatives; the result may miss a globally optimal or coherent sequence. section 4.1.1 section 4.3
- Unlike beam search, it does not maintain multiple candidate sequences; unlike sampling methods, it does not introduce stochastic token selection. section 4.1 section 4.3
Listing 4.1 Greedy search implementation
tokenizer = AutoTokenizer.from_pretrained(model_id,
use_auth_token=hf_token) #1
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
use_auth_token=hf_token
)
system_prompt = "You are a helpful assistant"
user_input = "Complete this sentence: In a world where AI has become
ubiquitous "
prompt = f"<|begin_of_text|><|start_header_id|>system<|
end_header_id|>{system_prompt}<|eot_id|><|start_header_id|>
user<|end_header_id|>
{user_input}<|eot_id|><|start_header_id|>assistant<|
end_header_id|>" #2
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)
greedy_output = model.generate(
input_ids=input_ids,
max_new_tokens=128,
do_sample=False,
pad_token_id=tokenizer.eos_token_id
)
The sentence-completion example configures generation with do_sample=False, a maximum token limit, and an end-of-sequence padding token. section 4.1.1
