At each generation step, greedy decoding chooses the token with the highest predicted probability. It is straightforward, but repeatedly choosing the most likely token can trap generation in high-probability loops. section 16.2.4
def greedy_search(preds): return ops.argmax(preds) compiled_generate(prompt, greedy_search)
Use argmax to select the most likely token, then pass that strategy into the generation loop. section 16.2.4
Greedy decoding does not consider future alternatives. Random sampling and top-K sampling offer other ways to choose tokens from the model’s predictions. section 16.2.4 section 16.6
