Top-k sampling generates each token by restricting selection to the k most probable next tokens. It introduces controlled randomness while excluding very unlikely choices that could derail a response. section 12.1.1
Use the parameter to adjust determinism and variety:
- Set top_k=1 for deterministic output. section 12.1.1
- Set top_k=40 to balance variety and quality. section 12.1.1
- Compare top-k with temperature and top-p when matching randomness to the application. section 12.1.1
Output-randomness configuration
A configuration can test top_k=1, top_k=40, and top_k=100 alongside temperature and top_p settings. Lower k narrows the candidate vocabulary; higher k permits more alternatives. section 12.1.1
