Computes the running total of sorted token probabilities so top-p filtering can find the smallest acceptable token set. section 4.5.1
Use it in the filtering workflow to:
- sort token probabilities from highest to lowest before accumulation. section 4.5.1
- accumulate each probability into a stepwise total with torch.cumsum. section 4.5.1
- compare the cumulative total against top_p to decide which tokens remain. section 4.5.1
Top-p cutoff
With top_p = 0.8, the cumulative values 0.4538 and 0.7290 stay under the threshold, while 0.8119 exceeds it, so two tokens are kept.
Listing 4.10 Computing the cumulative probability sum
sorted_probas, sorted_idx = torch.sort(toy_probas, descending=True) #1
cumsum = torch.cumsum(sorted_probas, dim=-1) #2
plt.bar(
torch.arange(len(sorted_probas)), sorted_probas,
alpha=0.5
)
plt.step(
torch.arange(len(cumsum)), cumsum,
where="mid", color="C1", label="Cumulative sum"
)
plt.ylim([0, 1])
plt.xlabel("Token rank (sorted by probability)")
plt.ylabel("Probability")
plt.show()
Shows the exact cumsum step on sorted probabilities and the matching cumulative-sum plot. section 4.5.1
