liveBook · Manning

Percentile

As explained in Rearchitecting LLMs

A percentile is a value below which a specified percentage of observations fall in an ordered distribution.

A percentile is a cutoff in an ordered distribution: the specified percentage of observations fall below it. Percentiles describe a distribution’s spread and tail without relying on an average. section 10.2.1 section 12.9

Percentiles serve two practical purposes here:

  • Set a heatmap color bound using the 99th percentile of absolute activations, reducing the influence of extreme values. section 10.2.1
  • Report request latency at p50, p95, and p99 to show typical performance and the slow tail. section 12.9

Latency and activation scale

For a hypothetical latency report, p95 is 62 ms for the self-hosted SLM and 620 ms for the API. In a separate visualization, the 99th percentile of absolute activations sets the heatmap’s color bound. section 10.2.1 section 12.9

Listing 10.4 Auxiliary functions for tokenization, binning and rendering
def get_token_labels(tokenizer, prompt):
    input_ids = tokenizer(prompt, return_tensors="pt")["input_ids"][0]
    start = 1 if input_ids[
        0] == tokenizer.bos_token_id else 0 #A
    labels = [
        tokenizer.convert_ids_to_tokens(
        int(t)).replace("▁", "").replace("Ġ", "") #B
        for t in input_ids[start:]
    ]
    return labels, start
 
def bin_neurons(tensor, bin_size):
    n_rows, n_neurons = tensor.shape
    n_bins = n_neurons // bin_size
    trimmed = tensor[:, : n_bins * bin_size] #C
    return trimmed.reshape(
        n_rows, 
        n_bins, 
        bin_size).mean(axis=2) #1
 
def plot_heatmap_panel(ax, matrix, bin_size, cmap, vmax_percentile, title, y_labels=None):
    abs_bound = float(np.percentile(np.abs(matrix), 
                        vmax_percentile)) #2
    norm = TwoSlopeNorm(vmin=-abs_bound, 
                        vcenter=0.0, 
                       vmax=abs_bound) #D
 
    im = ax.imshow(matrix, aspect="auto", 
            cmap=cmap, 
            norm=norm, interpolation="nearest")
 
    n_bins = matrix.shape[1]
    xtick_bins = np.linspace(0, 
            n_bins - 1, min(9, n_bins)).astype(int)
    ax.set_xticks(xtick_bins)
    
    ax.set_xticklabels(
            [str(int(b * bin_size)) for b in xtick_bins], 
                 fontsize=8, rotation=30)
    
    ax.set_xlabel("Neuron index", fontsize=10)
 
    if y_labels is not None:
        ax.set_yticks(range(len(y_labels)))
        ax.set_yticklabels(y_labels, fontsize=8)
 
    fig = ax.get_figure()
    cbar = fig.colorbar(im, ax=ax, fraction=0.03, pad=0.02)
    cbar.set_label(
        f"[-{abs_bound:.3f}, +{abs_bound:.3f}] (p{int(vmax_percentile)})", 
        fontsize=8)
 
    ax.set_title(title, fontsize=11, fontweight="bold")
    return im

See the auxiliary function that computes a percentile-based bound for the heatmap color scale. section 10.2.1

How Rearchitecting LLMs covers it

How deep it goes
Defines a percentile as an ordered-distribution cutoff and applies it to activation scaling and latency reporting.
Examples
Uses the 99th percentile for activation color scaling and p50, p95, and p99 for latency comparisons.
What you can do after
Evaluate activation heatmap scales and compare request latency across the distribution rather than by average alone.
How it connects
Connects percentiles to activation heatmaps and request-latency distributions.

Also explained in

Compare how 13 books explain percentile →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.