liveBook · Manning

Thread-count sweep

As explained in Quantization and Fast Inference

A thread-count sweep measures a workload across multiple configured thread counts to identify a suitable setting.

Measure decode throughput at multiple configured thread counts to find a suitable setting; do not assume the physical-core count is optimal. section 10.5.2

SMT can change the result

On an 11-core c3-standard-22 VM, throughput rose from 16.62 tok/s at 11 threads to 21.02 tok/s at 22, a 26% gain. It dipped 4.9% at 12 threads. Absolute rates vary across VMs, so treat these figures as an example, not a portable benchmark. section 10.5.2

Figure 10.12 Decode throughput on c3-standard-22 (Sapphire Rapids, 11 physical cores × 2 SMT) versus llama.cpp -t thread count, AVX-512+VNNI build, Q4_K_M Llama-2-7B, n=64 token generation, mean over 5 repetitions. Throughput climbs through the full vCPU count rather than peaking at the physical-core count: 16.62 tok/s at t=11 lifts to 21.02 tok/s at t=22, a 26% gain that the kernel extracts by recruiting SMT siblings. A local minimum at t=12 (15.81 tok/s, 4.9% below t=11) marks the first SMT-sibling activation under L1/L2 contention; throughput recovers past t=14 once enough cores share that pattern. The dashed line at t=11 marks the physical core count; the dashed line at t=22 marks the full vCPU count.

Compare decode throughput across the thread counts, including the dip at 12 and the rise through the full vCPU count. section 10.5.2

How Quantization and Fast Inference covers it

How deep it goes
Interprets repeated throughput measurements and SMT-related curve changes, while warning that absolute rates are VM-specific.
Examples
Compares 11- and 22-thread throughput on an 11-core VM and notes a dip at 12 threads.
What you can do after
Diagnose thread-count performance by comparing decode-throughput measurements across configured counts.
How it connects
Relates thread count to SMT, memory bandwidth, cache contention, and scheduling overhead.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.