Simultaneous multithreading (SMT) lets a physical CPU core host multiple hardware threads, called siblings. For CPU-bound model decode, it makes thread count a deployment choice: SMT siblings may add throughput, but can also contend for a core’s private caches. section 10.5 section 10.5.2
Tune thread count against the actual host rather than relying on the physical-core rule:
- Benchmark thread counts across physical cores and SMT siblings, using repeated runs; throughput can vary between shared-cloud VMs. section 10.5 section 10.5.2
- Watch for a dip when the first sibling is activated: on the measured 11-core host, 12 threads ran two threads on one core, increasing L1/L2 contention. section 10.5.2
Measured decode throughput
On one c3-standard-22 VM, throughput rose from 16.62 tok/s at 11 threads to 21.02 tok/s at 22, a 26% gain. The 12-thread result dipped 4.9% below the 11-thread result. section 10.5.2
Figure 10.12 Decode throughput on c3-standard-22 (Sapphire Rapids, 11 physical cores × 2 SMT) versus llama.cpp -t thread count, AVX-512+VNNI build, Q4_K_M Llama-2-7B, n=64 token generation, mean over 5 repetitions. Throughput climbs through the full vCPU count rather than peaking at the physical-core count: 16.62 tok/s at t=11 lifts to 21.02 tok/s at t=22, a 26% gain that the kernel extracts by recruiting SMT siblings. A local minimum at t=12 (15.81 tok/s, 4.9% below t=11) marks the first SMT-sibling activation under L1/L2 contention; throughput recovers past t=14 once enough cores share that pattern. The dashed line at t=11 marks the physical core count; the dashed line at t=22 marks the full vCPU count.

Compare the full thread-count curve, including the dip at 12 threads and the gain through 22 SMT threads. section 10.5.2


