liveBook · Manning

Simultaneous multithreading

As explained in Quantization and Fast Inference

SMT is a processor capability that allows multiple hardware threads to share the resources of one physical core.

Simultaneous multithreading (SMT) lets a physical CPU core host multiple hardware threads, called siblings. For CPU-bound model decode, it makes thread count a deployment choice: SMT siblings may add throughput, but can also contend for a core’s private caches. section 10.5 section 10.5.2

Tune thread count against the actual host rather than relying on the physical-core rule:

  • Benchmark thread counts across physical cores and SMT siblings, using repeated runs; throughput can vary between shared-cloud VMs. section 10.5 section 10.5.2
  • Watch for a dip when the first sibling is activated: on the measured 11-core host, 12 threads ran two threads on one core, increasing L1/L2 contention. section 10.5.2

Measured decode throughput

On one c3-standard-22 VM, throughput rose from 16.62 tok/s at 11 threads to 21.02 tok/s at 22, a 26% gain. The 12-thread result dipped 4.9% below the 11-thread result. section 10.5.2

Figure 10.12 Decode throughput on c3-standard-22 (Sapphire Rapids, 11 physical cores × 2 SMT) versus llama.cpp -t thread count, AVX-512+VNNI build, Q4_K_M Llama-2-7B, n=64 token generation, mean over 5 repetitions. Throughput climbs through the full vCPU count rather than peaking at the physical-core count: 16.62 tok/s at t=11 lifts to 21.02 tok/s at t=22, a 26% gain that the kernel extracts by recruiting SMT siblings. A local minimum at t=12 (15.81 tok/s, 4.9% below t=11) marks the first SMT-sibling activation under L1/L2 contention; throughput recovers past t=14 once enough cores share that pattern. The dashed line at t=11 marks the physical core count; the dashed line at t=22 marks the full vCPU count.

Compare the full thread-count curve, including the dip at 12 threads and the gain through 22 SMT threads. section 10.5.2

How Quantization and Fast Inference covers it

How deep it goes
Explains sibling-thread cache contention and measured throughput changes, without detailing processor implementation.
Examples
Measures llama.cpp decode from 11 to 22 threads, including a 12-thread dip on an 11-core VM.
What you can do after
Operate/configure inference thread counts by benchmarking SMT siblings instead of assuming physical-core count is optimal.
How it connects
Connects SMT siblings with physical cores, memory bandwidth, and L1/L2 cache contention.

Also explained in

Compare how 3 books explain simultaneous multithreading →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.