liveBook · Manning

Thread count

As explained in Quantization and Fast Inference

Thread count is the configured number of concurrent execution threads assigned to a computation or runtime.

Thread count sets how many threads llama.cpp uses for CPU inference via -t. One thread per physical core is a useful starting heuristic, but cloud CPU performance can favor using SMT siblings too. section 10.5 section 10.5.2

Tune the setting against the actual deployment:

  • Sweep -t for each model, format, and hardware combination; start with -t $(nproc), then choose from measured throughput. section 10.5.2
  • On the measured c3-standard-22, Q4_K_M Llama-2-7B decode rose from 16.62 tok/s at 11 threads to 21.02 tok/s at 22, a 26% gain. Throughput dipped at 12 threads before recovering past 14. section 10.5.2
  • Avoid pinning threads at high counts without evidence: on this host, pinning cost 25% by restricting scheduler balancing. section 10.5.2
Figure 10.12 Decode throughput on c3-standard-22 (Sapphire Rapids, 11 physical cores × 2 SMT) versus llama.cpp -t thread count, AVX-512+VNNI build, Q4_K_M Llama-2-7B, n=64 token generation, mean over 5 repetitions. Throughput climbs through the full vCPU count rather than peaking at the physical-core count: 16.62 tok/s at t=11 lifts to 21.02 tok/s at t=22, a 26% gain that the kernel extracts by recruiting SMT siblings. A local minimum at t=12 (15.81 tok/s, 4.9% below t=11) marks the first SMT-sibling activation under L1/L2 contention; throughput recovers past t=14 once enough cores share that pattern. The dashed line at t=11 marks the physical core count; the dashed line at t=22 marks the full vCPU count.

Compare decode throughput across thread counts, including the dip at 12 and the measured gain at 22. section 10.5.2

How Quantization and Fast Inference covers it

How deep it goes
Measures thread-count sweeps and explains SMT and scheduler effects, without establishing a universal optimum.
Examples
Compares 11 and 22 threads on c3-standard-22, measuring 16.62 and 21.02 tok/s respectively.
What it assumes
Assumes familiarity with physical cores, SMT siblings, and vCPUs when interpreting thread-count measurements.
What you can do after
Evaluate thread-count settings by sweeping -t and comparing measured throughput before deployment.
How it connects
Connects thread count to SMT, memory bandwidth, Linux scheduling, and decode throughput.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.