Thread count sets how many threads llama.cpp uses for CPU inference via -t. One thread per physical core is a useful starting heuristic, but cloud CPU performance can favor using SMT siblings too. section 10.5 section 10.5.2
Tune the setting against the actual deployment:
- Sweep -t for each model, format, and hardware combination; start with -t $(nproc), then choose from measured throughput. section 10.5.2
- On the measured c3-standard-22, Q4_K_M Llama-2-7B decode rose from 16.62 tok/s at 11 threads to 21.02 tok/s at 22, a 26% gain. Throughput dipped at 12 threads before recovering past 14. section 10.5.2
- Avoid pinning threads at high counts without evidence: on this host, pinning cost 25% by restricting scheduler balancing. section 10.5.2
Figure 10.12 Decode throughput on c3-standard-22 (Sapphire Rapids, 11 physical cores × 2 SMT) versus llama.cpp -t thread count, AVX-512+VNNI build, Q4_K_M Llama-2-7B, n=64 token generation, mean over 5 repetitions. Throughput climbs through the full vCPU count rather than peaking at the physical-core count: 16.62 tok/s at t=11 lifts to 21.02 tok/s at t=22, a 26% gain that the kernel extracts by recruiting SMT siblings. A local minimum at t=12 (15.81 tok/s, 4.9% below t=11) marks the first SMT-sibling activation under L1/L2 contention; throughput recovers past t=14 once enough cores share that pattern. The dashed line at t=11 marks the physical core count; the dashed line at t=22 marks the full vCPU count.

Compare decode throughput across thread counts, including the dip at 12 and the measured gain at 22. section 10.5.2
