A scheduler selects which runnable thread executes on a CPU. When several threads share one core, the OS gives them turns: they make progress concurrently without executing simultaneously. Different cores can run threads in parallel. Real schedulers use more complicated decisions than a fixed rotation, and a thread blocked on I/O may give up its turn early. section 9.2.1 section 7.5.1
Figure 9.2 The OS runs multiple threads at the same time (but not simultaneously) on a single CPU core by context-switching between them. In this example, every thread is run for exactly 4 milliseconds, and then the OS switches to another thread. In real-world systems, a thread might block to wait for I/O, for example, causing the OS to switch to another thread more quickly.
The three-thread timeline uses four-millisecond turns to make single-core scheduling visible; the fixed interval is illustrative, not a real scheduling rule. section 9.2.1
Scheduling matters to latency because a ready thread still waits until the OS runs it. A network request, for example, cannot be processed by its thread during that wait. Switching threads also has a cost: the OS saves one thread’s hardware state and restores another’s, while CPU state such as caches and page tables is affected. The direct cost can be on the order of microseconds. section 7.5.1
For a latency-sensitive application, consider the scheduler alongside the work threads do:
- Avoid running too many threads. Where appropriate, use thread affinity to restrict application threads to specific CPUs, or raise priority so the OS prefers them; neither removes scheduling decisions. section 7.5.1
- Watch lock holders. If a scheduled delay prolongs a lock hold, waiting threads can form a convoy, increasing tail latency and reducing throughput. Shorter critical sections and suitable scheduling policies can help, but strict latency requirements may call for wait-free synchronization. section 8.2.3
An OS thread scheduler is distinct from a userspace task scheduler. A task scheduler can choose which coroutine to resume after another yields, allowing useful work while asynchronous I/O is pending. section 9.2.3
