chapter four

4 Improving reasoning with inference-time scaling

 

This chapter covers

  • Prompting an LLM to explain its reasoning to improve answer accuracy
  • Modifying the text generation function to produce diverse responses
  • Improving reasoning reliability by sampling multiple responses

Reasoning performance and answer accuracy can be improved without retraining or modifying the model itself. As shown in the overview in figure 4.1, this chapter covers two inference-time-scaling methods. These methods operate during inference, when the model generates text. As you’ll see later in this chapter, both methods more than double the accuracy of the base model used in previous chapters.

We’ll first look at what inference-time scaling is, before we discuss the inference methods shown in figure 4.1.

Figure 4.1 A model of the topics covered in this book. This chapter focuses on inference-time-scaling techniques, which improve reasoning without additional training (stage 3). In particular, it extends the text generation function and implements a voting-based method to improve answer accuracy. The next chapter will introduce another inference-time-scaling approach where the model iteratively refines its own answers.

4.1 Introduction to inference-time scaling

There are two main strategies to improve reasoning:

  • Increasing training compute
  • Increasing inference compute (also known as inference-time scaling or test-time scaling)

4.2 Loading a pretrained model

4.3 Generating better responses with chain-of-thought prompting

4.4 Controlling output diversity with temperature scaling

4.4.1 Understanding the process of selecting the next token

4.4.2 Rescaling token scores (logits) via a temperature parameter

4.4.3 Sampling the next token from a probability distribution

4.4.4 Adding temperature scaling to the text generation function

4.5 Balancing diversity and coherence with top-p sampling

4.5.1 Selecting a subset of top-p tokens

4.5.2 Adding a top-p filter to the text generation function

4.6 Improving response accuracy with self-consistency

Summary