A logit is an unconstrained real-valued score that can be transformed into a probability with softmax. In GPT-2, logits score every vocabulary token at each input position before the next token is selected. section 10.2.1
When inspecting a language model, logits reveal:
- The output tensor’s dimensions: batch, sequence position, and vocabulary size. section 10.2.1
- The final-position scores used to compute next-token probabilities with torch.softmax. section 10.2.1
- The highest-ranked candidate tokens, such as ' new' and ' non' for 'NLP is a'. section 10.2.1
Inspecting GPT-2 logits
Calling GPT-2 directly returns output.logits. For an input with three tokens, output.logits.shape is [1, 3, 50257]; selecting output.logits[0, -1, :] isolates the scores for the next token. section 10.2.1
>>> output = vanilla_gpt2(input_ids=input_ids) >>> output.logits.shape ([1, 3, 50257])
Use the listing to inspect the batch, sequence, and vocabulary dimensions of a GPT-2 logit tensor. section 10.2.1
