Logit_bias

As explained in Generative AI in Action

Logit bias is a per-token generation setting that adds an offset to model logits before they are converted into probabilities.

Logit bias steers generated text by adding an offset to selected token logits before they become probabilities. Negative values suppress tokens, while positive values favor them; values from -100 to 100 provide strong control. section 3.3.2

Apply logit bias by:

• Use tiktoken to identify every relevant token variant, including capitalization and preceding-space forms. section 3.3.2
• Pass a dictionary mapping token IDs to bias values in the completion request. section 3.3.2
• Use -100 to effectively ban a token or 100 to make it exclusive, recognizing that smaller changes produce softer steering. section 3.3.2

Constraining pet salon names

A completion request can assign -100 to token IDs representing “purr,” “purrs,” and “meow,” then add a positive bias of 5 to selected “furry” tokens. The resulting suggestions avoid the banned terms and favor the targeted vocabulary.

Figure 3.2 The logit_bias parameter

Use the figure to inspect how changing token bias alters the probability distribution available during generation. section 3.3.2

How Generative AI in Action covers it

How deep it goes

Explains logit adjustment, token identification, bias ranges, and API usage without deriving the underlying mathematics.

Examples

Demonstrates banned pet-salon terms, token variants, negatively biased requests, and positively favored generated names.

What it assumes

Assumes familiarity with completion requests and introduces token IDs through the tiktoken library.

What you can do after

Apply it via a completion API to suppress or favor selected token sequences.

How it connects

Connects logit_bias with completion APIs, temperature, presence penalty, frequency penalty, and content safety filters.

liveBookmanning.com

Read Generative AI in Action on liveBook Get the book at manning.com

Related concepts

• parameterkind of · A model parameter is a numerical value, such as a weight or bias, that is learned or set to determine how a model computes outputs.
• generationconfigures · Generation is the computational production of outputs from supplied inputs or context, often by applying a model to produce predictions or new content.
• token iduses · A token ID is a numeric identifier assigned to a token in a vocabulary so it can be represented and processed numerically.
• tiktokenuses · tiktoken is an OpenAI-developed library that efficiently encodes text using model-specific byte-pair-encoding tokenizers and supports token counting.
• logitconfigures · A logit is a real-valued quantity representing an outcome’s log-odds or, in machine learning, an unnormalized model score used to derive probabilities.
• outputconfigures · An output is a result or response produced by a process, operation, or system, including data, content, actions, or signals.

This explainer was generated by Manning's knowledge graph from the text of Generative AI in Action. The section links point to the parts of the book it draws on.