liveBook · Manning

Logit

As explained in Timeless Algorithms, The Foundational Ideas

A logit is an unnormalized real-valued score that can be transformed into a probability.

A logit is an unnormalized numerical score produced before a function converts it into a probability or normalized weight. A logit can be positive or negative; it is not itself a probability. Its meaning depends on what the model is scoring. section 11.3.2 section 12.2.1

In binary classification, an output node can calculate a logit z = v₁h₁ + v₂h₂ + b, where h₁ and h₂ are hidden-node outputs, v₁ and v₂ are weights, and b is a bias. The sigmoid function σ(z) = 1/(1 + e⁻ᶻ) converts that score into a probability. A separate decision threshold can then turn the probability into an action. section 11.3.2

Pedestrian detection

With hidden outputs 0.75 and 0.52, weights 1.2 and −0.7, and bias 0.2, the output logit is approximately 0.74. Sigmoid converts it to approximately 0.68. With a decision threshold of 0.30, the positive classification triggers braking; calculating the prediction has not yet changed the model’s parameters. section 11.3.2

In self-attention, compatibility logits are scaled query–key dot products: sᵢ = (q · kᵢ)/√dₖ, where dₖ is the key-vector dimension. Scaling limits excessively large scores. Softmax converts the scores into weights: αᵢ = eˢⁱ/Σⱼeˢʲ. The weights sum to one and determine how much each value contributes to the resulting representation. section 12.2.1

Attention and vocabulary scores

For a pronoun query, scaled logits of 0.14 for “trophy” and 0.57 for “box” become attention weights of approximately 0.39 and 0.61. The larger weight indicates stronger compatibility, not a guarantee of correct interpretation. Separately, a language model produces logits over vocabulary candidates from its final prompt representation; softmax turns those output logits into probabilities of possible continuations. section 12.2.1 section 12.2.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Works through weighted output scores, scaled attention scores, and their conversion using sigmoid or softmax.
Examples
Calculates a pedestrian-detection score and compares pronoun attention scores for a trophy and a box.
What you can do after
Evaluate and understand how scores become probabilities or attention weights before a decision or representation is formed.
How it connects
Connects logits to sigmoid, softmax, binary classification, self-attention, and language modeling.

Also explained in

Compare how 24 books explain logit →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.