liveBook · Manning

DetectGPT

As explained in Introduction to Generative AI, Second Edition

DetectGPT is a zero-shot method that estimates whether text was generated by a language model by examining how model likelihoods change under perturbations.

DetectGPT is a zero-shot method for estimating whether language-model text was machine-generated without relying on labeled examples. It makes small changes, or perturbations, to a passage and observes how a language model scores those altered versions. Stable high scores may suggest generated text; greater variation may suggest human writing with more unexpected phrasing. section 6.9

Early experiments reported up to 95% accuracy across several open-source models, but later testing exposed a major limitation: performance fell when the scoring model differed from the model that generated the text. DetectGPT therefore provides evidence, not a dependable verdict, and should not serve as the sole basis for consequential decisions. section 6.9

How Introduction to Generative AI, Second Edition covers it

How deep it goes
Explains perturbations and model-score changes, but stops short of implementation details or a complete evaluation procedure.
Examples
Uses altered passages, open-source language models, and comparisons between scoring and generating models.
What it assumes
Defines zero-shot detection in context while assuming only a basic understanding of language-model text generation and scoring.
What you can do after
Evaluate detection claims while recognizing model mismatch, false confidence, and unreliable consequential use.
How it connects
Connects language-model detection with GLTR, classifier-based detection, and model generalization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Introduction to Generative AI, Second Edition. The section links point to the parts of the book it draws on.