GPT-2 made zero-shot ability hard to dismiss. The transformer language model could generate a news article from a headline or answer a reading-comprehension question without task-specific training. In a zero-shot setting, a model receives a task or query without worked examples in the input. Its response draws on what it learned during training rather than on a demonstration supplied with that query. section 1.2
That ability does not guarantee reliable answers to every phrasing of a question. GPT-4 can name Tom Cruise’s mother when asked directly, yet fail to identify Tom Cruise when asked whose mother Mary Lee Pfeiffer is. The reverse query is brittle without a hint, even though the forward association is available. Put the forward fact in the prompt first, and the model can answer the inverse: the test has shifted from zero-shot querying to in-context learning. section 7.5.2
Figure 7.8 GPT-4 correctly gives the name of Tom Cruise’s mother (left). But when prompted with her name, it fails to retrieve “Tom Cruise” (right). Models trained on “A is B” (for example, “Tom Cruise’s mother is Mary Lee Pfeiffer”) do not automatically infer that “B is A.”
The paired queries make the distinction visible: retrieving an association in one direction does not ensure zero-shot retrieval in the other. section 7.5.2
