When an AI system produces a harmful or surprising answer, people need more than the answer itself: they need some way to understand how it arose. Interpretability is the degree to which people can make sense of a system’s outputs. It supports efforts to find mistakes, assess bias, and hold systems accountable, but an explanation should not be mistaken for a complete account of a complex model. section 10.19
For a large language model, one starting point is to inspect which parts of the input it attended to while generating a message. Explaining the result remains difficult: the size of these models makes many existing explanation methods impractical to run. Researchers are also looking for identifiable features inside models that might permit more targeted changes, including more precise machine unlearning. That possibility is not the same as reliable, granular forgetting today. section 10.19 section 3.9
A feature that changed the answer
Researchers identified a feature associated with the Golden Gate Bridge in Claude 3 Sonnet. When they amplified its contribution, the model mentioned the bridge in responses where it was irrelevant. The experiment makes one internal influence visible; it does not establish that every output can be explained or controlled this way. section 3.9
Interpretability matters beyond research. As AI agents take on tasks with less supervision, users can become too willing to trust past performance. Confidence scores, visible intermediate steps, and approval requirements for sensitive actions can support oversight, though increasing autonomy makes that oversight harder. Governance also needs auditability and meaningful ways for affected people to challenge AI-generated outcomes—requirements that remain uneven in practice. section 8.23 section 3.18
