liveBook · Manning

Metacognition

As explained in AI Agents in Action, Second Edition

Metacognitive monitoring is the ongoing assessment of an agent's own knowledge and confidence while producing answers or taking actions.

An agent may produce a confident answer even when its evidence is incomplete. Metacognition is the ability to monitor its own thinking while acting: to recognize when it is confident, guessing, stuck, or outside its knowledge. Cognition concerns understanding the problem; metacognition concerns assessing that understanding. section 10.1.1 section 10.1.4

Three monitoring capabilities help an agent adjust its behavior:

  • Confidence calibration: assess whether confidence signals correlate with correctness. A verbal claim of confidence alone is not enough; signals can include token probabilities, retrieval scores, and evidence consistency. section 10.1.4
  • Stagnation detection: notice when repeated attempts are not producing new findings, then pivot instead of retrying until an iteration limit. section 10.1.4
  • Knowledge boundary awareness: recognize when available knowledge and retrieval are insufficient, and degrade gracefully rather than commit to a guess. section 10.3.6

Monitoring is especially useful when a request is ambiguous, evidence conflicts, or information is missing. In a cognitive agent architecture, an evaluation module monitors reasoning, while shared workspace state supports decisions about what to do next. section 10.1.1 section 10.6

Listing 10.11 12_boundary_awareness.py
class KnowledgeBoundary(str, Enum):
    WITHIN = "within_knowledge"
    EDGE = "edge_of_knowledge"
    OUTSIDE = "outside_knowledge"

def assess_knowledge_boundary(
    workspace: CognitiveWorkspace
) -> KnowledgeBoundary:
    signals = []

    # Check retrieval quality
    if workspace.findings:
        avg_relevance = sum(
            f.relevance_score for f in workspace.findings
        ) / len(workspace.findings)
        signals.append(avg_relevance)
    else:
        signals.append(0.0)                         #1

    # Check memory coverage
    if workspace.memory_hits:
        signals.append(0.8)                         #2
    else:
        signals.append(0.2)

    # Check confidence state
    signals.append(workspace.confidence)

    avg_signal = sum(signals) / len(signals)

    if avg_signal > 0.6:
        return KnowledgeBoundary.WITHIN
    elif avg_signal > 0.3:
        return KnowledgeBoundary.EDGE
    else:
        workspace.active_signal = AttentionSignal.LOW_CONFIDENCE
        return KnowledgeBoundary.OUTSIDE

Inspect a knowledge-boundary assessment that combines retrieval relevance, memory coverage, and workspace confidence into a boundary signal. section 10.3.6

How AI Agents in Action, Second Edition covers it

How deep it goes
Defines three monitoring capabilities and illustrates knowledge-boundary assessment with workspace signals.
Examples
Illustrates stagnation through repeated unchanged findings and knowledge-boundary assessment through retrieval relevance, memory coverage, and confidence.
What you can do after
Evaluate agent behavior for overconfident guesses, stalled progress, and insufficient knowledge.
How it connects
Connects metacognition to cognition, confidence calibration, reasoning, and a shared workspace.

Also explained in

Compare how 2 books explain metacognition →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Agents in Action, Second Edition. The section links point to the parts of the book it draws on.