liveBook · Manning

Heuristic

As explained in Designing AI Agents

A practical rule or strategy for proposing a solution without guaranteeing an optimal result.

A heuristic is a proposed practical lesson from a failure, not automatically trusted advice. An LLM can generate one from a recorded failure context, error, and fix; it remains a candidate until independent evaluation admits it to active recall. section 4.5.3

Keep failure records append-only and preserve provenance. Candidate retrieval can use a workload-supplied similarity threshold, but retrieved candidates are for review only. Coordinate journal and index writes in production, detecting partial failures. section 4.5.3

Listing 4.7 Failure Journal capture and candidate generation
from dataclasses import dataclass, field
import json
import time
 
@dataclass
class FailureEntry:
    context: str           # What the agent was trying to do
    error_type: str        # Exception class or error category
    error_message: str     # The actual error message
    fix: str              # What resolved the error
    candidate_heuristic: str  # Proposed lesson; not yet admitted
    tags: list[str]       # Semantic tags for retrieval
    timestamp: float = field(default_factory=time.time)
 
class FailureJournal:
    """Append-only failure evidence with candidate retrieval."""
 
    def __init__(self, llm, vector_db, journal_path: str):
        self.llm = llm
        self.vector_db = vector_db
        self.path = journal_path
 
    def record(self, context: str, error: str,
               fix: str) -> FailureEntry:  #A
        """Record failure evidence and propose a candidate lesson."""
        candidate_heuristic = self.llm.generate(
            f"Agent encountered this error:\n"
            f"Context: {context}\nError: {error}\nFix: {fix}\n\n"
            f"Propose a candidate heuristic, a possible lesson "
            f"for similar situations. One to two sentences."
        )
        entry = FailureEntry(
            context=context,
            error_type=self._classify(error),
            error_message=error, fix=fix,
            candidate_heuristic=candidate_heuristic,
            tags=self._auto_tag(context, error),
        )
        self._append(entry)  #B
        self.vector_db.upsert(
            text=f"{context} | {error} | {candidate_heuristic}",
            metadata={"fix": fix, "tags": entry.tags},
        )
        return entry

Shows a failure record containing context, error, fix, and a candidate heuristic that has not yet been admitted. section 4.5.3

Listing 4.8 Candidate consultation and helper methods
    def consult_candidates(self, current_context: str,
                min_score: float, k: int = 3):  #A
        """Return similar candidates for review, not admitted advice."""
        results = self.vector_db.search(current_context, top_k=k)
        return [self._to_entry(r) for r in results
                if r.score > min_score]
 
    def _append(self, entry: FailureEntry) -> None:
        """Append-only—never overwrite the journal."""
        with open(self.path, "a") as f:
            f.write(json.dumps({
                "context": entry.context,
                "error_type": entry.error_type,
                "error_message": entry.error_message,
                "fix": entry.fix,
                "candidate_heuristic": entry.candidate_heuristic,
                "tags": entry.tags,
                "timestamp": entry.timestamp,
            }) + "\n")
 
    def _classify(self, error: str) -> str:
        for prefix in ["TypeError", "ValueError", "ImportError",
                       "ModuleNotFoundError", "KeyError",
                       "FileNotFoundError", "TimeoutError"]:
            if prefix in error:
                return prefix
        return "UnclassifiedError"
 
    def _auto_tag(self, context: str, error: str) -> list[str]:
        return self.llm.generate(
            f"Generate 3-5 short tags for:\n"
            f"Context: {context}\nError: {error}\n"
            f"Comma-separated tags only."
        ).split(",")
 
    def _to_entry(self, result) -> FailureEntry:
        parts = result.text.split(" | ")
        return FailureEntry(
            context=parts[0] if len(parts) > 0 else "",
            error_type="retrieved",
            error_message=parts[1] if len(parts) > 1 else "",
            fix=result.metadata.get("fix", ""),
            candidate_heuristic=parts[-1],
            tags=result.metadata.get("tags", []),
        )

Shows similar candidates being returned for review, with results filtered by a similarity threshold. section 4.5.3

How Designing AI Agents covers it

How deep it goes
Implements candidate generation and retrieval, while leaving admission to independent evaluation.
Examples
Records failure context, error, fix, and candidate lessons, then retrieves similar candidates for review.
What you can do after
Evaluate candidate heuristics against admission evidence before allowing them into active recall.
How it connects
Connects heuristics to agent memory, semantic retrieval, and independent evaluation.

Also explained in

Compare how 12 books explain heuristic →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Agents. The section links point to the parts of the book it draws on.