liveBook · Manning

Cognitive attack surface

As explained in Build High Value Software Systems

The set of ways that AI-generated or AI-mediated information can introduce uncertainty, inconsistency, or harmful influence into human reasoning and software decisions.

A cognitive attack surface is the risk that AI-generated code weakens developers’ understanding of the software they maintain. Unlike a library dependency, AI-written code can look like familiar project code while hiding unfamiliar decisions. section 4.3.3

Ununderstood code is harder to debug, modify, and assess for failure modes. Its plausible micro-decisions can accumulate into cognitive debt and inconsistent patterns across features. section 4.3.3 section 4.6

A caching change

An AI-generated cache for an API endpoint contains a subtle race condition. When production slows weeks later, debugging takes hours because the developer must first understand code and decisions they did not make. section 4.3.3

Reduce the risk by making expectations explicit:

  • Use specifications and standards to constrain AI-generated choices. section 4.6
  • Review changes against standards, waste, and value; block merges for objective violations. section 4.6
Figure 4.6 Without defined standards, each AI-generated feature adopts different patterns, creating a maintenance burden and technical debt.

Shows how AI-generated features can adopt different patterns, increasing maintenance burden and technical debt. section 4.3.3

How Build High Value Software Systems covers it

How deep it goes
Traces hidden implementation decisions to debugging difficulty, cognitive debt, and inconsistent patterns.
Examples
Uses an API caching race condition and inconsistent patterns across AI-generated features.
What you can do after
Apply specifications, standards, and objective review criteria to constrain and assess AI-generated changes.
How it connects
Links AI dependencies to technical debt, software standards, specifications, and code review.

This explainer was generated by Manning's knowledge graph from the text of Build High Value Software Systems. The section links point to the parts of the book it draws on.