Chapter 4: Passive reconnaissance agents
This chapter introduction frames passive reconnaissance as an auditable, human-supervised pipeline that records safe evidence and prepares the reader for later enumeration and scoring work.
After this chapter you will be able to
- Auditable reconnaissance pipelines. Explain why a repeatable reconnaissance pipeline is preferable to scattered manual lookups, command-line runs, and one-off scripts. Describe how explicit stages and structured artifacts make the operation reviewable and reconstructable.
- Human-supervised AI assistance. Describe AI's intended role in the recon workflow as supporting professional judgment through suggestions, explanations, and summaries rather than replacing human decision-making. State that humans supervise execution and keep the agent within defined boundaries through safety gates.
- Passive and active reconnaissance. Distinguish passive reconnaissance from active reconnaissance by describing whether the target system is directly contacted, what kinds of sources or actions are involved, and the resulting visibility and authorization risks.
- Artifact-driven recon pipeline. Describe the AI recon agent as a conveyor-belt pipeline of small, well-defined stages that pass outputs forward and record persistent artifacts, enabling review, recovery, and traceability of commands, parameters, and timing.
- Human-controlled safety gates. Explain how pipeline gates enforce safety and accountability through controls such as time windows, approval checks, and scope filters, while allowing the AI to propose actions but reserving execution decisions for humans.
- Minimum viable recon pipeline. Describe the minimal AI recon agent as a minimum viable pipeline, including that the section frames it by deciding what belongs inside its scope and what is deliberately excluded.
- Scripts and pipeline structure. Distinguish a script from a pipeline, including how scripts serve as building blocks that a pipeline orchestrates in sequence.
- Pipeline workflow capabilities. Describe the structure a pipeline adds around recon logic: a stage input interface, structured JSON artifacts with context and timestamps, safety gates, and the ability to pause, review, or replay without losing context.
- Pipeline and agent autonomy. Distinguish a human-intervened, brittle pipeline from an AI agent that responds to environmental changes by reasoning, choosing a course of action, and invoking external tools without requiring a script.
- ReAct reconnaissance workflow. Describe how planning, acting, analyzing, and recording findings work together in a ReAct reconnaissance agent. Explain how planning uses objectives, current knowledge, scope boundaries, hypotheses, and operational considerations to choose a reconnaissance strategy.
- Persistent reconnaissance artifacts. Explain why model context and insights from a reconnaissance session are ephemeral and how an artifact preserves essential findings. Describe artifacts such as JSON files or vector databases as stored model outputs and context that can be retrieved in future sessions or used for model training.
- Reconnaissance activity boundaries. Distinguish truly passive reconnaissance from non-intrusive active reconnaissance and intrusive active WAF fingerprinting. State that HTTPS probing and reading response headers for WAF hints send requests to the target and are therefore active, even when they avoid payloads, defense triggering, and operational impact.
- Reconnaissance artifact contents. Describe reconnaissance artifacts as persistent, tangible outputs that store findings about an explored host, network, or system, including information such as host details, target address, and detected headers.
- JSONL recon artifact format. Explain why the section recommends storing reconnaissance findings in machine-readable JSONL files and identify recon.jsonl as the recon agent output artifact referenced throughout the book.
- Artifact-driven follow-on operations. Trace how stored JSONL artifacts enable monitoring systems to merge and search findings, support replay of agent actions for debugging and simulation, and supply later AI agents for summarization, prioritization, and reporting.
- Safety gate purpose. Explain that safety gates are decision checkpoints and control mechanisms that keep reconnaissance agents operating responsibly before they make network calls.
- Pre-execution gate pair. State the two required pre-execution gates: a time-window gate that permits runs only during approved hours, and an approval-or-automation gate that confirms intent before execution.
- Stage-to-artifact progression. Describe how functions correspond directly to agentic stages and how artifacts are appended as the agent progresses.
- Minimal recon pipeline foundation. Describe the program introduced in this section as a single self-contained pipeline that executes each reconnaissance-agent stage in sequence and saves the results as structured artifacts.
- Sandbox-only safety status. State that this minimal recon agent is a foundation rather than a production-ready system, and explain that it should be used for reconnaissance only in a sandbox until later components make the system safer and auditable.
- Python virtual environment isolation. Explain that a Python virtual environment is a self-contained copy of Python and the packages installed for a project, and state why activating it comes first for these exercises.
- Artifact reconnaissance record. Describe the Artifact class as a dataclass that stores structured reconnaissance information about a target host, including DNS data, HTTP metadata, TLS details, and general notes. Explain that each artifact is an auditable record of what was done and what was found.
- Append-only JSONL artifact output. Describe how write_jsonl appends artifacts to a configurable output file such as recon.jsonl as newline-delimited JSON, with one JSON object per line. Explain why this portable, serializable format supports incremental streaming and later processing with standard tools or AI agents.
- Reconnaissance pipeline stages. Describe how the pipeline moves from candidate-host generation and DNS resolution through header probing, WAF/CDN inference, optional TLS and content inspection, sanitization, and artifact recording.
- Low-impact web and TLS evidence. Describe the evidence gathered from a lightweight HTTPS HEAD request, TLS handshake, and optional robots.txt or sitemap retrieval, including why HEAD avoids downloading a response body and what ALPN and certificate SANs can reveal.
- Orchestration and composable artifacts. Explain how main() parses arguments, loops over candidate hosts, invokes the stateless probe functions, and writes one machine-readable JSONL artifact per host; distinguish the default lightweight scan from optional --tls and --content stages, and explain why this structure can be used by AI agents.
- Reconnaissance scope classification. Distinguish this pipeline's non-intrusive intent from truly passive reconnaissance: it avoids payloads, vulnerability probing, and authentication, but DNS, HEAD, and TLS interactions still reach the target and therefore sit at the low-impact end of active reconnaissance.
- Passive reconnaissance discipline. Explain why the pipeline begins with passive reconnaissance and describe the operational discipline it requires: one request per host, timeouts, complete recording, and remaining within scope.
- Recon JSONL artifact. Describe recon.jsonl as the new file produced in the project root after a successful run and explain why the subsection calls it the heartbeat of the pipeline: it records what the pipeline did.
- Append-only artifact history. Explain that repeated pipeline runs append new artifact lines rather than overwriting existing data, producing a chronological record.
- Recon artifact field meanings. Explain the anatomy of a recon-v1 artifact: its schema and host identify the record and tested fully qualified hostname, while DNS, HTTP header, WAF, TLS, notes, and timestamp fields preserve the observed reconnaissance context.
- Recorded context for AI reasoning. Explain why the section preserves these exact reconnaissance logs: later AI agents can reason over recorded context rather than guess the circumstances of each observation.
- Reconnaissance pattern queries. Describe the three quick queries used to inspect reconnaissance results: finding hosts with HTTP 200 responses, finding hosts whose waf_hint is true, and extracting unique header keys across hosts.
- Interpretation of scan patterns. Explain how the query output guides next steps: mostly empty 200 results can indicate that the seed list needs broader expansion, many responsive candidates indicate a rich attack surface, and WAF/CDN hints mean active testing may need a different approach.
- Artifacts as reasoning episodes. Explain how each artifact represents a mini-episode in the pipeline's reasoning process and how reading artifacts in chronological order reveals the system's discovery trail.
- Structured artifact reasoning. Explain why consistent JSONL fields enable an AI agent to reason across reconnaissance observations and make inferences beyond the pipeline's explicitly programmed checks.
- Consent boundaries for reconnaissance. Explain why every reconnaissance pipeline must operate within a clear boundary of consent, including the difference between authorized reconnaissance and activity that violates privacy or disrupts services.
- Configured operational scope. Describe how a pipeline's permitted targets can be defined before it runs using a scope.json file or an environment variable listing allowed domains, IP ranges, or network zones.
- Automatic scope enforcement. Explain that an AI in a larger system can check configured scope before acting, automatically deny a proposed out-of-scope target, and record that denial in an artifact.
- Human approval gates. Explain why early AI reconnaissance pipelines should pause for human permission before actions leave the machine, and describe each gate as a small ethics check.
- Recon artifact data handling. Describe how reconnaissance artifacts should be stored, retained, and disposed of because they can contain sensitive intelligence such as banners, internal hostnames, and API endpoints.
- Pre-write header sanitization. Explain that tokens, session IDs, and sensitive headers must be sanitized before write_jsonl() writes an artifact to disk, and distinguish full header removal from redaction.
- Human accountability for AI actions. Explain why an AI reconnaissance agent cannot own consequences and why the human operator remains accountable for every action it performs. State that the AI is advisory rather than autonomous.
- Responsible pipeline controls. Describe the three principles for integrating models into reconnaissance: explainable suggestions, least-privilege access, and auditing accepted as well as rejected suggestions as artifacts. Relate these principles to respecting documented scope, blocking and logging out-of-scope requests, and using approval gates.
What it builds on
- Pipelines: Information Routing Systems (chapter 1) · Pipelines as information-routing systems with defined stages.
- Artifacts provide precise decision-making in offensive security (chapter 1) · Artifacts as the units that move intelligence through an offensive-security workflow.
- Artifacts as agent interfaces (chapter 3) · Artifacts as interfaces between components or agents.
- Why agents matter in offensive security (chapter 1) · Why agents matter in offensive security.
- Safety and governance (chapter 2) · Safety and governance mechanisms for agent operation.
- Safety gates and authorization control (chapter 3) · Safety gates, authorization control, and human involvement in multi-agent systems.
- Artifacts and logging (chapter 2) · Artifacts and logging in the minimal agent.
- The minimal agent loop (chapter 2) · The minimal agent loop that acts, observes, and records.
Section by section
Introduction
This chapter introduction frames passive reconnaissance as an auditable, human-supervised pipeline that records safe evidence and prepares the reader for later enumeration and scoring work.
- Auditable reconnaissance pipelines: Explain why a repeatable reconnaissance pipeline is preferable to scattered manual lookups, command-line runs, and one-off scripts. Describe how explicit stages and structured artifacts make the operation reviewable and reconstructable.
Why it matters: An auditable pipeline lets a team answer where an observation came from and how it was obtained. - Human-supervised AI assistance: Describe AI's intended role in the recon workflow as supporting professional judgment through suggestions, explanations, and summaries rather than replacing human decision-making. State that humans supervise execution and keep the agent within defined boundaries through safety gates.
Why it matters: Keeping humans in charge ensures AI-assisted reconnaissance remains accountable and within authorized limits. - Passive recon pipeline scope: State the planned passive reconnaissance workflow: generate candidate hostnames for a root domain, resolve them with passive DNS, make one safe HEAD request for response headers, collect lightweight TLS metadata such as ALPN and SAN, and append observations to a versioned recon.jsonl artifact. Explain that observations are timestamped and tied to a reason so the work can be replayed, audited, paused, or extended without touching live application logic.
Why it matters: This bounded workflow demonstrates how to collect useful early evidence while preserving explainability and control.
Key terms: reconnaissance pipeline, explicit stages, structured outputs, audit trail, artifacts, BlackHat AI philosophy, human-in-the-loop, safety gates, defined boundaries, supervisors and escalators
4.1 Why an AI reconnaissance agent?
This section motivates a passive AI reconnaissance agent by contrasting passive and active recon and showing how staged, artifact-driven pipelines make recon repeatable, auditable, and bounded by human-controlled safety gates.
- Passive and active reconnaissance: Distinguish passive reconnaissance from active reconnaissance by describing whether the target system is directly contacted, what kinds of sources or actions are involved, and the resulting visibility and authorization risks.
Why it matters: Knowing this distinction helps you choose reconnaissance methods that fit the target scope and risk level. - Manual recon workflow limitations: Explain why a traditional multi-terminal reconnaissance workflow is fragile, including manual copying and parsing of outputs, missing lineage, and difficulty repeating or selectively rerunning useful work.
Why it matters: These limitations explain why a structured agent pipeline is more useful than merely running more reconnaissance tools. - Artifact-driven recon pipeline: Describe the AI recon agent as a conveyor-belt pipeline of small, well-defined stages that pass outputs forward and record persistent artifacts, enabling review, recovery, and traceability of commands, parameters, and timing.
Why it matters: Persistent artifacts turn isolated reconnaissance observations into a reviewable and repeatable operational process. - Human-controlled safety gates: Explain how pipeline gates enforce safety and accountability through controls such as time windows, approval checks, and scope filters, while allowing the AI to propose actions but reserving execution decisions for humans.
Why it matters: Autonomy is useful in offensive security only when it operates within enforceable human-defined boundaries.
Key terms: reconnaissance, passive reconnaissance, active reconnaissance, target system, scope, lineage tracking, repeatable, CI/CD pipeline, reconnaissance session, AI recon agent
4.2 The minimal agent architecture
Defines the minimal AI reconnaissance agent as a deliberately small, bounded intelligent workflow before implementation.
- Minimum viable recon pipeline: Describe the minimal AI recon agent as a minimum viable pipeline, including that the section frames it by deciding what belongs inside its scope and what is deliberately excluded.
Why it matters: A clear minimum scope makes the agent easier to understand and build before adding code or capabilities. - Miniature intelligent workflow: Explain that the minimal recon pipeline is presented as the most miniature possible version of an intelligent workflow.
Why it matters: This framing helps readers see the small agent as a purposeful workflow model rather than merely a reduced script.
Key terms: minimal AI recon agent, minimum viable pipeline, intelligent workflow
4.2.1 Pipelines vs. scripts
This section distinguishes standalone scripts, staged pipelines, and AI agents in passive reconnaissance workflows.
- Scripts and pipeline structure: Distinguish a script from a pipeline, including how scripts serve as building blocks that a pipeline orchestrates in sequence.
Why it matters: This distinction explains why a recon workflow can be more than a single command or self-contained automation task. - Pipeline workflow capabilities: Describe the structure a pipeline adds around recon logic: a stage input interface, structured JSON artifacts with context and timestamps, safety gates, and the ability to pause, review, or replay without losing context.
Why it matters: These capabilities make a reconnaissance workflow inspectable, controllable, and reusable rather than merely printing a result to the screen. - Pipeline and agent autonomy: Distinguish a human-intervened, brittle pipeline from an AI agent that responds to environmental changes by reasoning, choosing a course of action, and invoking external tools without requiring a script.
Why it matters: It clarifies why an AI agent is useful when inputs, queries, or external APIs change in ways that can break fixed pipelines.
Key terms: script, pipeline, stages, orchestrate, stage interface, structured artifact, JSON, context, timestamps, gates
4.2.2 The four building blocks to building ReAct agents
This section maps the four reusable ReAct-agent components to reconnaissance work and clarifies the boundary between passive, non-intrusive active, and intrusive active recon.
- ReAct reconnaissance workflow: Describe how planning, acting, analyzing, and recording findings work together in a ReAct reconnaissance agent. Explain how planning uses objectives, current knowledge, scope boundaries, hypotheses, and operational considerations to choose a reconnaissance strategy.
Why it matters: A recon agent is useful only when its actions follow an explicit strategy and its results feed back into later reasoning. - Persistent reconnaissance artifacts: Explain why model context and insights from a reconnaissance session are ephemeral and how an artifact preserves essential findings. Describe artifacts such as JSON files or vector databases as stored model outputs and context that can be retrieved in future sessions or used for model training.
Why it matters: Persisted findings let a future agent session build on prior reconnaissance rather than losing the reasoning when the session ends. - Reconnaissance activity boundaries: Distinguish truly passive reconnaissance from non-intrusive active reconnaissance and intrusive active WAF fingerprinting. State that HTTPS probing and reading response headers for WAF hints send requests to the target and are therefore active, even when they avoid payloads, defense triggering, and operational impact.
Why it matters: Accurately classifying recon activity helps operators understand scope, target visibility, and the risk of crossing into intrusive testing.
Key terms: ReAct agents, planning phase, acting phase, analyzes, context, reconnaissance strategy, ephemeral, artifact, JSON file, vector database
4.2.3 Artifacts provide persistent memory to an agent
This section explains how reconnaissance agents use persistent JSONL artifacts to retain findings and support later analysis, debugging, and reporting.
- Reconnaissance artifact contents: Describe reconnaissance artifacts as persistent, tangible outputs that store findings about an explored host, network, or system, including information such as host details, target address, and detected headers.
Why it matters: Persisted findings give the agent and subsequent workflow stages a concrete record of what reconnaissance discovered. - JSONL recon artifact format: Explain why the section recommends storing reconnaissance findings in machine-readable JSONL files and identify recon.jsonl as the recon agent output artifact referenced throughout the book.
Why it matters: A machine-readable artifact format lets both LLMs and operational tooling consume the same reconnaissance record. - Artifact-driven follow-on operations: Trace how stored JSONL artifacts enable monitoring systems to merge and search findings, support replay of agent actions for debugging and simulation, and supply later AI agents for summarization, prioritization, and reporting.
Why it matters: The value of persisted reconnaissance is that the same evidence can be reused, audited, and transformed by downstream stages rather than collected again.
Key terms: artifacts, persistent, reconnaissance findings, host, target address, detected headers, JSONL, machine-readable, LLMs, recon.jsonl
4.2.4 Safety gates
This section explains why passive-reconnaissance agents need safety gates and introduces time-window and approval-or-automation checkpoints before any network call.
- Safety gate purpose: Explain that safety gates are decision checkpoints and control mechanisms that keep reconnaissance agents operating responsibly before they make network calls.
Why it matters: Safety gates help keep an ostensibly passive workflow from creating operational, legal, ethical, or privacy harm. - Ungated reconnaissance risks: Describe the harms that can result when a reconnaissance agent runs without safety gates, including accidentally becoming active reconnaissance, triggering alerts, violating privacy laws, or entering dangerous, illegal, or unethical territory.
Why it matters: Naming the failure modes makes it clear why execution controls are necessary rather than optional. - Pre-execution gate pair: State the two required pre-execution gates: a time-window gate that permits runs only during approved hours, and an approval-or-automation gate that confirms intent before execution.
Why it matters: These gates translate authorization and operational timing into concrete conditions that must be satisfied before the agent acts. - Additional reconnaissance gates: Describe the suggested extensions to the two basic gates: quality-control gates that catch agent mistakes before execution and ethical-boundary gates that enforce privacy and authorization limits.
Why it matters: Additional gates can address errors and authorization concerns that timing and intent checks do not fully cover.
Key terms: safety gates, decision checkpoints, control mechanisms, network call, active reconnaissance, security alerts, privacy laws, production, ethical territory, time-window gate
4.2.5 Two scripts, one story
This subsection explains the deliberately simple two-script layout used to teach the minimal passive-reconnaissance agent and its artifact flow.
- Two-script implementation layout: Describe how the example divides the implementation between recon_pipeline.py and artifacts.py, including the responsibility of each file.
Why it matters: Knowing the file layout makes it easier to follow where the reconnaissance logic runs and where its records are written. - Approachable design tradeoff: Explain why the chapter uses a single main agent file plus a small artifact helper instead of a more modular file structure.
Why it matters: The simplified layout keeps attention on the agent's logic flow before introducing the complexity of larger workflow designs. - Stage-to-artifact progression: Describe how functions correspond directly to agentic stages and how artifacts are appended as the agent progresses.
Why it matters: This connection lets you read the pipeline as an ordered, transparent record of the agent's work. - Path toward extensible workflows: State how the simple implementation prepares readers for more complicated agentic workflows, such as using script flags to toggle optional TLS or content-discovery stages.
Why it matters: The small example establishes a foundation for adding optional stages without losing the workflow's structure.
Key terms: recon_pipeline.py, artifacts.py, JSONL, passive recon logic, single file, logic flow, modular, file management system, agentic stage, artifacts
4.3 Building the minimal AI recon pipeline
Introduces a self-contained minimal reconnaissance pipeline that runs agent stages sequentially and saves structured artifacts, while warning that it is only a sandbox foundation rather than a production-ready recon agent.
- Minimal recon pipeline foundation: Describe the program introduced in this section as a single self-contained pipeline that executes each reconnaissance-agent stage in sequence and saves the results as structured artifacts.
Why it matters: This establishes the concrete starting point for building an agentic reconnaissance workflow. - Sandbox-only safety status: State that this minimal recon agent is a foundation rather than a production-ready system, and explain that it should be used for reconnaissance only in a sandbox until later components make the system safer and auditable.
Why it matters: The warning prevents readers from treating an early instructional prototype as an authorized operational reconnaissance tool.
Key terms: recon agent, agent stage, structured artifacts, pipeline, production-ready agent, sandbox environment, safe and auditable agentic system
4.3.1 Getting started
This subsection explains why to activate a project-specific Python virtual environment before building the reconnaissance pipeline.
- Python virtual environment isolation: Explain that a Python virtual environment is a self-contained copy of Python and the packages installed for a project, and state why activating it comes first for these exercises.
Why it matters: Using an isolated environment prevents the exercise dependencies from interfering with other Python installations.
Key terms: Python virtual environment, system-level Python library, packages, dependencies
4.3.2 Starting the artifact helper
This section defines the Artifact record and JSONL-writing helper that persist passive reconnaissance observations for later agent reasoning and reporting.
- Artifact reconnaissance record: Describe the Artifact class as a dataclass that stores structured reconnaissance information about a target host, including DNS data, HTTP metadata, TLS details, and general notes. Explain that each artifact is an auditable record of what was done and what was found.
Why it matters: Artifacts preserve reconnaissance findings as usable evidence rather than leaving them as transient tool output. - Schema version and timestamp: Explain that the artifact schema identifies the version of the data format and that every artifact receives a UTC timestamp when it is written. Describe how the timestamp helper sets this writing time.
Why it matters: Versioned, timestamped records make stored reconnaissance data easier to interpret and audit over time. - Append-only JSONL artifact output: Describe how write_jsonl appends artifacts to a configurable output file such as recon.jsonl as newline-delimited JSON, with one JSON object per line. Explain why this portable, serializable format supports incremental streaming and later processing with standard tools or AI agents.
Why it matters: JSONL lets the pipeline accumulate findings incrementally in a format that people, tools, and later agents can all consume.
Key terms: Artifact, dataclass, DNS, HTTP metadata, TLS details, general notes, schema, UTC timestamp, _now_iso(), write_jsonl
4.3.3 Creating the passive reconnaissance pipeline
This section assembles a command-line reconnaissance pipeline from small, low-impact discovery and inspection functions that write one structured artifact per host.
- Reconnaissance pipeline stages: Describe how the pipeline moves from candidate-host generation and DNS resolution through header probing, WAF/CDN inference, optional TLS and content inspection, sanitization, and artifact recording.
Why it matters: Understanding the stages lets you explain what evidence the pipeline collects for each host and where it comes from. - Host discovery and deduplication: Explain how candidates() combines seed prefixes, including the empty prefix for the apex domain, with a root domain and deduplicates them, and how resolve() returns IP addresses and a CNAME or an empty result when DNS lookup fails.
Why it matters: This establishes which potential hosts exist before the pipeline spends effort collecting web-facing evidence. - Low-impact web and TLS evidence: Describe the evidence gathered from a lightweight HTTPS HEAD request, TLS handshake, and optional robots.txt or sitemap retrieval, including why HEAD avoids downloading a response body and what ALPN and certificate SANs can reveal.
Why it matters: These standard interactions provide useful reconnaissance signals while keeping the collection footprint small. - WAF detection and header sanitization: Explain that WAF/CDN detection uses broad partial signature matching in response-header names or values as an early-warning heuristic, and that sensitive response-header values are redacted before artifacts are saved while their names are retained.
Why it matters: The pipeline both flags likely traffic protections for later judgment and avoids persisting secrets in its reconnaissance logs. - Orchestration and composable artifacts: Explain how main() parses arguments, loops over candidate hosts, invokes the stateless probe functions, and writes one machine-readable JSONL artifact per host; distinguish the default lightweight scan from optional --tls and --content stages, and explain why this structure can be used by AI agents.
Why it matters: Composable tools and queryable artifacts allow an agent or later orchestrator to reason from collected evidence and choose hosts for deeper inspection. - Reconnaissance scope classification: Distinguish this pipeline's non-intrusive intent from truly passive reconnaissance: it avoids payloads, vulnerability probing, and authentication, but DNS, HEAD, and TLS interactions still reach the target and therefore sit at the low-impact end of active reconnaissance.
Why it matters: Accurately describing the operational footprint helps practitioners apply scope and authorization controls to even restrained collection.
Key terms: candidates(), resolve(), https_head(), inter_waf(), tls_peek(), robots_and_sitemap(), SEEDS, apex domain, getaddrinfo(), gethostbyname_ex()
4.3.4 Diving into the passive reconnaissance results
This subsection runs the passive reconnaissance pipeline, explains why passive and controlled automation comes first, and introduces recon.jsonl as the persistent record whose depth changes with optional flags.
- Passive reconnaissance discipline: Explain why the pipeline begins with passive reconnaissance and describe the operational discipline it requires: one request per host, timeouts, complete recording, and remaining within scope.
Why it matters: Starting slowly and visibly makes automated reconnaissance safer, controllable, and defensible. - Recon JSONL artifact: Describe recon.jsonl as the new file produced in the project root after a successful run and explain why the subsection calls it the heartbeat of the pipeline: it records what the pipeline did.
Why it matters: The recorded artifact provides the evidence needed to inspect reconnaissance activity and support later AI reasoning. - Optional reconnaissance depth: Describe how running the pipeline for example.com with no option, with --tls, or with --content changes the length and richness of recon.jsonl while leaving the pipeline's overall structure unchanged.
Why it matters: Comparing these runs shows that optional data depth can change without changing the underlying pipeline design.
Key terms: passive reconnaissance, deep enumeration, scanning, one request per host, timeouts, scope, recon.jsonl, artifact, project root, pipeline
4.4 Inspecting your reconnaissance results
This section introduces executing the recon pipeline and inspecting its append-only JSON-line artifacts as a chronological record of each run.
- Append-only artifact history: Explain that repeated pipeline runs append new artifact lines rather than overwriting existing data, producing a chronological record.
Why it matters: An append-only record lets you inspect what happened across runs instead of losing earlier reconnaissance evidence. - Artifact inspection purpose: Describe the immediate workflow of executing the runnable recon pipeline and inspecting the JSON-line artifacts it creates to read the story in the results.
Why it matters: Inspecting artifacts turns pipeline output into an understandable record of the reconnaissance process.
Key terms: append-only, artifact lines, chronological record, pipeline logging, recon pipeline, artifacts, JSON lines
4.4.1 Viewing results
This section shows how to view the pipeline’s recent JSON artifacts and interpret the fields that preserve passive reconnaissance context for later AI reasoning.
- Recon artifact inspection: Describe how to inspect what the passive reconnaissance pipeline captured by displaying the most recent JSON records with a JSON tool, and explain that each record represents captured reconnaissance context.
Why it matters: Viewing the raw records lets you verify what the pipeline actually collected before relying on its results. - Recon artifact field meanings: Explain the anatomy of a recon-v1 artifact: its schema and host identify the record and tested fully qualified hostname, while DNS, HTTP header, WAF, TLS, notes, and timestamp fields preserve the observed reconnaissance context.
Why it matters: A structured artifact gives later agents explicit evidence and context instead of forcing them to infer what happened. - Recorded context for AI reasoning: Explain why the section preserves these exact reconnaissance logs: later AI agents can reason over recorded context rather than guess the circumstances of each observation.
Why it matters: Recorded evidence makes downstream AI analysis more grounded in what the pipeline actually observed.
Key terms: JSON, artifact, recon-v1, schema, host, fully qualified hostname, a, cname, artifacts, logs
4.4.2 Checking for patterns
This subsection shows how to use quick grep and jq queries on passive reconnaissance results to identify responsive hosts, WAF/CDN-protected hosts, and technology fingerprints, then use the output to guide further investigation.
- Reconnaissance pattern queries: Describe the three quick queries used to inspect reconnaissance results: finding hosts with HTTP 200 responses, finding hosts whose waf_hint is true, and extracting unique header keys across hosts.
Why it matters: These lightweight checks turn raw reconnaissance output into immediate leads for deciding where to investigate next. - Interpretation of scan patterns: Explain how the query output guides next steps: mostly empty 200 results can indicate that the seed list needs broader expansion, many responsive candidates indicate a rich attack surface, and WAF/CDN hints mean active testing may need a different approach.
Why it matters: Interpreting patterns lets an operator prioritize and adapt work instead of treating reconnaissance output as an undifferentiated list. - Artifact narrative reconstruction: Explain that reconnaissance artifacts form a story of what the pipeline examined, what it found, and how it found it, allowing a later AI agent to reconstruct that reasoning from the artifacts.
Why it matters: A reconstructable trail makes reconnaissance findings more understandable and usable by both people and later agents.
Key terms: grep, jq, 200 status code, waf_hint, header keys, seed list, candidate, attack surface, WAF, CDN
4.4.3 Clean up and prepare for next steps
This subsection advises archiving old runs while preserving the durable findings artifact for later interpretation and expansion.
- Archived reconnaissance artifacts: Describe why old pipeline runs can be archived and why the remaining durable artifact file should be preserved as the evidence from a passive reconnaissance run.
Why it matters: Keeping durable artifacts while archiving old runs leaves a tidy workspace without losing findings that can be traced, shared, or extended.
Key terms: archive old runs, durable artifact file, pipeline findings, evidence
4.5 Reading the pipeline like a story
This section frames the recon.jsonl artifact log as a narrative record of a reconnaissance operation rather than raw output.
- Reconnaissance artifact narrative: Explain how the lines in recon.jsonl can be read as a story of a reconnaissance operation, with each artifact recording what the pipeline planned, did, saw, and concluded.
Why it matters: Reading artifacts as a narrative helps you interpret the operation's intent, context, and outcome instead of treating the log as disconnected raw data.
Key terms: recon.jsonl, artifact, pipeline, intent, context, outcome
4.5.1 Artifacts as episodes of reasoning
This subsection explains that chronologically ordered artifacts preserve the traces of each reasoning stage, letting readers reconstruct the reconnaissance pipeline's discovery process.
- Artifacts as reasoning episodes: Explain how each artifact represents a mini-episode in the pipeline's reasoning process and how reading artifacts in chronological order reveals the system's discovery trail.
Why it matters: Artifacts let you inspect how reconnaissance conclusions emerged rather than seeing only a final result.
Key terms: artifact, pipeline, reasoning process, discovery, traces
4.5.2 Example: following the trail
This section shows how clearly structured reconnaissance artifacts let analysts and AI agents connect observations into meaningful findings and host clusters.
- Structured artifact reasoning: Explain why consistent JSONL fields enable an AI agent to reason across reconnaissance observations and make inferences beyond the pipeline's explicitly programmed checks.
Why it matters: Clean, structured artifacts give an AI agent the contextual facts it needs to identify meaningful security findings. - Reconnaissance trail interpretation: Describe the conclusions drawn from the consecutive reconnaissance entries: an internet-exposed development server, a Cloudflare-backed CDN or production front, and an authenticated staging environment that still advertises a sitemap; distinguish a WAF hint from a confirmed WAF fingerprint.
Why it matters: Interpreting related reconnaissance records as a trail turns isolated host facts into leads worth investigating. - Shared-IP host clustering: State how filtering hosts by a shared IP address can reveal multiple subdomains belonging to a single deployed application across several domains, and explain why that cluster is valuable for red teams and defenders.
Why it matters: Shared-backend clusters reveal relationships between domains that may otherwise look like unrelated reconnaissance targets.
Key terms: JSONL format, structured artifacts, downstream reasoning, observations, development server, CDN host, Cloudflare, production front, staging environment, sitemap
4.6 Safety, scope, and ethics
This section introduces responsible AI reconnaissance as a practice bounded by consent, ethics, and rules rather than technical capability alone.
- Consent boundaries for reconnaissance: Explain why every reconnaissance pipeline must operate within a clear boundary of consent, including the difference between authorized reconnaissance and activity that violates privacy or disrupts services.
Why it matters: Consent is what makes reconnaissance a legitimate security activity rather than harmful or illegal behavior. - Responsible AI reconnaissance rules: Describe why responsible AI reconnaissance requires rules as well as code, since the same pipeline can support defensive attack-surface mapping or be misused to harm others.
Why it matters: AI automation can amplify both legitimate security work and misuse, so responsible operating rules must guide its application.
Key terms: clear boundary of consent, authorized reconnaissance, illegal scanning, AI reconnaissance systems, responsibly, rules, code, attack surface
4.6.1 Scope defines the battlefield
This subsection makes explicit scope configuration and enforcement a built-in design control for passive reconnaissance pipelines.
- Configured operational scope: Describe how a pipeline's permitted targets can be defined before it runs using a scope.json file or an environment variable listing allowed domains, IP ranges, or network zones.
Why it matters: Explicit scope gives the pipeline a machine-readable boundary for where it may operate. - Automatic scope enforcement: Explain that an AI in a larger system can check configured scope before acting, automatically deny a proposed out-of-scope target, and record that denial in an artifact.
Why it matters: Automatic rejection and recorded decisions prevent unauthorized actions while leaving an auditable trace of enforcement. - Scope as system design: Justify treating scope as a required part of system design rather than a policy assumption, including its contractual role in professional red-team operations and its disciplinary role in personal or educational labs.
Why it matters: Treating scope as configuration turns an uncontrolled script into a system with enforceable operational boundaries.
Key terms: scope.json, environment variable, allowed domains, IP ranges, network zones, scope enforcement, proposed target, artifact line, pipeline denies, scope
4.6.2 Gates keep humans in the loop
This section explains how permission gates keep early reconnaissance agents under human control and turn ethical intent into enforced code.
- Human approval gates: Explain why early AI reconnaissance pipelines should pause for human permission before actions leave the machine, and describe each gate as a small ethics check.
Why it matters: Human approval gates prevent an agent from treating autonomous action as permission to act outside authorized scope. - Ethical intent as code: Describe how encoding gates makes ethical intent and professional operating habits enforceable defaults rather than informal reminders.
Why it matters: Putting constraints in code makes responsible behavior repeatable even when an operator is busy or a workflow is automated.
Key terms: gates.py, gate, human in the loop, ethics check, ethics as code, MITRE Engenuity’s ATT&CK Evaluations, OSCP exam environments
4.6.3 Handling and storing data safely
This section explains how to protect sensitive reconnaissance artifacts through local storage, pre-write sanitization, retention limits, and evidence-grade handling.
- Recon artifact data handling: Describe how reconnaissance artifacts should be stored, retained, and disposed of because they can contain sensitive intelligence such as banners, internal hostnames, and API endpoints.
Why it matters: Careful artifact handling prevents passive reconnaissance output from becoming an unnecessary data exposure. - Pre-write header sanitization: Explain that tokens, session IDs, and sensitive headers must be sanitized before write_jsonl() writes an artifact to disk, and distinguish full header removal from redaction.
Why it matters: Sanitizing before persistence limits exposure because sensitive data is much harder to contain after it reaches disk. - Artifacts as audit evidence: Explain why reconnaissance JSONL artifacts must be handled like vulnerability data: they are audit trails that may be reviewed for compliance or used in legal disclosure.
Why it matters: Viewing artifacts as evidence encourages handling controls proportionate to their compliance and legal consequences.
Key terms: recon artifacts, recon.jsonl, confidential intelligence, archive, delete artifacts, set-cookie, sanitize_headers(), redaction, write_jsonl(), JSONL
4.6.4 AI models and responsibility
This section establishes that human operators remain accountable for AI-assisted reconnaissance and defines the controls that make a recon pipeline responsible and auditable.
- Human accountability for AI actions: Explain why an AI reconnaissance agent cannot own consequences and why the human operator remains accountable for every action it performs. State that the AI is advisory rather than autonomous.
Why it matters: Human accountability prevents AI assistance from becoming an excuse for unauthorized or harmful reconnaissance activity. - Responsible pipeline controls: Describe the three principles for integrating models into reconnaissance: explainable suggestions, least-privilege access, and auditing accepted as well as rejected suggestions as artifacts. Relate these principles to respecting documented scope, blocking and logging out-of-scope requests, and using approval gates.
Why it matters: These controls make AI-assisted reconnaissance reviewable, constrained to authorization, and safe to operate professionally. - Artifact-based reasoning audit trail: Explain how artifacts create a machine-readable narrative and audit trail of the system's reasoning. State that artifact records must not contain personal or customer data and can later be revisited, summarized, or extended.
Why it matters: A safe artifact trail lets operators inspect and reuse AI-assisted reasoning without exposing sensitive data.
Key terms: human operator, accountable, advisory, autonomous, Explainability first, least privilege, Audit everything, artifacts, documented scope, out-of-scope request
4.7 Summary
This summary consolidates how a safe ReAct-based passive reconnaissance pipeline gathers, records, interprets, and governs low-impact target intelligence.
Concepts explained in this chapter
- reconnaissance attackReconnaissance that gathers information through ordinary protocol interactions without exploiting vulnerabilities, authenticating, or delivering attack payloads.
- active reconnaissanceReconnaissance that sends packets or requests to interact directly with the target, even when the interaction is low impact.
- HEADAn HTTP request using the HEAD method to retrieve response metadata and headers without transferring the response body.
- JSONLA newline-delimited JSON format in which each line contains one complete JSON object.
- AI agentA software system that applies artificial intelligence to interpret inputs, reason, and take actions, potentially using external tools or data.
- header sanitizationThe process of removing or masking secret values in HTTP headers while retaining non-sensitive header metadata.
- reconnaissance agentAn AI agent specialized in gathering and interpreting information about systems, networks, and exposed services.
- schemaA schema is a formal specification of the structure, fields, and data types permitted in data exchanged between software components.
- chain-of-thought promptingA prompting and reasoning technique that elicits intermediate steps to connect a problem statement with a conclusion or action.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
