liveBook · Manning

HEAD

As explained in AI Agents for Offensive Security

An HTTP request using the HEAD method to retrieve response metadata and headers without transferring the response body.

HEAD is a lightweight HTTPS probe that requests a target’s root path while collecting response headers and an HTTP status note, without retrieving the response body. In reconnaissance, it provides a low-footprint observation step after hostname discovery and resolution, helping keep traffic minimal while producing evidence that can be recorded and reviewed. section 4.3.3 section 4.0

A practical HEAD stage has a narrow contract: send HEAD / with a declared User-Agent and timeout; normalize returned header names to lowercase; return a header dictionary plus notes containing either status information or an error type; and close the connection even when probing fails. This makes expected network failures observable rather than silently discarded. section 4.3.3

Response headers become useful pipeline inputs when they are handled as structured observations:

  • Record the status code alongside the headers so the result retains basic request context. section 4.3.3
  • Inspect header names and values for broad WAF/CDN signatures; treat a match as an early warning, not definitive profiling, because the heuristic favors false positives. section 4.3.3
  • Sanitize sensitive values such as Set-Cookie, Authorization, and X-API-Key before persistence. section 4.7
  • Append timestamped observations to structured JSONL artifacts so later stages can query, audit, replay, or extend the work. section 4.0 section 4.7

Minimal safe probe

For each resolved candidate hostname, issue one HTTPS HEAD request to /. Store lowercase response headers and a note such as status:200; if the request fails, store an error note instead. Run a WAF/CDN header heuristic on the observation, redact sensitive header values, and persist the resulting record. Content retrieval, such as GET /robots.txt, remains separately gated to preserve the default lightweight behavior. section 4.3.3 section 4.7

Listing 4.5 HTTPS probing for header extraction
 
def https_head(host: str, timeout: int = 4, user_agent: str = "ReconSnap/1.0") -> Tuple[Dict[str, str], List[str]]:
    """Send a safe HTTPS HEAD / and return (headers, notes)."""
    headers: Dict[str, str] = {}
    notes: List[str] = []
    try:
        ctx = ssl.create_default_context()
        conn = http.client.HTTPSConnection(host, 443, context=ctx, timeout=timeout)
        conn.request("HEAD", "/", headers={"User-Agent": user_agent})
        res = conn.getresponse()
        headers = {k.lower(): v for k, v in res.getheaders()}
        notes.append(f"status:{res.status}")
    except Exception as e:
        notes.append(f"error:{type(e).__name__}")
    finally:
        try:
            conn.close()  # type: ignore[has-type]
        except Exception:
            pass
    return headers, notes

The implementation shows the probe’s operational boundary: HTTPS on port 443, HEAD /, configurable timeout and User-Agent, lowercase headers, status/error notes, and cleanup in finally. section 4.3.3

HEAD is low-impact rather than fully passive: it touches the target, unlike techniques that rely entirely on third-party data. Use it only within configured scope and operational safeguards; block and log out-of-scope requests, retain human approval where required, and keep every action explainable and auditable. section 4.7

How AI Agents for Offensive Security covers it

How deep it goes
Implements an HTTPS HEAD helper through request construction, normalized headers, status/error notes, and connection cleanup; defers definitive WAF profiling.
Examples
Demonstrates one HTTPS HEAD / probe with a configurable timeout and User-Agent, returning headers plus status or error notes.
What you can do after
Build it from scratch: implement a bounded HTTPS HEAD stage that records sanitized, auditable header observations.
How it connects
Connects HEAD observations to DNS resolution, WAF/CDN detection, TLS inspection, JSONL artifacts, scope enforcement, and ReAct.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Agents for Offensive Security. The section links point to the parts of the book it draws on.