liveBook · Manning

JSONL

As explained in AI Agents for Offensive Security

A newline-delimited JSON format in which each line contains one complete JSON object.

JSONL, or JSON Lines, stores one complete JSON object on each line. That small rule makes it a practical durable-artifact format for agent pipelines: observations can be appended as work progresses, read incrementally, and traced back into an ordered account of actions and results. section 2.5.6 section 4.3.2

Treat each line as an auditable event or finding, not as an unstructured log message. In passive reconnaissance, an artifact can capture a host, resolved addresses, HTTP headers, TLS details, notes, a schema version, and a UTC timestamp. Consistent field names and types let downstream tools and AI agents distinguish observations without guessing where records begin or end. section 4.2.3 section 4.3.2

A useful JSONL artifact contract includes:

  • A schema identifier for format versioning, such as recon-v1. section 4.3.2
  • Target and observation fields, such as host, addresses, headers, TLS data, and notes. section 4.2.3
  • A timestamp set when the record is written, preserving event order and review context. section 4.3.2
  • One append-only record per observation or pipeline stage, rather than a rewritten aggregate document. section 4.2.5
Listing 4.1 Setting the artifact schema for storing structure recon data
from dataclasses import dataclass, asdict, field
from datetime import datetime, timezone
from typing import Dict, List, Optional
import json
import os
 
# Output path can be overridden via env var
OUT = os.environ.get("RECON_OUT", "recon.jsonl")
 
 
@dataclass
class Artifact:
    schema: str = "recon-v1"
    host: str = ""
    a: List[str] = field(default_factory=list)                 #A
    cname: Optional[str] = None                                #B
    headers: Dict[str, str] = field(default_factory=dict)      #C
    waf_hint: Optional[bool] = None                            #D
    tls: Optional[Dict] = None                                 #E
    notes: List[str] = field(default_factory=list)             #F
    ts: str = ""                                               #G

This artifact schema shows a compact record contract for reconnaissance: versioned structure, target identity, network and HTTP evidence, optional TLS and WAF hints, notes, and a timestamp. section 4.3.2

Write JSONL by serializing one record, writing a newline, and appending it to a configurable output path. Flushing or closing after writes makes the trace durable during a run. This supports streaming results to later stages, merging artifacts from multiple probes, searching them with standard tools, and replaying actions for debugging or simulation. section 2.5.6 section 4.3.2

Reading a reconnaissance trail

Three consecutive records can show a public development host returning 200, a CDN-facing host with a Cloudflare header, and a staging host returning 401 while advertising robots.txt. A human or AI agent can compare these structured fields to form hypotheses, while treating a WAF-related header hint as an indicator rather than a definitive fingerprint. Filtering records with jq or Python can also reveal hosts sharing an IP address. section 4.5.2

JSONL artifacts may contain sensitive headers, internal hostnames, endpoints, tokens, or session identifiers. Sanitize before writing: remove or redact sensitive headers before the JSONL write call. Store artifacts locally, keep them out of public repositories, and archive or delete them after an assessment. Because records may support compliance review or legal disclosure, handle them as confidential evidence. section 4.6.3 section 2.5.6

How AI Agents for Offensive Security covers it

How deep it goes
Defines line-delimited records and implements schemas, append writers, timestamps, streaming, and safety controls without specifying a universal JSONL standard.
Examples
Demonstrates a Python artifact logger, a versioned reconnaissance record, consecutive host records, and jq or Python IP-clustering analysis.
What it assumes
Assumes familiarity with JSON-like structured data and Python utilities while introducing JSONL writing through small helper code.
What you can do after
Operate/configure append-only JSONL artifact logging and safely prepare structured reconnaissance records for review, debugging, and AI analysis.
How it connects
Connects JSONL to artifacts, agent pipelines, reconnaissance, audit logging, schemas, timestamps, and LLM reasoning.

Also explained in

Compare how 4 books explain JSONL →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Agents for Offensive Security. The section links point to the parts of the book it draws on.