liveBook · Manning

Archive

As explained in Self-Improving Agents

An archive is a stored collection of past candidates retained during a search process.

A search can lose a useful candidate simply because it scores poorly today. An archive stores candidates and their measured signals so the search can rank past work, recover an earlier branch, or inspect how a result was reached. In an agent improvement loop, the stored candidate might be a version of the system prompt. section 2.1.4 section 4.3.1

Two operations are easy to conflate:

  • Record and rank: add scored prompt variations during search; call best() to identify the top artifact. A SQLite-backed archive can persist them, beginning with a genesis prompt. section 2.1.4
  • Promote: explicitly call promote() before making a selected artifact available to the agent. Finding a better candidate does not itself deploy it. section 2.1.4
  • Revisit: in a Darwin Gödel machine search, keep every variant and sometimes sample a parent from the whole archive rather than only the leaders. Preserve each child's score and parent pointer. section 4.3.5

Why keep a losing branch?

On a toy landscape, a greedy climber stops at a local peak of height 6 because its next step is downhill. Archive search retains the lower valley points as possible parents and reaches a peak of height 10. In a prompt-search run, the same pattern lets a search revisit a low-scoring branch and find wording that excludes both cancelled and returned orders. section 4.3.4 section 4.3.5

Listing 4.8 Never discard, and sample by exploit-or-explore (06_revenue_dgm_loop.py)
def sample_parent(archive):                          #A
    ranked = sorted(archive, key=lambda e: e["score"],
                    reverse=True)
    if random.random() < EXPLOIT_P:   # usually exploit
        return random.choice(ranked[:3])   # a top candidate
    return random.choice(archive)     # else explore any member
 
for it in range(iters):
    parent = sample_parent(archive)                  #B
    diagnosis = await reflect(parent["content"], parent["traj"])
    child = await mutate(parent["content"], diagnosis)
    score, _, traj = await measure_full(child)
    child_art = parent["art"].mutate(new_content=child,  #C
                                     created_by="dgm")
    await store.record(                              #D
        Variant(artifact=child_art, parent=parent["art"],
                search_method="dgm"),
        GapMeasurement(score=score),
    )
    archive.append({"art": child_art, "content": child,
                    "score": score, "traj": traj})   #E

The loop shows the key implementation choice: usually select a leading parent, sometimes select any archived parent, then score and record its child without truncating the archive. section 4.3.5

Retention has a cost. An ever-growing archive means more iterations and model calls when the search repeatedly explores old branches. Spend that budget when a reliable score makes recovery across a deceptive landscape valuable; for an easier or noisier task, a smaller population can be the wiser choice. SQLite is convenient for the initial persistent archive, while a longer-lived production deployment may warrant a database such as Postgres. section 4.3.5 section 2.1.4

How Self-Improving Agents covers it

How deep it goes
Shows archive attachment and a parent-sampling loop, including retention, scores, and lineage; does not detail database internals.
Examples
Contrasts a greedy peak of 6 with an archive-reached peak of 10, then revisits a low-scoring revenue-prompt branch.
What it assumes
Assumes familiarity with agent prompts and scored candidate search; starts archive use from an initial prompt artifact.
What you can do after
Operate/configure it to persist prompt variants, distinguish best() from promote(), and sample old candidates during search.
How it connects
Connects prompt artifacts to measured signals, evolutionary search, quality-diversity sampling, and deployment promotion.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Self-Improving Agents. The section links point to the parts of the book it draws on.