liveBook · Manning

Url discovery

As explained in Learn AI Data Engineering

URL discovery is the process of finding and selecting a web address for a particular product or other target item.

URL discovery finds the product page to scrape for each input record. Start with a search key formed from the brand and product name, retrieve several candidate pages, then select one stable, relevant URL. It sits between data preparation and HTML fetching: a weak choice sends later extraction work to reviews, forums, category pages, or the wrong variant. section 10.2.2 section 10.4 section 11.1

A repeatable discovery workflow separates retrieval from judgment:

  • Build a search key by combining the brand and product name, such as “GORUCK GR1 26L.” section 10.2.2
  • Call a search API with that key and retain a small candidate set containing each result’s title, URL, snippet, and search position. section 11.4.1
  • Rank candidates for the intended product, preferring official manufacturer pages and individual product pages over review sites, forums, and category pages. section 10.4 section 12.3.2
  • Return the best URL with a confidence level and brief reasoning; treat an empty candidate set as a normal no-result outcome. section 12.3.2

From search key to candidate URLs

For “GORUCK GR1 26L,” a search can return candidates from goruck.com, Amazon, gear-review sites, and Reddit. The useful output is not the whole result page: it is a compact candidate list that preserves the title, URL, snippet, and position needed for a later ranking decision. If no API key is available, create a three-to-five-item candidate list from browser results and run the same ranking step. section 11.4.1

Manual discovery bundles several judgment calls: turn the product name into a query, assess relevance and credibility, choose a source, and capture a URL that can be scraped later. That is manageable once but becomes inconsistent and exhausting across hundreds of spreadsheet rows. A search API plus LLM ranking preserves human-directed selection criteria while making the decision more consistent at scale. section 10.4 section 11.0 section 12.3.2

Listing 11.1 Searching for product URLs with SerpAPI
import os  #A
import requests  #B
import pandas as pd
 
SERPAPI_KEY = os.getenv("SERPAPI_KEY")  #C
 
def search_product_urls(search_key: str, num_results: int = 5) -> list[dict]:  #D
    """Search for product page candidates using SerpAPI."""
    params = {  #E
        "q": search_key,
        "api_key": SERPAPI_KEY,
        "num": num_results,
        "engine": "google",
    }
    resp = requests.get("https://serpapi.com/search", params=params)  #F
    resp.raise_for_status()
    data = resp.json()
    
    candidates = []  #G
    for result in data.get("organic_results", []):
        candidates.append({
            "title": result.get("title", ""),
            "url": result.get("link", ""),
            "snippet": result.get("snippet", ""),
            "position": result.get("position", 0),
        })
    return candidates  #H
 
# Example usage
search_query = "GORUCK GR1 26L"
print(f"\nSearching for: '{search_query}'\n")
 
candidates = search_product_urls(search_query)  #I
 
# Display as DataFrame
df_candidates = pd.DataFrame(candidates)
print(f"Found {len(candidates)} candidate URLs:\n")
display(df_candidates[['position', 'title', 'url']])
 
print(f"\n✓ {len(candidates)} URLs ready for AI ranking")

Inspect the search function for the practical retrieval contract: query parameters, API-key configuration, HTTP failure handling, and conversion of organic results into candidate dictionaries. section 11.4.1

Figure 10.1 The Product Enrichment Pipeline transforms raw product data into fully attributed catalog entries through six sequential stages. The process begins with Data Preparation, where source records are cleaned and standardized, followed by URL Discovery to identify relevant product pages across the web. HTML Fetching retrieves the page content, which then feeds into Image Extraction to capture product visuals. The AI Extraction stage leverages machine learning to parse and interpret unstructured content, extracting key attributes like dimensions, materials, and specifications. Finally, Target Mapping aligns the enriched data to the destination schema, producing catalog-ready records.

View URL discovery in the full enrichment flow, where its chosen page becomes the input to HTML fetching, image extraction, AI extraction, and target mapping. section 10.2.2

How Learn AI Data Engineering covers it

How deep it goes
Works through search-key construction, API candidate retrieval, ranking, confidence reporting, retries, and no-result handling.
Examples
Uses the GORUCK GR1 26L search key and candidates from goruck.com, Amazon, review sites, and Reddit.
What it assumes
Assumes product records and search keys while offering a manual candidate-list fallback when no SerpAPI key is available.
What you can do after
Apply it via a search API and LLM ranking to return a best product URL, confidence, and reasoning.
How it connects
Connects data preparation, search APIs, LLM ranking, HTML fetching, AI extraction, and target mapping.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Learn AI Data Engineering. The section links point to the parts of the book it draws on.