Web crawler
A web crawler is an automated program that retrieves and processes web resources, follows links, and discovers content for indexing or analysis.
Grokking AI Algorithms, Second Edition
A web crawler is software that automatically visits web pages and follows links to collect or index web content.
- How it is used
- Indexes web pages through recursive link following and collects broad public text for language-model training.
- How deep it goes
- Defines crawling’s recursive process and practical scale, but stops short of implementation details and crawler engineering.
- Examples
- Uses recursive page traversal, Google’s 1998 architecture, and noisy public-internet text collected for language models.
- What you can do after
- Evaluate crawling as a scalable way to index web content or gather language-model data, including its need for substantial cleaning.
Read how Grokking AI Algorithms, Second Edition explains web crawler →
Modern Angular
A web crawler is an automated program that traverses web pages, extracts information, and discovers links for indexing or analysis.
- How it is used
- Explains why search visibility and SEO become concerns for public Angular websites rendered as single-page applications.
- How deep it goes
- Defines crawling and traces the SPA indexing problem to initial HTML and JavaScript execution, without detailing crawler internals.
- Examples
- Uses a legal company targeting “lawyer in ABQ” and an Angular index.html document containing only app-root and script references.
- What you can do after
- Evaluate how client-side rendering affects search indexing and public-site traffic.
Related concepts
- bloom filteruses · A Bloom filter is a space-efficient probabilistic data structure for testing set membership that permits false positives but not false negatives.
- aiohttpuses · aiohttp is a Python library for asynchronous HTTP client and server applications built on asyncio.
- Beautiful Soupuses · Beautiful Soup is a Python library for parsing HTML and XML documents, navigating their parsed structures, and extracting or modifying their contents.
- social mediaused by · Social media is a class of online services that enables users to create, share, discover, and interact with content and with one another.
- JavaScriptcontrasts with · JavaScript is a high-level, dynamically typed, multi-paradigm programming language standardized as ECMAScript and used for web and general-purpose programming.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.