Implement a web crawler pipeline covering URL discovery, fetching, parsing, and storage.
SkillScore82Maintained
Automatic
◆Data Engineer pack
Last verified 8/28/2026
What it changes
Passed 18 of 22 test cases with the skill, and 12 of 22 without it.
Without skill55%
With skill82%
Checks it passed that the model alone missed
Removes trailing slashes.
Enforces a default 1 second crawl delay.
Applies a default crawl delay of 1 second.
and 8 more in the full report
6 fixed12 passed either way4 still failing0 broke
Pass-rate lift
+27 pp
Tokens
+25%
Agent turns
0%
Response time
-20%
Model gemini-3.6-flashMeasured 8/28/2026v118 of 22 cases passed with skill
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 6 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 6
Prompt
The `robots.txt` file for `https://forum.community.org` allows path crawling but omits an explicit `Crawl-delay` directive. What default delay interval must be enforced between HTTP requests to this domain?
What was checked, without and with the skill
Checked
Without skill
With skill
Applies a default crawl delay of 1 second.
Missed
Passed
case-11 · gemini-3.6-flash · 8/28/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md81 lines3 files
Web Crawler
Overview
A web crawler systematically traverses web pages by discovering URLs, fetching content, parsing HTML, and storing results. Uses BFS or priority-based frontier management. Performance is I/O-bound, typically limited by politeness constraints rather than compute.
When to Use
Trigger conditions:
Building a site audit tool to discover all pages and their link structure
Collecting structured data from websites at scale
Mapping site architecture for SEO analysis
When NOT to use:
When you need data from a single API endpoint (use HTTP client directly)
When a sitemap.xml provides all needed URLs (parse sitemap instead)
Algorithm
IRON LAW: Respect robots.txt and Rate Limits
A crawler MUST:
1. Parse and obey robots.txt before crawling any path
2. Enforce crawl-delay (default 1s if unspecified)
3. Identify itself with a descriptive User-Agent
Ignoring these is unethical and will get your IP blocked.
Phase 1: Input Validation
Parse seed URLs, fetch and parse robots.txt for each domain, set crawl scope (same-domain, subdomain, or cross-domain).
Gate: Valid seed URLs, robots.txt rules loaded, scope defined.
Phase 2: Core Algorithm
Initialize URL frontier with seed URLs (priority queue or FIFO)
Dequeue URL, check: not visited, allowed by robots.txt, within scope
Fetch page with timeout and retry logic, respect crawl-delay