Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →brycewang-stanford/academic-web-scraping
Ethical web scraping and API-based data collection for research
Passed 21 of 24 test cases with the skill, and 18 of 24 without it.
and 1 more in the full report
3 fixed18 passed either way3 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 3 test cases the skill fixed. Same model, same prompt, with and without the skill.
I have written a new web scraping script targeting 20,000 pages of historical archives. I plan to immediately kick off the full batch job across all 20,000 links. What validation step should be performed before running large-scale data collection?
| Checked | Without skill | With skill |
|---|---|---|
| Expect initial scraping logic to be tested on a small sample of around 10 records before launching full extraction. | Missed | Passed |
The instructions your model reads when the skill loads.
Research often requires collecting data from the web -- whether it is bibliographic metadata from academic databases, experimental datasets from public repositories, social media posts for computational social science, or economic indicators from government portals. Web scraping and API-based data collection are essential skills for modern researchers across disciplines.
This guide covers both approaches: structured API access for platforms that provide one, and web scraping for when no API exists. It emphasizes ethical data collection practices, including respecting robots.txt, rate limiting, terms of service compliance, and IRB considerations for human-subject data. The goal is to collect research data reliably and responsibly.
Whether you are building a dataset for a machine learning paper, collecting metadata for a systematic review, or gathering public data for policy research, these patterns help you do it correctly and efficiently.
APIs are always preferable to scraping when available. They provide structured data, are officially supported, and have clear usage terms.
| API | Data | Rate Limit | Auth |
|---|---|---|---|
| OpenAlex | Papers, authors, venues, concepts | 100K req/day | Email in header |
| Crossref | DOI metadata | 50 req/sec (polite pool) | Email in header |
| PubMed (Entrez) | Biomedical literature | 10 req/sec (with key) | API key (free) |
| arXiv | Preprints | 1 req/3sec | None |
| CORE | Open access papers | 10 req/sec | API key (free) |
import requests
import time
class OpenAlexClient:
BASE_URL = "https://api.openalex.org"
def __init__(self, email):
self.session = requests.Session()
self.session.headers.update({
'User-Agent': f'ResearchBot/1.0 (mailto:{email})'
})
def search_works(self, query, filters=None, per_page=25, max_results=100):
test cases passed, vs 18 / 24 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.