Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →brycewang-stanford/web-scraping-ethics-guide
Scrape web data ethically and legally for research purposes
Passed 18 of 21 test cases with the skill, and 15 of 21 without it.
and 3 more in the full report
3 fixed16 passed either way3 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 2 test cases the skill fixed. Same model, same prompt, with and without the skill.
If a researcher calls `assess_irb_requirements(data_type='product_prices', contains_pii=False)`, what values should be returned for `irb_required` and `recommendation`?
| Checked | Without skill | With skill |
|---|---|---|
| Sets irb_required to indicate the project is possibly exempt subject to IRB verification. | Missed | Passed |
| Recommends submitting an IRB exemption request. | Missed | Passed |
The instructions your model reads when the skill loads.
A skill for collecting web data ethically and legally for research purposes. Covers robots.txt compliance, rate limiting, legal frameworks, data privacy considerations, and practical scraping techniques that respect website operators and comply with institutional review requirements.
1. Respect robots.txt and Terms of Service
- Check robots.txt before scraping any site
- Review the site's ToS for explicit prohibitions
- When in doubt, contact the site operator
2. Minimize server impact
- Use rate limiting (1-2 requests per second maximum)
- Scrape during off-peak hours when possible
- Cache responses to avoid redundant requests
- Use conditional requests (If-Modified-Since headers)
3. Collect only what you need
- Define your data requirements before scraping
- Do not scrape personal data without ethical justification
- Anonymize or pseudonymize personal information
4. Attribution and transparency
- Set a descriptive User-Agent header with contact info
- Be prepared to identify yourself if contacted
- Credit data sources in publications
5. Institutional compliance
- Check if your IRB/ethics board requires approval for web data
- Follow your institution's acceptable use policy
- Consider data protection regulations (GDPR, CCPA)
test cases passed, vs 15 / 21 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.