Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →sediman-agent/robots-txt
When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawlers," "GPTBot," "allow/disallow," "disallow path," "crawl directives," "user-agent," "block Googlebot," "fix robots.txt," "robots.txt blocking," or "search engine crawling." For indexing, use indexing.
Passed 19 of 22 test cases with the skill, and 14 of 22 without it.
and 4 more in the full report
5 fixed14 passed either way3 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 5 test cases the skill fixed. Same model, same prompt, with and without the skill.
We noticed `ChatGPT-User` crawling our site and want to block it via a `Disallow: /` directive under `User-agent: ChatGPT-User` in our robots.txt file. Draft the recommended configuration for our web server root.
| Checked | Without skill | With skill |
|---|---|---|
| The output omits ChatGPT-User from the robots.txt file. | Missed | Passed |
| The output explicitly notes that server-side controls like WAF or IP rate-limiting are required to block ChatGPT-User. | Missed | Passed |
The instructions your model reads when the skill loads.
Guides configuration and auditing of robots.txt for search engine and AI crawler control.
When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output.
Check for project context first: If .claude/project-context.md or .cursor/project-context.md exists, read it for site URL and indexing goals.
Identify:
https://example.com)| Point | Note |
|---|---|
| Purpose | Controls crawler access; does NOT prevent indexing (disallowed URLs may still appear in search without snippet) |
| Advisory | Rules are advisory; malicious crawlers may ignore |
| Public | robots.txt is publicly readable; use noindex or auth for sensitive content. See indexing |
| Tool | Controls | Prevents indexing? |
|---|---|---|
| robots.txt | Crawl (path-level) | No—blocked URLs may still appear in SERP |
| noindex (meta / X-Robots-Tag) | Index (page-level) | Yes. See indexing |
| nofollow | Link equity only | No—does not control indexing |
test cases passed, vs 14 / 22 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.