Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →Use when applies to any public website. Use when auditing technical SEO foundations or diagnosing crawl coverage gaps.
.claude/skills/thedaviddias-robots-txt/SKILL.md| Test case | Without → With | Effect | Δ tokens | Δ turns |
|---|---|---|---|---|
| case-01 | ✗→✓ | ▲ Improved | -18% | 0% |
| case-16 | ✗→✓ | ▲ Improved | 6% | 0% |
| case-02 | ✓→✓ | ＝ Same ✓ | 35% | 0% |
| case-03 | ✓→✓ | ＝ Same ✓ | -7% | 0% |
| case-04 | ✓→✓ | ＝ Same ✓ | -10% | 0% |
robots.txt is the first file crawlers fetch; misconfigured directives can silently block search engines from crawling your entire site, killing organic visibility.
robots.txt at /robots.txt on the production domain, returning HTTP 200Sitemap: directive pointing to your XML sitemapDisallow: / on a live siteFetch /robots.txt on the live domain and verify it returns HTTP 200, uses correct User-agent / Disallow / Allow syntax, and includes a Sitemap: directive pointing to the XML sitemap. Check for accidental Disallow: / directives.
Create or update robots.txt at the web root with valid directives. Add a live Sitemap: line for the production sitemap URL. Remove any Disallow: / rules that block the whole site or resources needed for rendering.
Explain how robots.txt controls crawler access, why an accidental Disallow: / can delist a site, and why CSS/JS must remain accessible for rendering-based indexing.
Review metadata generation, rendered HTML, structured data, and response headers related to Publish a robots.txt file. Flag exact routes or templates where search-facing output violates the rule, and describe how to verify the final page output.
For full implementation details, code examples, and framework-specific guidance, see references/rule.md.
Rule page: https://frontendchecklist.io/en/rules/seo/robots-txt
| Case | Status | Duration (ms) | Turns | Tokens | Tool calls | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Without | With | Δ | Without | With | Δ | Without | With | Δ | Without | With | Δ | ||
case-01 | fail→pass | 16,596 | 10,573 | -36% | 1 | 1 | 0% | 2,950 | 2,419 | -18% | 0 | 0 | — |
case-02 | pass→pass | 4,330 | 4,073 | -6% | 1 | 1 | 0% | 848 | 1,147 | +35% | 0 | 0 | — |
case-03 | pass→pass | 9,072 | 6,620 | -27% | 1 | 1 | 0% | 1,921 | 1,793 | -7% | 0 | 0 | — |
case-04 | pass→pass | 10,114 | 7,681 | -24% | 1 | 1 | 0% | 2,077 | 1,869 | -10% | 0 | 0 | — |
case-05 | pass→pass | 8,768 | 7,572 | -14% | 1 | 1 | 0% | 1,697 | 1,811 | +7% | 0 | 0 | — |
case-06 | pass→pass | 10,704 | 5,208 | -51% | 1 | 1 | 0% | 1,854 | 1,309 | -29% | 0 | 0 | — |
case-07 | pass→pass | 11,604 | 8,831 | -24% | 1 | 1 | 0% | 1,987 | 1,861 | -6% | 0 | 0 | — |
case-08 | pass→pass | 4,622 | 4,556 | -1% | 1 | 1 | 0% | 953 | 1,250 | +31% | 0 | 0 | — |
case-09 | pass→pass | 5,428 | 5,186 | -4% | 1 | 1 | 0% | 1,066 | 1,381 | +30% | 0 | 0 | — |
case-10 | pass→pass | 5,703 | 4,976 | -13% | 1 | 1 | 0% | 1,111 | 1,300 | +17% | 0 | 0 | — |
case-11 | pass→pass | 6,542 | 5,438 | -17% | 1 | 1 | 0% | 1,179 | 1,419 | +20% | 0 | 0 | — |
case-12 | pass→pass | 9,151 | 7,106 | -22% | 1 | 1 | 0% | 1,738 | 1,712 | -1% | 0 | 0 | — |
case-13 | pass→pass | 7,610 | 6,673 | -12% | 1 | 1 | 0% | 1,321 | 1,604 | +21% | 0 | 0 | — |
case-14 | pass→pass | 7,529 | 6,917 | -8% | 1 | 1 | 0% | 1,383 | 1,517 | +10% | 0 | 0 | — |
case-15 | pass→pass | 10,865 | 6,316 | -42% | 1 | 1 | 0% | 1,810 | 1,455 | -20% | 0 | 0 | — |
case-16 | fail→pass | 7,207 | 5,509 | -24% | 1 | 1 | 0% | 1,322 | 1,404 | +6% | 0 | 0 | — |
case-17 | pass→pass | 4,200 | 4,437 | +6% | 1 | 1 | 0% | 841 | 1,214 | +44% | 0 | 0 | — |
case-18 | pass→pass | 15,295 | 15,517 | +1% | 1 | 1 | 0% | 2,425 | 2,951 | +22% | 0 | 0 | — |
case-19 | pass→pass | 7,946 | 5,935 | -25% | 1 | 1 | 0% | 1,357 | 1,465 | +8% | 0 | 0 | — |
case-20 | pass→pass | 3,770 | 2,491 | -34% | 1 | 1 | 0% | 671 | 798 | +19% | 0 | 0 | — |
case-21 | pass→pass | 13,724 | 9,181 | -33% | 1 | 1 | 0% | 2,419 | 2,075 | -14% | 0 | 0 | — |
case-22 | pass→pass | 8,753 | 6,876 | -21% | 1 | 1 | 0% | 1,635 | 1,616 | -1% | 0 | 0 | — |
case-23 | pass→pass | 6,134 | 7,118 | +16% | 1 | 1 | 0% | 1,103 | 1,678 | +52% | 0 | 0 | — |
DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 23 cases were attempted. The headline lift of +9 percentage points is the difference between those two pass rates over the 23 comparable cases.
Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.
Other measured skills in the registry, with their headline benchmark lift.