Technical SEO audit across 9 categories: crawlability, indexability, security, URL structure, mobile, Core Web Vitals, structured data, JavaScript rendering, and IndexNow protocol.
SkillScore95Maintained
Automatic
◆Marketing & Content pack
Last verified 8/15/2026
What it changes
Passed 21 of 22 test cases with the skill, and 15 of 22 without it.
Without skill68%
With skill95%
Checks it passed that the model alone missed
Includes IndexNow as a category in the breakdown table.
Suggests verifying DNS resolution and public site availability.
Includes the section heading ### Critical Issues (fix immediately).
and 4 more in the full report
6 fixed15 passed either way1 still failing0 broke
Pass-rate lift
+27 pp
Tokens
+89%
Agent turns
0%
Response time
-9%
Model gemini-3.6-flashMeasured 8/15/2026v221 of 22 cases passed with skill
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 6 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 6
Prompt
We have configured Google API credentials in our environment and want to run a PageSpeed Insights test CLI command to pull real user CrUX field data and lab metrics for `https://example.com`. What script should be executed?
What was checked, without and with the skill
Checked
Without skill
With skill
Recommends running python3 scripts/pagespeed_check.py for PageSpeed Insights and CrUX data.
Missed
Passed
case-21 · gemini-3.6-flash · 8/15/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md186 lines2 files
Technical SEO Audit
Categories
1. Crawlability
robots.txt: exists, valid, not blocking important resources
XML sitemap: exists, referenced in robots.txt, valid format
Noindex tags: intentional vs accidental
Crawl depth: important pages within 3 clicks of homepage
JavaScript rendering: check if critical content requires JS execution
Crawl budget: for large sites (>10k pages), efficiency matters
AI Crawler Management
As of 2025-2026, AI companies actively crawl the web to train models and power AI search. Managing these crawlers via robots.txt is a critical technical SEO consideration.
Known AI crawlers:
Crawler
Company
robots.txt token
Purpose
GPTBot
OpenAI
GPTBot
Model training
ChatGPT-User
OpenAI
ChatGPT-User
Real-time browsing
ClaudeBot
Anthropic
ClaudeBot
Model training
PerplexityBot
Perplexity
PerplexityBot
Search index + training
Bytespider
ByteDance
Bytespider
Model training
Google-Extended
Google
Google-Extended
Gemini training (NOT search)
CCBot
Common Crawl
CCBot
Open dataset
Key distinctions:
Blocking Google-Extended prevents Gemini training use but does NOT affect Google Search indexing or AI Overviews (those use Googlebot)
Blocking GPTBot prevents OpenAI training but does NOT prevent ChatGPT from citing your content via browsing (ChatGPT-User)
~3-5% of websites now use AI-specific robots.txt rules
Example, selective AI crawler blocking:
# Allow search indexing, block AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended