When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawlers," "GPTBot," "allow/disallow," "disallow path," "crawl directives," "user-agent," "block Googlebot," "fix robots.txt," "robots.txt blocking," or "search engine crawling." For indexing, use indexing.
Receba novas skills de Pesquisa e Web toda segunda
1 email curto, só com as skills novas de Pesquisa e Web. 4 minutos de leitura, sem spam, cancela com 1 clique.
Você confirma o email no primeiro envio. Sem spam. Cancela com 1 clique.
Ler descrição completaRecolher descrição
SEO Technical: robots.txt
Guides configuration and auditing of robots.txt for search engine and AI crawler control.
When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output.
Scope (Technical SEO)
Robots.txt: Configure Disallow/Allow, Sitemap, Clean-param; audit for accidental blocks
Crawler access: Pat
[Description truncada. Veja o README completo no GitHub.]
