robots-txt

Name: robots-txt
Rating: 5 (1 reviews)
Author: bg-szy

When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawlers," "GPTBot," "allow/disallow," "disallow path," "crawl directives," "user-agent," "block Googlebot," "fix robots.txt," "robots.txt blocking," or "search engine crawling." For indexing, use indexing.

1stars

Updated 13 days ago

View on GitHub ↗

How to add

/plugin marketplace add bg-szy/TOP-SKILLS

The exact command may vary by repository. Check the README on GitHub.

For the skill author

Drop this on your repo README

Shows your skill is listed on Skillteca, generates a backlink and trackable traffic.

[![Listada na Skillteca](https://www.skillteca.com.br/api/badge/robots-txt/svg)](https://www.skillteca.com.br/skills/robots-txt?utm_source=badge&utm_medium=readme&utm_campaign=badge)

#ai

Related skills

See all in Pesquisa e Web →

dev-browser

60.3k

Automates browser interactions with persistent page state. Use for navigating websites, filling forms, taking screenshots, extracting web data, testing web apps, or automating browser workflows.

Pesquisa e Web#testby code-yeongyu

agent-browser

55.6k

Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to "open a website", "fill out a form", "click a button", "take a screenshot", "scrape data from a page", "test this web app", "login to

Pesquisa e Web#ai#testby shanraisshan

understand-chat

45.5k

Use when you need to ask questions about a codebase or understand code using a knowledge graph.

Pesquisa e Webby Lum1104

understand-dashboard

45.5k

Launch the interactive web dashboard to visualize a codebase's knowledge graph.

Pesquisa e Webby Lum1104

Category alert

Get new Pesquisa e Web skills every Monday

One short email with only the new Pesquisa e Web skills. 4 minutes of reading, no spam, unsubscribe with one click.

You confirm your email on the first send. No spam. Unsubscribe with one click.

SEO Technical: robots.txt

Guides configuration and auditing of robots.txt for search engine and AI crawler control.

When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output.

Scope (Technical SEO)

Robots.txt: Configure Disallow/Allow, Sitemap, Clean-param; audit for accidental blocks
Crawler access: Pat

[Description truncada. Veja o README completo no GitHub.]

ShareX LinkedIn

Comments · No comments

No comments yet. Be the first.