SlipItIn/.agents/skills/agent-ready-cloudflare/robots-txt/SKILL.md

name, description

name	description
agent-ready-robots-txt	Sub-skill de agent-ready-cloudflare: Implement robots.txt

Implement robots.txt

Publish a valid robots.txt at your site root per RFC 9309.

Requirements

• Serve /robots.txt as text/plain with HTTP 200
• Include User-agent directives with Allow/Disallow rules
• Reference your sitemap if one exists: Sitemap: https://example.com/sitemap.xml

Cloudflare

AI Crawl Control can manage your robots.txt from the dashboard, including AI-specific bot rules.

Validate

POST https://isitagentready.com/api/scan
Content-Type: application/json

{"url": "https://YOUR-SITE.com"}

Check that checks.discoverability.robotsTxt.status is "pass".