SlipItIn/.agents/skills/agent-ready-cloudflare/ai-rules/SKILL.md

name, description

name	description
agent-ready-ai-rules	Sub-skill de agent-ready-cloudflare: Implement AI Bot Rules in robots.txt

Implement AI Bot Rules in robots.txt

Add explicit User-agent entries for AI crawlers in your robots.txt per RFC 9309.

Requirements

• Add User-agent blocks for AI-specific bots: GPTBot, OAI-SearchBot, Claude-Web, Google-Extended, Amazonbot, anthropic-ai, Bytespider, CCBot, Applebot-Extended
• Set Allow and/or Disallow rules matching your content policy
• A wildcard User-agent: * block alone is not sufficient — explicit AI bot entries are required

Cloudflare

AI Crawl Control lets you manage AI bot rules from the dashboard without editing robots.txt manually.

Validate

POST https://isitagentready.com/api/scan
Content-Type: application/json

{"url": "https://YOUR-SITE.com"}

Check that checks.botAccessControl.robotsTxtAiRules.status is "pass".