Move skills to .agent directory
This commit is contained in:
31
.agents/skills/agent-ready-cloudflare/robots-txt/SKILL.md
Normal file
31
.agents/skills/agent-ready-cloudflare/robots-txt/SKILL.md
Normal file
@@ -0,0 +1,31 @@
|
||||
---
|
||||
name: agent-ready-robots-txt
|
||||
description: >
|
||||
Sub-skill de agent-ready-cloudflare: Implement robots.txt
|
||||
---
|
||||
# Implement robots.txt
|
||||
|
||||
Publish a valid robots.txt at your site root per
|
||||
[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309).
|
||||
|
||||
## Requirements
|
||||
|
||||
- Serve `/robots.txt` as `text/plain` with HTTP 200
|
||||
- Include `User-agent` directives with `Allow`/`Disallow` rules
|
||||
- Reference your sitemap if one exists: `Sitemap: https://example.com/sitemap.xml`
|
||||
|
||||
## Cloudflare
|
||||
|
||||
[AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/)
|
||||
can manage your robots.txt from the dashboard, including AI-specific bot rules.
|
||||
|
||||
## Validate
|
||||
|
||||
```
|
||||
POST https://isitagentready.com/api/scan
|
||||
Content-Type: application/json
|
||||
|
||||
{"url": "https://YOUR-SITE.com"}
|
||||
```
|
||||
|
||||
Check that `checks.discoverability.robotsTxt.status` is `"pass"`.
|
||||
Reference in New Issue
Block a user