🤖 Robots.txt
🗺️ Sitemap
📖 AI Crawlers
📋 Website Template
🤖 AI Crawler Management
📝 Custom Rules
📄 Generated robots.txt
Click "Generate" to create your robots.txt
🔍 URL Tester
🌐 Sitemap Settings
📝 URLs (one per line)
📄 Generated sitemap.xml
Click "Generate Sitemap" to create your sitemap.xml
📖 AI Crawler Reference Guide (Updated Feb 2026)
Comprehensive list of AI bot user-agents. Data sourced from official documentation and web research.
| User-Agent | Vendor | Type | Description |
|---|