πŸ€– Robots.txt
πŸ—ΊοΈ Sitemap
πŸ“– AI Crawlers

πŸ“‹ Website Template

πŸ€– AI Crawler Management

πŸ“ Custom Rules

πŸ“„ Generated robots.txt

Click "Generate" to create your robots.txt

πŸ” URL Tester

🌐 Sitemap Settings

πŸ“ URLs (one per line)

πŸ“„ Generated sitemap.xml

Click "Generate Sitemap" to create your sitemap.xml

πŸ“– AI Crawler Reference Guide (Updated Feb 2026)

Comprehensive list of AI bot user-agents. Data sourced from official documentation and web research.

User-AgentVendorTypeDescription