Robots.txt Generator — Help & Documentation

Generate custom robots.txt files to control search engine crawling. Includes pre-built templates for WordPress, e-commerce, and other common platform types.

Overview: Generate custom robots.txt files to control search engine crawling behavior. Includes templates for WordPress, e-commerce, and various website types.

Features

Pre-built templates for common platforms
Custom rule creation
Syntax validation and error checking
Sitemap URL integration
Bot-specific directives
Comments and documentation support

Robots.txt Directives

Directive	Description	Example
User-agent	Specifies which bot the rules apply to	User-agent: *
Disallow	Prevents crawling of specified paths	Disallow: /admin/
Allow	Explicitly allows crawling (overrides Disallow)	Allow: /public/
Sitemap	Points to XML sitemap location	Sitemap: https://example.com/sitemap.xml
Crawl-delay	Sets delay between requests in seconds	Crawl-delay: 10

Common Templates

WordPress

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap.xml

E-commerce

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /search?
Sitemap: https://yourstore.com/sitemap.xml

Best Practices

Do

Block admin and private areas
Include sitemap URL
Use specific bot directives when needed
Test your robots.txt file
Keep it simple and clear

Don't

Block important SEO pages
Use robots.txt for security purposes
Block CSS or JavaScript files
Forget to update after site changes
Use wildcard patterns incorrectly

Testing Your Robots.txt

Important Testing Steps

Use Google Search Console's robots.txt Tester

Test specific URLs and user agents

Check for syntax errors

Verify sitemap accessibility

Monitor for crawl errors after deployment

Common Bots

Googlebot: Google's web crawler
Bingbot: Bing's search crawler
Slurp: Yahoo's web crawler
DuckDuckBot: DuckDuckGo crawler
FacebookExternalHit: Facebook's link preview bot