Robots.txt Tester
Fetch and analyze a site's robots.txt. Test whether specific URLs are blocked or allowed.
We'll fetch /robots.txt from this domain
Enter URLs to test against the robots.txt rules (one per line, max 50)
Like this tool? Meet the full platform
SECrawl is the SEO platform behind these free tools — crawl thousands of pages, run 40+ SEO, AI-readiness, performance and security reports, and chat with the results to find what to fix first. Start free, no credit card.