Robots.txt Tester

Robots.txt Tester checks a URL or path against the rules in a robots.txt file you paste, for the crawler you name. It selects the matching user-agent group, evaluates every Allow and Disallow rule with * wildcards and $ end anchors, and reports the deciding rule, its length and whether the URL is allowed or blocked. Nothing is fetched.

A single misplaced slash or wildcard in robots.txt can block important pages from being crawled. Testing a draft before publishing it shows exactly which rule decides each URL.

1. Paste the contents of your robots.txt file.
2. Enter a full URL or a path that starts with /.
3. Enter the crawler’s user-agent token, such as Googlebot, or * for all crawlers.
4. Press Test URL.
5. Read the result, matching group and deciding rule, then copy or download the summary.

• Case-insensitive user-agent groups with fallback to *
• Groups for the same crawler combined, as the standard requires
• * wildcards and $ end-of-path anchors
• Longest matching rule wins; Allow wins a tie
• Crawl-delay and Sitemap lines shown for information
• Parser notes for unknown fields and misplaced rules

• Test a page you want crawled and a page you want blocked after every change.
• Remember that robots.txt rules are case-sensitive for paths.
• Use noindex rather than robots.txt to keep crawlable pages out of results.
• Keep a “User-agent: *” group as a sensible default.