robots.txt Generator

robots.txt Generator creates a clean, correctly formatted robots.txt file from rule groups you define. Each group can target one or more user-agents and contain Disallow and Allow paths, an optional Crawl-delay and a comment, and you can add Sitemap lines for the whole file.

A typo in robots.txt can accidentally ask crawlers to skip your whole site, or leave in place rules you meant to remove. Building the file from structured inputs with validation keeps the syntax clean. Remember that robots.txt is guidance for crawlers, not access control.

1. Start with the default group for all crawlers (User-agent: *).
2. Add Disallow paths for areas crawlers should skip, and Allow paths for exceptions.
3. Add more rule groups for specific crawlers, and reorder or remove groups as needed.
4. Enter your sitemap URL and an optional comment.
5. Check the live preview and fix any highlighted fields.
6. Copy the text or download robots.txt and upload it to your site root.

• Multiple user-agent rule groups — add, remove and reorder
• Allow, Disallow, Crawl-delay and comment per group
• One or more Sitemap lines
• Validation for paths, user-agents, Crawl-delay and sitemap URLs
• Live preview with correct line breaks
• Copy or download as robots.txt

• Keep the file short and specific; start every path with /.
• Do not block CSS and JavaScript files that pages need to render.
• Never list secret URLs — the file is public and can be ignored by crawlers.
• Use a noindex robots meta tag, not Disallow, to keep a crawlable page out of search results.
• After uploading, open your-domain/robots.txt in a browser to confirm it is live.