Robots.txt Generator

Generate a robots.txt file to control how search engines crawl your site

User-agent

Disallow paths (one per line)

Allow paths (one per line)

Sitemap URL

Crawl Delay (seconds)

Generated robots.txt

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

Frequently Asked Questions

What is robots.txt?▼

robots.txt is a text file that tells search engine crawlers which pages on your site they can and cannot access. It lives at the root of your domain (example.com/robots.txt).

Can robots.txt block a page from Google?▼

robots.txt can prevent crawling, but it doesn't guarantee a page won't appear in search results. Use a noindex meta tag instead to prevent indexing.