Robots.txt Generator

Write a robots.txt file by listing the paths crawlers should avoid and adding your sitemap. It runs in your browser.

How to use it

1. Enter the user agent, or leave * for all crawlers.
2. List the paths to block, one per line, or tick “Allow all”.
3. Add your sitemap URL.
4. Copy the text into a file named robots.txt at the root of your site.

Examples

User-agent: *
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

“Allow all” gives User-agent: * and an empty Disallow:, which blocks nothing.

Good to know

robots.txt is a request, not a lock: well-behaved crawlers follow it, anyone can read it, and listing a path shows where private areas are. It does not remove a page from search results; to keep a page out, use a noindex tag and let it be crawled so the tag can be seen. A line like Disallow: / blocks the whole site, so test after changes, and avoid blocking the CSS and JavaScript your pages need.

See the guide and the glossary entry.

FAQs