To create a robots.txt file, set your allow and disallow rules and sitemap URL in the Robots.txt Generator and copy the result to a file named robots.txt in your site root. It tells crawlers which paths they may fetch.
Follow along with the free Robots.txt Generator, which runs in your browser, or browse more tools in this category.
How to use the Robots.txt Generator
- Choose the user agent, or use all crawlers.
- Add allow and disallow rules.
- Add your sitemap URL.
- Save the output as robots.txt at the root of your domain.
Worked example
User-agent: * with Disallow: /admin/ and Sitemap: https://example.com/sitemap.xml blocks the admin area and points crawlers to your sitemap.
Tips and common mistakes
- robots.txt is a request, not security; do not use it to hide private pages.
- Disallowing a page does not remove it from search if it is linked elsewhere.
- Never block CSS or JavaScript that pages need to render.
Frequently asked questions
Where does robots.txt go?
At the root of the domain, such as example.com/robots.txt.
Does it stop pages being indexed?
Not reliably. Use noindex for that.
Is it private?
Yes.