What it contains
Lines starting with User-agent name a crawler (or * for all), Disallow lists paths to avoid, and Sitemap points to your sitemap.
What it does not do
- It does not hide content: anyone can read the file.
- It does not remove a page from search results. A blocked page can still be listed if other sites link to it.
- A mistaken
Disallow: /blocks the whole site.
Try it
The Robots.txt Generator writes the file, and the guide shows examples.
Common misunderstanding
To keep a page out of search results, use a noindex tag instead, and let the page be crawled so the tag can be seen.