How to use the Robots.txt Generator
- 1
Choose the default rule and optional crawl delay.
- 2
Add your sitemap URL and any restricted directories.
- 3
Create the file, then upload it as /robots.txt in your site root.
About the Robots.txt Generator
The Robots.txt Generator builds a valid robots.txt from simple choices: a default rule for all robots, a crawl delay, your sitemap URL, per-bot rules for the main search engines and any folders you want to keep crawlers out of.
Last updated
Frequently asked questions
Does robots.txt hide pages from Google?
It stops crawling, not indexing. To keep a page out of results use a noindex tag instead.
Where must the file be?
At the root of the host: https://example.com/robots.txt.
Is crawl-delay respected?
Bing and some others respect it; Google ignores it.