Robots.txt Guide: Syntax, Rules and Practical Examples

A robots.txt file is a plain-text file at the root of your domain (for example https://example.com/robots.txt) that tells crawlers which URLs they may and may not request. It controls crawling, not indexing: a blocked URL can still appear in search results if other pages link to it. Use it to keep bots out of low-value areas such as internal search results and admin paths, and to point crawlers to your sitemap.

Key takeaways

• robots.txt must live at the root of each host and protocol: https://example.com/robots.txt.
• It blocks crawling, not indexing. Use a noindex meta tag to keep a page out of results.
• Never block CSS and JavaScript files that pages need to render.
• Crawlers follow the most specific group that matches their user-agent, and the most specific (longest) matching rule.
• robots.txt is public and voluntary; it is not a security tool.

How robots.txt works

Before a well-behaved crawler fetches pages on a host, it requests /robots.txt and reads the rules. The file is made of groups. Each group starts with one or more User-agent lines, followed by Allow and Disallow rules. A Sitemap line can appear anywhere and applies to the whole file.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

This says: every crawler may access everything except /wp-admin/, but may still fetch admin-ajax.php inside it. The sitemap location is given as a full URL.

The rules were formalized as an internet standard in 2022, RFC 9309, which is a useful reference if you need exact matching behavior.

The directives you will actually use

Google ignores Crawl-delay and Noindex lines in robots.txt. Some other crawlers, such as Bing, do read Crawl-delay. Google stopped supporting noindex in robots.txt in 2019, so it will not remove pages from Google.

Pattern matching

Paths are matched from the start of the URL path and are case-sensitive. Two special characters are widely supported:

• * matches any sequence of characters.
• $ marks the end of the URL.

# Block any URL containing a session parameter
Disallow: /*?sessionid=

# Block PDF files only
Disallow: /*.pdf$

# Block /private and everything under it (also /private-notes!)
Disallow: /private

Note the last example: Disallow: /private also matches /private-notes.html because matching is by prefix. Use /private/ with a trailing slash if you only mean the folder.

Which rule wins

For Google and crawlers following RFC 9309, the most specific rule (the one with the longest matching path) wins. If an Allow and a Disallow are equally specific, Allow wins.

User-agent: *
Disallow: /shop/
Allow: /shop/sale/

Here /shop/sale/shoes is allowed, because /shop/sale/ is the longer match.

Groups work differently: a crawler picks only the single group that best matches its name and ignores the rest. If you have a User-agent: Googlebot group, Googlebot ignores the User-agent: * group completely. Repeat any shared rules in both.

Practical examples

Allow everything

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

An empty Disallow means nothing is blocked. Having no robots.txt at all (a 404) also means "crawl everything".

Typical small business or blog

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Disallow: /cart/
Disallow: /checkout/

Sitemap: https://example.com/sitemap.xml

Online store with filter parameters

User-agent: *
Disallow: /*?*sort=
Disallow: /*?*color=
Disallow: /*?*price=
Disallow: /account/

Sitemap: https://example.com/sitemap-products.xml
Sitemap: https://example.com/sitemap-categories.xml

Faceted navigation can create huge numbers of near-duplicate URLs. Blocking the parameters you never want crawled keeps crawlers focused on real category and product pages.

Block a specific crawler

User-agent: ExampleBot
Disallow: /

User-agent: *
Disallow:

This only works for crawlers that obey robots.txt. Bad bots ignore it, and you need server or firewall rules to stop them.

Staging site

User-agent: *
Disallow: /

This keeps compliant crawlers out, but it does not guarantee the staging URLs stay out of results if they get linked. Password protection is the reliable option. And remember to remove this file when the staging site goes live; it is one of the most common causes of a new site not being indexed.

If you would rather not write the file by hand, the Robots.txt Generator builds one from a form that you can then edit.

robots.txt vs noindex vs other controls

robots.txt is one of several tools for controlling how search engines treat your URLs, and each does a different job. Picking the wrong one is the source of most robots.txt problems.

A header-based noindex looks like this in the server response:

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex

A useful rule: if you want Google to know something about a page (that it is a duplicate, or should not be indexed), Google has to be allowed to crawl it. Only use robots.txt when you do not want the URL fetched at all.

Common mistakes

• Using robots.txt to remove pages from Google. A blocked page cannot be crawled, so Google never sees a noindex tag on it. To deindex a page, allow crawling and add <meta name="robots" content="noindex">.
• Blocking CSS, JavaScript or image folders. Google renders pages like a browser. If it cannot load your stylesheets and scripts, it may misjudge layout and content.
• Hiding sensitive URLs. robots.txt is public. Listing /secret-admin-panel/ in it tells everyone where to look.
• Wrong location. https://example.com/blog/robots.txt is ignored. Each subdomain needs its own file, so shop.example.com does not use the rules from example.com.
• Wrong case. Disallow: /Admin/ does not block /admin/.
• Serving the file as HTML or with errors. It should be plain text and return a 200. If robots.txt returns a 5xx error, Google may stop crawling the site until it can read the file.

How to test your robots.txt

1. Open https://yourdomain/robots.txt in a browser and confirm it loads as plain text.
2. Check the robots.txt report in Google Search Console to see the version Google fetched and any parsing errors.
3. Use URL Inspection in Search Console on an important page to confirm it is not "Blocked by robots.txt".
4. Run a page through the Search Engine Spider Simulator to see the text and links a crawler can read once it is allowed in.
5. After any change, spot-check your key templates: home page, a category, a product or post, and a CSS file.

Quick checklist

• File is at the root of each host, returns 200 and is plain text.
• Only low-value paths are disallowed: admin, cart, internal search, endless filters.
• CSS, JavaScript and images used by pages are crawlable.
• A Sitemap line with a full URL is included.
• No staging-style Disallow: / left on the live site.
• Pages you want out of search use noindex, not robots.txt.