How robots.txt Works

robots.txt is a plain text file at the root of a host — https://example.com/robots.txt — that tells crawlers which paths they may fetch. It has existed since 1994 as an informal convention and was finally standardised in 2022 as RFC 9309, the Robots Exclusion Protocol. It is a request, not access control: well-behaved crawlers follow it, but nothing stops a client from ignoring it.

Where the file lives and what it covers

It must be at the root of the host, named exactly robots.txt in lowercase. A file in a subdirectory is ignored.
It applies only to the exact protocol, host and port it was served from. https://example.com, https://www.example.com and https://shop.example.com each need their own file.
It must be UTF-8 plain text. Google parses up to 500 KiB and ignores anything beyond that.
If the file returns 404 (or another 4xx), crawlers assume there are no restrictions. If it returns a 5xx or is unreachable, RFC 9309 says crawlers should assume everything is disallowed — so a broken server config can quietly halt crawling of an entire site.
Crawlers cache the file, typically for up to 24 hours, so changes are not instant.

Groups and rules

A robots.txt file is a list of groups. Each group starts with one or more User-agent lines followed by Allow and Disallow rules:

# Rules for every crawler without a more specific group
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?orderby=

# Googlebot gets its own group and ignores the * group entirely
User-agent: Googlebot
Disallow: /internal-search/

# Several user agents can share one group
User-agent: GPTBot
User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap_index.xml

Which group applies

A crawler picks the group whose User-agent value matches its product token most specifically (case-insensitive). If none matches, it uses the * group. Crucially, groups do not inherit from each other. In the example above, Googlebot obeys only Disallow: /internal-search/ and is free to crawl /cart/. If you want a bot-specific group to include the general rules, repeat them.

If the same user agent appears in several groups, the rules are combined into one. Rules are paths and always start with /; values are case-sensitive, so /Admin/ does not block /admin/.

Which rule wins

Within a group, the order of lines doesn't matter. The crawler finds every rule that matches the URL path and uses the one with the longest matching path (the most specific). If an Allow and a Disallow rule are equally long, Allow wins.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# /wp-admin/options.php     → blocked  (matches only Disallow: /wp-admin/)
# /wp-admin/admin-ajax.php  → allowed  (Allow rule is longer, so it wins)

An empty Disallow: line means “nothing is disallowed”. Disallow: / blocks the whole site.

Wildcards: * and $

RFC 9309 defines two special characters in paths:

* matches any sequence of characters, including none.
$ anchors the rule to the end of the URL.

Disallow: /*?s=          # WordPress search result URLs
Disallow: /*.pdf$         # any URL ending in .pdf
Disallow: /tag/*/feed/    # tag feeds
Allow: /*.css$            # explicitly allow stylesheets

Every rule is already a prefix match, so a trailing * (/private*) is redundant. Regular expressions are not supported.

The Sitemap line

Sitemap: is not part of any group and can appear anywhere in the file. It must be an absolute URL, and you can list several. It is a hint for discovery; submitting the sitemap in Google Search Console and Bing Webmaster Tools is still worthwhile. If you need to create a sitemap file, the sitemap generator builds one.

Crawl-delay is a non-standard directive. Bing and some other crawlers honour it; Google ignores it.

Crawling is not indexing

This is the most misunderstood part of robots.txt. Disallow stops a crawler from fetching a URL. It does not stop the URL from being indexed. If other pages link to a blocked URL, Google can still list it in results — typically with no description, because it never saw the content.

To keep a page out of search results:

Add <meta name="robots" content="noindex"> to the page, or send an X-Robots-Tag: noindex HTTP header (useful for PDFs and other non-HTML files).
Make sure the URL is not blocked in robots.txt. If the crawler can't fetch the page, it never sees the noindex.
Once the page has dropped out of the index, you may block it if crawl budget matters — but usually there is no need.

For truly private content, use authentication. robots.txt is public, so listing secret paths in it advertises them. Google's robots.txt introduction covers the same point. The meta tag generator outputs the correct robots meta tag.

robots.txt in WordPress

The virtual file

WordPress does not ship a physical robots.txt. When a request for /robots.txt reaches WordPress and no file exists on disk, it generates one on the fly. On a public site running core sitemaps, the output is:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

SEO plugins such as Yoast SEO and Rank Math let you edit this virtual output from the admin and swap in their own sitemap URL. In code, use the robots_txt filter:

<?php
add_filter( 'robots_txt', function ( $output, $public ) {
    if ( $public ) {
        $output .= "Disallow: /*?s=\n";
    }
    return $output;
}, 10, 2 );

If you upload a real robots.txt file to the web root, the web server serves it directly and the virtual file (and plugin settings for it) are bypassed. That is a common reason edits in an SEO plugin “don't work”.

“Discourage search engines from indexing this site”

The checkbox under Settings → Reading sets the blog_public option to 0. Since WordPress 5.3, it no longer relies on robots.txt; it outputs a noindex, nofollow robots meta tag on every page. That is the correct approach for keeping a site out of the index, but it is still only a request. Staging sites should be protected with HTTP authentication or IP restrictions as well — otherwise they can be crawled, linked and occasionally indexed. And remember to untick the box at launch; the WordPress launch checklist includes this step.

AI crawler user agents

Many AI companies publish user-agent tokens you can address in robots.txt. Some are used for training data, some for search features and some for fetching pages on a user's request, and companies separate them differently. Commonly referenced tokens include:

GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI)
ClaudeBot, Claude-SearchBot and Claude-User (Anthropic)
Google-Extended — a control token, not a separate crawler, that governs use of content for Google's AI models; it does not affect Google Search
Applebot-Extended (Apple), PerplexityBot, CCBot (Common Crawl), meta-externalagent (Meta)

# Opt out of model training while staying in normal search
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

Token names and their purposes change, so check each vendor's current documentation before relying on a list. Blocking a search-oriented AI agent can remove your site from that product's answers and citations.

Testing

Google Search Console → Settings → robots.txt shows the versions Google fetched, when, and any parse errors. You can request a recrawl after an urgent fix.
URL Inspection in Search Console tells you whether a specific URL is blocked by robots.txt.
Fetch the live file with curl -i https://example.com/robots.txt to confirm the status code is 200 and the content type is text/plain.

Common mistakes

Blocking CSS and JavaScript (for example Disallow: /wp-includes/ or /wp-content/plugins/). Google renders pages like a browser; without these files it may misjudge layout and mobile-friendliness.
Deploying a staging robots.txt containing Disallow: / to production. Check the file immediately after every launch or migration.
Using Disallow to de-index pages, which prevents crawlers from seeing noindex.
Expecting groups to inherit rules from User-agent: *.
Relative sitemap URLs or a sitemap on a different protocol than the site.
Forgetting subdomains, which each need their own file.

The robots.txt generator produces a valid file with WordPress-friendly defaults, optional AI crawler rules and your sitemap URL.