WordPress robots.txt Generator

What is robots.txt and what should WordPress put in it?

robots.txt is a plain-text file at the root of your domain that tells crawlers which URL paths they may fetch. Each group starts with one or more User-agent lines followed by Disallow and Allow rules; the most specific (longest) matching rule wins, and * and $ work as wildcards. A Sitemap line points crawlers to your XML sitemap.

The most important thing to understand is that robots.txt controls crawling, not indexing. A disallowed URL can still appear in Google if other pages link to it — Google just cannot see its content. To keep a page out of search results, let it be crawled and add noindex instead. Blocking a page in robots.txt actually prevents Google from ever seeing that noindex.

For WordPress, less is more. The default virtual file disallows /wp-admin/ but allows admin-ajax.php, which front-end features rely on. Never block /wp-content/ or /wp-includes/: Google renders pages like a browser and needs your CSS, JavaScript and images. Sensible extras are internal search results (an infinite space of thin pages) and, for WooCommerce, cart, checkout, account and add-to-cart URLs.

How to use the WordPress robots.txt Generator

1. Choose the platform preset — WordPress, WooCommerce or Generic — and enter your site URL for absolute sitemap links.
2. Toggle the default rules, internal search blocking and, if you wish, the AI crawler group. Untick individual bots you want to allow.
3. Add custom user-agent groups for specific crawlers. Every path must start with / or *; invalid paths are flagged and skipped.
4. Pick the sitemap your site uses: WordPress core (wp-sitemap.xml) or Yoast SEO / Rank Math (sitemap_index.xml), plus any extra sitemaps.
5. Download robots.txt, upload it to your web root and test it in Google Search Console’s robots.txt report.

Example

A typical WooCommerce robots.txt:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?add-to-cart=*
Disallow: /?s=
Disallow: /search/

Sitemap: https://shop.example.com/sitemap_index.xml

Virtual vs physical robots.txt in WordPress

WordPress generates robots.txt on the fly through do_robots() and the robots_txt filter — there is no file on disk. SEO plugins hook into the same filter or offer an editor. As soon as you upload a real robots.txt to the web root, the server serves that file and WordPress’s virtual version (and any plugin edits) is ignored.

If Settings → Reading → Discourage search engines is ticked, WordPress outputs noindex meta tags instead of a blocking robots.txt since version 5.3 — remember to untick it at launch. See the Launch Checklist.

About blocking AI crawlers

• GPTBot, ClaudeBot, CCBot, Bytespider and similar bots collect training data. Google-Extended and Applebot-Extended are control tokens: they do not crawl on their own but tell Google and Apple not to use your content for AI models, without affecting search.
• ChatGPT-User, OAI-SearchBot and PerplexityBot fetch pages to answer users. Blocking them can remove you from AI answers and citations.
• robots.txt is a voluntary standard. Reputable operators honour it; for hard enforcement use firewall or CDN bot rules.