AI Visibility Report

Free guide · last checked 16 Oct 2026

WordPress robots.txt for AI crawlers

WordPress answers /robots.txt itself until a real file exists at that path. The default text does not mention GPTBot, ClaudeBot or PerplexityBot. Those crawlers then follow the same public-page rules as everyone else.

Short version: the virtual file disallows the admin path and allows admin-ajax.php. Settings → Reading → Discourage search engines from indexing this site is a noindex request, not a training opt-out. Add a named User-agent group with the robots_txt filter, or put a real robots.txt in the site root. Core does not generate llms.txt.

1. What core serves

do_robots() in WordPress core sends Content-Type: text/plain; charset=utf-8 and prints:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

The paths come from admin_url(), so a site in a subdirectory gets that subdirectory in the path. The robots_txt filter receives this string and a boolean, $public, from the blog_public option. Plugins and themes change the file through that filter. The changelog for do_robots() says WordPress 5.3 removed the old Disallow: / output.

Open https://your-site/robots.txt before you add anything. If you see only those lines, core is answering, and GPTBot is not named. The free AI crawler & raw-HTML checker reads that response.

2. Leave "Discourage search engines" alone

The Reading settings screen documents Settings → Reading and the checkbox Discourage search engines from indexing this site. Since WordPress 5.3, checking it causes a robots meta tag of noindex, nofollow in the head when the theme calls wp_head(). Through version 5.2, the same checkbox made /robots.txt return User-agent: * and Disallow: /, and the docs say that older behavior only applied when WordPress was in the site root and no physical robots.txt existed.

The screen also says neither state blocks access. Search engines choose whether to honor the request. The checkbox does not name GPTBot, and it asks search engines not to index the site. That is the wrong control if you still want to be fetched for AI search.

3. Name the crawlers in the filter

Append groups. Do not replace the default output unless you mean to. A crawler with its own User-agent group uses that group. Google's robots.txt specification, which describes the group model these files follow, says the rules are the ones in the matching group. Repeat a path you still want blocked, such as the admin path, inside the new group.

add_filter( 'robots_txt', function( $output, $public ) {
    $output .= "\nUser-agent: GPTBot\nDisallow: /\n";
    $output .= "\nUser-agent: ClaudeBot\nDisallow: /\n";
    $output .= "\nUser-agent: OAI-SearchBot\nAllow: /\nDisallow: /wp-admin/\nAllow: /wp-admin/admin-ajax.php\n";
    $output .= "\nUser-agent: PerplexityBot\nAllow: /\nDisallow: /wp-admin/\nAllow: /wp-admin/admin-ajax.php\n";
    return $output;
}, 10, 2 );

Put that in a small plugin, or in a child theme's functions.php if you accept that switching themes removes it. Adjust the admin path if admin_url() on your site is not /wp-admin/. Fetch /robots.txt after you deploy it. If a plugin already prints a full file at a later priority, your lines can be overwritten. The filter's second argument is there so you can see it. This snippet does not treat $public as a reason to disallow the whole site.

4. A real file in the root

WordPress's documented Apache .htaccess for pretty permalinks contains:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]

An existing file is not rewritten to index.php, so do_robots() does not run for it. A robots.txt you upload to the WordPress root is the file crawlers get on that setup. The Reading settings note agrees that the old virtual Disallow: / behavior did not apply when a robots.txt file existed. If you still see only the two default lines after uploading, the file is not in the directory the web server uses as the site root, or the host is not using those rewrite rules. Read the URL again rather than assuming.

The same condition is how a plain-text llms.txt gets served. WordPress core has no llms.txt generator in do_robots() or the Reading screen. Create a UTF-8 text file at the installation root. WordPress sets text/plain for the virtual robots response. A static .txt file uses whatever type your server assigns. Confirm the body is the text you uploaded, not a theme's HTML 404.

5. Prices in the HTML

A theme prints the post or page through PHP templates, and that output is the HTML response. A price you typed into the content is in that HTML. A price that a page builder or plugin writes into the page only after JavaScript runs is not in the first response. The reliable test is the raw HTML, not the rendered browser tab. Paste the pricing URL into the checker.

JSON-LD has to be in that same response. A <script type="application/ld+json"> printed from wp_head is in the HTML. A tag injected by a browser script is not. Core does not require a particular schema type. Match the visible price if you add an Offer.

In our 14 Oct 2026 check, GPTBot was disallowed on 1 of 56 readable robots.txt files. The common case in that sample was a file that did not name the AI crawlers at all. 29 of 46 usable homepages had no price-like text in the raw HTML.

6. Verify

  1. Open /robots.txt and confirm the GPTBot group is the one you saved, and that OAI-SearchBot still has an allow for the public path.
  2. Open /llms.txt if you added one.
  3. Run the checker on the homepage and the pricing URL.

Want the full check?

Every AI Visibility Report ($49 PDF, generated by our software) includes a technical check of the HTML and robots.txt, plus what Perplexity and Grok answer for 10 buyer questions. Or request a free mini report.

More platform guides

Sources

Last checked 16 Oct 2026 against WordPress developer documentation. The robots_txt example is the documented filter, with crawler names added. It is not a file WordPress ships.

Independent guide; not affiliated with WordPress or Automattic. Product names are trademarks of their owners.