AI Visibility Report

Free guide · last checked 16 Oct 2026

How to block or allow AI bots on Cloudflare

If the domain is proxied through Cloudflare, a dashboard setting can stop an AI crawler even when the origin's robots.txt allows it. The steps below are the controls Cloudflare's own docs describe, and how to read the robots.txt those controls actually serve.

Short version: open /robots.txt and look for # BEGIN Cloudflare Managed content. Behavior policies live at Security Settings → Configure AI bot policies (Search, Agent, Training). A per-crawler block lives in AI Crawl Control → Crawlers → Action. robots.txt is a request. The Block action is an edge rule. /llms.txt is not created by either of those features.

1. Read the file Cloudflare is serving

Request https://your-domain/robots.txt with a cache-busting query if you have just changed a setting. Two different things can be in that response:

If the origin has no robots.txt, Cloudflare's docs say the managed feature creates a file and serves it. On the Free plan, a domain with no robots.txt of its own and without the managed feature still receives the Content Signals Policy text when a crawler asks for robots.txt. That text defines search, ai-input and ai-train. It does not set them to yes or no.

An edge block does not have to appear as Disallow. AI Crawl Control enforces Block with a WAF custom rule. The free AI crawler & raw-HTML checker reads the robots.txt response and the HTML. It does not show that WAF rule. Use the checker for the file and the page, and the Crawlers tab for the edge action.

2. Search, Agent and Training

Cloudflare's bot docs (page updated 1 Jul 2026) say every customer can treat three behaviors separately. A single bot can have more than one behavior.

PolicyWhat Cloudflare says it covers
SearchCrawlers that collect or index content to answer questions about it later.
AgentAutomated activity acting in real time on a person's behalf, such as chat fetch bots and browser-use agents.
TrainingCrawlers taking content to train or fine-tune a model, including mixed-purpose crawlers used for both Training and Search.

Each setting has three mitigations: Block (on all pages), Block on pages with ads, and Allow (do not block). The documented path is Security Settings → Configure AI bot policies.

The same page says a legacy control, Block AI bots, is deprecating on 15 Sep 2026. It blocks verified bots classified as crawling for AI training, plus some unverified bots that behave similarly, and it excludes mixed-purpose bots used for both Training and Search. That control is at Security Settings → Block AI bots.

The same page, still dated 1 Jul 2026 when we read it on 16 Oct 2026, says that on 15 Sep 2026 new domains get a new default: Training and Agent blocked on pages that display ads, Search left allowed, and mixed-purpose crawlers that combine Search and Training blocked by every configuration that blocks AI training, including the legacy option. The page tells customers they can opt out before that date. Open the policies on your zone and read the values that are saved there. This guide does not assume a default for a zone we have not looked at.

3. Put the choice into robots.txt

Cloudflare's robots.txt setting (page updated 8 Oct 2026) says Bot Preference Sync writes managed directives when at least one of Search, Agent or Training is set to Block or Block on pages with ads, or when Training is set to Disallow. robots.txt compliance is voluntary. The page says to use AI Crawl Control when you need a technical block.

Documented steps for training directives:

  1. Go to Security Settings.
  2. Filter by Bot traffic.
  3. Select Configure AI bot policies.
  4. For Training, select Disallow.
  5. Turn on Enable Bot Preference Sync.
  6. Select Save.

With that sync on, Training set to Disallow adds the managed directives and enables edge blocking for AI training crawlers. Mixed-use crawlers are excluded from that edge block. Search and Agent add their managed directives when set to Block or Block on pages with ads.

If the origin robots.txt returns HTTP 200, Cloudflare prepends the managed block and then your file. The published example of that managed block includes:

User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

That example does not list OAI-SearchBot or PerplexityBot. Treat the example as the sample Cloudflare published, then read your own URL. The served lines follow the policies you saved, not this screenshot.

The same page says managed robots.txt adds use=reference next to the existing search=yes, ai-train=no line. use=immediate means interact but store nothing, use=reference means index, excerpt and link back, and use=full means summarize and reproduce. Content Signals are preferences in the file. They are not the WAF block. Managed robots.txt is available on all plans.

Cloudflare notes that Google Search Console may report Syntax not understood for Content Signals. Their docs say they have observed no impact on crawling rates or SEO from those reports.

4. Which name is which

Cloudflare's AI Crawl Control bot reference (page updated 23 Apr 2026) lists these user agents. The reference points at the Radar bots directory for the current list. A later note on the verified-bots page (updated 1 Jul 2026) says the separate "AI Search" category is kept for older rules, and that new search crawlers are classified under Search. Use the table as the names and the categories printed on that reference, and use your zone's Action column for what is actually blocked.

CrawlerOperatorCategory on the referenceUser agent
GPTBotOpenAIAI CrawlerGPTBot
OAI-SearchBotOpenAIAI SearchOAI-SearchBot
ChatGPT-UserOpenAIAI AssistantChatGPT-User
ClaudeBotAnthropicAI CrawlerClaudeBot
Claude-SearchBotAnthropicAI SearchClaude-SearchBot
Claude-UserAnthropicAI AssistantClaude-User
PerplexityBotPerplexityAI SearchPerplexityBot
Perplexity-UserPerplexityAI AssistantPerplexity-User

On the Free plan, AI Crawl Control matches these crawlers by user agent. Paid Bot Management adds detection IDs. The plan table on the get-started page says Allow and Block controls exist on all plans. Free-plan metrics cover the past 24 hours.

5. Block or allow one crawler

  1. Connect the domain so traffic proxies through Cloudflare. AI Crawl Control's get-started page lists that as a prerequisite.
  2. Open AI Crawl Control.
  3. Open the Crawlers tab. The table is crawlers that are requesting the domain.
  4. In Action, choose Block or Allow.

Blocking creates or updates one WAF custom rule named AI Crawl Control, under Security → Security rules, filtered to custom rules. The rule is added at the end of existing custom rules. Cloudflare's WAF page for this product says:

Allowing OAI-SearchBot or PerplexityBot in this table does not, by itself, rewrite your origin robots.txt. Fetch /robots.txt after you save.

6. llms.txt

The managed robots.txt feature and AI Crawl Control do not publish an llms.txt. Serve a plain-text file at /llms.txt from the origin that Cloudflare proxies. On Cloudflare Pages, that means the file has to be in the deployed output. Then request the URL. An HTML page at that path, or a 404, is not a text file. In our 14 Oct 2026 check of 60 AI and SaaS homepages, 33 of 59 responding hosts returned a plain-text /llms.txt.

7. Pricing and other text in the HTML

Cloudflare does not render your application. If the origin's first HTML response has no price and no product description, a crawler that does not run JavaScript does not get them from a Cloudflare setting. Fix that on the platform that generates the HTML (see the other guides on this site), then run the checker on the public pricing URL. In that same 60-site check, 29 of 46 usable homepages had no price-like text in the raw HTML.

If the response is a challenge page, the checker says so. Cloudflare's AI Crawl Control docs say an Allow in the Crawlers tab can still lose to another security rule. That is the place to look when the HTML a browser sees is not the HTML a crawler is allowed to fetch.

8. Verify

  1. Open /robots.txt and read both the managed block and your own groups.
  2. On the Crawlers tab, read Action for GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot if they are listed.
  3. Paste the homepage and the pricing URL into the AI crawler & raw-HTML checker.

Want the full check?

Every AI Visibility Report ($49 PDF, generated by our software) includes a technical check of the HTML and robots.txt, plus what Perplexity and Grok answer for 10 buyer questions. Or request a free mini report.

More platform guides

Sources

Last checked 16 Oct 2026. Setting names and examples below are taken from those pages on that date. Dashboard labels can move. If a label on your account does not match, use the page contents rather than this guide.

Independent guide; not affiliated with Cloudflare, OpenAI, Anthropic or Perplexity. Product names are trademarks of their owners.