AI Visibility Report

Free guide · last checked 16 Oct 2026

Webflow robots.txt for GPTBot and other AI crawlers

Webflow publishes a robots.txt at the root of the domain. You can let two named groups of bots through, or write a rule for one user agent. Nothing in those settings applies until you publish.

Short version: Site settings → SEO → Indexing. Search-engine crawlers and AI bots are allowed by default. OAI-SearchBot is in the search-engine list. GPTBot, ClaudeBot and PerplexityBot are in the AI-bot list. One bot needs its own User-agent lines. llms.txt is a separate upload and is served on the custom domain only.

1. Read the published file

Webflow's help center says the file lives at the top of the domain, for example yourdomain.com/robots.txt. Open it on the domain you care about. The webflow.io staging host can have its own file (see below). The free AI crawler & raw-HTML checker fetches that URL and scores GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot for the path you paste.

Webflow says not every bot honors the file. It is a public request, not access control. Do not use it to hide a private page. Webflow's indexing article says to leave sensitive pages unpublished or password-protected, and to use the sitemap indexing toggle rather than Disallow when you want a page dropped from search indexes.

2. The two traffic-control toggles

In Site settings → SEO → Indexing, Webflow can write robots.txt rules for two groups. Help center text updated 29 Apr 2026 says traffic from both groups is allowed by default. Turning a toggle off asks that group not to crawl or index the site. You have to publish before it takes effect.

ToggleUser agents Webflow lists in that group
Search engine crawlersYandexBot, Googlebot, Bingbot, Baiduspider, OAI-SearchBot, Sogou, DuckDuckBot, MojeekBot, SeznamBot, Qwantify, Yeti, Slurp, 360Spider
AI botsmeta-externalagent, ChatGPT-User, ClaudeBot, PerplexityBot, GPTBot, GoogleOther, Timpibot, SBIntuitionsBot, Bytespider, AI2Bot, Amazon Q, Claude-User, CCBot, Diffbot, YandexGPT, MistralBot, cohere-ai, youBot, NeevaAI

The split matters. Turning off AI bots is how Webflow documents a request that GPTBot, ClaudeBot and PerplexityBot stay out. It also includes ChatGPT-User and Claude-User. It does not include OAI-SearchBot. OAI-SearchBot is in the search-engine group, with Googlebot. Turning off the search-engine group asks Googlebot to stay out as well.

Webflow's help article does not say the two toggles can allow one bot inside a group and block the rest. For that, write the user agent yourself.

3. One user agent

Webflow's robots.txt article (updated 19 May 2026) says you add rules at Site settings → SEO → Indexing, then click Save and publish. A rule is a User-agent line plus Allow or Disallow. / means the whole site.

Their pattern for "allow every bot except one" is:

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /

To ask GPTBot to stay out of training use while leaving the search crawler allowed, name both. This is the same shape as Webflow's "allow one bot, disallow another" examples, with OpenAI's tokens:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Publish, then read /robots.txt on the live domain. Webflow does not document what happens if a toggle and a hand-written rule disagree. The published file is the one to trust.

Webflow adds a Sitemap: line by default. The switch that removes it is labeled Remove sitemap.xml from robots.txt.

Once a robots.txt exists, Webflow says it cannot be removed completely. To allow crawling again, replace the rules with:

User-agent: *
Disallow:

The empty value after Disallow: is what their troubleshooting section shows. Save and republish. If the old rules are still on the published site, their article says to contact support.

4. Content-Signal is not a block

The same Indexing screen can add a Content-Signal line and a Content-Signal response header. Webflow's example is:

Content-Signal: ai-train=no, search=yes, ai-input=no

They describe the header as separate from robots.txt: the file is about access, the header is about training, search indexing, and use in AI answers. They also say this is a proposed extension to RFC 9309 and is not an accepted standard. Do not treat it as a substitute for Disallow.

5. The webflow.io subdomain

To keep the staging host out of search indexes: Site settings → SEO → Indexing, set Staging indexing to off, save, and publish. Webflow says this publishes a robots.txt only on the subdomain. The help article says you need a Site plan or a paid Workspace for that control.

For one page, Page settings → SEO settings → Sitemap indexing off adds a noindex meta tag and drops the page from the sitemap. Webflow says the toggle only works when the auto-generated sitemap is enabled. For one CMS item, the same switch is under the item's details, and the article says that control is on paid Site plans.

6. Upload llms.txt

Webflow's upload article (updated 25 Aug 2025) says:

  1. Go to Site settings → SEO → LLMs.txt.
  2. Click Upload file.
  3. Choose a UTF-8 file under 100 KB.
  4. Click Save changes and publish.

After that, the file is at https://your-domain/llms.txt on the custom domain. Webflow says it is not published to the staging domain. Their article calls the format experimental and says it does not guarantee that an LLM will read it. They also say the product does not publish the per-page .html.md files the proposal describes. A normal HTML page linked from llms.txt is what they tell you to rely on.

Check the custom domain, not something.webflow.io/llms.txt. The checker warns when the URL is missing, empty, or HTML.

7. Prices and text in the published HTML

Webflow's HTML structure article says a heading, paragraph and image are written as tags in the body, and that those tags are created when you add that content in the product. A price you type into a text element is part of that published HTML. A price that exists only inside an image, or only inside a script that runs in the browser, is not the same thing. Fetch the page without JavaScript, or paste the pricing URL into the checker.

JSON-LD goes in Page settings → Schema markup (generate or paste), then Save and publish. Webflow says markup from that field is in the <head> of the published page, on the custom domain and on the staging domain. Markup you already added in custom code keeps working. Their custom-code article says head and footer code are published into the HTML, up to 50,000 characters in each site-wide slot and each page slot, and that the sections accept HTML, CSS and JavaScript, not PHP or other server-side languages. Site-wide custom code requires a Core, Growth, Agency or Freelancer Workspace, or an active Site plan. Preview is not the published site.

In our 14 Oct 2026 check, 29 of 46 usable homepages had no price-like text in the raw HTML. The platform does not fix that if the price was never in the document.

8. Verify

  1. Publish to the custom domain.
  2. Open /robots.txt and /llms.txt on that host.
  3. Paste the homepage and the pricing page into the checker.

Want the full check?

Every AI Visibility Report ($49 PDF, generated by our software) includes a technical check of the HTML and robots.txt, plus what Perplexity and Grok answer for 10 buyer questions. Or request a free mini report.

More platform guides

Sources

Last checked 16 Oct 2026 against Webflow Help Center.

Independent guide; not affiliated with Webflow. Product names are trademarks of their owners.