Original data · 14 Oct 2026

We checked 60 SaaS & AI tool sites: how many let AI crawlers read them

On 14 Oct 2026 we requested 60 public homepages of well-known AI products and AI-featured SaaS, using the same checks as our free crawler tool. 59 responded. 46 were usable HTML. 1 of 56 (1.8%) with a readable robots.txt disallowed GPTBot. OAI-SearchBot was disallowed on 1 of 56 (1.8%) and PerplexityBot on 1 of 56 (1.8%). Both are Poe, whose request ended at https://poe.com/login?redirect_url=%2F (HTTP 403). 1 of 46 (2.2%) usable homepages had fewer than 50 visible words in the raw HTML.

How many homepages disallow each crawler

Denominator for every bar in this section: 56 homepages whose robots.txt we could apply (HTTP 200 text, or no robots file so crawlers are allowed by default). 3 responded but the robots file could not be used. 1 homepage fetches failed, so those hosts were not scored here. A disallow is a robots.txt rule for the final homepage path. It is not a firewall test.

Search and user-triggered crawlers

Training crawlers and training opt-out tokens

Google-Extended and Applebot-Extended are training opt-out tokens. A disallow there does not, by itself, block ordinary Google or Apple search crawling. GPTBot, ClaudeBot, CCBot and Bytespider are training crawlers in our checker.

ChatGPT-User is the user-triggered fetch agent. OpenAI says it may ignore robots.txt. Claude-SearchBot was not part of this run; the checker this study reused does not score that token.

What was in the raw HTML

Usable homepages: 46 of 60. A page counts as usable when the homepage returned HTTP 200–399, looked like HTML, and was not classified as a bot challenge. 10 responses were classified as bot challenges. 3 other fetched pages returned HTTP 400 or higher. 0 returned a non-HTML body. JavaScript was not executed.

Bar widths are each row's own percentage. The llms.txt and sitemap.xml rows use the 59 homepages that responded, not the 46 usable HTML pages. The other rows use usable HTML pages.

Of the 29 usable homepages with no price-like text in the visible HTML, 23 linked to a URL containing “pricing”, and 10 had the substring “pricing” in the /sitemap.xml body we fetched. We did not open those pricing URLs.

52 of 56 (92.9%) homepages with a readable robots.txt listed at least one Sitemap: URL. We only downloaded /sitemap.xml, so a sitemap that lives only at another path is not counted as a usable sitemap above.

What we observed on individual sites

These sentences use the measured fields only. Named products were flagged as widely known in the sample file before the crawl. Every other product is in the table and the CSV.

• Poe was the only site where OAI-SearchBot or PerplexityBot was disallowed. The homepage request followed redirects to https://poe.com/login?redirect_url=%2F and returned HTTP 403. The winning rule for those tokens was * / Disallow: /login. On that same path the checker also counted a disallow for OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Applebot-Extended, CCBot, Bytespider. That robots.txt also contains a Disallow: / line. This run scored the redirected URL only.
• Canva disallowed GPTBot for / (GPTBot, Disallow: /). The same path was disallowed for ClaudeBot, CCBot, Bytespider, Applebot-Extended. OAI-SearchBot and PerplexityBot were allowed for that path. The homepage response was HTTP 403, so its HTML was not counted as usable page content.
• Google-Extended was disallowed on 0 of 56 checked paths.
• Meta AI (https://ai.meta.com/) returned HTTP 200 with 9 visible words. There was no H1. The HTML was cut off at the checker's 512 KB limit. JSON-LD types in that response: WebPage, Organization, Brand, Corporation, ImageObject, WebSite. Runway (https://runway.com/) returned HTTP 200 with 100 visible words. The H1 was “Building Real-World Intelligence”. The HTML was cut off at the checker's 512 KB limit. JSON-LD types in that response: Organization, Person, ContactPoint, SoftwareApplication, WebSite.

Methodology

• When: 2026-10-14 01:25 to 2026-10-14 01:27 UTC.
• Sample: 60 homepages written to sample.json before the crawl. Big recognizable AI products and popular AI apps with a public homepage. Not a random sample.
• Requests, in order, one site at a time: the homepage, then /robots.txt, then /llms.txt, then /sitemap.xml on the final homepage origin. 500 ms between those requests, 1000 ms between sites. No retries. If the homepage did not respond, the other three files were not requested. Redirects are followed (up to four). robots.txt is scored for the final path, which can be a login URL rather than the address we typed.
• User-Agent: AIVisibilityReport-Study/1.0 (+https://ai-biz-site.pages.dev/guides/ai-crawler-access-study-2026; one-time public homepage study; contact: tkdumplingsage@outlook.com)
• Timeout: 8 seconds per request, including DNS and up to four redirects, same budget as the live checker.
• Logic: functions/api/check.js (runCheck with sequential fetches). Visible words ignore script, style and SVG. Under 50 words is the checker's fail line; under 150 is its warning line. Price-like text uses the checker's currency and “per month” patterns. JSON-LD types are read from application/ld+json in the raw HTML. A robots.txt disallow is scored only when a rule matches the final path. No robots.txt (HTTP 404 or 410) counts as allowed by default, not as a block.
• Not done: no browser, no second crawl as GPTBot or ClaudeBot, no pricing-page fetch, no sitemap URL other than /sitemap.xml.

The 60 homepages

Per-site results

Allow and Block are robots.txt outcomes for the final path. Words are visible words in the raw HTML. “Price in HTML” is price-like visible text on that homepage. The CSV has the rule text and the rest of the crawlers.

Homepage fetch failed (1): Google Gemini (The host did not respond.).

Limitations

• The list was chosen by us from publicly marketed products. It over-represents well-known AI brands. It does not estimate the whole web, or small SaaS as a class. Our earlier note on 61 small SaaS homepages is a different sample: Can AI crawlers read your homepage?
• One pass, from one network, on 14 Oct 2026. A timeout or a bot wall is what this client saw, not proof the site always behaves that way.
• robots.txt is a request, not a lock. We did not send the crawlers' own user-agents, and we did not test CDN or firewall rules that block those agents.
• A low word count means the initial HTML is thin. It does not measure the page after JavaScript. A short page can also be complete.
• No price on the homepage is common when pricing lives on another URL. 23 of the 29 homepages with no visible price linked to a pricing URL. We did not fetch those pages, so this study does not say their pricing pages lack prices.
• The price pattern can miss “contact sales” and can match a number that is not a plan price. The matched strings are in the CSV.
• /llms.txt that returned HTML, or anything other than HTTP 200 plain text, counts as not present. Same rule as the checker.
• xAI has not published a crawler token. Grok is omitted from the block rates. The results JSON still records the checker's notes for the unofficial names GrokBot and xAI-SearchBot.

A practical fix checklist

1. Decide training and search separately. In this sample, GPTBot was disallowed on 1 of 56 (1.8%) readable robots files. OAI-SearchBot was disallowed on 1 of 56 (1.8%) and PerplexityBot on 1 of 56 (1.8%); The single search-crawler disallow is described under findings. If you want answer-time crawlers to read the site, allow OAI-SearchBot and PerplexityBot on the public pages. A GPTBot, ClaudeBot, CCBot, Bytespider, Google-Extended or Applebot-Extended disallow is a training choice, not the same switch.
2. Put the description in the first HTML. 1 of 46 (2.2%) usable homepages were under 50 visible words, and 2 of 46 (4.3%) were under 150. 5 of 46 (10.9%) had no H1. Server-render or pre-render the homepage, or at least ship a text H1 and a paragraph that says what the product is.
3. Put prices in HTML on the page that states them. 29 of 46 (63%) usable homepages had no price-like visible text. Check the pricing URL itself with the free checker. Keep the plan names and numbers in the HTML, not only in a script.
4. Add JSON-LD that matches the visible page. 28 of 46 (60.9%) usable homepages had Organization, Product, or SoftwareApplication. 27 of 46 (58.7%) had Organization, 1 of 46 (2.2%) had Product, and 14 of 46 (30.4%) had SoftwareApplication. A useful block names the product and includes an Offer when you publish a price.
5. Publish a real sitemap. 47 of 59 (79.7%) responding homepages had a usable /sitemap.xml. 52 of 56 (92.9%) readable robots files listed a Sitemap URL. Either path is fine; an HTML page at /sitemap.xml does not count.
6. llms.txt is optional. 33 of 59 (55.9%) responding homepages served a plain-text file. It is cheap to add and it is not, by itself, evidence an assistant reads it. See what we can actually say about llms.txt.
7. Look past robots.txt. 10 responses in this run were bot challenges. If a firewall challenges every non-browser agent, a permissive robots.txt does not make the page readable.

Raw data

• results.csv — one row per homepage, with the robots rule text
• results.json — the checker output for each site, including the robots.txt text we stored (truncated at 12,000 characters) and the aggregates used on this page
• sample.json — the list, fixed before the crawl

More guides

• Can AI crawlers read your homepage?
• How to get your SaaS recommended by AI assistants
• Why AI assistants get your SaaS pricing wrong
• How to check whether AI assistants recommend your SaaS

Independent study; not affiliated with the products measured or with OpenAI, Anthropic, Google, Perplexity, Apple, Common Crawl or ByteDance. Product names are trademarks of their owners. Counts describe this list on this date.