Free tool · no sign-up
AI crawler & raw-HTML checker
Paste one public page. We fetch it without running JavaScript, then read robots.txt, llms.txt and sitemap.xml on the same host. Nothing is stored, and there is no account.
- robots.txt for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, and xAI/Grok
- How much visible text is in the HTML before JavaScript runs
- Whether a price appears in that HTML
- JSON-LD blocks and their types
- Whether /llms.txt and a sitemap exist
See if assistants actually name you
This page only checks whether a crawler can read the HTML. The free mini report asks Perplexity 3 buyer questions and emails a short PDF: whether you were mentioned, who was named instead, and which pages were cited.
How to read a result
- Fail means a search crawler is blocked (OAI-SearchBot or PerplexityBot), the HTML is almost empty without JavaScript, or there is no JSON-LD.
- Warn means a training opt-out (GPTBot, ClaudeBot, CCBot, Bytespider, Google-Extended, Applebot-Extended), thin text, no price on this URL, a missing llms.txt or sitemap, or no official Grok user-agent. Blocking a training crawler is a normal choice. It does not by itself remove you from AI search.
- Pass means nothing to change for that row.
Under 50 visible words is a fail. Under 150 is a warning. The count skips script, style and SVG. No price on the homepage is only a warning: pricing often lives on its own URL, so check that page too. xAI has not published a crawler user-agent, so the Grok row cannot be a straight allow or block.
robots.txt is a request, not a lock. Official crawler notes: OpenAI, Perplexity. Related: Can AI crawlers read your homepage?