Free guide · 2 Oct 2026

Can AI crawlers read your homepage?

Many SaaS homepages are built with JavaScript frameworks that render the text in the browser. That's fine for people and for Google. But several AI crawlers fetch the HTML without running JavaScript, so for them a page like that can be close to empty.

Which crawlers run JavaScript?

In its analysis of crawler traffic, "The rise of the AI crawler", Vercel reported that none of the major AI crawlers it studied rendered JavaScript, naming OpenAI's (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic's ClaudeBot, Meta's, ByteDance's and PerplexityBot. They noted that Google's Gemini uses Googlebot's infrastructure, which does render JavaScript, and that AppleBot renders too. Crawler behaviour can change, so treat this as a reason to check rather than a permanent rule. Either way, text that's already in the HTML is readable by every crawler.

What we saw in 61 small SaaS homepages

On 1 Oct 2026 we ran our technical check on the homepages of 65 small SaaS products with public revenue listed on TrustMRR. 61 returned a normal page; 4 failed or served a bot challenge and were excluded. Fetching the HTML without JavaScript:

This sample isn't random (it's products we screened as potential customers), so the counts describe this group, not the whole web. "No price-like text" can be fine for products that keep pricing on a separate page. Check that page too.

The pattern is the useful part. In this group, robots.txt wasn't the problem: nobody blocked the main AI search crawlers. The more common issue was a homepage that says almost nothing until JavaScript runs.

The 5-minute test

1. Count the words. curl -s https://yoursite.com | sed 's/<[^>]*>/ /g' | wc -w gives a rough count, including some script noise. If it's in the dozens rather than hundreds, look closer.
2. Look for your one-line description. curl -s https://yoursite.com | grep -i "your category words". Is the sentence that says what you are and who it's for in the HTML?
3. Check pricing. Do the same for your pricing page and a plan name or price.
4. Disable JavaScript in your browser (Developer tools → Settings → Disable JavaScript) and reload. What you see is roughly what a non-rendering crawler gets.
5. Check robots.txt and your CDN/firewall for the search crawlers listed in our checklist. Bot protection can block crawlers even when robots.txt allows them.

How to fix it

• Server-side rendering or static generation for the homepage, pricing, features and comparison pages. Most frameworks support it (for example Next.js, Nuxt, SvelteKit, Astro), often per route.
• Pre-rendering if a full migration isn't practical: generate static HTML snapshots of your key marketing pages at build time.
• At minimum, put a plain-text H1 and a one-paragraph description (category, audience, differentiator, price basics) in the initial HTML, even if the rest of the page hydrates later.
• Add JSON-LD (Organization, SoftwareApplication with Offer) in the HTML head, matching the visible page.

Google's JavaScript SEO guide is a good reference for rendering strategies. Even though Googlebot renders JavaScript, it notes that "server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript."

More guides

• How to check whether AI assistants recommend your SaaS
• Why AI assistants get your SaaS pricing wrong
• How to get your SaaS recommended by AI assistants: a checklist

Sources

• Vercel, The rise of the AI crawler: vercel.com/blog/the-rise-of-the-ai-crawler
• Google Search Central, Understand JavaScript SEO basics: developers.google.com
• OpenAI, Overview of OpenAI Crawlers: developers.openai.com/api/docs/bots
• Perplexity, Perplexity Crawlers: docs.perplexity.ai
• Our data: technical checks of 65 homepages listed on TrustMRR, run 1 Oct 2026 with our own checker (no JavaScript, standard browser user agent); 61 usable.

Independent guide; not affiliated with OpenAI, Anthropic, Google, Perplexity, xAI or Vercel. Product names are trademarks of their owners.