AI Visibility Report

Free guide · last checked 16 Oct 2026

Next.js robots.txt and llms.txt

The App Router has a file convention for robots.txt. It does not have one for llms.txt. Prices belong in the HTML Next.js prerenders. This page does not describe a Vercel switch that blocks AI crawlers, because the Next.js and Vercel docs checked for this guide don't document one.

Short version: add app/robots.ts with one rule object per crawler. Put a static public/llms.txt, or a route handler that returns text/plain. Exclude both URLs from a catch-all proxy. Render the price on the server. If Cloudflare proxies the domain, the edge rules are in the Cloudflare guide, not in next.config.

1. app/robots.ts

Next.js documents two forms, both in the root of app. A static robots.txt, or robots.js / robots.ts that returns a MetadataRoute.Robots object. The convention shipped in 13.3.0. robots.js is a special route handler, cached unless it uses a request-time API or a dynamic config.

The docs' own example of "one group per bot" passes an array of rules. The same shape, with the crawler tokens you care about:

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: 'OAI-SearchBot',
        allow: '/',
        disallow: '/private/',
      },
      {
        userAgent: 'PerplexityBot',
        allow: '/',
        disallow: '/private/',
      },
      {
        userAgent: 'GPTBot',
        disallow: '/',
      },
      {
        userAgent: 'ClaudeBot',
        disallow: '/',
      },
    ],
    sitemap: 'https://example.com/sitemap.xml',
  }
}

Next.js prints one User-Agent block per object. Their sample output for a Googlebot object plus an Applebot object is two separate blocks, not one shared group. Repeat disallow: '/private/' on every crawler that should skip that path. A named group does not inherit the * group. Google's robots.txt specification is the model for that.

The public-folder docs say static metadata files such as robots.txt should use the special files in app, not public. Use the app convention for robots. The metadata files page also lists sitemap.xml as its own convention (sitemap.js or a static file).

Since 16.3.0, a rule can include other for directives outside the robots exclusion standard. Next.js copies the keys through and does not validate them. This guide does not put a content-signal line there. Webflow's docs, not Next.js, are the place that calls that header a proposal rather than a standard.

2. llms.txt

The metadata files index lists robots, sitemap, manifest, icons and Open Graph images. It does not list llms.txt. Two documented ways to serve an extra text file:

export function GET() {
  const body = `# Example\n\n> One-line description.\n\n- [Pricing](https://example.com/pricing)\n`
  return new Response(body, {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  })
}

Put that in app/llms.txt/route.ts. Use the static file or the route, not both. This guide does not claim which one wins if you add both.

Vercel documents a proxy that must let /robots.txt, /sitemap.xml and /llms.txt through before any rewrite. The Next.js metadata page says that if you use proxy.ts, the matcher should exclude metadata files. Vercel's page says that in Next.js 16 the middleware.ts filename was renamed to proxy.ts. A matcher that rewrites every path will answer /llms.txt with whatever that proxy returns. Exclude the three paths, deploy, and request them.

3. Prices in the HTML

Next.js's server and client components page says layouts and pages are Server Components by default, and that Client Components and the React Server Component payload are used to prerender HTML. The HTML is what shows the route before JavaScript runs. On a static export, Server Components run during next build and the result is written into the page's HTML file. Client Components are also prerendered to HTML during that build. Browser APIs such as window are not available there.

That is not the same as "a Client Component is invisible." Text that is in the component's initial render can be in the HTML. Text that appears only after useEffect, or after a fetch that runs in the browser, is not in the first response. Put the price in a Server Component, or otherwise in the render that produces the initial HTML. Then fetch the deployed pricing URL with JavaScript off, or paste it into the checker.

JSON-LD belongs in that HTML too. A <script type="application/ld+json"> rendered by the server is visible to the checker. A script inserted after hydration is not.

In our 14 Oct 2026 check, 1 of 46 usable homepages had fewer than 50 visible words in the raw HTML, and 29 of 46 had no price-like text. A framework that can prerender still serves an empty shell if the price is loaded only in the browser.

4. Vercel and anyone in front of it

Deploying the app to Vercel serves the app metadata routes and public files as part of the Next.js deployment. The docs checked for this page describe those files. They do not describe a Vercel control named for GPTBot or a default that blocks AI crawlers. If a firewall, bot challenge or a Cloudflare zone sits in front of the deployment, the HTML and robots.txt a crawler receives can differ from what next dev shows. The checker should be pointed at the public URL. Edge blocks are covered in the Cloudflare guide when that is the proxy.

5. Verify

  1. Deploy, then open /robots.txt and /llms.txt on the production host.
  2. Confirm GPTBot and OAI-SearchBot are separate groups, and that a private path you care about is repeated in the groups you allow.
  3. Paste the pricing URL into the checker.

Want the full check?

Every AI Visibility Report ($49 PDF, generated by our software) includes a technical check of the HTML and robots.txt, plus what Perplexity and Grok answer for 10 buyer questions. Or request a free mini report.

More platform guides

Sources

Last checked 16 Oct 2026. The GPTBot rules are the documented rules array, with those user agents filled in. Next.js does not ship those user agents for you.

Independent guide; not affiliated with Vercel or Next.js. Product names are trademarks of their owners.