How AI crawlers read your site and what to set up technically

AI crawlers such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot fetch the raw HTML of your page, do not run JavaScript and follow the rules in your robots.txt per bot name. If you want ChatGPT, Claude or Perplexity to read and cite you, you arrange three things: your robots.txt lets the right bots in, your content is already in the HTML your server sends, and your firewall does not quietly hold the bots back. An llms.txt file is optional, not a requirement.

This article explains how that process works, which bots are around in October 2026 (checked against the documentation of OpenAI, Anthropic, Perplexity and Google) and how to set up each part.

How does your content end up in an AI answer?

Your content reaches an AI model in two ways: through training and through retrieval. The difference decides where you have influence.

Training. During training the model reads huge amounts of web text and bakes that knowledge in. After training that knowledge does not change until the next model version. Here you only decide whether you take part, through the training bots in your robots.txt.

Retrieval. When someone asks ChatGPT with search, Perplexity or Google AI Mode a question, the system fetches pages at that moment and writes the answer from them. This is called RAG: Retrieval-Augmented Generation. Here you have the most influence, because the system reads what is on your site now.

The retrieval process roughly goes like this:

1. Fetch. A crawler requests your page and gets the HTML your server returns.
2. Parse. The system removes navigation, footer and other noise and keeps the main text, similar to reader mode in your browser.
3. Split and store. The text is divided into pieces and each piece is turned into an embedding: a series of numbers that captures the meaning. Texts with similar meaning get similar numbers, even when the words differ. “What does a CRM cost?” and “price of customer management software” end up close together.
4. Select and cite. For a question, the system looks for the pieces that fit best, writes an answer and chooses which sources to name.

Steps 1 and 2 are technical: the rest of this article is about them. In steps 3 and 4 your text matters most. You do not need to chop your content into small pieces artificially; Google says literally in its guide on generative AI features that this is not necessary. What does correlate with citations: a Semrush study of 304,805 URLs cited by language models found the biggest differences in clarity and summarization (+32.8%), E-E-A-T signals (+30.6%), question-and-answer format (+25.5%) and section structure (+22.9%). That is correlation, not proven cause, but it points the same way: start every section with an answer that stands on its own. More in content chunking for AI citations.

Which AI crawlers visit your site?

There are three kinds of AI bots: training bots that collect content for new models, search bots that build an index for AI search results, and user bots that fetch a page because a user asks for it at that moment. Each kind has its own name in your robots.txt.

These are the names as the providers document them (as of October 2026, sources: OpenAI, Anthropic, Perplexity, Google):

User agent	From	Purpose	If you block it
GPTBot	OpenAI	Model training	Your content is no longer used for training. No effect on ChatGPT search.
OAI-SearchBot	OpenAI	Index for search in ChatGPT	You no longer appear as a source in ChatGPT search answers.
ChatGPT-User	OpenAI	Fetches a page when a user asks for it	According to OpenAI, robots.txt may not apply here, because the user makes the request.
OAI-AdsBot	OpenAI	Checks landing pages of ads in ChatGPT	Only relevant if you advertise in ChatGPT.
ClaudeBot	Anthropic	Model training	Future content from your site is left out of training.
Claude-SearchBot	Anthropic	Index for search in Claude	Less visible in Claude’s search results.
Claude-User	Anthropic	Fetches a page when a user asks for it	Claude cannot open your page when someone asks for it.
PerplexityBot	Perplexity	Index for Perplexity; according to Perplexity not for training	You do not appear in Perplexity results.
Perplexity-User	Perplexity	Fetches a page when a user asks for it	According to Perplexity, this bot generally ignores robots.txt.
Googlebot	Google	Search index, also the basis of AI Overviews and AI Mode	You disappear from Google. Almost never what you want.
Google-Extended	Google	Not a separate crawler, only a token: use for Gemini training and grounding in the Gemini apps and Vertex AI	No effect on Google Search.
Bingbot	Microsoft	Bing index, also the basis of Copilot	You disappear from Bing and Copilot.

There are also training tokens from other parties, such as Applebot-Extended (Apple) and CCBot (Common Crawl, an open dataset widely used for AI training).

The most common mistake follows directly from this table: blocking GPTBot assuming it controls “ChatGPT”. GPTBot is training only. For visibility in ChatGPT search, OAI-SearchBot is what counts, and OpenAI keeps those two explicitly separate.

Setting up robots.txt per AI bot

Your robots.txt decides per bot name which parts of your site a crawler may fetch. For most small businesses that want to be found, the advice is simple: let all search and user bots in, and make a deliberate choice about training.

Option 1: allow everything

If you want maximum visibility and have no objection to training, this is enough:

User-agent: *
Disallow: /admin/

Sitemap: https://www.yoursite.com/sitemap.xml

All bots, including OAI-SearchBot, Claude-SearchBot and PerplexityBot, may then fetch everything except /admin/.

Option 2: visible in AI answers, not for training

# Keep training bots out
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# All other bots, including OAI-SearchBot,
# Claude-SearchBot and PerplexityBot
User-agent: *
Disallow: /admin/

Sitemap: https://www.yoursite.com/sitemap.xml

Google-Extended is left out on purpose. That token covers grounding in the Gemini apps besides training: the use of your content to answer a question. If you block it, you also choose less visibility there. If you want that, add a User-agent: Google-Extended block with Disallow: /.

Pitfall: rules do not stack

A crawler picks the block that most specifically matches its name and ignores the rest. That is in the robots.txt standard, RFC 9309. So if you add a User-agent: OAI-SearchBot block with Allow: /, the Disallow rules under User-agent: * no longer apply to that bot. That is why option 2 lets the search bots fall under *: then they follow the same exclusions as everyone else.

AI Overviews and AI Mode are not set in robots.txt

Googlebot crawls for the regular results and for AI Overviews and AI Mode at the same time. If you want to be in Google but not in those AI features, you have been able to use the generative AI control in Search Console since August 31, 2026. According to Google that choice does not affect your regular rankings. nosnippet and data-nosnippet also remain available to limit per page or per text block what Google may show.

Test after every change

One typo in robots.txt can close off a whole folder. Check the file after every change in the robots.txt report in Google Search Console (under Settings) and in Bing Webmaster Tools. OpenAI says a change for search takes about 24 hours to take effect.

Does your firewall block AI bots without you knowing?

Often, yes. Firewalls, CDNs and security plugins can stop AI bots before they even read your robots.txt. Cloudflare has blocked AI crawlers by default on new domains since July 1, 2025 (MIT Technology Review). If your site is behind Cloudflare, check in the dashboard which bots are allowed.

Two things to watch:

• Allow bots by name and verify by IP. Any crawler can call itself GPTBot. The providers therefore publish their IP addresses: OpenAI per bot (for example openai.com/searchbot.json), Perplexity in perplexity.com/perplexitybot.json and Anthropic in claude.com/crawling/bots.json. For Googlebot and Bingbot a reverse DNS lookup works: a real Googlebot belongs to googlebot.com or google.com.
• Do not block by IP when you mean robots.txt. Anthropic warns that blocking its IP addresses also stops the bot from reading your robots.txt, so your opt-out does not work reliably.

Why AI crawlers do not see your JavaScript

AI crawlers download the HTML and move on; they do not build the page the way a browser does. An analysis by Vercel and MERJ showed that GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot do not run JavaScript. They sometimes fetch JavaScript files, but do not execute them. Googlebot (and so Gemini) does render. Anything that only appears on your page after JavaScript runs does not exist for ChatGPT, Claude and Perplexity.

The difference is in how your site produces HTML:

Method	What the server sends	Readable for AI crawlers
CSR (client-side rendering)	An empty shell, for example <div id="root"></div>, plus JavaScript	No
SSR (server-side rendering)	Full HTML, generated per request	Yes
SSG (static generation)	Full HTML, built in advance	Yes
ISR (incremental regeneration)	Full HTML, refreshed in the background	Yes

To a crawler there is no difference between SSR, SSG and ISR: all three deliver complete HTML. Choose between them based on how often your content changes.

What this means per platform:

• WordPress builds pages on the server by default. You are usually fine there, unless a plugin or page builder fills in content with JavaScript after loading.
• Next.js renders on the server by default with the App Router (Server Components). Watch components with "use client" that fetch their data only in the browser.
• Nuxt defaults to server-side rendering (ssr: true).
• Astro builds static HTML by default and only loads JavaScript for separate interactive parts.
• A React or Vue app without server rendering, such as a default Vite setup, sends an empty page. This is the case where you really need to change something.

Typical mistakes: text only fetched in a useEffect or onMounted, a page that only shows “Loading…” until the script is done, and titles or meta tags only set in the browser. Prerendering or dynamic rendering for bots works as a stopgap, but gives you two versions of a page that can drift apart.

On WordPress you usually have little trouble here, because WordPress serves plain HTML. What you do need to arrange there is in WordPress SEO.

How to test what a crawler sees

Fetch your page without JavaScript and see what remains. If your main text is not there, an AI crawler does not see it either.

# The HTML as a crawler receives it
curl -s https://yoursite.com/page | head -100

# Is a sentence from your main text in the HTML?
curl -s https://yoursite.com/page | grep -i "a sentence from your text"

# Does a bot get something different from a browser (for example due to your firewall)?
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot/1.0" https://yoursite.com/page

Without a terminal it works too: choose “View page source” in your browser (Ctrl+U or Cmd+Option+U). That shows the raw HTML, not the built page. The technical scan runs the same check and also shows whether your robots.txt allows AI bots.

Checking whether bots really visit

Your server logs are the only place where you see whether AI crawlers actually fetch your site; they do not appear in Google Analytics. Search for the user agents and watch the status codes:

grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|PerplexityBot" access.log \
  | awk '{print $9, $7}' | sort | uniq -c | sort -rn | head -20

Many 403 codes point to a firewall, 5xx codes to a server that cannot keep up, and many 404s to outdated links. That last one happens more than you think: in the Vercel and MERJ analysis, 34.8% of ChatGPT’s requests ended on a 404 page, against 8.2% for Googlebot. So keep your redirects and sitemap up to date when you change URLs. How to measure people who arrive through AI is in tracking ChatGPT traffic in Google Analytics.

llms.txt: what it is and what it does not do

llms.txt is a Markdown file in the root of your domain that gives language models a short overview of your site and your most important pages. It is cheap to make and does no harm, but there is no evidence that it gets you into AI answers more often.

The format was proposed by Jeremy Howard in September 2024. Only an H1 with the name of your site is required. Below it, optionally, come a summary as a blockquote, some explanation, and lists of links under H2 headings. A section called “Optional” contains links a model may skip when it has little room.

# Golden Crust Bakery

> Artisan bakery in Portland with two shops and delivery to businesses in the area.

## Key pages

- [Our bread and pastries](https://www.example.com/menu): bread, pastries and custom cakes.
- [Business delivery](https://www.example.com/business): lunch and meetings, order before 3 p.m.
- [Shops and hours](https://www.example.com/shops): addresses and hours of both locations.

## Optional

- [About us](https://www.example.com/about): the story of the bakery.

What the numbers say:

• Google writes that you do not need an llms.txt for Google Search, including AI Overviews and AI Mode, and that the file neither helps nor hurts your visibility (Google).
• Ahrefs looked at 137,210 domains: 28% had an llms.txt, and 97% of those received not a single request in May 2026. Of the requests that did come in, most were from SEO tools and coding agents, not AI search engines.

If you do make one, keep it small and current: only pages that exist and that you really want cited, with one sentence per link that matches the destination. An llms.txt with dead links or old prices is worse than no file. And it does not replace robots.txt: robots.txt decides who may come in, llms.txt only explains what is there.

Summary: what to set up

1. Robots.txt: search and user bots allowed, a deliberate choice about GPTBot, ClaudeBot and Google-Extended.
2. Firewall and CDN: AI bots not blocked by default.
3. Rendering: your main text, titles and meta tags are in the HTML the server sends.
4. Logs: you see that the bots visit and get a 200 status.
5. llms.txt: optional, small and current.

Once the technical side is in order, the next question is whether AI models actually mention you. To go through everything at once, use the technical GEO checklist. For structured data there is a separate guide to JSON-LD and schema markup.

Read also

• How AI models choose their sources
• Getting found in AI search engines
• What is GEO?