AI crawlers
By Thibault · Updated
AI crawlers are bots run by AI companies like OpenAI, Anthropic and Google. They fetch web pages to train models, to build a search index for AI answers, or to read a page live when a user asks a question.
Why do AI crawlers matter for SEO?
AI crawlers decide whether your pages can show up in ChatGPT, Claude and Gemini answers. If a bot can't reach or read your page, the AI model can't cite it, however good your content is. I see AI models as a trust channel. Being quoted in an answer builds credibility before anyone visits your site.
What are the main AI crawlers?
Each AI company runs several bots, and each bot has its own job. Blocking one has different effects from blocking another:
- GPTBot (OpenAI): collects content that may be used to train OpenAI's foundation models.
- OAI-SearchBot (OpenAI): surfaces websites in ChatGPT search. OpenAI says sites that opt out of it will not be shown in ChatGPT search answers.
- ChatGPT-User (OpenAI): visits a page when a user asks for it. OpenAI says robots.txt rules may not apply because the user starts the action.
- ClaudeBot (Anthropic): collects content that may be used for training.
- Claude-SearchBot and Claude-User (Anthropic): index pages for search and fetch pages for user questions.
- Google-Extended (Google): a robots.txt token, not a separate bot. It controls whether content Google crawls can be used to train Gemini models and for grounding in Gemini Apps and Vertex AI.
How do AI crawlers work?
AI crawlers request your HTML much like Googlebot does, but most of them don't run JavaScript. A Vercel analysis found that none of the major AI crawlers it studied rendered JavaScript, including GPTBot and ClaudeBot. They downloaded JavaScript files but didn't execute them. If your content only appears after client-side rendering, these bots get an empty page.
You control them in robots.txt with their user agent names. Anthropic says its bots follow robots.txt and support Crawl-delay. Google says Google-Extended has no effect on inclusion or ranking in Google Search.
An example robots.txt setup
Say a publisher wants to appear in ChatGPT search but keep its articles out of model training. It disallows GPTBot and allows OAI-SearchBot. With that setup the site stays eligible for ChatGPT search answers and tells OpenAI not to use its content for training. OpenAI notes that robots.txt changes can take about 24 hours to apply to search.
Frequently asked questions
Should I block AI crawlers?
It depends on what you want. Blocking training bots like GPTBot or ClaudeBot keeps your content out of future training sets. Blocking search bots like OAI-SearchBot or Claude-SearchBot can remove you from AI answers, so most sites that want visibility leave those open.
Does blocking Google-Extended hurt my Google rankings?
No. Google states that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. It only controls use of your content for Gemini training and grounding.
Can AI crawlers read JavaScript content?
Most can't. Vercel's analysis found that the major AI crawlers fetch JavaScript files without executing them. Serve your main content in the initial HTML if you want AI models to read it.
