Tornado SEO

GPTBot

By Thibault · Updated

Definition

GPTBot is the OpenAI web crawler that collects public content OpenAI may use to train its generative AI foundation models. You control it in robots.txt, and that setting is separate from the bots behind ChatGPT search.

Why does GPTBot matter for SEO?

GPTBot decides one thing: whether your content can be used to train OpenAI's future models. OpenAI says GPTBot is used to make its generative AI foundation models more useful and safe. If a model learns your brand, your method and your expertise during training, it can mention you even when it is not browsing the web. I see that as long term visibility in a channel people trust.

How does GPTBot work?

GPTBot identifies itself with a user agent that contains GPTBot and a link to openai.com/gptbot. OpenAI publishes its IP ranges at openai.com/gptbot.json, so you can check in your server logs that a visit claiming to be GPTBot is genuine. OpenAI runs three separate bots, and each setting is independent of the others:

  1. GPTBot crawls content that may be used to train the models.
  2. OAI-SearchBot finds the websites that appear in ChatGPT search. Sites that block it are not shown in ChatGPT search answers.
  3. ChatGPT-User visits a page when a user asks ChatGPT or a Custom GPT to. OpenAI says robots.txt rules may not apply to it because a user starts the action.

How do you block or allow GPTBot in robots.txt?

You block GPTBot with its own group in robots.txt: the line User-agent: GPTBot followed by Disallow: /. To block only part of the site, replace the slash with a folder, for example Disallow: /members/. Blocking GPTBot tells OpenAI not to use your content for training. Your site stays in ChatGPT search, because that depends on OAI-SearchBot.

Example: a publisher that blocks training but keeps ChatGPT search

A news site wants to keep its articles out of training but still get cited and clicked in ChatGPT. Its robots.txt blocks GPTBot and allows OAI-SearchBot. As a result, the articles can appear in ChatGPT search answers with a link, and OpenAI is told not to train on them. The mistake I see most often is the reverse: someone copies a rule from a forum that blocks every OpenAI bot, and the site drops out of ChatGPT search without anyone noticing.

Thibault's take My advice: allow GPTBot unless your content is your product, such as paid articles or proprietary data. For most businesses, being known by AI models builds trust that you cannot buy later. Then check three things. Make sure your robots.txt does not block OAI-SearchBot by accident. Make sure your server logs show OpenAI visits from the published IP ranges. And make sure your key pages put their content in the HTML. Accessibility comes before visibility. There is no absolute in SEO, so decide one page type at a time.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot only covers training. Whether you appear in ChatGPT search answers depends on OAI-SearchBot, which has its own group in robots.txt.

How can I check that a visit really comes from GPTBot?

Compare the IP address in your server logs with the ranges OpenAI publishes at openai.com/gptbot.json. Anyone can fake a user agent, so the IP check is the test to trust.

Should I block GPTBot?

It depends on your business model. I allow it on most sites, because being part of what AI models know builds trust in your brand. I block it when the content itself is what you sell.

Sources

  1. developers.openai.com/api/docs/bots

Stay ahead of every search update

Google, ChatGPT, Gemini, Claude: what changed, why it matters, and what to do about it. Every morning, in 3 minutes, free.

What the...?