GPTBot
By Thibault · Updated
GPTBot is the OpenAI web crawler that collects public content OpenAI may use to train its generative AI foundation models. You control it in robots.txt, and that setting is separate from the bots behind ChatGPT search.
Why does GPTBot matter for SEO?
GPTBot decides one thing: whether your content can be used to train OpenAI's future models. OpenAI says GPTBot is used to make its generative AI foundation models more useful and safe. If a model learns your brand, your method and your expertise during training, it can mention you even when it is not browsing the web. I see that as long term visibility in a channel people trust.
How does GPTBot work?
GPTBot identifies itself with a user agent that contains GPTBot and a link to openai.com/gptbot. OpenAI publishes its IP ranges at openai.com/gptbot.json, so you can check in your server logs that a visit claiming to be GPTBot is genuine. OpenAI runs three separate bots, and each setting is independent of the others:
- GPTBot crawls content that may be used to train the models.
- OAI-SearchBot finds the websites that appear in ChatGPT search. Sites that block it are not shown in ChatGPT search answers.
- ChatGPT-User visits a page when a user asks ChatGPT or a Custom GPT to. OpenAI says robots.txt rules may not apply to it because a user starts the action.
How do you block or allow GPTBot in robots.txt?
You block GPTBot with its own group in robots.txt: the line User-agent: GPTBot followed by Disallow: /. To block only part of the site, replace the slash with a folder, for example Disallow: /members/. Blocking GPTBot tells OpenAI not to use your content for training. Your site stays in ChatGPT search, because that depends on OAI-SearchBot.
Example: a publisher that blocks training but keeps ChatGPT search
A news site wants to keep its articles out of training but still get cited and clicked in ChatGPT. Its robots.txt blocks GPTBot and allows OAI-SearchBot. As a result, the articles can appear in ChatGPT search answers with a link, and OpenAI is told not to train on them. The mistake I see most often is the reverse: someone copies a rule from a forum that blocks every OpenAI bot, and the site drops out of ChatGPT search without anyone noticing.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot only covers training. Whether you appear in ChatGPT search answers depends on OAI-SearchBot, which has its own group in robots.txt.
How can I check that a visit really comes from GPTBot?
Compare the IP address in your server logs with the ranges OpenAI publishes at openai.com/gptbot.json. Anyone can fake a user agent, so the IP check is the test to trust.
Should I block GPTBot?
It depends on your business model. I allow it on most sites, because being part of what AI models know builds trust in your brand. I block it when the content itself is what you sell.
