Glossary

GPTBot

By the Citationly team · Updated

Definition

GPTBot is OpenAI's web crawler for collecting content that may be used to train its generative AI foundation models.

OpenAI’s crawler documentation says GPTBot is used to make its generative AI foundation models more useful and safe, and that it crawls content that may be used in training those models.

GPTBot is a separate user agent from OAI-SearchBot, the crawler for ChatGPT’s search features. Because robots.txt rules are written per user agent, blocking GPTBot does not by itself block OAI-SearchBot. A site can opt out of training and still allow search, or allow both.

Whether to allow GPTBot is a business choice. Allowing it means your public pages may become part of what future models learn, which can shape what they know about your brand even when they answer without searching. Blocking it keeps your content out of that training. Either way, check that your robots.txt says what you intend:

User-agent: GPTBot
Allow: /

Read robots.txt for AI crawlers for the full picture, or check your site with the free checker.

Sources

  1. OpenAI: Overview of OpenAI crawlers