A single line in a text file can decide whether ChatGPT is able to cite your website. That file is robots.txt. This guide explains what it is, what OpenAI’s crawlers do, how a crawler reads the file, and the mistakes that most often lock brands out by accident.
What robots.txt is
robots.txt is a plain text file at the top level of your site, at yourdomain.com/robots.txt. It lists rules for crawlers: which parts of the site each one may read. The rules are defined in an internet standard, RFC 9309, published in September 2022.
Two things to keep in mind:
- It is a request, not a lock. Crawlers that follow the standard obey it. It doesn’t stop anyone determined to ignore it.
- It controls reading, not ranking. Allowing a crawler doesn’t make anyone cite you. Blocking one can stop it.
OpenAI’s crawlers
OpenAI documents its crawlers on one page, Overview of OpenAI crawlers. The three that matter for most brands:
| Name | What it does | If you block it |
|---|---|---|
| OAI-SearchBot | Used to surface websites in ChatGPT’s search features | Your site won’t be shown in ChatGPT search answers (it can still appear as a plain navigational link) |
| GPTBot | Crawls content that may be used to train OpenAI’s models | Signals that your content shouldn’t be used for training |
| ChatGPT-User | Visits a page when a ChatGPT user’s request needs it; not used for automatic crawling | OpenAI says robots.txt rules may not apply, because a user asked; it isn’t used to decide whether you appear in search |
OpenAI also lists OAI-AdsBot, which only visits pages submitted as ads on ChatGPT.
Two details from OpenAI’s page are worth remembering:
- Changes aren’t instant. It can take about 24 hours after you update robots.txt for OpenAI’s systems to adjust.
- Search and training are separate decisions. OpenAI’s own example is a site that allows OAI-SearchBot, so it can appear in search, while disallowing GPTBot.
How a crawler reads your file
robots.txt is made of groups. Each group starts with one or more User-agent lines naming crawlers, followed by Allow and Disallow rules for paths. RFC 9309 sets out how a crawler picks its rules:
- It looks for a group with its own name, matched without caring about upper or lower case. If several groups name it, their rules are combined.
- If no group names it, it uses the
*group, the rules for all crawlers. - The most specific rule wins. If
/products/is disallowed but/products/plans/is allowed, the longer, more specific rule applies to pages under/products/plans/. - On a tie, allow wins.
- If no rule matches a page, the page is allowed.
One rule surprises people: if robots.txt itself fails with a server error (a 5xx error), the standard says crawlers must assume everything is disallowed. If the file simply doesn’t exist (a 404), crawlers may read anything.
Common setups
Allow everything (the default). No robots.txt, or this:
User-agent: *
Allow: /
Allow ChatGPT search and AI training explicitly:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
Allow ChatGPT search, opt out of training:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /
Keep private areas out of everyone’s reach while leaving public pages open:
User-agent: *
Disallow: /account/
Disallow: /checkout/
Mistakes that lock brands out
- A blanket block.
User-agent: *withDisallow: /blocks every crawler without its own group, including OAI-SearchBot. This often comes from a staging site whose robots.txt was copied to the live site. - Blocking “all AI bots” with a plugin. Some plugins add a long list of AI crawlers to block. Check whether OAI-SearchBot is on it. If it is, you’ve opted out of ChatGPT search, not only training.
- A robots.txt that errors. If the file returns a server error, crawlers that follow the standard treat the whole site as off-limits.
- A block upstream. Your CDN or firewall can refuse AI crawlers before they ever read robots.txt. Ask your developer what your bot-protection settings do with OpenAI’s user agents.
- Assuming ChatGPT-User covers search. It doesn’t. Search eligibility depends on OAI-SearchBot.
Should you block GPTBot?
That’s a business decision, and there isn’t one right answer.
- Reasons to allow it: you may want your brand and products to be well known to future models, which matters for answers given without web search. Being in training data is no promise of being named, but being absent from it may leave the model knowing little about you.
- Reasons to block it: you may not want your content used for training, for legal, licensing or competitive reasons.
Either way, blocking GPTBot doesn’t affect whether you can appear in ChatGPT search. That is OAI-SearchBot’s job.
Check yours
Open your robots.txt now, or paste your address into our free checker. It reports, for each crawler, whether it is allowed, blocked, or not mentioned (and so allowed by default). Then work through the rest of our 15-minute checklist.
How Citationly helps
Getting robots.txt right makes sure ChatGPT can read you. Whether it names you when buyers ask is a separate question, and that is what a Citationly audit measures, answer by answer. Book a demo to see a report.