# AI crawlers in robots.txt: which to allow, which to block

> GPTBot, OAI-SearchBot, ClaudeBot and the rest each do a different job. What blocking each one costs you, and three robots.txt policies you can copy.

Published 2026-10-08 by Site Speed Labs.
Canonical: https://www.sitespeedlabs.com/blog/ai-crawlers-robots-txt

A lot of site owners have one rule in their head about AI: either "block the AI bots" or "let them in". The trouble is that there is no single AI bot. OpenAI, Anthropic, Perplexity, Google and Apple each send several, and each one does a different job. One collects pages to train future models. Another builds the index that an AI search tool cites from. A third fetches a page only because a person just asked about it.

Blocking the wrong one can take you out of the answers people see in ChatGPT or Perplexity, while doing nothing about the training you meant to opt out of. This post explains what each crawler is for, what you lose by blocking it, and three robots.txt policies you can copy.

## What robots.txt is, in one paragraph

robots.txt is a plain text file at the root of your site (for example `yoursite.com/robots.txt`). It lists crawlers by name, which the file calls a "user agent", and says which parts of the site each may visit. It is a request, not a lock. Well-behaved crawlers read it and obey; nothing stops a badly behaved one. Every company quoted in this post says its own crawlers respect it, with one important exception we cover below.

## The three jobs an AI crawler can do

Once you sort the crawlers by job, the decision gets much simpler.

**Training.** The crawler collects pages that may be used to train a future model. Nobody is asking about your business when this happens, and the model will not link back to you because of it. Blocking a training crawler is how you say "don't learn from my content".

**Search.** The crawler builds an index that an AI search product looks things up in when someone asks a question. This works like Google's own crawler: if you are in the index, the assistant can find you, quote you and link to you. Blocking a search crawler takes you out of those answers.

**User-triggered fetching.** Someone pastes your link into a chat, or asks a question and the assistant decides to read your page right then. The company sends a fetcher on that person's behalf. It is closer to a browser than a crawler, and that is why some companies say robots.txt may not apply to it.

## Which crawler does which job

These are the names you put in robots.txt, taken from each company's own documentation.

| Company | Training | Search | User-triggered |
| :-- | :-- | :-- | :-- |
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | (none listed) | PerplexityBot | Perplexity-User |
| Google | Google-Extended (a control, not a crawler) | Googlebot | (not separate) |
| Apple | Applebot-Extended (a control, not a crawler) | Applebot | (not separate) |

Sources: [OpenAI's crawler page](https://developers.openai.com/api/docs/bots), [Anthropic's crawler article](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), [Perplexity's crawler page](https://docs.perplexity.ai/guides/bots), [Google's list of common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) and [Apple's Applebot page](https://support.apple.com/en-us/119829).

A few details from those pages are worth knowing before you change anything.

**OpenAI.** Disallowing GPTBot "indicates a site's content should not be used in training generative AI foundation models". OAI-SearchBot is the one that decides whether you appear in ChatGPT's search answers; OpenAI says sites that opt out of it are not shown in those answers, though they can still appear as plain navigational links. For ChatGPT-User, OpenAI says that "because these actions are initiated by a user, robots.txt rules may not apply". OpenAI also says a robots.txt change takes about 24 hours to reach its search systems.

**Anthropic.** Restricting ClaudeBot signals that the site's future material should be left out of training. Claude-SearchBot crawls to improve search results. Disabling Claude-User "prevents our system from retrieving your content in response to a user query". Anthropic says its bots honour robots.txt.

**Perplexity.** PerplexityBot surfaces and links sites in Perplexity's search results, and Perplexity says it is not used to crawl content for training foundation models. Perplexity-User, its user-triggered fetcher, "generally ignores robots.txt rules" because a person asked for the page.

**Google.** Google-Extended is not a separate crawler. Google crawls with its usual crawlers and reads the Google-Extended rule to decide whether the content may be used to train and ground Gemini models. Google states that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". It also does not take you out of AI Overviews. Google says pages appear in [AI Overviews and AI Mode](https://developers.google.com/search/docs/appearance/ai-features) on the same terms as ordinary search results: indexed, and allowed to show a snippet. The only controls there are the normal search ones (`nosnippet`, `max-snippet`, `noindex`), and they affect your regular listing as well.

**Apple.** Applebot-Extended does not crawl anything. It controls whether pages Applebot already collected for Siri, Spotlight and Safari may train Apple's models. Apple says pages that disallow it "can still be included in search results".

## What blocking each one costs you

The pattern from the table: blocking a training crawler costs you very little that you can see today, and blocking a search crawler costs you visibility right away.

| If you block | You stop | You keep |
| :-- | :-- | :-- |
| GPTBot, ClaudeBot, Google-Extended, Applebot-Extended | Use of future crawls for model training (by companies that honour it) | ChatGPT, Claude and Perplexity search, Google Search, AI Overviews |
| OAI-SearchBot, Claude-SearchBot, PerplexityBot | Appearing in those assistants' search answers | Google Search |
| ChatGPT-User, Claude-User, Perplexity-User | Some on-request reads, where the company honours it | Everything else |
| Googlebot | Google Search, including AI Overviews | Almost nothing, for most businesses |

Note the wording in the first row. A training opt-out applies to what is collected from now on. It does not reach into models that have already been trained.

## Three policies you can copy

Most small businesses want one of these three. Pick one on purpose, then write it down.

**1. Open to everything.** You want to be found by every assistant and you don't mind your pages being used for training. If your robots.txt has no rules for these crawlers, you already have this policy. Naming them makes it explicit:

```
User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /
```

**2. Cited, but not trained on.** This is the policy most owners have in mind when they say "block AI". You stay in AI search answers and opt out of training:

```
User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /
```

The search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fall under the `*` group and stay allowed. One caution: a crawler only obeys the group that names it. If you write a named group for, say, OAI-SearchBot, it ignores your `*` rules entirely, so any `Disallow` lines you rely on (an admin area, a checkout page) need repeating in that group.

**3. Closed to AI entirely.** You list every crawler in the table under `Disallow: /`, except Googlebot and Applebot if you still want Google and Apple search. Be clear about the trade: you disappear from ChatGPT, Claude and Perplexity search, and the user-triggered fetchers may still read a page someone links them to. You stay in Google's AI Overviews unless you also restrict snippets in Google Search.

If you want to state your intent in words as well as rules, [Content Signals](https://contentsignals.org) adds lines such as `search=yes, ai-input=yes, ai-train=no` to robots.txt. It is new and voluntary, and we explain how we use it on this site in [what "AI-ready" means for a website](/blog/making-this-site-readable-by-ai-agents).

## How to find out what your site says today

Open `yoursite.com/robots.txt` in a browser. Look for `User-agent:` lines naming any of the crawlers above, and for `Disallow: /` under them. A `Disallow: /` under `User-agent: *` blocks every crawler that has no group of its own, AI and search alike.

If the file looks fine but AI tools still can't reach you, robots.txt may not be the cause. Three common places a block hides:

- **Your CDN or firewall.** Since July 2025 Cloudflare has [asked every new domain at sign-up whether to allow AI crawlers](https://www.cloudflare.com/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/), with blocking as the default, and it now sorts AI bots by what they do: [search, agent and training](https://developers.cloudflare.com/bots/concepts/bot/). A block there turns crawlers away before they ever read robots.txt, and it does not show up in the file.
- **A plugin or theme setting.** SEO and security plugins on WordPress and similar platforms can write robots.txt rules for you. If your file has rules nobody remembers adding, that is usually where they came from.
- **A rule copied from a template.** Lists of "AI bots to block" get copied from site to site, and some mix search crawlers in with training ones.

Changes are not instant. OpenAI quotes about 24 hours. Google says recrawling can take days or longer.

## What robots.txt cannot do

It cannot enforce anything. It is a request that the companies above say they honour. Crawlers that don't identify themselves, or that pretend to be someone else, are a firewall problem, not a robots.txt problem.

It cannot undo the past. Opting out of training covers what is collected from now on.

It cannot make an assistant read you well. A crawler that is allowed in still needs to find your words in the page. If your text only appears after JavaScript runs in a browser, a crawler that does not run JavaScript sees a nearly empty page. That is a separate question from permission, and a common one.

## Check yours in a minute

Our [free check](/audit) reads your robots.txt and tells you which of the main AI crawlers it shuts out of your whole site: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended. It also fetches your page the way a crawler does, with no browser and no JavaScript, and counts the words it can read. If you would rather have it set up for you, the [AI-ready setup](/ai-ready) covers the crawler policy along with the rest.

---

Want to know where your own site stands? Free plain-language check: https://www.sitespeedlabs.com/audit