---
title: "Let ChatGPT and Claude find your site without training on it"
description: "OpenAI and Anthropic each run a training bot, a search bot and a user-request bot, and each has its own name in robots.txt. Block the training bot, keep the search bot open, and your site still shows up in answers."
url: https://ihsanalapsi.dev/articles/ai-crawlers-robots-txt
lang: en
date: 2026-11-04
updated: 2026-11-04
tags: ["AI", "AI SEO", "Web development"]
author: Ihsan Alapsi (إحسان العبسي)
---
# Let ChatGPT and Claude find your site without training on it

You want your site to show up when someone asks ChatGPT or Claude about your field, but you do not want your content in model training. Many sites block "all AI" with two lines and vanish from the answers too. By the end of this article you will have a robots.txt that keeps the two apart.

## What is robots.txt, and where did it come from?

robots.txt is a text file at the root of your site that tells bots what they may read, and each bot looks for the lines under its own name.

The method is far older than AI: [the RFC 9309 specification](https://www.rfc-editor.org/rfc/rfc9309.html) says it specifies and extends a method originally defined by Martijn Koster in 1994. That spec came out in September 2022, and it requires the file at `/robots.txt` in the top-level path of the site, in lowercase.

## Why can one line drop your site from AI answers?

One line can drop you because each company now runs more than one bot, and each bot has a job: one collects for training, one builds the search index, and one opens a page because a user asked for it.

The human mistake here is treating them as one bot. You block everything because you do not want training, so you block search with it, and the answers start quoting your competitor instead. Or the other way round: you open everything because you want to show up in search, and your content goes into training without you noticing.

## What are the AI bots called, and what does each one do?

Each company gives its training bot one name and its search bot another, so you can block one and open the other.


| Company   | Training bot    | Search bot       | User-request bot |
| --------- | --------------- | ---------------- | ---------------- |
| OpenAI    | GPTBot          | OAI-SearchBot    | ChatGPT-User     |
| Anthropic | ClaudeBot       | Claude-SearchBot | Claude-User      |
| Google    | Google-Extended |                  |                  |


- According to [OpenAI's page on its bots](https://developers.openai.com/api/docs/bots), disallowing GPTBot means your content should not be used in training its models, and OAI-SearchBot is the one that surfaces sites in ChatGPT's search features; the company recommends allowing it. ChatGPT-User acts when a user asks for something, and robots.txt rules may not apply to it.
- According to [Anthropic's help article on its bots](https://support.claude.com/en/articles/8896518), blocking ClaudeBot signals that your future content should be left out of training, blocking Claude-SearchBot stops your site being indexed for search, and blocking Claude-User stops your pages being fetched when a user asks.
- According to [Google's list of its common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), Google-Extended is a name in robots.txt only, with no bot of its own, and it controls whether your content is used to train Gemini models and to ground them with content from the search index at prompt time. It has no effect on your inclusion in Google Search or on your ranking.

## How does a bot read your file?

A bot looks for the group under its name and follows it, and only if no group matches its name does it follow the `*` group.

This is the moment it clicks: the `*` group is not a set of shared rules added to every bot. The spec says a bot follows the `*` group only when no group matches its name. So if you write a group for OAI-SearchBot and open everything to it, it no longer sees the `Disallow: /admin/` you put under `*`. And name matching ignores upper and lower case.

The second point: the file is not a lock. The spec itself says its rules are not a form of access authorization. A well-behaved bot follows it, and a sensitive page needs a login.

## How do you write a robots.txt that blocks training and keeps search?

You write a group for each training bot with `Disallow: /`, and a group for each search bot that repeats the `*` rules you want.

```
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/

User-agent: Claude-SearchBot
Allow: /
Disallow: /admin/

User-agent: *
Disallow: /admin/
```

1. Replace `/admin/` with the paths you do not want indexed, and repeat them in every search group.
2. Put the file at the root of every subdomain you want the rules on. Anthropic asks for this plainly: one file per subdomain.
3. Check that it is served: `curl -s https://example.com/robots.txt`.
4. Wait. OpenAI says its search systems can take about 24 hours to pick up the change, and the spec asks bots not to use a cached copy older than 24 hours unless the file cannot be reached.

If a bot is loading your server rather than something you want to block, Anthropic says it supports `Crawl-delay` for ClaudeBot, even though that line is not part of the spec.

## Where do you start now?

Open your site's `robots.txt` now and look for a group that blocks search bots by mistake, or a group for one bot that forgets the `*` rules. Then decide on training for each company on its own.

If your goal is to show up in ChatGPT answers, also read [what the Cloudflare and OpenAI pilot taught us](/articles/chatgpt-search-cloudflare-discovery-pilot) about how it finds your pages, and the [AI SEO guide](/products/ai-seo-guide) for the steps after robots.txt. All our articles on it are under [the AI SEO topic](/articles?tag=ai-seo).

What do you want for your content: training, or only showing up in answers?

## FAQ

**If I block GPTBot, does my site vanish from ChatGPT search?**

No. OpenAI keeps search apart from training: the search bot is OAI-SearchBot, and it decides whether your site shows up in search results. Block GPTBot and keep OAI-SearchBot open.

**Does robots.txt actually stop a bot from reading my pages?**

No. The spec itself says its rules are not access authorization. A bot that respects it follows it, and a page nobody should see needs a login, not a line in robots.txt.

**How long does a change take to apply?**

OpenAI says its search systems can take about a day after you edit the file. The spec asks bots not to use a cached copy older than a day unless the file cannot be reached.

**Is blocking AI bots by IP address better?**

Anthropic says blocking IP addresses may not work correctly, because it stops the bot from reading robots.txt itself, so it never learns you opted out. Start with the file.