--- title: "How does ChatGPT find your pages? What a Cloudflare-OpenAI pilot found" description: "Allow OAI-SearchBot in robots.txt and in your firewall, and clean up dead and duplicate links. The Cloudflare and OpenAI pilot shows why, but indexing doesn't guarantee ChatGPT cites you." url: https://ihsanalapsi.dev/articles/chatgpt-search-cloudflare-discovery-pilot lang: en date: 2026-10-21 updated: 2026-10-24 tags: ["AI SEO", "AI"] author: Ihsan Alapsi (إحسان العبسي) --- # How does ChatGPT find your pages? What a Cloudflare-OpenAI pilot found On October 15, 2026, Cloudflare and a team from OpenAI published the first results of a pilot that helped OpenAI's search crawler find pages it didn't know about. A crawler is a program that visits pages and reads them so it can add them to a search index. Nobody said which sites took part, but what they found about dead and duplicate links applies to every site. By the end of this article you'll know what to check in robots.txt and the firewall, what to fix in your links, and what nobody can promise you. ## What is the Cloudflare and OpenAI pilot? The Cloudflare and OpenAI pilot is a research pilot that started in July 2026: Cloudflare gives OpenAI's search crawler signals about pages from its network, and OpenAI evaluates and ranks on its own. According to Cloudflare, the sample was about 1% of its network, and it helped OpenAI discover and index "hundreds of millions" of new pages. The index is the database a search engine looks through when someone asks a question, and a page that isn't in it can't be found by that search. Cloudflare [announced the pilot on July 8, 2026](https://www.cloudflare.com/press/press-releases/2026/cloudflare-announces-research-pilot-with-openai/) and said it was built on information from participating sites on its network. The results came out in [a post on the Cloudflare blog](https://blog.cloudflare.com/cloudflare-openai-discovery-agentic-web/) written by two people from Cloudflare and two from OpenAI, and its numbers stop at the 1% and "hundreds of millions". The names of the participating sites, how often these pages were cited in ChatGPT answers and how many visits they sent, and how much crawling hit the servers are all unpublished. [A Search Engine Journal report](https://www.searchenginejournal.com/chatgpt-search-can-draw-on-more-pages-after-a-cloudflare-pilot/593720/) noted the same gaps on October 20. ## What signals does Cloudflare send to OpenAI? The signals Cloudflare sends to OpenAI are of three kinds: filtering for links that lead nowhere, change signals that say when a page is worth another visit, and signals that flag a single harmful page without penalizing the whole domain. According to Cloudflare, the signals are information about the pages (metadata), not their content. | Kind | What it catches, according to the post | What it means for your site | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | Filtering | Missing files requested over and over, sites that redirect almost any URL to the homepage, and URLs that lead to copies of the same content | The crawler skips these links before fetching them, so its time goes to your real pages | | Change | When a page is worth another visit, and when it isn't | Less load from pages that haven't changed | | Harm (still being tested) | A single compromised page on a healthy site, like a fake login form an attacker planted | The page gets flagged on its own, and the domain isn't penalized. Early results hint that it could help identify pages that should stay out of the index | The post also says new pages can raise crawling at first, because they have to be checked and indexed, and the goal is for the crawler to keep discovering and keeping up with less work. And at the end of the post: > "A signal about a page doesn’t grant permission to access it. This research uses metadata about the web for search and discovery, rather than sharing page content for model training." That means a signal about a page doesn't give anyone permission to reach it, and the pilot uses information about the web for search and discovery, not page content for model training. So the decision stays in your hands, in robots.txt and the firewall. A week before the pilot was announced, in [a July 1, 2026 post](https://blog.cloudflare.com/making-ai-search-smarter/) that announced its signals program for answer engines, Cloudflare wrote that the program is limited to search, that it isn't sharing any content, and that nothing is used for model training. That applies only to what Cloudflare sends. OAI-SearchBot itself reads your pages if you allow it. ## How do I get my site to show up in ChatGPT search? For your site to show up in ChatGPT search, OAI-SearchBot has to be able to reach your pages: allowed in robots.txt, and not blocked by the firewall or the CDN. OpenAI says sites that block OAI-SearchBot don't appear in ChatGPT's search answers. OpenAI has three bots that matter to you here, each with its own job, according to [the official bots page](https://developers.openai.com/api/docs/bots): | Bot | Its job | If you block it | | ------------- | ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------- | | OAI-SearchBot | Fetches sites for search results in ChatGPT | Your site doesn't appear in search answers, but it may still appear as a navigation link | | GPTBot | Crawls content that may be used to train OpenAI models | A signal that your content shouldn't be used for training | | ChatGPT-User | Visits a page when a user asks ChatGPT or a Custom GPT | robots.txt may not apply to it because the user made the request, and it doesn't decide whether you appear in search | Each setting is independent of the other. You can allow OAI-SearchBot and block GPTBot, and if you allow both, OpenAI may use the same crawl for both jobs. So the GPTBot decision stands on its own: you make it based on whether you want your content used for training, and search isn't affected by it. After any change to robots.txt, OpenAI's search systems need about 24 hours to adjust to it. ### Check robots.txt Open `robots.txt` on your site (at the end of the domain: `/robots.txt`) and go through it in order: 1. Look for `OAI-SearchBot`. If there's `Disallow: /` under it, that's the line blocking you. 2. If there's no group with its name, the bot follows the `User-agent: *` group. Then `Disallow: /` under it blocks every bot, the search bot included. 3. If you made a group with the bot's name, repeat the `Disallow` lines for your private pages in it. According to [Google's explanation of the robots.txt standard](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt), "Only one group is valid for a particular crawler": the bot reads its own group only, and doesn't combine it with the `*` group. This example allows search and blocks training, the same combination OpenAI gives as an example. If you want to allow training too, remove the GPTBot group: ``` User-agent: OAI-SearchBot Allow: / Disallow: /admin/ User-agent: GPTBot Disallow: / User-agent: * Allow: / Disallow: /admin/ Sitemap: https://example.com/sitemap.xml ``` ### Check the firewall and Cloudflare A web application firewall (WAF) is a layer in front of your site that inspects every request and decides whether it gets through. If the WAF blocks OAI-SearchBot, robots.txt doesn't help you, because the bot never reaches the page. OpenAI recommends allowing requests from the IP addresses it publishes for OAI-SearchBot in the file `https://openai.com/searchbot.json`. If you make an allow rule, base it on the user agent and the IP together: the user agent is text written by whoever sends the request, and anyone can put the bot's name in it. [Perplexity recommends the same approach](https://docs.perplexity.ai/guides/bots) for its bots. If your site is on Cloudflare, there are two places to check: - [AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/): available on all plans, it shows you which AI services are visiting your site, lets you allow or block each crawler separately, and tracks who follows robots.txt. - The Search, Agent and Training settings: since [July 1, 2026](https://blog.cloudflare.com/content-independence-day-ai-options/) Cloudflare splits AI traffic into these three categories on all plans, the free one included. Keep Search on Allow. If you want to block training and stay in search, Cloudflare says in [a September 15, 2026 post](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) that the Disallow AI Training setting blocks crawlers whose only job is training, OpenAI's training crawler among them, without affecting search. ## What do you do with links that lead nowhere? Links that lead nowhere are the same ones the pilot filters out: a URL that doesn't exist returns a normal page or redirects to the homepage, and the same content lives at more than one URL. Fix them on your site even if you're not on Cloudflare, because every request to them is a crawler visit wasted. This waste isn't new. In [a measurement Vercel ran with MERJ](https://vercel.com/blog/the-rise-of-the-ai-crawler) and published on December 17, 2024, the OpenAI crawler Vercel called "ChatGPT" (which, in its volume numbers, is GPTBot, the training crawler) spent 34.82% of its requests on 404 pages, and another 14.36% following redirects. Googlebot in the same measurement spent 8.22% on 404 pages and 1.49% on redirects. The measurement doesn't break out numbers for OAI-SearchBot, the search crawler, on its own. 1. Try a URL that doesn't exist on your site and see what it returns: ```bash curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" https://example.com/this-page-does-not-exist ``` The right answer is `404` with no redirect URL. If you get `200`, you have a soft 404: an error page that tells the crawler it's a real page. And if you get `301` or `302` with the homepage URL, your site does what the post described. 2. For a page that moved, redirect it with `301` to its own new page, not to the homepage. For a page that was deleted and has no replacement, let it return `404` or `410`. 3. If the same page opens at more than one URL (with and without a trailing `/`, or with tracking parameters), pick one URL, and make the others redirect to it or put `rel="canonical"` on them. 4. Open the Page indexing report in Google Search Console. [The report gives you a reason](https://support.google.com/webmasters/answer/7440203) for each page that wasn't indexed, and the reasons include "Soft 404", "Page with redirect" and "Duplicate without user-selected canonical". These reasons come from Google, not ChatGPT, but they're the same kinds the post talks about, and it's the easiest place to find them. ## Which pages are worth ChatGPT finding? The pages worth ChatGPT finding are the ones with specific information someone is looking for: a user manual, a menu, opening hours, current requirements and fees. The examples from the Cloudflare and OpenAI pilot itself are of this kind. The post gives three examples: a washing machine manual for a specific model, current passport requirements and fees, and a restaurant's menu and opening hours. It says pages like these are useful even if few sites link to them. The washing machine manual is information about one specific product, so if your site is a store or a B2B marketplace, like [SyMarket](/work#symarket-app), which I built, your product pages are the same kind. To get these pages found and keep what they say accurate: 1. Give each piece of information its own page at a fixed URL: the menu, opening hours, prices, and the delivery or returns policy. 2. Keep the information as text in the HTML the server returns. In the same Vercel and MERJ measurement (December 2024), none of OpenAI's crawlers (OAI-SearchBot, ChatGPT-User and GPTBot) ran JavaScript, so anything that only appears after JavaScript runs in the browser may never reach them. 3. When the information changes, edit the page itself instead of making a new page, because the old URL is the one the crawler knows. And in the pilot, change signals are what tell the crawler when a page is worth another visit. 4. Put the page in the sitemap and link to it from your site's pages, so it doesn't stay a page nobody reaches except by chance. ## Does indexing guarantee that ChatGPT cites my site? 1. Indexing puts your page among the sources ChatGPT can draw from, but OpenAI evaluates and ranks pages on its own. And the Cloudflare post didn't publish a single number about citations or visits for the pages that were found. [A study by AirOps](https://airops.com/report/influence-of-retrieval-fan-out-and-google-serps-on-chatgpt-citations), published on March 12, 2026, found that only 15% of the pages ChatGPT fetched while answering were cited in the final answer. The study covers 15,000 questions and 548,534 pages, and it's one company's measurement on its own sample, not a number from OpenAI. If you want to measure it on your side, count the visits that carry `utm_source=chatgpt.com`. That's the advice of Cedric Huesler from Adobe, because by his measurement most ChatGPT clicks arrive with no referrer. And between October 8 and 15, 2026, according to [a Search Engine Journal report on October 15](https://www.searchenginejournal.com/chatgpt-traffic-drop-appears-to-be-a-tracking-problem/593157/), links inside the text of answers came out without the tag and without a referrer, so their visits may be counted as Direct in GA4. Huesler said OpenAI confirmed to him that it was a bug, and that the fix started rolling out on October 15, and on that same day tagged clicks came back to about 95 to 100% of the same hour a week earlier. As of the day of the report, OpenAI's release notes hadn't mentioned the fix. Mark that week in your reports, so the drop isn't read as lost traffic. ## Where do you start now? Start with your site's robots.txt today, because it's the fastest check, and a single line in it can block search entirely. After that comes the firewall, then dead links, and last your useful pages. 1. Open `/robots.txt` and make sure OAI-SearchBot isn't blocked, either by name or by the `*` group, and decide on GPTBot separately. 2. Open your firewall settings, or AI Crawl Control and the Search setting if your site is on Cloudflare, and see whether a rule blocks search bots. 3. Try a URL that doesn't exist with the `curl` command above, and make sure it returns `404`. 4. Write a list of the pages that hold specific information about your business, and make sure each one has a fixed URL and its text is in the HTML. If you want these checks and others in a single order, [Chapter 4 of the AI SEO Guide](/products/ai-seo-2027-chapter-4) has ten checks that don't touch your site, including robots.txt and the missing-URL test, and a 22-item table so you know where you stand. And the [AI SEO Guide · 2027 Edition](/products/ai-seo-guide) is an Arabic reference for preparing your site so answer engines can reach it, understand it and quote it. What does your site return right now when you open a URL that doesn't exist on it? ## FAQ **Does my site have to be on Cloudflare for ChatGPT to find it?** No. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing the IP addresses it publishes, and that applies to any site. The pilot is an extra way for pages to be found, and it was tested on about 1% of Cloudflare's network. **If I block GPTBot, does it affect my visibility in ChatGPT search?** No, according to OpenAI. Each setting is independent: GPTBot is for training and OAI-SearchBot is for search, and you can allow the second and block the first. If you allow both, OpenAI may use the same crawl for both jobs. **How do I join the Cloudflare and OpenAI pilot?** There's no published way for site owners to join. Cloudflare's July 1, 2026 post said it plans to open this capability widely before the end of 2026, with no date, and the October 15 post gave no date either. **After I edit robots.txt, how long until anything changes?** About 24 hours for OpenAI's search systems to adjust, according to the official bots page. That's only the time to read the change, not a date for your pages to appear in answers. **Does Cloudflare give OpenAI the content of my pages?** No, according to Cloudflare's July 1, 2026 post: the program is limited to search, Cloudflare isn't sharing any content, and nothing is used to train models. But OAI-SearchBot itself reads your pages if you allow it. **What is ChatGPT-User, and should I allow it?** ChatGPT-User may visit a page when a user asks ChatGPT a question. OpenAI says robots.txt may not apply to it because the user made the request, and that it doesn't decide whether your site appears in search. For search, the setting that counts is OAI-SearchBot.