- GPTBot collects training data for ChatGPT — block it to opt out of training
- OAI-SearchBot powers real-time ChatGPT search — allow it to get cited
- Google-Extended controls Gemini training and Google AI Overview citations
- All three are separate user-agents; you can allow one while blocking another
AI search is no longer a future scenario. Google AI Overviews now appear on roughly half of all searches, and ChatGPT serves hundreds of millions of weekly users with real-time browsing. The robots.txt file you wrote in 2022 almost certainly doesn’t address any of it.
A single missing directive can mean your content trains AI models without consent — or gets excluded from AI search entirely, including the citations that would send traffic back. This guide covers every bot and gives you copy-paste directives.
Last updated: July 2026 — verify current tokens against OpenAI’s and Google’s official docs, which change occasionally.
Why This Matters Now (2026)
Every site owner now faces two parallel search ecosystems. Traditional search — Google and Bing — links back to your site. AI search — ChatGPT, Perplexity, Google AI Overviews — cites your content, with or without a click.
Industry tracking put AI Overviews on about 48% of Google searches in early 2026, and ChatGPT reported roughly 900 million weekly active users. Ignoring how AI crawlers reach your site now means opting out of a channel that already sits above your traditional listing for many queries.
The Complete Bot Reference
| Bot | Company | What it does | If blocked |
|---|---|---|---|
| GPTBot | OpenAI | Collects pages to train ChatGPT and future models | Excluded from training. Does NOT affect ChatGPT search. |
| OAI-SearchBot | OpenAI | Crawls to serve real-time ChatGPT search answers | Won’t appear in ChatGPT search. Users can’t find you there. |
| Google-Extended | Controls use for Gemini and AI Overviews | Excluded from AI Overviews/Gemini. Does NOT affect Google ranking. | |
| Googlebot | Standard search indexing | You disappear from Google Search entirely. Almost never block. | |
| CCBot | Common Crawl | Public crawl used by many AI training datasets | Excluded from Common Crawl, an upstream training source. |
| PerplexityBot | Perplexity | Crawls for real-time Perplexity answers | Won’t appear in Perplexity search. |
| Applebot-Extended | Apple | Controls Apple Intelligence training data | Excluded from Apple Intelligence training. |
The Key Distinction: Training vs Search
This is the mistake most sites make: they see “GPTBot,” block everything from OpenAI, then wonder why ChatGPT never cites them.
Training bots (GPTBot, CCBot, Applebot-Extended) collect data to build models. Blocking them keeps your content out of future training sets — a valid choice for unique or proprietary content.
Search bots (OAI-SearchBot, PerplexityBot, Google-Extended) power real-time answers that cite your site. Blocking them removes you from AI search results. For content sites that depend on discovery, that is self-defeating.
You can split the difference: block GPTBot (no training without consent) but allow OAI-SearchBot (get cited in ChatGPT answers). Most AI-forward content sites in 2026 take exactly this approach.
Setting Up Your robots.txt
Here is a recommended configuration for a content site that wants AI citations but not training-data harvesting:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /tag/ Disallow: /search/ User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: Googlebot Allow: / Sitemap: https://designcopy.net/sitemap_index.xml
The
Disallow: / under GPTBot blocks all pages from OpenAI training. The Allow: / under OAI-SearchBot lets ChatGPT still find and cite you in search. These directives are independent — one never affects the other, which is exactly what lets you opt out of training while staying in AI search.This config allows Google-Extended so you stay eligible for AI Overview citations. If you would rather keep your content out of Gemini training entirely, change that block to Disallow: / — you lose AI Overview eligibility in exchange.
What each section does
Catch-all wildcard (*). Standard directives for every crawler: block admin areas, tag archives, and internal search pages, while allowing the admin-ajax endpoint that many plugins need.
GPTBot. Blocks everything with Disallow: /. OpenAI cannot use your content to train future models. This is your training opt-out, and it does not touch ChatGPT search.
CCBot. Blocks Common Crawl. Many training datasets pull from Common Crawl dumps, so this closes a secondary training pipeline that the GPTBot directive alone doesn’t cover.
Applebot-Extended. Opts your content out of Apple Intelligence training without affecting standard Applebot indexing for Siri and Spotlight.
OAI-SearchBot and PerplexityBot. Allowed, so your content can surface when users ask ChatGPT or Perplexity questions in your niche. This is how AI search sends discovery traffic back to you.
Google-Extended. Allowed here, keeping you eligible for AI Overview citations. The trade-off is that Google may also use that data to improve Gemini — each site owner weighs citation traffic against the training contribution.
Googlebot. Always allow. Blocking it removes you from traditional Google Search entirely, which is almost never the right call.
“But I want AI Overview citations”
Then keep Google-Extended set to Allow: /, as above. Google AI Overviews cite external sources, and those citations carry clicks. If you publish factual, well-structured content with clear answer blocks, AI Overviews can become a meaningful traffic source — at the cost of contributing that content to Gemini’s training.
What Does NOT Work
Blocking GPTBot doesn’t remove you from ChatGPT — it only affects future training; content already in the model and live browsing via OAI-SearchBot remain. Blocking OAI-SearchBot doesn’t affect Bing — the crawlers are independent despite the partnership. robots.txt is a request, not enforcement — well-behaved bots obey it; malicious scrapers ignore it.
Should You Block or Allow? A Decision Framework
| Your goal | GPTBot | OAI-SearchBot | Google-Extended | CCBot |
|---|---|---|---|---|
| Maximize all traffic, don’t mind training | Allow | Allow | Allow | Allow |
| Get AI citations, block training (recommended) | Block | Allow | Allow | Block |
| Full opt-out from the AI ecosystem | Block | Block | Block | Block |
For most content sites in 2026, the sweet spot is block training bots (GPTBot, CCBot) but allow search bots (OAI-SearchBot, PerplexityBot, Google-Extended). You keep your content out of training pipelines without sacrificing AI-discovery traffic. Once this is set, focus on structure — see how to get cited by ChatGPT.
Testing Your Setup
Three free ways to verify robots.txt is working: check Google Search Console’s crawl stats for Googlebot activity; use OpenAI’s robots.txt validator to confirm GPTBot and OAI-SearchBot see the right directives; and simply visit yoursite.com/robots.txt to confirm the sections are present.
Frequently Asked Questions
Does blocking GPTBot affect my Google rankings?
No. GPTBot is entirely separate from Googlebot. Blocking it has zero impact on your Google search positions, indexing, or ranking.
How do I check if AI bots are crawling my site?
Check server access logs for user-agents containing “GPTBot,” “OAI-SearchBot,” or “Google-Extended.” Most hosting panels expose raw logs, and CDNs like Cloudflare log bot traffic in analytics.
Can I allow specific pages and block others for AI training?
Yes. Instead of Disallow: /, use Disallow: /proprietary-data/ with Allow: /blog/. Granular directives protect unique content while still contributing general content to training if you choose.
What’s the difference between Google-Extended and noindex?
Google-Extended affects only Gemini training and AI Overviews. A noindex tag affects all Google services, including traditional search. Don’t use noindex as a substitute for Google-Extended control.
If I block everything, can users still find me?
Yes, through traditional Google Search if Googlebot is allowed. But you won’t appear in ChatGPT, Perplexity, or Google AI Overviews — for a content site, that means losing an emerging traffic channel.
Is robots.txt enough to prevent AI scraping?
It stops well-behaved crawlers (Google, OpenAI, Anthropic, Perplexity). It does not stop unauthorized scrapers. For sensitive content, add Cloudflare’s AI scraper controls, rate limiting, and authentication.
Last updated: July 2026. Bot user-agent tokens and behavior change — check OpenAI and Google’s official documentation for the latest. Related: AEO complete guide · best AI SEO tools.
한국어 
English