<\/p>\n
AI search is no longer a future scenario. Google AI Overviews now appear on roughly half of all searches, and ChatGPT serves hundreds of millions of weekly users with real-time browsing. The robots.txt file you wrote in 2022 almost certainly doesn’t address any of it.<\/p>\n
A single missing directive can mean your content trains AI models without consent \u2014 or gets excluded from AI search entirely, including the citations that would send traffic back. This guide covers every bot and gives you copy-paste directives.<\/p>\n
Last updated: July 2026 \u2014 verify current tokens against OpenAI’s and Google’s official docs, which change occasionally.<\/p>\n
Every site owner now faces two parallel search ecosystems. Traditional search \u2014 Google and Bing \u2014 links back to your site. AI search \u2014 ChatGPT, Perplexity, Google AI Overviews \u2014 cites your content, with or without a click.<\/p>\n
Industry tracking put AI Overviews on about 48% of Google searches in early 2026, and ChatGPT reported roughly 900 million weekly active users. Ignoring how AI crawlers reach your site now means opting out of a channel that already sits above your traditional listing for many queries.<\/p>\n
| Bot<\/th>\n | Company<\/th>\n | What it does<\/th>\n | If blocked<\/th>\n<\/tr>\n<\/thead>\n | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
GPTBot<\/strong><\/td>\n| OpenAI<\/td>\n | Collects pages to train ChatGPT and future models<\/td>\n | Excluded from training. Does NOT affect ChatGPT search.<\/td>\n<\/tr>\n | OAI-SearchBot<\/strong><\/td>\n | OpenAI<\/td>\n | Crawls to serve real-time ChatGPT search answers<\/td>\n | Won’t appear in ChatGPT search. Users can’t find you there.<\/td>\n<\/tr>\n | Google-Extended<\/strong><\/td>\n | Google<\/td>\n | Controls use for Gemini and AI Overviews<\/td>\n | Excluded from AI Overviews\/Gemini. Does NOT affect Google ranking.<\/td>\n<\/tr>\n | Googlebot<\/strong><\/td>\n | Google<\/td>\n | Standard search indexing<\/td>\n | You disappear from Google Search entirely. Almost never block.<\/td>\n<\/tr>\n | CCBot<\/strong><\/td>\n | Common Crawl<\/td>\n | Public crawl used by many AI training datasets<\/td>\n | Excluded from Common Crawl, an upstream training source.<\/td>\n<\/tr>\n | PerplexityBot<\/strong><\/td>\n | Perplexity<\/td>\n | Crawls for real-time Perplexity answers<\/td>\n | Won’t appear in Perplexity search.<\/td>\n<\/tr>\n | Applebot-Extended<\/strong><\/td>\n | Apple<\/td>\n | Controls Apple Intelligence training data<\/td>\n | Excluded from Apple Intelligence training.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n | The Key Distinction: Training vs Search<\/h2>\nThis is the mistake most sites make: they see “GPTBot,” block everything from OpenAI, then wonder why ChatGPT never cites them.<\/p>\n Training bots<\/strong> (GPTBot, CCBot, Applebot-Extended) collect data to build models. Blocking them keeps your content out of future training sets \u2014 a valid choice for unique or proprietary content.<\/p>\n Search bots<\/strong> (OAI-SearchBot, PerplexityBot, Google-Extended) power real-time answers that cite your site. Blocking them removes you from AI search results. For content sites that depend on discovery, that is self-defeating.<\/p>\n You can split the difference: block GPTBot (no training without consent) but allow OAI-SearchBot (get cited in ChatGPT answers). Most AI-forward content sites in 2026 take exactly this approach.<\/p>\n Here is a recommended configuration for a content site that wants AI citations but not training-data harvesting:<\/p>\n <\/p>\n This config allows Google-Extended so you stay eligible for AI Overview citations. If you would rather keep your content out of Gemini training entirely, change that block to Catch-all wildcard (*).<\/strong> Standard directives for every crawler: block admin areas, tag archives, and internal search pages, while allowing the admin-ajax endpoint that many plugins need.<\/p>\n GPTBot.<\/strong> Blocks everything with CCBot.<\/strong> Blocks Common Crawl. Many training datasets pull from Common Crawl dumps, so this closes a secondary training pipeline that the GPTBot directive alone doesn’t cover.<\/p>\n Applebot-Extended.<\/strong> Opts your content out of Apple Intelligence training without affecting standard Applebot indexing for Siri and Spotlight.<\/p>\n OAI-SearchBot and PerplexityBot.<\/strong> Allowed, so your content can surface when users ask ChatGPT or Perplexity questions in your niche. This is how AI search sends discovery traffic back to you.<\/p>\n Google-Extended.<\/strong> Allowed here, keeping you eligible for AI Overview citations. The trade-off is that Google may also use that data to improve Gemini \u2014 each site owner weighs citation traffic against the training contribution.<\/p>\n Googlebot.<\/strong> Always allow. Blocking it removes you from traditional Google Search entirely, which is almost never the right call.<\/p>\n Then keep Google-Extended set to <\/p>\n |