llms.txt vs robots.txt vs Google-Extended: What Actually Stops GPTBot, ClaudeBot, and PerplexityBot
Quick Answer
- llms.txt does not block anything. It is a content-guidance file that points AI models to your clean markdown — access control is not its job.
- robots.txt is what controls access — but only for crawlers that choose to honor it, and you must name each user-agent (GPTBot, ClaudeBot, Google-Extended) explicitly.
- Some AI bots ignore robots.txt. Server- or WAF-level blocking (Cloudflare’s AI bot rules) is the only hard stop.
- Google-Extended lets you opt out of Gemini and Vertex training without losing Google Search rankings — they are separate user-agents.
Three files claim to manage how AI systems use your site, and they do completely different things. People set up one, assume they are covered, and stay exposed.
After configuring all three controls on a content site and watching the server logs, the picture got clear fast. Here is what each one actually does.
What is llms.txt and does it actually block crawlers?
No — llms.txt does not block crawlers, and that confusion costs people real protection. It is a proposal for a /llms.txt file that points language models to a curated, markdown version of your key pages.
The idea, introduced by Jeremy Howard of Answer.AI in 2024, is to give models clean context instead of forcing them to parse cluttered HTML, navigation, and ad markup.
Think of it as a sitemap for reasoning, not a gate. It says “here is the good stuff, in a format you can read.” It never says “stay out.”
How is llms.txt different from robots.txt?
robots.txt is an access-control file; llms.txt is a content-delivery file. One manages permission, the other manages presentation.
robots.txt has been a web standard since 1994 and is read by nearly every serious crawler. llms.txt is a 2024 community proposal with no enforcement and inconsistent adoption.
| Feature | robots.txt | llms.txt |
|---|---|---|
| Purpose | Allow / disallow access | Point models to clean content |
| Status | Established standard (1994) | Community proposal (2024) |
| Blocks bots? | Yes, if honored | No |
| Location | /robots.txt | /llms.txt |
You can run both at once. They are not rivals — they solve separate problems.
Which AI crawlers obey robots.txt, and which ignore it?
The major named bots publicly commit to honoring robots.txt, but enforcement varies and unnamed scrapers routinely skip it. Naming each user-agent is the only reliable approach.
OpenAI runs three separate agents: GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for live user fetches. Blocking one does not block the others.
Anthropic‘s ClaudeBot handles training-data collection, while Claude-User fetches pages a person asks about in real time. Perplexity uses PerplexityBot for indexing and a separate user-triggered fetcher.
The takeaway is uncomfortable but useful. robots.txt manages the cooperative bots. It does nothing against the ones that do not cooperate.
Which AI bot is which? A user-agent reference
Each major AI company runs more than one crawler, and they serve different functions. Blocking the training bot while allowing the search bot is the common goal, so the names matter.
Here is the practical map of the user-agents you will see most often in 2026 server logs:
- GPTBot (OpenAI) — collects training data.
- OAI-SearchBot (OpenAI) — indexes pages for ChatGPT search.
- ChatGPT-User (OpenAI) — fetches a page live when a user asks about it.
- ClaudeBot (Anthropic) — collects training data.
- Claude-User (Anthropic) — real-time fetch for a user’s question.
- PerplexityBot (Perplexity) — indexes for the answer engine.
- Google-Extended (Google) — the token that governs Gemini and Vertex AI training.
- CCBot (Common Crawl) — feeds an open dataset that many other models train on.
Common Crawl deserves special attention. Its CCBot feeds an open archive that countless downstream models reuse, so blocking CCBot cuts off a wide supply chain in one line.
What is Google-Extended, and why is it separate from Googlebot?
Google-Extended is a control token that opts your content out of Gemini and Vertex AI training without affecting how Googlebot ranks you in Search. Google split them on purpose.
If you disallow Googlebot, you vanish from Google Search. If you disallow only Google-Extended, you keep your rankings but withhold your content from generative model training.
That distinction matters for publishers who want traffic but not free training data. You can have one without surrendering the other.
According to OpenAI’s published policies, google-Extended does not impact a site’s inclusion or ranking in Google Search — it solely governs whether content helps improve Gemini apps and the models that power them.
How do I set up llms.txt and robots.txt correctly?
Put each file at your root domain and name user-agents explicitly. Below is a working robots.txt pattern for opting out of major AI training while staying visible in search.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Googlebot Allow: /
That config keeps Googlebot indexing you while shutting out OpenAI’s GPTBot, Anthropic’s ClaudeBot, Common Crawl’s CCBot, and Gemini training. Place it at yoursite.com/robots.txt.
For llms.txt, the format is markdown: an H1 with your site name, a short blockquote summary, then linked sections pointing to your best pages.
Disallow: / under the wrong user-agent can quietly deindex your whole site.How do I block AI bots on WordPress specifically?
WordPress serves a virtual robots.txt by default, so you cannot just upload a file — you either edit it through your SEO plugin or replace it with a real static file. Both work; pick one.
RankMath and Yoast each expose a robots.txt editor under their tools menu, where you paste the user-agent rules directly. Changes go live without touching server files.
For server-level enforcement on a Contabo or similar VPS, add user-agent deny rules in the Nginx or Apache config, or use a security plugin like Wordfence to block by user-agent and IP.
What about Cloudflare and server-level AI blocking?
Server- and WAF-level rules are the only controls that physically stop a bot, because they reject the request before content is served. robots.txt asks; the firewall enforces.
Cloudflare offers a one-click “Block AI Bots” toggle and a managed rule that uses behavioral signals — not just declared user-agents — to catch crawlers masquerading as browsers.
For teams without Cloudflare, the same effect comes from user-agent and IP rules at the Nginx, Apache, or application layer. The principle holds: a returned 403 is enforcement, a robots.txt line is etiquette.
The trade-off is maintenance. Bot identities change, IP ranges rotate, and aggressive blocking can catch legitimate tools. Hard blocking buys control at the cost of upkeep.
How do I verify a crawler is really GPTBot and not a faker?
Confirm the request’s IP against the operator’s published ranges, because the user-agent string itself is trivial to spoof. Anyone can label a scraper “GPTBot” — the IP cannot lie as easily.
OpenAI, Anthropic, and Perplexity each publish the IP ranges and verification methods for their bots. A request claiming to be GPTBot from outside OpenAI’s published block is an impostor you can treat as hostile.
For Googlebot and Google-Extended, run a reverse DNS lookup on the IP and confirm it resolves to a googlebot.com or google.com host, then forward-resolve back to the same IP.
Will blocking AI crawlers hurt my search visibility?
Not if you block the right bots. Training crawlers and search crawlers are separate, so you can refuse training while staying fully indexed and rankable.
The real risk is a careless rule. A blanket Disallow: / under Googlebot, or a server rule that catches Googlebot’s IPs, will pull you out of Google Search entirely.
There is also a strategic question. As AI answer engines send a growing share of referral clicks, some publishers now stay open to search-oriented bots like OAI-SearchBot and PerplexityBot while blocking only training collection.
One thing worth saying plainly: none of these files stop a model from quoting you when a user pastes your URL. Real-time fetchers like ChatGPT-User and Claude-User act on a person’s request, not a crawl schedule.
Key Takeaway
Use llms.txt to be cited, robots.txt to politely manage cooperative AI bots, Google-Extended to opt out of Gemini training while keeping rankings, and Cloudflare or server rules when you need a real wall. They stack — they do not replace each other.
Frequently asked questions
Does llms.txt help my SEO rankings?
Not directly. Google has not confirmed using llms.txt as a ranking input. Its value is in feeding cleaner content to AI assistants that may cite you, not in moving Google Search positions.
Will blocking GPTBot remove me from ChatGPT search results?
Not necessarily. GPTBot is for training. ChatGPT’s live search uses OAI-SearchBot and ChatGPT-User. To stay searchable but untrained, block GPTBot and allow the search agents.
Is robots.txt legally binding on AI companies?
No. robots.txt is a voluntary convention, not law. Cooperative companies honor it; enforcement against those who ignore it relies on technical blocking or separate legal action.
Do I need both llms.txt and robots.txt?
If you want AI citations and access control, yes. They serve different goals and run independently. Most sites benefit from configuring both deliberately rather than relying on either alone.
How do I check which AI bots are hitting my site?
Read your raw server access logs and filter by user-agent strings such as GPTBot, ClaudeBot, and PerplexityBot. Cloudflare Analytics and Search Console crawl stats add a higher-level view.
Last updated: 2026-05-28
한국어 
English 