AI Crawlers Explained: GPTBot, ClaudeBot & How to Manage Them
Every AI answer about your industry starts with a crawler fetching a page. Some of those bots gather text to train models; others fetch your page live to answer a question and cite a source. Confuse the two and you can quietly disappear from AI search while thinking you only protected your content. This guide explains the major AI crawlers in 2026, what each one actually does, and how to manage their access through robots.txt so you stay visible where it counts.
Why AI crawlers suddenly matter
For two decades, bot management meant one question: can Googlebot reach my pages? In 2026 that question has fractured into a dozen. A growing share of buyers now ask an AI assistant before they ever open a search results page, and the answer they receive is assembled from pages that an AI crawler was allowed to fetch. If the right bot cannot reach you, you are not ranked low in the AI answer; you are simply absent from it.
This is the uncomfortable shift for anyone who treats crawler control as a security checkbox. The instinct, especially after years of headlines about AI scraping, is to lock everything down. But blocking indiscriminately is how brands erase themselves from ChatGPT, Perplexity, and Google's AI answers without ever noticing, because nothing in their analytics screams about it. Understanding which bot does what is now a core part of optimizing your site for AI search engines, not a niche technical concern.
The distinction that changes everything: training vs retrieval
If you remember one thing from this guide, make it this. AI crawlers fall into two fundamentally different jobs, and treating them as one category is the single most expensive mistake in AI bot management.
- Training crawlers collect text to train or improve a model. They visit broadly, store what they find, and there is no immediate link back to your site. Examples include GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, and Bytespider. Whether you allow them is a content-strategy decision.
- Retrieval and search crawlers fetch your page in real time to answer a specific user question right now, and they almost always cite the source. Examples include OAI-SearchBot, PerplexityBot, Perplexity-User, and the engines feeding Google's AI Overviews. Blocking these directly costs you live citations.
The trap is that the same operator often runs both. OpenAI uses GPTBot for training and OAI-SearchBot for live search retrieval. If you copy a popular robots.txt snippet that disallows every OpenAI agent, you have not just opted out of training; you have opted out of being cited in ChatGPT's answers. The block is silent, your traffic dashboard looks normal, and you only discover the problem when a competitor is being recommended and you are not.
Blocking a training crawler protects your content. Blocking a retrieval crawler removes you from the answer. They look almost identical in robots.txt, so the difference between visibility and invisibility is often a single line you copied without reading.
The major AI crawlers, at a glance
Here are the bots that matter most in 2026, with the operator behind each, its declared user-agent, and its primary job. User-agent strings are case-sensitive in practice, so match them exactly when you write rules.
| User-agent | Operator | Primary job |
|---|---|---|
| GPTBot | OpenAI | Training crawler for model improvement |
| OAI-SearchBot | OpenAI | Real-time search retrieval for ChatGPT answers |
| ChatGPT-User | OpenAI | On-demand fetch when a user action triggers a page visit |
| ClaudeBot | Anthropic | Crawler for training and powering Claude |
| Claude-SearchBot | Anthropic | Retrieval to surface and cite sources in answers |
| Claude-User | Anthropic | On-demand fetch from a user request inside Claude |
| PerplexityBot | Perplexity | Indexing crawler for the Perplexity answer engine |
| Perplexity-User | Perplexity | Live fetch to answer a specific user query |
| Google-Extended | Opt-out control for Gemini and Vertex AI training | |
| CCBot | Common Crawl | Open web archive used widely as training data |
| Applebot-Extended | Apple | Opt-out control for Apple AI training |
| Bytespider | ByteDance | Aggressive training crawler tied to TikTok's parent |
Two things are easy to miss in this table. First, Google-Extended and Applebot-Extended are not crawlers at all; they are tokens you use to opt out of AI training while leaving the normal Googlebot and Applebot indexing untouched. Second, the regular search engines still matter for AI: Googlebot feeds AI Overviews and Bingbot underpins much of what reaches ChatGPT through Microsoft's index. Block those classic crawlers and you damage both your search rankings and your AI presence in one move.
How robots.txt actually controls these bots
robots.txt is a plain text file at the root of your domain that tells well-behaved crawlers what they may fetch. It works the same way for AI bots as it always has for search engines: you declare a user-agent, then list the paths it is allowed or disallowed to crawl. The key word is well-behaved. Compliance is voluntary, so reputable operators honor it and rogue scrapers may ignore it entirely.
The most common goal in 2026 is the nuanced one: stay open to the bots that cite you, and make a deliberate choice about the ones that train on you. Here is a robots.txt that allows the real-time search crawlers while opting out of the major training crawlers, which is a sensible default for a brand that wants AI visibility but is cautious about feeding model training.
# Allow real-time AI search and retrieval crawlers
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Opt out of training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# Keep classic search crawlers fully open
User-agent: Googlebot
Allow: /
# Default for everything else
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlIf your goal is the opposite, maximum reach with no training concerns, you simply allow everything and let every compliant bot through. And if you genuinely want to wall off all AI use, you disallow the training and the search crawlers alike, accepting that you forfeit AI citations. The point is that each line is a choice. Copying someone else's file without mapping it to your own goal is how brands end up with rules that contradict their strategy.
Most crawlers obey the most specific user-agent group that matches their name, not the wildcard. So a named group for GPTBot overrides your User-agent: * block. Always set rules per bot rather than assuming the catch-all covers them, and never assume a global Disallow stops a bot that has its own named group.
The real trade-off of blocking training crawlers
Blocking training bots feels like the safe, protective choice, and for some publishers it genuinely is. If your content is your product, paywalled, or licensed, keeping it out of training corpora is reasonable and increasingly tied to licensing negotiations. But for most marketing-led sites, the calculus is different and the cost is subtler than it looks.
The case for blocking is straightforward: you protect proprietary content, you avoid feeding a model that competes with you, and you keep leverage if you later want to license access. The case against is mindshare. Models trained on your content learn that your brand exists, what you do, and how you talk about your category. That ambient familiarity shapes how an assistant describes your space even when it is not citing a live page. Opt out entirely and you can become the brand the model has never heard of.
Blocking training crawlers is a content-protection decision. Blocking search crawlers is a marketing decision. Far too many teams make the second by accident while trying to make the first.
There is no universal right answer. A media company protecting an archive should weigh blocking seriously. A SaaS company that wants to be the default recommendation in its category usually should not block training crawlers at all, because being learned is the entire point. What no one should do is block both kinds of bot reflexively and call it security.
How this connects to llms.txt
robots.txt tells crawlers what they may not touch. It does not help them understand what is worth reading. That gap is what the emerging llms.txt convention tries to fill: a curated file that points AI systems to your most important, cleanest content rather than leaving them to crawl everything blindly. Think of robots.txt as the gate and llms.txt as the welcome map.
The two are complementary, not interchangeable. You still need robots.txt to govern access, and llms.txt to guide attention once access is granted. If you are allowing the search crawlers in, giving them a clear path to your best material is the natural next step; our walkthrough on how to set up llms.txt for your project covers the format and what to include. Pairing a deliberate robots.txt with a thoughtful llms.txt is how you move from merely permitting AI access to actively shaping what AI learns from you.
Verifying crawler access in your server logs
You cannot manage what you do not measure, and AI crawler behavior is invisible in most analytics tools because bots do not run JavaScript or fire your tags. The ground truth lives in your raw server access logs. That is where you can confirm whether GPTBot, OAI-SearchBot, PerplexityBot, and ClaudeBot are actually reaching your pages, and what status codes they receive.
- Filter by user-agent. Grep your access logs for each bot name and confirm it is appearing at all. Silence from a bot you expect to see is a red flag worth investigating.
- Check the status codes. A healthy crawl returns 200. A wall of 403 means something is blocking the bot, and a run of 429 means you are rate-limiting it, often unintentionally.
- Watch the crawl paths. Confirm the bots are reaching your important pages, not just your homepage, and that no key section is being skipped because of a stray disallow.
- Track changes over time. Re-check the logs after every robots.txt, CDN, or firewall edit, so you catch a regression the day it happens rather than a quarter later.
Server-log analysis is also how you connect access to outcomes. Knowing the search bots reach you is the precondition for the citation tracking covered in our guide to tracking and measuring GEO performance; access without measurement is guesswork, and measurement without access is impossible.
The accidental block: firewalls, CDNs, WAFs, and plugins
Here is the failure mode that catches the most teams. Their robots.txt is perfect, deliberate, and well-reasoned, and AI crawlers still cannot get in, because something upstream is silently rejecting them before robots.txt is ever consulted. These accidental blocks are far more common than intentional ones.
- CDN bot protection. Many CDNs ship with bot-mitigation rules that lump AI crawlers in with malicious scrapers and serve them a challenge page or a 403. The default settings are often the culprit.
- Web application firewalls. A WAF rule meant to stop scraping can match legitimate AI user-agents, especially aggressive blanket rules that target anything that looks like automation.
- Security plugins. CMS plugins frequently add their own bot blocklists that quietly include AI crawlers, and they update those lists without telling you.
- Rate limiting. Tight per-IP limits can throttle a legitimate crawl into a stream of 429 responses, which a bot may interpret as a soft block and back off from.
The lesson is that robots.txt is necessary but not sufficient. If your logs show a bot getting blocked despite an allow rule, look up the stack at your CDN, firewall, and plugins. This kind of misconfiguration is one of the most common findings in a technical audit, and it is invisible until someone looks, which is part of why understanding what GEO is includes owning the full request path, not just the file at your domain root.
Handling aggressive crawlers and crawl rate
Not every bot is polite. Some crawlers, Bytespider being the most cited example, are known for crawling aggressively enough to strain a server, and not every operator respects a crawl-delay directive. Managing access is not only about allow and disallow; it is also about volume.
If a legitimate bot is hammering your origin, you have options short of an outright block. You can serve it through your CDN cache so the load never reaches your origin, apply a generous rate limit that slows it without triggering a hard refusal, or restrict it to the sections that matter and disallow expensive, low-value paths like faceted search URLs. For genuinely abusive or non-compliant scrapers that ignore robots.txt, a server-level or WAF block by user-agent or IP range is the appropriate tool, since a robots.txt rule they already ignore will not help. The goal is to protect performance while keeping the citing bots welcome.
Common mistakes to avoid
Most AI crawler problems are not exotic. They are a handful of recurring errors, and avoiding them puts you ahead of the majority of sites.
- Blocking search bots while meaning to block training bots. The headline mistake of this guide, and the most expensive. Always separate OAI-SearchBot from GPTBot in your thinking and your file.
- Copying a robots.txt from a blog post. A snippet built for a paywalled publisher is wrong for a brand chasing visibility. Map every line to your own goal.
- Mistyping user-agent strings. Bots match their exact name. A misspelled or wrongly cased user-agent silently does nothing, leaving you with rules you think are active but are not.
- Disallowing Googlebot or Bingbot. These classic crawlers feed AI Overviews and ChatGPT. Blocking them is a double loss across search and AI at once.
- Forgetting the upstream stack. A clean robots.txt cannot rescue you from a CDN or WAF that blocks bots first. Check the whole path.
- Set and forget. The roster of AI crawlers changes constantly. A file you wrote in 2024 is missing bots that matter in 2026, including the search retrieval agents that did not exist when you last looked.
Crawler access is only one layer of being chosen by AI. Structured data helps engines parse what they fetch, and our guide to schema markup for GEO covers that side; meanwhile, what you do once the right bots can reach you determines whether you are quoted, which is the subject of our piece on getting your content cited by AI. Access is the entry ticket, not the prize.
Not sure which AI crawlers can actually reach your site?
We will audit your robots.txt, CDN, and server logs in a free 30-minute session and tell you exactly which AI bots are blocked, which are citing you, and what to change. No upsell.
Get Your Free AuditFrequently asked questions
What is an AI crawler?
An AI crawler is an automated bot operated by an AI company that fetches pages from your site. Some, like GPTBot and ClaudeBot, gather text to train large language models. Others, like OAI-SearchBot and PerplexityBot, retrieve pages in real time to answer a user question and cite sources. They identify themselves with a user-agent string, and you can allow or block each one in robots.txt.
Should I block GPTBot?
It depends on your goal. GPTBot collects content to train OpenAI models, so blocking it keeps your work out of future training data. But GPTBot is not the same bot that fetches pages to answer live questions in ChatGPT search. If you block GPTBot you protect your content, while if you also block OAI-SearchBot you can quietly remove yourself from ChatGPT answers. Most brands chasing visibility allow the search bot and decide on the training bot separately.
What is the difference between a training crawler and a search crawler?
A training crawler, such as GPTBot, ClaudeBot, Google-Extended, or CCBot, collects content that may be used to train or improve a model, with no immediate link back to your site. A search or retrieval crawler, such as OAI-SearchBot or PerplexityBot, fetches your page to answer a specific user question right now and usually cites you. Blocking training crawlers protects content, while blocking search crawlers costs you live citations and visibility.
How do I block AI crawlers in robots.txt?
Add a user-agent group for the bot you want to control and a disallow rule beneath it. For example, a group with User-agent: GPTBot followed by Disallow: / tells GPTBot not to fetch any page. You can add a separate group for each crawler and mix allow and disallow rules. Place robots.txt at the root of your domain, keep the user-agent strings exact, and remember that compliance is voluntary, so well-behaved bots obey but rogue ones may not.
Does blocking AI crawlers hurt my SEO?
Blocking dedicated AI crawlers like GPTBot or CCBot does not affect classic Google rankings, because those bots are separate from Googlebot. The real risk is blocking the wrong bot. If you disallow Googlebot or the AI search crawlers that feed AI answers, you can lose both organic rankings and AI visibility at once. Audit your rules carefully so you only restrict the crawlers you actually intend to restrict.
How do I verify that AI crawlers can access my site?
Check your server access logs and filter for the user-agent strings of the bots you care about, such as GPTBot, OAI-SearchBot, PerplexityBot, and ClaudeBot. Confirm they are returning 200 status codes rather than 403 or 429 responses. Also review your CDN, firewall, and WAF dashboards, since many accidental blocks happen there rather than in robots.txt. Testing robots.txt rules in a validator and watching logs after each change is the most reliable check.