AI Crawler Checker: Can GPTBot and ClaudeBot Crawl Your Site?

By Linus Li · Updated 2026-10-05 · Free, runs in your browser

Rate this tool1 rating so far

Enter a domain or paste a robots.txt. The tool applies RFC 9309 to decide whether 30+ AI and search crawlers may fetch the path you enter, flags common configuration mistakes and gives you the fix.

Key takeaways

The tool applies RFC 9309 to decide whether 30+ AI and search crawlers may fetch the path you enter, flags common configuration mistakes and gives you the fix.
AI crawlers fall into AI search indexing, user-triggered fetches and model training, and blocking each kind affects a different outcome.
Each crawler carries a source reliability label and each finding an evidence level; the tool gives the site no overall score.
The robots.txt is analysed in your browser; only when you enter a domain does our server request robots.txt once.

When to use it and how

The tool covers three situations: checking before launch that robots.txt does not shut out AI search by accident; letting ChatGPT or Perplexity cite your content without handing over training data; and taking over a site and needing to see quickly how it treats each AI crawler.

Enter a domain at the top and click “Fetch robots.txt”. You can also paste the file, or click “Load example” to see a configuration with typical mistakes.
Under “Path to test”, enter the page path you care about, such as /blog/ or /products/abc.
Read “Diagnostics” on the right first. Each finding expands to show the reason, an evidence level and a fix you can copy.
Then read the crawler table. Each row shows allowed or blocked, the matched rule and line number, the group used and the source reliability of that crawler. “Blocked only” hides the rest.
To change the file, pick an intent preset at the bottom left and click “Merge into the robots.txt above and check”. The tool keeps your other rules and Sitemap lines, replaces only AI crawler groups and checks again. Replaced lines stay in place as comments, so you can check whether any path rule or Crawl-delay should be kept.

The robots.txt is analysed in your browser and never uploaded. Only when you enter a domain does our server request https://domain/robots.txt once; the FAQ explains why.

Three kinds of AI crawler

AI companies usually run several crawlers, each with its own job. Blocking one affects a different outcome:

Purpose	Examples	What blocking does
AI search index	OAI-SearchBot, Claude-SearchBot, PerplexityBot, Kimi-SearchBot	Your pages stop appearing, with citations, in that product’s search answers
User-triggered fetches	ChatGPT-User, Claude-User, Perplexity-User, Kimi-User	The assistant cannot read a page a user pastes; several vendors say these fetches may ignore robots.txt
Model training	GPTBot, ClaudeBot, CCBot, KimiBot, Meta-ExternalAgent	Content stays out of future training data; search citations are unaffected

There are also two control tokens: Google-Extended and Applebot-Extended. They are not crawlers. Googlebot and Applebot still do the fetching, and the token only decides whether content may be used for training. Google’s crawler documentation states that Google-Extended does not affect inclusion or ranking in Google Search.

Google AI Overviews and AI Mode are part of Google Search and use the index Googlebot builds. Copilot uses the Bing index. Blocking Googlebot or Bingbot therefore removes you from classic search and from those AI features at the same time.

Source reliability and evidence levels

AI crawler names change often, and public lists copy each other; some names have never been confirmed by the company that supposedly runs them. Each crawler carries a source label that links to its source:

Official docs: the operator publishes the name and purpose on its own site, as OpenAI, Anthropic, Google and Kimi do.
Third-party report: there is no public operator documentation to check, or third-party measurements contradict the operator. Bytespider is the example: ByteDance says it follows robots.txt, but TollBit’s H1 2026 data showed it reaching disallowed URLs on about half of the European sites that named it.
Unverified: the name appears only in third-party lists and we found no documentation from the operator, as with DeepSeekBot, Doubaobot and ERNIEBot. These names may not exist or may not be what the operator actually uses. Listing them does not affect other crawlers, but do not rely on them.

Every diagnostic also carries an evidence level, the same scale used across this site: A Confirmed (official documentation or public statement), B Documented, C Patented, D Speculative. The tool gives no overall score, because whether to allow a crawler depends on your goal and has no universal answer.

Mistakes the tool checks for

Blocking GPTBot and assuming you left ChatGPT search

GPTBot handles training only. ChatGPT search results depend on OAI-SearchBot, and links a user pastes are fetched by ChatGPT-User. If your goal is “cited but not trained on”, blocking only GPTBot is correct. To leave ChatGPT search entirely, add:

1 2	User-agent: OAI-SearchBot Disallow: /

The reverse, blocking OAI-SearchBot while allowing GPTBot, keeps you out of search while still allowing training. The tool marks that as a warning.

A crawler with its own group stops reading `*`

Under RFC 9309 a crawler reads only the group that matches its name, and falls back to User-agent: * only when none does. For example:

User-agent: *
Disallow: /cart

User-agent: PerplexityBot
Disallow: /blog/drafts/

PerplexityBot may now crawl /cart. To apply both rules, copy Disallow: /cart into the PerplexityBot group. The tool lists the rules each dedicated group is missing and writes the completed group for you.

`User-agent: *` blocks the whole site, AI search included

Staging sites and old sites often carry User-agent: * / Disallow: /. It blocks every crawler without its own group, including Googlebot, Bingbot and all AI search crawlers. If the goal is to stop unknown bots only, give each crawler you want an explicit Allow: / group.

Expecting Google-Extended to opt you out of AI Overviews

Google-Extended controls Gemini model training and grounding in Gemini Apps and Vertex AI. AI Overviews and AI Mode draw on the Google Search index; you can limit them only with page-level nosnippet, max-snippet, data-nosnippet or noindex, which also changes your regular snippets. Google says so in its AI features documentation.

Content-Signal and Content-Usage

robots.txt originally said only whether a crawler may fetch a URL. Since 2025 two ways of stating how content may be used have appeared, and the tool parses and lists both:

Content-Signal, proposed by Cloudflare, sits inside a User-agent group, for example Content-Signal: search=yes, ai-input=yes, ai-train=no. The three keys cover search indexing, use as input to AI answers and model training, and each takes yes or no. Sites with Cloudflare managed robots.txt get a block starting with # BEGIN Cloudflare Managed content inserted automatically, and the tool detects it.
Content-Usage comes from an IETF AIPREF working group draft, for example Content-Usage: train-ai=n, optionally preceded by a path. Its keys and values are still changing.

Both are declarations, not access control. Google has said publicly that Content-Signal has no effect on its crawlers or models, and the crawler documentation of OpenAI, Anthropic and Perplexity does not mention either. Allow and Disallow rules still decide what gets fetched.

Matching rules

The tool follows RFC 9309 and Google’s implementation:

A crawler uses the group whose User-agent matches its name (case-insensitive), or the User-agent: * group if none does. When several groups match, their rules are combined.
Within the group, the rule with the longest matching path wins; when Allow and Disallow are equally long, Allow wins.
* matches any characters and $ marks the end of the path. An empty Disallow: blocks nothing.
/robots.txt itself is always allowed.

For practitioners: details that trip people up

Consecutive User-agent lines form one group and share the rules below them. Google’s open-source parser starts a new group only after an Allow or Disallow line, so Crawl-delay, Sitemap, Content-Signal and similar lines in between do not separate two runs of User-agent lines. A User-agent: Bingbot followed only by Crawl-delay and then the next User-agent line merges both groups. The tool evaluates it that way and flags it.
Google ignores Crawl-delay; Bing, Yandex and Anthropic support it.
A 4xx robots.txt means no restrictions; a 5xx means RFC 9309 crawlers treat the whole site as disallowed. The domain fetch tells the two apart.
Amazon says Amzn-SearchBot, when it has no rules of its own, follows the rules you give other search bots. That differs from the RFC 9309 fallback; the tool shows the RFC result and notes it on that row.
Cloudflare managed robots.txt is served in front of your origin file. Same-name groups in both parts merge under RFC 9309, and when Allow: / and Disallow: / of equal length meet, Google’s implementation picks Allow.
robots.txt controls crawling, not indexing. A blocked URL with external links can still appear in results without a snippet.

FAQ

The tool says allowed, so why can’t AI reach my site?

robots.txt is only a declaration. A CDN “block AI bots” switch (Cloudflare has one), WAF rules or user-agent blocking on the server reject requests regardless of robots.txt. Most AI crawlers also do not run JavaScript, so client-side-only content looks empty to them.

I blocked ChatGPT-User or Perplexity-User. Why do they still show up in my logs?

OpenAI, Perplexity, Kimi, Amazon and Meta all document that fetches triggered by a user question may not follow robots.txt, because they count as opening a page on the user’s behalf. If you really need to stop them, block by user agent and the published IP ranges at your CDN or server.

How do I get cited by AI search without allowing training?

Allow AI search and user-triggered crawlers, and block training crawlers plus the Google-Extended and Applebot-Extended control tokens. The “Get cited by AI search, no training” preset writes this for you and can merge it into your existing file.

What about Chinese AI crawlers?

Among the Chinese AI products checked for this tool, only Kimi publishes crawler documentation: Kimi-SearchBot (search), Kimi-User (user-triggered) and KimiBot (training). ByteDance states that Bytespider follows robots.txt, but third-party measurements keep finding it on disallowed URLs. We found no crawler docs for DeepSeek, Doubao or ERNIE, so the names in circulation are marked “Unverified”. We found nothing public for Tencent Yuanbao, so it is not listed. Baidu Search still crawls with Baiduspider.

How does fetching a domain work?

The browser’s same-origin policy stops scripts on one site from reading another, so our server requests https://domain/robots.txt for you and returns only that file. Fetching is rate limited (30 per IP per hour), and the same site checked again within 10 minutes is served from cache. If a site serves different content to servers and browsers, open its robots.txt yourself and paste it.

How often is the crawler list updated?

It follows each company’s published crawler documentation, is reviewed every quarter and updated whenever a vendor announces a change; the date is at the top of the page. The label next to each crawler links to its source so you can check it yourself.