Robots.txt checker for AI crawlers
Enter your store's domain and, if you like, a product URL. The checker reads your live robots.txt and shows, for every major AI crawler, whether it may fetch the page and which line decides. Training crawlers, AI search crawlers and user-triggered fetchers are listed separately, each with the vendor's own documentation and the date we checked it.
Which AI crawlers should be in your robots.txt?
These are the ten crawlers the checker tests, grouped by what the provider says each one does. Whether to let them in is your store's choice.
| Crawler | What it does | What the provider says | robots.txt | Source |
|---|---|---|---|---|
| Training | ||||
| GPTBotOpenAI | What it doesCrawls content that may be used to train OpenAI's generative foundation models. | What the provider saysOpenAI says blocking GPTBot signals that your content should not be used for training. | robots.txtOpenAI documents it as controlled by robots.txt. | SourceOpenAI's documentationLast checked November 7, 2026 |
| ClaudeBotAnthropic | What it doesCollects content that may contribute to training Anthropic's models. | What the provider saysAnthropic says blocking ClaudeBot signals that future material should be excluded from its training data. | robots.txtAnthropic says its bots follow robots.txt. | SourceAnthropic's documentationLast checked November 7, 2026 |
| Google-ExtendedGoogle | What it doesControls whether content is used to train Gemini and for grounding in Gemini Apps and Vertex AI. | What the provider saysGoogle says Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. | robots.txtGoogle says it is a standalone product token in robots.txt, with no user agent of its own. | SourceGoogle's documentationLast checked November 7, 2026 |
| AI search | ||||
| OAI-SearchBotOpenAI | What it doesUsed for ChatGPT's search features. | What the provider saysOpenAI says sites that block OAI-SearchBot are not shown in ChatGPT search answers, but can still appear as navigational links. | robots.txtOpenAI documents it as controlled by robots.txt and says a change takes about 24 hours. | SourceOpenAI's documentationLast checked November 7, 2026 |
| Claude-SearchBotAnthropic | What it doesIndexes pages to improve search quality for Claude's users. | What the provider saysAnthropic says blocking Claude-SearchBot may reduce your visibility in users' search results. | robots.txtAnthropic says its bots follow robots.txt. | SourceAnthropic's documentationLast checked November 7, 2026 |
| PerplexityBotPerplexity | What it doesSurfaces and links sites in Perplexity's search results. | What the provider saysPerplexity says PerplexityBot is not used to crawl content for AI foundation models. | robots.txtPerplexity documents it as controlled by robots.txt and says a change can take up to 24 hours. | SourcePerplexity's documentationLast checked November 7, 2026 |
| GooglebotGoogle | What it doesCrawls for Google Search, including its AI features such as AI Overviews. | What the provider saysGoogle says robots.txt rules for Googlebot control crawling for Search, including its AI features. | robots.txtGoogle says robots.txt rules for Googlebot control its crawling. | SourceGoogle's documentationLast checked November 7, 2026 |
| User-triggered fetchers | ||||
| ChatGPT-UserOpenAI | What it doesVisits pages when users ask ChatGPT or a Custom GPT, and for GPT Actions. | What the provider saysOpenAI says ChatGPT-User is not used to decide whether content appears in search. | robots.txtOpenAI says “robots.txt rules may not apply” to it. | SourceOpenAI's documentationLast checked November 7, 2026 |
| Claude-UserAnthropic | What it doesFetches pages when users ask Claude. | What the provider saysAnthropic says blocking Claude-User prevents Claude from fetching your pages when users ask. | robots.txtAnthropic says its bots follow robots.txt. | SourceAnthropic's documentationLast checked November 7, 2026 |
| Perplexity-UserPerplexity | What it doesFetches pages at a user's request. | What the provider saysPerplexity says Perplexity-User fetches pages when a user asks for them. | robots.txtPerplexity says it “generally ignores robots.txt rules.” | SourcePerplexity's documentationLast checked November 7, 2026 |
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
OpenAI uses three user agents for the web, plus one for ads, and describes each one differently.
- GPTBot crawls content that may be used to train OpenAI's generative foundation models. OpenAI says blocking it signals that your content should not be used for training.
- OAI-SearchBot is used for ChatGPT's search features. OpenAI says sites that block it are not shown in ChatGPT search answers, but can still appear as navigational links.
- ChatGPT-User visits pages when users ask ChatGPT or a Custom GPT, and for GPT Actions. OpenAI says robots.txt rules may not apply to it, and that it is not used to decide whether content appears in search.
- OAI-AdsBot visits only pages submitted as ads, and OpenAI says its data is not used to train foundation models. The checker leaves it out.
OpenAI also says that if both GPTBot and OAI-SearchBot are allowed, it may use one crawl for both purposes, and that a change to robots.txt can take about 24 hours to apply in search.
Does Google-Extended affect Google Search or AI Overviews?
Google says Google-Extended is a standalone product token with no user agent of its own. It controls whether your content is used to train Gemini and for grounding in Gemini Apps and Vertex AI. In Google's words, it does not impact a site's inclusion in Google Search, nor is it used as a ranking signal.
Google's page on AI features and your website says robots.txt rules for Googlebot control crawling for Search, including its AI features. It does not say that Google-Extended affects AI Overviews. That is why the checker shows Googlebot as a row of its own.
Do AI crawlers respect robots.txt?
Each provider says something different, so the checker shows what each one says for each crawler.
- OpenAI describes what blocking GPTBot and OAI-SearchBot does, and says robots.txt rules may not apply to ChatGPT-User.
- Anthropic says its bots follow robots.txt.
- Perplexity says Perplexity-User fetches pages at a user's request and generally ignores robots.txt rules.
- Google says robots.txt rules for Googlebot control crawling for Search, and that Google-Extended is a token in robots.txt.
robots.txt is not access control. The standard, RFC 9309, says its rules are not a form of access authorization. The checker sees only the file, not what a crawler does.
How do I block AI training but allow AI search?
Here is an example that blocks the training crawlers and lets the AI search crawlers in. It is an example, not a recommendation: whether to allow training is your store's choice.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /Each group starts with its User-agent lines. Under RFC 9309, a crawler follows the group that names it and falls back to the * group only when no group does. Googlebot is not in the example, so it keeps following your * group.
How do I edit robots.txt on Shopify?
Shopify creates robots.txt for your store. To change it, Shopify says to add a robots.txt.liquid template:
- In your Shopify admin, go to Online Store.
- Open Edit code for your theme.
- Choose Add a new template.
- Select robots.
- Choose Create template.
Shopify warns: “Incorrect use of the feature can result in loss of all traffic.” It is an unsupported customization, so Shopify Support doesn't help with the change.
Shopify's default file already blocks paths such as /admin, /cart and /checkout.
WooCommerce and Centra handle the file in other ways. Check your platform's own documentation.
What does your robots.txt not tell you?
The checker reads only the file. It can't tell you:
- whether a CDN, firewall or bot protection stops a crawler that robots.txt allows,
- whether an AI service has actually fetched, indexed or mentions your store,
- whether a change brings more mentions, citations or sales.
What ChatGPT, Gemini and Google's AI Overviews actually say when buyers ask is what Ansedd measures.
Questions about robots.txt
What is robots.txt used for?
robots.txt is a text file at the root of your site, such as yourstore.com/robots.txt. It tells crawlers which paths they may fetch, with a group of rules per crawler. The format is the Robots Exclusion Protocol, RFC 9309, and its rules are not access control.
How do I fix “blocked by robots.txt”?
Find the line that blocks the page: enter your domain and the page's address above, and the checker shows the deciding line for each crawler. Then remove or narrow that Disallow rule, or add a more specific Allow rule. Under RFC 9309 the most specific rule wins, and on a tie Allow wins. On Shopify you edit robots.txt.liquid.
How do I open my robots.txt file?
Add /robots.txt to your domain in a browser, such as https://yourstore.com/robots.txt. To change it, use your platform's settings or the file on your server. On Shopify, that is the robots.txt.liquid template.
What is Google-Extended?
A product token Google reads in robots.txt, with no user agent of its own. Google says it controls whether your content is used to train Gemini and for grounding in Gemini Apps and Vertex AI, and that it does not affect inclusion or ranking in Google Search.
What is ClaudeBot?
Anthropic's crawler for content that may contribute to training. Anthropic says blocking ClaudeBot signals that future material should be excluded from its training data. Claude-SearchBot and Claude-User are separate user agents, for search and for fetches when users ask Claude.
What is a robots.txt tester or validator, and is this one?
A tester shows whether a crawler may fetch a URL; a validator checks the file's syntax. This checker is a tester for AI crawlers: it reads your live robots.txt and evaluates it under RFC 9309 for ten crawlers, on your start page and one product page. It does not list syntax errors.
What happens if my site has no robots.txt?
Under the robots.txt standard, a missing file (a 4xx answer) means crawlers may fetch everything. A server error (5xx) is different: crawlers then assume the whole site is blocked.
How long does it take before a change applies?
OpenAI says about 24 hours for ChatGPT search, and Perplexity says up to 24 hours. The robots.txt standard says crawlers should not use a cached copy for more than 24 hours.
Your robots.txt decides who may read your store. It doesn't tell you what AI answers when buyers ask. See what ChatGPT, Gemini and Google's AI Overviews recommend in your category.
Book a demo