Free tool

Can AI crawlers reach your store?

Type your store's address. We read its robots.txt the way each search, AI and training crawler does, and show which ones it lets in, with the line that decides. Free, with no signup.

We read two files: your robots.txt, and your homepage to find your platform and a product page.

The crawlers we check

Each crawler reads robots.txt for its own name first, then the "User-agent: *" group if your file has no group for it. Applebot reads a Googlebot group before that one, as Apple documents.

Which AI crawlers to allow, and the mistakes that cost the most

Search crawlers

These read pages for search indexes, and the answers built on those indexes. Block one and that engine can't read your pages for its results.

Crawler	What it feeds
`Googlebot`	Google Search, including AI Overviews and AI Mode
`Bingbot`	Bing and Microsoft Copilot
`OAI-SearchBot`	OpenAI's search results
`Claude-SearchBot`	Claude's search results
`PerplexityBot`	Perplexity's search results
`Applebot`	Apple's Siri and Spotlight results

Fetches for a person

These open a page when someone asks an assistant to read it. Some operators say robots.txt may not apply to them, so this shows what your file asks, not what they'll do.

Crawler	What it feeds
`Claude-User`	Pages Claude opens when a person asks
`ChatGPT-User`	Pages OpenAI opens when a person asks; OpenAI says robots.txt may not apply
`Perplexity-User`	Pages Perplexity opens when a person asks; Perplexity says it generally ignores robots.txt

Training crawlers

These collect pages to train models. Blocking them doesn't block the search crawlers above.

Crawler	What it feeds
`GPTBot`	OpenAI's model training
`ClaudeBot`	Anthropic's model training

Opt-out tokens

Not crawlers: names you can put in robots.txt to opt out of one use of your pages, without blocking the crawler that reads them.

Crawler	What it feeds
`Google-Extended`	Gemini training and grounding, not Search, AI Overviews or AI Mode

Questions

Does blocking GPTBot keep my store out of AI search?

No. GPTBot collects pages for OpenAI's model training. OAI-SearchBot is the crawler for OpenAI's search results, and robots.txt treats them separately, so you can block one and allow the other.

Does Google-Extended control AI Overviews?

No. AI Overviews and AI Mode are part of Google Search, and Google says robots.txt rules for Googlebot are the control there. Google-Extended isn't a crawler: it's a token that controls whether Google uses your pages to train Gemini models and to ground Gemini answers, and it doesn't affect Google Search.

What if my robots.txt is missing or returns an error?

A missing file (HTTP 404) lets every crawler crawl every page. A server error (HTTP 5xx) tells crawlers to treat the whole site as blocked (RFC 9309); Google stops crawling, then falls back to the last copy it read. Google counts HTTP 429 as a server error too.

Should I give each AI crawler its own group?

Only to treat it differently. A crawler with a group of its own ignores the "User-agent: *" group entirely, so its group has to repeat any Disallow lines you still want, such as cart and checkout.

What can't this check see?

Firewalls and CDNs can turn crawlers away whatever robots.txt says. Cloudflare, for one, lets a site block AI search, agent and training crawlers across the whole site. This check reads robots.txt only, so check those settings too.

Does this check follow my robots.txt?

Yes. It always reads robots.txt, as every crawler does, and reads your homepage only if your robots.txt lets EcomIndex or every crawler in.

Crawlers can reach you. Do the answers name you?

Getting crawled is the first step. The AI Shelf Report asks OpenAI, Perplexity and Claude 20 questions your buyers would ask and counts which brands the answers name, yours and your competitors'. $99, delivered within 2 business days.

Get your report: $99 See a sample report

Sources

Checked on Nov 6, 2026. Crawler names and rules change. If something here is out of date, email hello@ecomindex.ai.

EcomIndex isn't affiliated with or endorsed by any company named on this page. The crawler, product and company names here are trademarks of their owners.