Which AI crawlers to allow in your store's robots.txt

Every crawler reads your robots.txt before anything else. That one file decides which AI engines can read your pages for their answers and which can collect them for training. It's easy to get wrong in a way that hides your store from the engines you want. This guide sorts the AI crawlers by what they do, says which ones a store usually lets in, lists the mistakes that cost the most, and shows where to edit the file on Shopify, WooCommerce and BigCommerce.

To see what your robots.txt allows today, run our free crawler check on your store first.

Four kinds of crawler

The crawlers fall into four groups, and the groups matter more than the names:

• Search crawlers read pages for a search index, and the answers built on it. Block one and that engine can't read your pages for its results.
• Training crawlers collect pages to train models. Blocking one doesn't block the same company's search crawler.
• Fetchers that act for a person open a page when someone asks an assistant to read it. Some operators say robots.txt may not apply to them.
• Opt-out tokens aren't crawlers. They're names you put in robots.txt to opt out of one use of your pages, without blocking the crawler that reads them.

The crawlers, by company

What the operators say about them:

• Google. Googlebot crawls for Google Search, which includes AI Overviews and AI Mode, and Storebot-Google crawls for Google Shopping. Google-Extended has no user agent of its own: it controls whether Google uses your pages to train Gemini models and ground Gemini answers, and doesn't affect Google Search (Google: common crawlers). Google's user-triggered fetchers generally ignore robots.txt (Google: user-triggered fetchers).
• Microsoft. Bingbot is Bing's standard crawler (Bing: crawlers), and Copilot's answers rest on Bing's crawling and indexing (Bing Webmaster Guidelines).
• OpenAI. OAI-SearchBot is for ChatGPT search and GPTBot for training, and each is set separately. ChatGPT-User fetches pages when a user asks, OpenAI says robots.txt may not apply to it, and it isn't used to decide what appears in ChatGPT search. OpenAI also runs OAI-AdsBot, which visits the landing pages of ChatGPT ads and isn't used for training (OpenAI: crawlers).
• Anthropic. Claude-SearchBot is for search, ClaudeBot collects content that may be used for training, and Claude-User fetches pages when someone asks Claude. Anthropic says its crawlers follow robots.txt (Anthropic).
• Perplexity. PerplexityBot is for Perplexity's search results and isn't used to train AI models. Perplexity-User fetches for a user and generally ignores robots.txt (Perplexity crawlers).
• Apple. Applebot powers search in Spotlight, Siri and Safari, and its data may also be used to train Apple's models. Applebot-Extended doesn't crawl: disallowing it opts your pages out of that training. If your robots.txt has no rules for Applebot but has rules for Googlebot, Applebot follows the Googlebot rules (Apple: About Applebot).
• Amazon. Amzn-SearchBot is for search experiences such as Alexa and isn't used for training. Amazonbot may be used to train Amazon's AI models. Amzn-User acts on user requests and may not follow every robots.txt rule (Amazon: Amazonbot).
• Meta. meta-webindexer improves Meta AI's search results, meta-externalagent is used for purposes such as training AI models, and meta-externalfetcher fetches for users and may bypass robots.txt (Meta: web crawlers).
• Common Crawl. CCBot builds Common Crawl's open web dataset, and you block it with a User-agent: CCBot group (Common Crawl: CCBot). Common Crawl says its dataset has become one of the most widely used sources of training data for large language models (Common Crawl: about).

Which ones a store should let in

Search crawlers: let them all in. They're how your pages get into AI answers and search results. That means Googlebot, Storebot-Google, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, Amzn-SearchBot and meta-webindexer.

Fetchers that act for a person: leave them alone. They open a page because a shopper asked about it. Some don't follow robots.txt anyway.

Training crawlers: your call. Blocking GPTBot, ClaudeBot, Amazonbot, meta-externalagent or CCBot keeps your pages out of those training sets from then on. It doesn't take you out of the same companies' search answers, because those come from the search crawlers above. Google and Apple handle training with opt-out tokens instead: disallow Google-Extended or Applebot-Extended and Googlebot and Applebot keep crawling for search.

To block one training crawler and nothing else, give it a group of its own:

User-agent: GPTBot
Disallow: /

Leave the search crawlers out of the file, or covered only by your User-agent: * group, unless you mean to treat one differently.

A new option from 2027. A proposal in an IETF Internet-Draft, still a work in progress, adds a way to state a training opt-out for every company that reads it, with a group for the name __usage_ai-training (IETF draft):

User-agent: __usage_ai-training
Disallow: /

It isn't a crawler, and it doesn't change how any crawler reads the rest of your file. Microsoft says it will honor this group from February 1, 2027, and that it won't affect your eligibility for regular Bing results (Bing: AI preferences). Google says it intends to support it in early 2027 (Google: common crawlers). Anthropic says it has recognized a "no AI training" signal from the same IETF work since October 30, 2026 (Anthropic).

The mistakes that cost the most

1. Blocking the wrong OpenAI crawler. Blocking GPTBot stops training, not ChatGPT search. Blocking OAI-SearchBot to stop training takes your site out of ChatGPT's search answers.
2. A group with only Allow lines. A crawler that finds a group naming it ignores the User-agent: * group entirely (RFC 9309). So a group like User-agent: PerplexityBot followed by Allow: / also opens the cart, checkout and account pages your * group closes. Repeat those Disallow lines in the crawler's own group.
3. A robots.txt that errors. If the file answers with a server error, crawlers treat the whole site as blocked (RFC 9309), and Google counts HTTP 429 as a server error too (Google: HTTP status codes). A missing file (HTTP 404) is fine: every crawler may crawl every page.
4. Blocking Googlebot to keep out of AI Overviews. That takes you out of Google Search. Google-Extended doesn't control AI Overviews either. Google's controls for what an AI feature can show are its preview controls, such as nosnippet and max-snippet (Google: AI features and your website).
5. Using robots.txt to hide a page. Google says a page blocked in robots.txt can still show up in its results, without a description, if other sites link to it. To keep a page out, use noindex or a password instead (Google: robots.txt introduction).
6. Forgetting Applebot. With no Applebot group, Applebot follows your Googlebot group, so a rule written for Google applies to Apple too.
7. A firewall that says no. CDNs and firewalls can turn crawlers away whatever robots.txt says. Cloudflare, for one, lets a site block AI search, agent and training crawlers across the whole site (Cloudflare). Check those settings as well as the file.

Where to edit robots.txt on your platform

Save a copy of your current file (yourstore.com/robots.txt) before you change anything, and check it again afterwards.

Shopify. Shopify writes your robots.txt for you. Its defaults keep crawlers out of the admin, cart, checkout, account and order pages, and out of filtered and sorted collection pages. To change it, go to Online Store, open your theme's ... menu, choose Edit code, then Add a new template, choose robots, and create it. That adds a robots.txt.liquid template, which replaces the generated file (Shopify Help Center: editing robots.txt.liquid). Shopify warns that this is an unsupported customization its Support can't help with, and that a mistake can cost you all your traffic. It recommends building on its default rules in Liquid rather than pasting in a fixed file, because fixed rules go out of date (Shopify: robots.txt.liquid). Shopify also notes that your product data reaches AI shopping channels such as ChatGPT and Copilot through Shopify Catalog, separately from robots.txt (Shopify Help Center).

WooCommerce. If there's no robots.txt file in your site's root folder, WordPress serves one it generates. The usual way to edit it is your SEO plugin:

• In Yoast SEO, go to Tools, then File editor (Yoast).
• In Rank Math, turn on Advanced Mode, then go to General Settings, then Edit robots.txt. Rank Math edits the generated file, so delete any robots.txt file in your root folder first (Rank Math).

BigCommerce. Go to Settings, then Website, and scroll to Search Engine Robots. BigCommerce merges your rules with its defaults, which keep crawlers out of the cart, checkout, account, search and similar pages. It also adds rules you can't see in the control panel, including a group that keeps known high-volume and AI crawlers out of cart, checkout, account, admin and filter pages, so check yourstore.com/robots.txt for the full file. BigCommerce says you don't need any entries to let AI crawlers in (BigCommerce Help Center: robots.txt).

Questions

Should I block AI crawlers? Block training crawlers if you'd rather your pages weren't used for training. Don't block search crawlers if you want to be in AI answers.

Does blocking GPTBot remove me from ChatGPT? No. GPTBot is OpenAI's training crawler; ChatGPT search uses OAI-SearchBot, and OpenAI sets them separately.

How fast does a change take effect? OpenAI says about 24 hours for ChatGPT search, and Google generally caches a robots.txt for up to 24 hours (Google: robots.txt specification). Shopify applies your edit at once, but crawlers may take a while to notice.