Is Your Shopify Theme Blocking GPTBot, ClaudeBot and PerplexityBot? Here's How to Check

Marco runs the dev team at a Shopify Plus agency in Toronto. Last Tuesday morning a client account manager forwarded him a Slack message: their D2C cookware brand had been getting cited inside ChatGPT for “induction-ready Dutch ovens” for nine months and the citations had silently dropped to zero across the last 14 days. Revenue from AI-source sessions had been tracking at 1.4 percent of total and had collapsed to under 0.2 percent. Marco’s first guess was a model update on OpenAI’s side. He pulled the access logs, filtered for the four bots his team tracks, and watched 73 percent of GPTBot requests come back as 500 errors. The store had pushed a theme update on a Thursday. The errors started Friday morning.

Why bot-blocking is the silent visibility killer for Shopify stores

The brand never lost its Google rank. GSC still showed impressions trending flat. The competitor that took its citation slot in ChatGPT did so without any visible content investment on their side, because the slot opened up the moment Marco’s client started returning 5xx codes to GPTBot.

Three things happen when a Shopify theme starts misbehaving for AI crawlers. The traffic from search holds because Googlebot uses a different infrastructure path. The conversion data inside GA4 looks normal. The dashboard inside the brand’s BI tool says everything is green. And meanwhile, the model surface is dropping you out of every retrieval pass it runs.

We have audited 14 Shopify stores in the last 90 days for crawler issues. Six had at least one of the major AI bots returning 5xx for over 50 percent of requests. Two had a paywalled app silently blocking GPTBot via user-agent filtering that nobody on the brand side knew was installed. One had blocked PerplexityBot by accident through a custom robots.txt edit a previous agency had made 18 months earlier.

The fix is rarely complicated. It just has to be looked for, because nothing else in the analytics stack will tell you. Honestly, you have to go looking.

The nine AI crawlers every Shopify merchant should map

The list as of May 2026. Map all of them, or your visibility data has holes.

GPTBot is OpenAI’s training and retrieval crawler. The user-agent string includes “GPTBot” and the bot announces itself in robots. The most common one we see blocked accidentally.

ChatGPT-User is the user-agent when ChatGPT browses to your store on behalf of an active user query. Different from GPTBot. Some themes treat them the same and shouldn’t.

OAI-SearchBot is OpenAI’s search crawler powering the ChatGPT search index. Newer than GPTBot, less universally recognized by middleware.

ClaudeBot is Anthropic’s crawler. There is also a separate Claude-User identity for browse-on-behalf sessions. Often missed by stores that only configured GPTBot.

PerplexityBot is Perplexity’s primary indexer. The brand we audited last week had been silently blocking this for a quarter because of a misconfigured CDN rule.

Google-Extended is Google’s opt-out user-agent for AI training and Gemini. Blocking it does not affect your Google Search ranking, but blocking it removes you from Gemini citations. Most brands want it on.

Bytespider is ByteDance’s crawler (TikTok and Doubao). Becoming relevant for brands selling into Asia-Pacific markets. Opinions vary on whether to allow it. Default: opt-in if you ship internationally.

CCBot is the Common Crawl bot. Indirectly feeds multiple LLM training datasets. Allow it.

Applebot-Extended is Apple’s opt-out for Apple Intelligence and Siri. Same principle as Google-Extended. Blocking it does not affect Apple search but removes you from Apple’s generative answers.

That is the nine. Treat the list as a checklist on every quarterly audit. The crawler landscape will change. The discipline of mapping it does not.

How to audit your robots.txt without breaking theme defaults

Shopify’s default robots.txt is reasonable. It allows the major search and AI bots, blocks /admin and /checkout, and includes a sitemap declaration. The problem is that brands edit it without understanding what they are touching.

The audit starts at https://yourstore.com/robots.txt. Pull the file with curl and check four things. Are GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, CCBot, and Applebot-Extended either Allowed or absent from a Disallow list? Are there any user-agent entries you don’t recognize? Is the sitemap declaration pointing at the right URL? And are there any wildcard Disallow lines that might be sweeping in AI bots accidentally?

If the file has been customized, it lives under Online Store, Themes, Customize robots.txt.liquid on Plus stores. We have lost count of how many times an old developer added a “User-agent: *” with a “Disallow: /products” line they meant for a specific case three years ago. The line was right at the time. The current effect of the line is wrong.

Then check whether the brand has any robots-managing app installed. We hit one last month: a “bot protection” app called Cloudsec Shield (no longer in the App Store) that was silently blocking GPTBot at the app proxy. The brand had been paying $39 a month for it for 14 months. Cancellation took 4 minutes. The citation rate came back inside a week. The official Shopify robots.txt customization guide covers the safe edit patterns.

Reading server logs to confirm crawlers are actually getting through

Robots.txt is intent. Server logs are reality.

Shopify does not expose raw access logs through the UI. There are three ways to get them. Web Pixel events can fire on bot-suspicious sessions and forward to a data warehouse. A Cloudflare worker in front of the store can log to R2 or to BigQuery. And for Plus clients, the access log export is available via the partner channel.

What you are looking for: across each of the nine bots, count requests per day for the last 30 days, and check the status code distribution. Healthy looks like 200 (60-90 percent), 304 (5-30 percent for sites with proper caching headers), occasional 404. Unhealthy looks like 500 (anything over 2 percent), 403 (anything outside known blocked paths), 429 (rate limit), or unexpectedly 301-chained.

When we ran this for Marco’s client, the breakdown was 73 percent 500s and 22 percent 429s for GPTBot specifically. ClaudeBot was clean. The theme update had introduced a server-side render path that timed out under bot concurrency and Shopify’s CDN was rate-limiting on top.

You also want to confirm bot identity, because some traffic claims to be a bot and isn’t. We cross-reference the source IP against the known ranges published by OpenAI, Anthropic, Perplexity, and Google. Unverified traffic gets dropped from the analysis. The shopify.dev developer reference on storefront access patterns is the canonical guide for the log shape you should expect.

Fixing 500 errors and rate limits that block ChatGPT and Claude

The two failure modes account for 80 percent of what we see.

500 errors usually come from theme code that times out under bot load. Bots crawl wider and faster than humans. A product template that does five Liquid loops over collection metafields will render fine for a user and time out under a 20-request-per-second crawl. The fix is template-side: cache the loop output, render via Hydrogen or a static section, or move the expensive logic to a Function.

500s also come from third-party apps that hook into product page render. App-injected scripts that hit a remote API on every request will cascade into 500s when the remote API rate-limits or the app’s CDN flakes. The fix is to identify the app via the Network tab and either move the call to a deferred script or replace the app.

Rate limits (429s) come from Shopify’s CDN protections firing on what looks like burst traffic. GPTBot tends to crawl in bursts. The default Shopify CDN policy is permissive but theme-level apps can install tighter custom rules. The audit is: walk the app list, identify any “bot protection” or “security” app, and confirm what rules it has active. We have had to uninstall four such apps across our client base this year.

A note on Cloudflare. If the brand has Cloudflare in front of Shopify, the rate-limit decision often happens there, not at Shopify. Check Cloudflare’s bot management settings. Bot management can be on by default for some plans and will silently challenge or block AI crawlers. Whitelist the nine user-agents explicitly.

What “allow vs disallow” should look like for a privacy-aware brand

The question we get most often. Should we allow all bots, or selectively block them?

For most Shopify catalog brands, allow all nine. The visibility gain from being inside ChatGPT, Claude, Perplexity, and Gemini answers far exceeds whatever privacy or competitive concern motivates blocking. If a brand is worried about training data use, the cleanest move is to allow GPTBot, ChatGPT-User, ClaudeBot, OAI-SearchBot, and PerplexityBot for retrieval (these enable citations) and to block Google-Extended and Applebot-Extended for training (those don’t affect retrieval-time citations).

For brands with sensitive customer or pricing data exposed on non-public pages, the Disallow line should be path-scoped, not bot-scoped. Disallow /admin, /checkout, /account, and any custom paths that should not be public. The bots will read the rest.

We had one brand argue strongly for blocking all AI bots on principle. We ran the numbers. They were sitting at 3.1 percent of revenue from AI sources at the time. Blocking would have cost them roughly $180K in annualized revenue against a privacy concern that did not have a specific operational threat behind it. They reversed the decision inside a week.

An agency-grade audit checklist we now run on every onboard

The checklist sits in our onboarding doc. Twelve items. We run it for every new Shopify client before any AEO or visibility work starts.

The opening six are robots and policy: pull robots.txt, list all user-agents, identify any custom Disallow lines, check for any installed bot-management apps, confirm Cloudflare bot settings if present, and document the platform’s expected baseline. None of this is hard. All of it gets skipped if the engagement is rushed.

The middle four are log-based: confirm log access path, count requests per bot per day, check distribution by status code, check distribution by path.

The last two are validation: hit the store with each of the nine bot user-agents via a swap-UA curl, and confirm response code 200 and content-type. Anything else: ticket the brand-side dev team with the exact failure.

A clean audit takes us about four hours. A dirty one (apps to remove, theme code to refactor, Cloudflare rules to rewrite) runs ten to fifteen hours. We do it as a fixed-fee diagnostic because the variance in scope is too wide to estimate hourly.

Questions we get every week

Will allowing AI bots hurt my Shopify performance? Not measurably. The nine bots collectively account for under 4 percent of total request volume on every store we have measured. Performance hits are usually template inefficiency surfacing under bot load, not the bots themselves being heavy. Fix the template, not the bot policy.

Do I need a separate llms.txt file if I configure robots.txt correctly? Yes. robots.txt controls access. llms.txt curates what the model should read first and how to represent your store. They are complementary, and we ship both on every AEO engagement.

How often should the audit run? Quarterly at minimum, and after any theme update or app install. Theme updates are the single most common cause of new 500-error patterns we see. The 14-day shadow that takes citations offline after a theme push is the most preventable visibility loss in our portfolio.

Can I block competitors’ AI agents while allowing the major bots? You can try, but most competitive scraping happens via residential proxy networks that do not announce themselves with a known user-agent. Block-listing won’t reach them. The defensible move is differentiation on schema, catalog quality, and category positioning rather than blocking.

What we keep telling clients

Bot management is plumbing. It is not the strategic work.

What it does is make the strategic work possible. If your robots.txt is blocking GPTBot, your llms.txt rollout will not produce citations. If your theme is returning 500s to PerplexityBot, your schema cleanup is invisible to the models that matter. The plumbing has to be right before the rest of the program shows results. That is the order we run engagements in: audit and fix bots first, ship the model manifest second, schema and feeds third, prompt tracking last.

Marco’s client got the 500s fixed by Wednesday of the same week. Theme code that did a chained Liquid loop over metafields got refactored into a Function. Cloudflare’s rate-limit rule got loosened. By the Friday of the following week, citation rate had climbed back to 1.1 percent of revenue. By day 30, it was sitting at 1.8 percent, above the pre-theme-push baseline. The brand’s CFO sent Marco a one-line Slack: “What just happened?” That was a good week.

We are telling every new Shopify client that walks in: run the bot audit before anything else. Twelve items, four to fifteen hours, fixed fee. It is the cheapest, fastest, highest-leverage visibility win on the engagement plan. The teams that skip it spend six months wondering why their AEO program isn’t producing returns. The ones that run it have answers inside two weeks.

To run a 12-point AI crawler audit on your Shopify store, book a 30-minute scoping call with Monkey Man and we will quote a fixed-fee diagnostic before the end of the call.