How to Stop Your Shopify AI Chatbot From Giving Customers Wrong Answers

Priya runs Earthen, a four-year-old natural skincare DTC brand on Shopify Plus, roughly $4M ARR, 60% of orders shipping to the EU. Her support team caught the bot telling a German customer the return window was 90 days. Earthen’s policy is 30 days globally and 14 days for EU under the standard withdrawal rule. The customer screenshotted the chat, threatened a chargeback, and Earthen honored the 90 days because the bot had said it in writing. That single ticket cost about €180 in product and €40 in return shipping, and the team logged seven similar incidents the same week.

So Priya called us. We did what we always do on these calls. We opened her Shopify admin, the chatbot’s training pipeline, and her help center, side by side.

Why bots hallucinate on a Shopify store isn’t really a model problem

The first thing every founder assumes when a bot gives wrong info is that the underlying language model is broken. It isn’t, usually. We hear a version of this on every onboarding call, and the cleanest framing came from a Shopify Plus operator we worked with last spring: “Most chatbot mistakes aren’t actually AI problems. They’re knowledge-base problems.”

That isn’t a slogan. It’s the diagnosis 80% of the time.

When a modern support bot answers a question, it goes through three steps in roughly this order. It pulls relevant context from your store’s data sources, the model writes a response grounded in that context, and a guardrail layer (or the system prompt) decides whether to send it. If the wrong answer is wrong-but-specific (real policy numbers, real product specs, just incorrect), the issue is almost always step one. The model wrote a confident answer using bad context.

If the bot says “We accept 90-day returns” and your real policy is 30, that means somewhere in the data the bot reads, there’s a page that says 90 days. A legacy returns FAQ from 2023. A duplicate product page nobody archived. A help center article a contractor wrote and forgot. The model didn’t invent the number. It found it.

The knowledge base most stores don’t realize they have

When we say “knowledge base”, merchants usually picture a Notion doc or a help center. The actual knowledge base your bot reads is much wider than that. It’s every product description on the storefront, every shipping policy version that lives anywhere on shop.app, every Klaviyo template that mentions a number, every PDF return label your old 3PL hosted on a public S3 bucket Google indexed in 2024.

This is the part most teams skip. The bot doesn’t read “the FAQ”. It reads whatever your retrieval layer pulls in, which is usually a mix of: the help center, the storefront pages, the product catalog, recent order data, and whatever PDFs and policy pages it was seeded with at setup. Half of those sources were last edited by someone who no longer works there.

Two questions to ask before you touch the bot:

• What sources is the retrieval layer actually indexing right now?
• When was each one last updated, and who owns it today?

If you can’t answer both for every source in the index, that’s the work. The model is fine. The library it reads from is the problem.

Auditing the answers before customers find them

The audit is unglamorous and it’s the only thing that works. Pull the last 30 days of bot transcripts. Filter for the ones flagged by customers, the ones that ended in a handoff, and the ones where the customer’s next message was “that’s wrong” or “are you sure”. That’s your starter set. Usually it’s between 40 and 200 conversations depending on store size.

Read every one against your written policy doc. Grade each as correct, incorrect, or partially correct. Track which source the bot pulled from in the response (Gorgias, Tidio, Lyro, Intercom Fin, and the rest all log this). The patterns show up fast. On Earthen, 62% of the wrong answers came from one outdated returns page that nobody had archived. Three sources caused 90% of the incidents.

Audit cadence we use on new clients: 100 conversations the first week, then 50 a week thereafter for the first two months, then 25 a week ongoing. Light enough that one support lead can do it in 90 minutes.

Restructuring returns, shipping, and product data so the bot can’t drift

Once the audit is done, the cleanup is mostly editorial work. Every fact the bot might be asked about lives in exactly one canonical place. Returns policy is one page. EU-specific returns is one section under that page, called out as the regional override. Shipping rates table is one page, sourced from the same logic your checkout uses. Product specifications live in metafields on the product, not in description HTML.

A practical structure we hand clients on Shopify Plus:

• One markdown policy doc per topic (returns, shipping, sizing, gift cards, subscriptions, B2B).
• Each doc has a single “last verified” date at the top, a single owner, and a single regional override section.
• The retrieval layer is pointed at that doc folder and nothing else. The old PDF and the old help center pages get archived, redirected, or noindexed.

This is the part where a founder usually winces and asks “is that really enough material for the bot to work?” Yes. Most stores have 6 to 10 policy topics. A clean canonical doc for each is roughly 800 to 1,500 words. The bot needs accurate context, not a library.

Shopify’s own help docs on policy structure are a reasonable starting template. The point isn’t the template though. The point is “one source of truth per topic, owned by a human, dated.”

Teaching the bot to say “let me check”

This is the single biggest move and it’s the one most stores skip. The fastest reduction in wrong answers isn’t better RAG. It’s a better escape hatch.

Your bot should refuse to answer in three situations. When the user’s question doesn’t match anything in retrieval with high confidence. When it touches a policy that varies by region and the user’s region is ambiguous. When the answer would commit the store to something with money or legal weight (refund promises, custom discounts, “we’ll waive the fee”).

In each case, the bot says something like: “I want to make sure I get this right for your order. I’m passing this to a human teammate who’ll respond within the hour.” Then it escalates. The customer never sees a wrong number because the bot never wrote one.

The implementation is usually 20 lines of system prompt plus a confidence threshold in the retrieval layer. Most platforms let you tune this. Gorgias calls it “human handoff triggers.” Tidio Lyro calls it “fallback intent.” Intercom Fin has confidence-based routing built in. The vocabulary differs. The behavior is the same.

Tracking wrong-answer rate so the fix actually sticks

You don’t actually own this metric until it’s on a weekly dashboard. Otherwise the audit happens once, the team feels good, and three months later the bot is hallucinating again because someone added a holiday-shipping promo page and forgot to update the canonical doc.

Define wrong-answer rate as: (audited responses graded incorrect or misleading) divided by (total audited responses). Sample 50 to 100 conversations per week. Track it next to your other support KPIs (CSAT, AHT, deflection rate). Target under 2% on a real audit. Anything above 5% means the source data has drifted again.

Klaviyo’s own benchmark report on AI support puts industry-average wrong-answer rates somewhere between 4% and 11% across DTC merchants who don’t actively audit. That’s the baseline. Below 2% is achievable and defendable.

And honestly, this is also the metric that makes the case to the CFO. Wrong-answer rate sits next to ticket volume on the dashboard, and the financial cost of each incident is calculable from the audit (refunds issued, shipping eaten, churn from the customer who never came back).

A four-week accuracy hardening plan

Week one is the audit. Pull 100 transcripts, grade them, identify the top three drift sources. Get the wrong-answer rate baseline. Don’t touch the bot yet.

Week two is canonical-doc cleanup. Rewrite the top three policy docs the bot pulled wrong answers from. Archive the duplicates. Point the retrieval layer at the cleaned folder only. Verify each policy doc has a single dated owner.

Week three is guardrails. Add the “let me check” fallback. Tune confidence thresholds. Write five region-specific overrides into the system prompt (EU returns, US tax-free states, AU/NZ shipping bands, UK post-Brexit duty, anything else local to your operation). Re-test the original 100 transcripts against the new bot. Measure the delta.

Week four is the monitoring layer. Set up the weekly 50-conversation sample. Add wrong-answer rate to the support dashboard. Assign one person to own the audit. Set a recurring 30-minute review on the support lead’s calendar.

That’s it. Four weeks, no rebuild. Earthen ran this plan in May and the wrong-answer rate went from 7.1% to 1.4% in 28 days. Two months later it’s holding at 1.6%. Two policy docs drifted in that time and the weekly audit caught both inside seven days.

What we keep telling clients

The temptation when a bot gets caught lying is to start shopping for a smarter bot. Sometimes that’s right. Most times it’s the expensive answer to the wrong question.

What actually fixes this is dull. It’s a returns policy doc with one owner and a “last verified” date. It’s a Slack channel where support flags every wrong answer the same day it happens. It’s a 30-minute weekly review where someone reads ten transcripts. None of it scales by buying software. All of it scales by deciding the policy library is a real asset, not a side project.

Priya ran the four-week plan in May. Her team caught the drift sources, rewrote the returns doc, added the EU override, and put the audit on the weekly support meeting. The bot didn’t get smarter. The library got cleaner and the bot stopped getting cornered into making things up. The week we wrapped, her support lead messaged me: “It’s basically just answering questions correctly now. Wild.”

That’s the bar. Not “the bot is impressive.” Just “the bot is correct, and we’d know within seven days if it stopped being.”

Questions we get every week

Do we need to switch chatbot platforms to fix this?

Almost never. We’ve fixed accuracy problems on Gorgias, Tidio Lyro, Intercom Fin, Re:amaze, and a few custom Shopify-native setups using the same playbook. The platform matters less than the source data and the guardrails. Switch only if the platform genuinely can’t expose retrieval confidence or block low-confidence responses.

How much of this can the bot vendor’s onboarding team handle?

The bot setup, yes. The canonical-doc cleanup and audit cadence, no. Vendor onboarding teams ship a working bot. They don’t own your policy library or your weekly QA. Both of those are merchant-side work and they’re the work that actually moves wrong-answer rate.

What if we don’t have enough volume for a useful audit?

If you’re under 50 conversations a month, audit every one. The work is the same. Smaller stores actually have an easier time of this because there are fewer policy docs and fewer regional overrides to keep clean. The four-week plan compresses to about two weeks.

Can we just turn the bot off for risky topics like refunds?

Yes, and many stores do. Refunds and order-specific issues route to humans by default; the bot handles tracking, product Q&A, and general shipping questions. That’s a fine architecture. Just make sure the bot doesn’t try to answer “where’s my refund” from cached policy text. Make it route, not respond.

How long before we can trust the bot on policy questions again?

After the four-week plan, give it another four weeks of weekly audits before you increase exposure. Trust is rebuilt by measurement, not by belief. If wrong-answer rate sits under 2% for eight straight weeks across all topic categories, the bot has earned the broader scope.

If you want help running this audit and rebuilding your policy library on Shopify, talk to us about a chatbot accuracy diagnostic and we’ll benchmark your wrong-answer rate, clean the canonical docs, and hand your team the weekly dashboard inside four weeks.