What Shopify Sidekick Can't Tell You About Your Numbers
A merchant asked Sidekick a simple margin question and shipped the wrong ad budget off the answer. Here's where the AI assistant helps, where it fails, and what we use instead.
Priya runs finance for a home-fragrance brand on Shopify Plus, roughly $3.2M in trailing revenue across a website and two retail pop-ups. Last quarter she asked Sidekick a question she’d normally hand to an analyst: what was the contribution margin on the candle line in June? The answer came back in four seconds, clean and specific, 41%. She set the July ad budget against it. The real figure, once her ops lead rebuilt it by hand, was closer to 28%. Sidekick hadn’t refused the question. It had answered a slightly different one and made the number look finished.
That gap, thirteen points of margin, is the whole problem in miniature. Not that the assistant is useless. That it’s confidently wrong in the exact places where being wrong costs money.
Where it actually earns its keep
We’re not here to trash the tool. On a good day Sidekick saves real time, and we tell clients to use it for the things it’s built for.
It’s genuinely good at navigation. “Where do I change my shipping zones?” gets you the right settings page faster than clicking through the admin. It’s good at single-metric lookups with no filtering: total sales yesterday, order count this week, your top product by units. It drafts things well, a product description, a discount-code name, a first pass at a policy page. And it’s a decent front door to Shopify’s own help docs, summarizing what the Sidekick help pages already say without making you read all of them.
Those are the jobs where a small error either can’t happen or doesn’t matter. Nobody ships an ad budget off a product-description draft.
The place it quietly falls apart
The trouble starts the moment a question needs more than one step. A merchant we ran a discovery call with last month put it about as bluntly as it gets: the assistant is close to useless for anything past the simplest task, and it gets basic things inside Shopify’s own system wrong.
We hear a version of that every few weeks, so we went and stress-tested it. The failures cluster in three spots.
Filtered questions are the first. “Sales from returning customers in the UK, excluding wholesale” is the kind of thing you’d build in a report with three conditions. Ask Sidekick and you’ll often get a number, but you can’t see which conditions it actually applied, and sometimes it silently drops one.
Multi-step math is the second. Anything shaped like “revenue minus COGS minus shipping, as a percentage” asks the model to hold several values and combine them correctly. That’s where Priya’s 41% came from, a plausible-looking answer to a calculation it didn’t fully carry out.
Then there’s the quiet third one: definitions. Ask about “profit” and you have no idea whether it means gross sales, net of discounts, net of refunds, or something with COGS in it. The answer won’t tell you which. So the number might be technically real and still answer a question you didn’t ask.
Why a language model is bad at exact arithmetic
It helps to know why this happens, because it tells you when to trust the thing.
Sidekick is built on a large language model. A language model predicts the most plausible next piece of text given everything before it. That’s a fantastic engine for language and a shaky one for arithmetic, because “plausible” and “correct” are not the same thing when the answer is a specific number. The model is optimized to sound right.
So a wrong analytics answer doesn’t look wrong. It arrives with the same tidy phrasing, the same decimal precision, the same calm confidence as a correct one. There’s no tremble in the voice. Compare that to a broken report, which usually looks broken, throws an error, returns blank, shows an obviously insane figure. A hallucinated metric shows you 41% and moves on.
Shopify has been layering in more structured retrieval so Sidekick pulls from your actual store data rather than guessing, and that’s helped on the simple lookups. But the moment a question needs joining tables, applying filters in a specific order, or doing several operations in sequence, you’re back to trusting a text-prediction engine with your accounting. That’s not a knock on the engineering. It’s just the wrong tool held the wrong way.
The sixty-second check before you trust a number
Here’s the habit we install on every team that leans on Sidekick. Any number that’s going to drive a decision gets reproduced somewhere deterministic before anyone acts on it. It takes about a minute.
Open Analytics, go to Reports, and rebuild the same question with an actual filtered report. Shopify’s reporting docs walk through the filters if you haven’t lived in there. If the report agrees with Sidekick, great, you’ve lost sixty seconds and bought certainty. If it doesn’t, you just caught a thirteen-point error before it became a budget.
For anything with math in it, the check is even simpler: ask Sidekick to show its work, the components, not just the result. If it can’t lay out revenue, COGS, shipping and the subtraction, the percentage it gave you is a guess wearing a lab coat.
The rule we say out loud on kickoff calls: Sidekick can point you at a number, but the number that decides spend gets confirmed in a report. Every time.
One more tell worth knowing. If you ask the same filtered question twice, slightly reworded, and get two different answers, that’s the model improvising rather than reading. A native report doesn’t do that. It returns the same figure because it’s running the same query against the same rows. When two phrasings disagree, believe neither and go build the report.
Matching the job to the right tool
Most of the pain we see comes from one thing, using Sidekick for jobs it was never meant to do. Different questions want different tools, and once a team sorts that out the frustration mostly evaporates.
| The job | Reach for | Why |
|---|---|---|
| ”Where do I change X?” / quick single lookup | Sidekick | Fast, low stakes, hard to get wrong |
| Exact filtered figure (region, customer type, date) | Native Analytics reports | You see and control every filter |
| Custom or repeatable exact query | ShopifyQL | Deterministic, auditable, reusable |
| Anything a board or investor sees | BI connector (warehouse + dashboard) | Governed definitions, one source of truth |
ShopifyQL is the piece most merchants skip. It’s a query language for your store data that gives you an exact, repeatable answer, the same question returns the same number every time, and you can read the query to see what it did. For a recurring question like weekly contribution margin by collection, that beats re-asking a chatbot and hoping.
The stack we actually set up for clients
For a store doing under a million, native reports plus the sixty-second check usually covers it. Sidekick stays in the toolbox for lookups and drafting, and nothing that touches money gets trusted without a report behind it.
Past a couple million in revenue, or once someone’s presenting numbers to people who write checks, we move the important metrics out of any AI assistant entirely. Store data flows into a warehouse, definitions get written down once (what “profit” means, what “returning customer” means, how refunds are handled), and a dashboard reads from that. The AI layer, if it’s there at all, sits on top as a way to ask questions in plain language, but the arithmetic lives underneath in something deterministic.
That’s the pattern that holds up: model as translator, math as machinery. Sidekick will keep getting better at the lookups, and we expect the retrieval to tighten over the next year. But we don’t expect a text-prediction engine to become the thing you close your books against, and we’re not building anyone’s finance stack on that bet.
The setup cost is smaller than people fear. Most of the work is agreeing on definitions, and that’s a one-afternoon conversation between whoever owns finance and whoever owns growth. Once “profit” means one thing across the whole company, the tooling almost picks itself, and the assistant becomes a nice way to ask that governed data a question in English instead of a nag you don’t quite trust.
What we keep telling clients
The mistake isn’t using Sidekick. It’s trusting it uniformly, treating a margin calculation the same as a “where’s my settings page” question. Those are different risk levels and they deserve different tools.
So we draw one line and make everyone hold it. Lookups and drafts, use the assistant freely, it’ll save you real time. Anything that decides a budget, an inventory order, or lands on a slide, confirm it in a report or a query first. Sixty seconds. That single habit turns Sidekick from a liability into an actual time-saver, because you stop fearing it and start using it for what it’s good at.
Priya’s team runs it that way now. Sidekick is still open in a tab all day for quick questions. But the candle-line margin, the one that set the ad budget, lives in a ShopifyQL query her ops lead built once. This quarter’s number was 29%, confirmed before a dollar moved. The thirteen-point surprise doesn’t happen anymore, because nobody’s asking a chatbot to do the accounting.
Questions we get every week
Is Sidekick just bad, then? No, it’s good at a specific set of jobs and bad at another set. Navigation, single-metric lookups and drafting are genuinely useful. Filtered questions, multi-step math and anything with a fuzzy definition are where it slips, and those happen to be the questions that decide money.
How do I know if a Sidekick number is wrong? You usually can’t tell by looking, which is the trap. The only reliable check is to reproduce it somewhere deterministic, a native filtered report or a ShopifyQL query, and see if the figures match. Build that into any decision that touches spend or inventory.
What should I use instead for real reporting? Native Analytics reports for exact filtered figures, ShopifyQL for repeatable custom queries, and a proper BI connector once numbers are going in front of a board. Keep Sidekick for the quick stuff and let the deterministic tools own anything that matters.
Will this get better as Sidekick matures? The lookups will keep improving, and Shopify keeps tightening how it pulls from your real store data. But exact multi-step arithmetic is a structural weak spot for language models, so we’d keep the verification habit regardless of how polished the assistant feels.
If you want a reporting stack you can actually defend at a board meeting, talk to us about a short analytics audit and we’ll map every number that matters to a tool that gets it right every time.