Build vs Buy: Choosing an AI Fraud-Detection Stack for Shopify Stores (Signifyd, Native, or Custom)

Devon runs delivery at a six-person Shopify agency, and last month a client cornered him on a call. The client, a supplements brand doing about $6M a year, had just eaten three chargebacks in two weeks on high-value orders. The question on the table was blunt: do we buy Signifyd, or do you build us something smarter?

Devon’s honest first answer was “neither, yet.” But that’s a hard thing to say to a founder who just lost four grand and wants a wall built by Friday.

The reflex in that moment is to reach for the most powerful-sounding option, which usually means a paid AI vendor or, for the technically ambitious, a custom model. Both can be right. Both are also frequently the wrong call for the store actually sitting in front of you, and an agency that recommends the heavy option by default ends up selling complexity its clients don’t need and can’t maintain.

So this is the framework we use to decide, and how we keep the recommendation honest.

What Shopify’s native fraud analysis already covers

Before anyone reaches for an app, it’s worth being precise about what the platform hands you for free, because a lot of agencies under-rate it.

Shopify’s built-in fraud analysis runs every order through a set of checks and returns a low, medium, or high risk rating. It evaluates the obvious signals: address verification, CVV match, IP geolocation versus billing and shipping, proxy and VPN detection, and order velocity off a given card or device. On Shopify Payments stores it layers in network-level signals from across Shopify’s order volume, which is a genuinely large dataset.

For a store with a normal catalog and normal order values, that’s often enough to catch the clumsy fraud, and the clumsy fraud is most of it. The native rating plus a tight review habit handles a surprising share of cases at zero added cost.

What it doesn’t do is decide for you. Native analysis flags and rates, but it won’t auto-approve or auto-cancel with any guarantee behind it, and it won’t reimburse you when a “low risk” order turns out to be a stolen card. The standard advice you’ll hear if you ask around is some version of “Shopify has several fraud prevention apps available, including AI-driven solutions like Signifyd,” and that advice skips the more important question of whether this particular store has outgrown what it already owns.

When native is enough and when it isn’t

The honest dividing line isn’t a revenue number, it’s a risk profile. Two stores doing identical GMV can sit on opposite sides of it.

A store selling $40 candles to repeat domestic customers can run on native analysis plus a human glancing at the occasional flag, pretty much indefinitely. The fraud volume is low, the average order is low, and a rare bad order stings but doesn’t threaten the business. Bringing in a paid vendor there is buying insurance against a fire in a stone house.

The picture flips when a few things stack up. High average order value, so each fraudulent order hurts and each false decline costs a real sale. Catalogs that fraudsters love, electronics, gift cards, luxury, anything resellable fast. Heavy international or cross-border volume where the native geo-signals get noisier. And order volume high enough that no human can review the flags by hand without falling behind. When several of those are true at once, native analysis stops being a safety net and starts being a list of warnings nobody has time to act on.

That’s the moment to bring in help. Not before.

The chargeback guarantee is the actual product

Here’s the thing agencies miss when they compare fraud apps on detection accuracy. For most paid vendors, detection is the feature and the guarantee is the product.

When a tool like Signifyd or NoFraud approves an order and it later comes back as a fraudulent chargeback, the better plans reimburse your client for the loss. That’s the real shift. It’s not that the model is dramatically smarter than Shopify’s, though it often is on hard cases, it’s that someone else has agreed to eat the downside of a wrong call. That single change rewires how a merchant behaves, because the fear comes out of the approve decision.

Think about what fear costs. A nervous merchant cancels good orders, declines travelers and gift-buyers, and bleeds margin from false positives that never show up on any dashboard. A guarantee lets them approve confidently and pushes the risk of being wrong onto a balance sheet built to absorb it. So when you’re scoping a vendor for a client, you’re not really buying a better algorithm. You’re buying the removal of a judgment call your client keeps getting wrong in the expensive direction.

Read the guarantee terms closely though, because they vary a lot. Some cover only orders the vendor explicitly approved, some carve out card-not-present edge cases, and some have claim processes strict enough that the “guarantee” is thinner than the marketing suggests.

Comparing the realistic options

Most agency decisions come down to four shapes, not forty products. Here’s how they actually differ for a Shopify store.

Option	Best fit	Guarantee	Lift to run
Native fraud analysis	Low-risk catalogs, modest AOV, manageable volume	None	Low, but needs a human review habit
Fraud Filter / rules apps	Stores wanting custom block rules on top of native	None	Medium, you maintain the rules
Managed vendor (Signifyd, NoFraud)	High AOV, risky catalog, or volume past manual review	Yes, on approved orders	Low after setup, vendor owns the model
Custom scoring model	Portfolios or very high-risk catalogs at scale	You build and own it	High, ongoing ML and ops cost

The middle two are where stores spend the most time agonizing and where the answer is usually simplest. A rules app like Shopify’s Fraud Filter is great if your client has a specific, knowable pattern to block (a region, a product, a velocity threshold) but it carries no guarantee and the rules rot if nobody tends them. A managed vendor costs more per order but offloads both the model and the financial risk, which is what stressed merchants are actually paying to escape.

Custom sits at the far end, and it’s the one agencies most often pitch and least often should.

Integration and latency at checkout

Whatever you recommend has to survive contact with the checkout, and this is where a clean-looking plan can quietly tank conversion.

A fraud tool can score in one of two places, and the difference matters. Pre-authorization scoring at checkout can block an order before it’s placed, which sounds ideal until you realize every millisecond of latency there is friction on your highest-intent moment. Post-authorization scoring reviews the order just after it’s placed and cancels or holds the bad ones, which keeps checkout fast but means you’re refunding rather than blocking. Most managed vendors lean post-auth for exactly this reason, they’d rather protect conversion and clean up after than slow the buy button.

For agency work the integration cost is real and worth scoping honestly. A managed vendor on Shopify is usually a near-plug-in install, app plus a little configuration, and you’re live in a day. A custom model is a different animal entirely: you’re standing up a scoring service, wiring it into checkout or order-create webhooks, handling the latency budget, and owning every false decline it produces at 2am. The native option, of course, has zero integration cost because it’s already running.

So the integration question isn’t just “can we connect it.” It’s “who owns this thing in six months, and do they have the time.” For most clients, the answer quietly rules custom out.

Total cost of ownership, not the sticker price

The fee is the number everyone fixates on and the least useful one for making the call. You have to price the whole system.

A managed vendor typically charges a percentage of GMV or a per-order fee, and on paper that can look steep against a $6M run rate. But you net it against three things the fee buys back: the chargeback losses it covers, the good orders your client stops wrongly cancelling, and the staff hours nobody spends reviewing flags by hand anymore. We’ve watched a “expensive” vendor come out clearly positive once the recovered false-decline margin alone was tallied, because false declines are almost always a bigger silent loss than fraud itself.

Native analysis looks free, and the license is. The hidden cost is the human time to review flags well and the losses that slip through when nobody has time to review at all. That cost is low for a small store and balloons with volume, which is exactly why the native option stops penciling out at scale even though the price tag never changes.

Custom is the one whose true cost agencies routinely lowball. The build is the cheap part. The expensive part is the forever part: retraining the model as fraud patterns shift, monitoring it, handling its mistakes, and keeping a data pipeline alive. For a single store that math basically never closes. It only starts to work when you can amortize that ongoing cost across a whole portfolio of stores, or across a catalog so high-risk and high-volume that even a fractional improvement over a vendor pays a salary. Short of that, you’re building a science project your client will be maintaining long after the engagement ends.

An agency scorecard for picking per client

When we’re deciding for a specific store, we don’t argue philosophy. We score five things and let the pattern point at the answer.

Average order value and catalog risk come first, because they set the stakes of any single bad order. Then order volume, because that decides whether a human can realistically review flags or not. Then the client’s own operational maturity, meaning do they have anyone who will actually run a review SOP or maintain rules, because a tool nobody tends is worse than no tool. And finally their appetite for risk on their own books, because a founder who loses sleep over chargebacks values a guarantee far more than the spreadsheet says they should.

Score low across those and the answer is native plus a review habit, and you should talk them out of spending money. Score high on stakes and volume but low on internal time, and a managed vendor with a guarantee is almost always the move. Reserve custom for the rare client who’s both a portfolio operator and sitting on a catalog risky enough to justify owning the whole stack. Most stores, honestly, land in the first two buckets, and the agencies that admit that build more trust than the ones forever pitching the big build.

What we keep telling clients

The temptation, on the agency side and the merchant side both, is to equate spending more with being safer. Fraud marketing leans hard on fear, and fear sells the heaviest option in the room.

The better instinct is to match the tool to the actual risk and the actual team. A store that can run on native analysis and a ten-minute daily review habit should do exactly that, and pocket the money. A store getting hammered on high-value orders with nobody left to review them needs a guarantee far more than it needs a cleverer algorithm. And the store that genuinely warrants a custom model is rare enough that if you’re not sure, it isn’t that store.

We tell clients to buy the financial outcome they want, not the technology that sounds the most advanced. The question is never “what’s the best fraud AI.” It’s “what’s the smallest, most maintainable thing that moves this specific risk off our plate,” and the answer is usually less than they came in expecting to spend.

Devon didn’t sell that supplements client a custom model, and he talked them out of the most expensive vendor tier too. He scored the account, saw high AOV and a resellable catalog but a capable ops person and modest volume, and put them on a managed vendor’s mid plan with a chargeback guarantee. The three-grand months stopped, the false declines dropped, and the client kept the build budget for something that actually grows the business.

Questions we get every week

Is Shopify’s built-in fraud analysis good enough on its own? For a lot of stores, yes. If your average order value is modest, your catalog isn’t a fraud magnet, and someone actually reviews the flagged orders, the native rating handles most of the clumsy fraud at no extra cost. It stops being enough when high order values, risky products, or sheer volume mean nobody can keep up with the flags by hand.

What am I really paying for with a vendor like Signifyd or NoFraud? Mostly the chargeback guarantee, not just the detection. The model is often sharper than native on hard cases, but the thing that changes your client’s behavior is that the vendor reimburses approved orders that turn out fraudulent. That moves the financial risk off your client’s books, which is what they’re actually buying.

When does building a custom fraud model make sense? Rarely, and almost never for a single store. The build is cheap but the upkeep is brutal (retraining, monitoring, and owning every false decline forever), so it only pencils out when you can spread that ongoing cost across a portfolio of stores or a catalog risky and high-volume enough that a small edge over a vendor pays for itself.

Will a fraud app slow down my checkout? It can, depending on where it scores. Tools that block orders before they’re placed add latency at checkout, while most managed vendors score just after the order is placed to protect conversion and then cancel or hold the bad ones. Scope which model a vendor uses before you install it, because checkout speed is too valuable to trade away quietly.

If you’re weighing native, a vendor, or a custom build for a client and want a second read on the math, we’ll score the decision with you.