Reading Shopify's fraud analysis without over-reacting

What the tool is doing

Shopify's fraud analysis examines each order against a range of signals and produces a risk level: low, medium or high. Behind that label sits a set of individual indicators, and those indicators are far more useful than the label they roll up into.

The system is pattern matching. It is asking how closely this order resembles orders that turned out to be fraudulent. That is a genuinely useful question, but it produces a probability rather than a finding, and treating a probability as a verdict is where stores lose good customers.

This post is about reading the indicators. For the procedure once an order is flagged, see what to do in the first ten minutes.

The three families of signal

Payment signals: weight these heavily

Whether the billing address matched what the card issuer holds, whether the card security code was correct, and whether several cards were attempted on one order or account.

These are the strongest indicators available, because they are hard for a fraudster to satisfy and easy for a legitimate customer to satisfy. Someone using their own card generally knows their own billing address.

The caveat: address verification is not supported uniformly across every country and card type, so a non-match sometimes reflects the check being unavailable rather than a genuine mismatch. Worth knowing before you treat every failed check as damning, particularly on international orders.

Location signals: weight these lightly

Distance between billing and shipping address, the country the IP resolves to, and whether the connection came through an anonymous proxy or VPN.

This family generates most of the false positives, for reasons that have nothing to do with fraud:

• Gifts. Shipping to a different address is the entire point of buying someone a present.
• VPNs. Increasingly ordinary. Plenty of people leave one on permanently, and some corporate laptops route everything through one by default.
• Travel and expatriates. A card issued in one country, a person living in another.
• Corporate networks. Traffic that egresses somewhere unrelated to the person sitting at the desk.

A location signal on its own is weak evidence. A store cancelling orders on distance or VPN alone is losing legitimate revenue at a rate it cannot see, because declined good customers do not write in to complain, they just do not come back.

Behaviour signals: weak alone, meaningful together

Order velocity, repeated failed attempts, a new account placing an unusually large first order.

Each has an innocent explanation. Someone whose first payment failed will retry. A customer who found you through a viral post is new by definition. What matters is accumulation: several behaviour signals alongside a payment signal is a genuinely different situation from any one of them alone.

The rule that prevents most bad decisions

Count the families, not the indicators.

Four location signals is one family and still weak evidence. One payment signal plus one behaviour signal is two families and deserves attention. The overall risk level does not distinguish between these, which is exactly why opening the detail is worth the thirty seconds.

Add customer history as a fourth input that can override the rest. A returning customer with delivered orders and no chargebacks is strong evidence, and it is evidence the fraud analysis weights differently from how you should.

The risk level tells you to look. The indicators tell you what to do. Acting on the first without reading the second is how stores cancel on their own customers.

The new false positive

An emerging pattern worth naming, though it is early enough that you should treat it as a hypothesis to check rather than an established fact.

Automated shopping agents, which place orders on a person's behalf, produce a signal profile that looks remarkably like fraud: a brand-new account, an order completed within seconds, and a connection from a data centre rather than a residential address. That is close to a textbook fraud fingerprint, and it is generated by an entirely legitimate purchase.

If you notice high-risk flags on orders that otherwise look completely ordinary, where the basket is sensible and the customer responds normally to a verification message, this is a plausible explanation. Lean harder on payment signals and customer history, and lean less on how the connection arrived.

What the analysis does not see

Three things you know and it does not, which is why the tool is an input rather than a decision maker.

Your products. Resale value drives fraud. If your catalogue is low-value or hard to resell, your genuine fraud rate is lower than a generic model assumes, and you can afford to be more relaxed.

Your customers. If you sell to a specific community, an order that looks statistically odd may be entirely normal for you. A store selling to expatriates will see billing and shipping mismatches constantly.

The conversation. If the customer emailed beforehand asking a question about sizing, that context exists in your inbox and nowhere in the fraud analysis. It is often the single most exonerating piece of evidence available.

Calibrating over time

Keep a short record of what you cancelled and what you shipped, and revisit it a few months later. Two questions:

Did anything you shipped result in a chargeback? If none did, you are almost certainly cancelling too aggressively and can loosen.

Did anyone you cancelled get in touch, confused? Each of those is a customer you rejected. Most will not contact you, so treat every one you hear from as representing several you did not.

Most stores that do this exercise discover they are far more cautious than their actual fraud exposure warrants, because the cost of a chargeback is visible and itemised while the cost of a rejected customer is invisible.

When to add something on top

The built-in analysis is enough for most stores, and adding tooling before you have a problem is a cost without a benefit. Three situations genuinely change that.

High-value or highly resellable products. If a single fraudulent order is painful rather than annoying, the economics of extra verification shift considerably.

Volume that makes manual review impossible. Reading indicators on ten flagged orders a week is fine. On a hundred a day it is not, and the failure mode becomes rubber-stamping, which is worse than no review.

A chargeback rate that is actually hurting you. Note the word actually. Check the number before acting on the feeling, because the memory of one bad chargeback distorts risk perception for months.

Absent those, the better investment is usually a clearer internal procedure rather than another tool. Most stores lose more to inconsistent handling than to fraud itself.

What to record on the order

Whatever you decide, note which indicators fired and what you concluded. Not for compliance, for calibration.

Three months of notes lets you answer the only question that matters here: were we right? Look at what you shipped and check for chargebacks, then look at what you cancelled and ask whether the reasoning holds up in hindsight. Without notes, that review is impossible and you will keep whatever caution level you started with, correct or not.

Talking to your team about it

Fraud decisions get made by whoever is on shift, and their instincts are usually more cautious than yours, for an understandable reason: the downside of shipping to a fraudster is vivid and personal, while the downside of cancelling on a real customer is invisible and lands on somebody else.

Left alone, that asymmetry produces steady over-cancellation. Nobody is ever criticised for the order they refused.

Two things counteract it. Say explicitly that cancelling a good customer is a real cost, and that you would rather occasionally lose an order to fraud than routinely reject genuine buyers. People need to hear that from the person whose money it is, because they will not assume it.

Then make the decision reviewable rather than final. If someone is unsure, the answer should be to hold and ask, not to cancel to be safe. Holding preserves both options; cancelling destroys one of them and does it in a way that is hard to reverse gracefully.

Common questions