Shopify Sidekick vs Stats Compass: can Sidekick tell if a promotion made money?

Same question, same answer

Stats Compass doesn't generate its numbers: they come from fixed statistical methods, so the same question on the same data always gives the same answer, and you can see how each one was worked out. Every figure in the app opens "How this was calculated": its definition, the version of that definition, and how many orders it was computed from.

You can also use Stats Compass through an AI assistant. The assistant then writes the sentences and chooses what to ask for, but each figure Stats Compass's tools return is worked out the same way every time.

What we tested

We asked Sidekick and Stats Compass the same five questions on a Shopify development store with generated orders, so the right answers are known. It holds 4,181 orders from April 2025 to October 2026 and two discount promotions:

• SUMMER40: 40% off, 14 May to 3 June 2026, 310 orders. It brought more sales but lost money: the discount cost more margin than the extra sales earned.
• FLASH20: 20% off, 15 to 21 June 2026, 100 orders.

We asked Sidekick in the Shopify admin and only asked questions; we didn't let it take any action. The Stats Compass answers are from its analysis of the same store.

Question by question

Can Shopify Sidekick tell me whether a discount made money?

It can work out what a promotion's orders brought in after discounts and recorded costs, and it is careful to say what it doesn't know. Asked plainly, it didn't compare the promotion with what the store would have sold anyway, and it said so: "it doesn't tell us how many sales would have happened without the promotion." So it called both promotions "promising", though one lost money. When we asked for the comparison, it made a sensible one.

How does Stats Compass decide whether a discount made money?

It estimates what the store would have sold without the promotion, from the weeks before it and the same time of year, and measures the promotion against that. Each result has a range and a verdict: expected, real change, noise, or not enough data yet. It also works out how much more revenue the promotion needed to break even, the largest discount the extra sales could afford, and how many new customers it brought.

Do they agree?

When the question includes the comparison, closely. On SUMMER40's lost margin, Sidekick said £1,361 less against the 21 days before, and Stats Compass £1,377 less against a season-adjusted expectation. On break-even, both counted the discount once. The difference is in what each does when the comparison isn't in the question.

When should I use Sidekick, and when Stats Compass?

• Sidekick: General questions about your store and tasks in your Shopify admin.
• Stats Compass: whether a promotion made money compared with what would have happened anyway, whether this week's numbers are normal for the time of year, and a weekly verdict without having to ask.

They work side by side. Stats Compass only reads your store, so it doesn't replace Sidekick.

How fair was this test?

• Generated data. The orders were generated so the answers are known.
• One leading question. We added the comparison to the second question's wording. The third and fourth followed it in the same conversation, so Sidekick may have carried that comparison forward.
• Each question was asked once. Stats Compass's answers aren't generated: they come from fixed, versioned calculations, so the same question on the same data gives the same answer every time, and every figure can be traced to how it was worked out. Sidekick's answers are generated, so, like any generated answer, they can differ from one asking to the next.