If you run A/B tests on your store, the most important decision isn’t what you test. It’s what number decides the winner. Pick the wrong one and you can roll out a string of winners that leave you with less money.
We judge every test on one number: revenue per session, or RPS. Here’s what it is and why we use it.
What revenue per session is
Revenue per session is the revenue a version of a page brings in, divided by the number of sessions that saw it. That’s all it is. If one version brought in more money for every visitor who saw it, it’s the better version.
Some tools call it revenue per visitor. The idea is the same: money in, divided by people through the door.
Why conversion rate isn’t enough
Conversion rate only counts how many sessions ended in an order. It doesn’t care how big the order was. So any test that pushes people towards smaller orders can look like a win.
Say a test swaps your bundle offer for a cheaper single item up front. More people buy, so conversion rate goes up. But they each spend less, and the total can go down. On conversion rate alone, you’d ship it.
It works the other way too. A test that gets people to build a bigger bundle might convert slightly fewer sessions and still bring in more money. Judge it on conversion rate and you’d throw away a winner.
Why average order value isn’t enough either
The opposite mistake is to chase order value on its own. You can push average order value up by making it hard to buy just one thing, and lose the people who only wanted one. Order value looks great. Revenue doesn’t.
Revenue per session counts both at once: how many people buy, and how much they spend. That’s why it’s the truest single measure of a test.
What it looks like on real tests
Every result we quote is in revenue per session. Charge’s dynamic subscription cart lifted it by 12%. Bunker’s dynamic cart upsells lifted it by 11%. N-Pro’s animated mechanism section lifted it by 12%, and Sassy Saints’ offer builder by 7%.
Carts and offer builders are exactly the kind of test where conversion rate on its own would mislead you, because they change what people buy as well as whether they buy. Revenue per session catches both.
A few things to watch
- Give tests enough traffic. Revenue per session moves around more than conversion rate, because a few big orders can swing it. The less traffic a page gets, the longer a test needs.
- Look at the parts too. When a test wins on revenue per session, check whether it came from more orders, bigger orders or both. It tells you what to test next.
- Keep the brand in view. A number can’t tell you if a design cheapens the brand. That’s why we score every design for brand before it’s built, not after.
If you’d like tests run this way on your store, see our Shopify A/B testing service.
