Most online stores guess at their shipping rates. Should it be 39 kr or 19 kr? Should free shipping start at 399 kr or 499 kr? The discussion usually ends up being based on gut feelings, because no one can see what the other option would have yielded.
A split test of shipping rates replaces guesswork with an experiment. You choose one shipping method, set a new price, and let Shoporama show it to half of your visitors, while the other half sees your current price. Afterward, you’ll get the results in kroner.
What the feature does
A shipping test always has exactly two variants: “As is ” and “New shipping price.” Traffic is split 50/50 between them.
Each visitor is assigned one of the two variants and sees the same price throughout the entire process: on the product page, in the cart, at checkout, and if they return later. The assignment remains in effect for up to 30 days. The price of the order is exactly what the customer was shown. Even the free shipping bar in the cart in the Montana and Delaware themes—the one that says “Spend 120 kr more and get free shipping”—follows the variant’s threshold. If you’ve opted out of a shipping method that still uses the country’s lower threshold, the bar will display that threshold because it still applies to that method.
The new shipping price applies only to the shipping method you’ve selected. Your other shipping methods retain their prices, and subscription renewals and upsell orders always use the store’s standard prices.
If you’re testing the free shipping threshold, you choose which shipping methods it should apply to. If you raise the threshold for one method, the customer will simply switch to another method with the old threshold, and the test will mainly measure how many customers switch shipping methods. Therefore, all methods are selected by default, except for those with a higher threshold than the one you’re testing.
Four steps, and you’re all set
The setup consists of four steps: 1. Which shipping method do you want to test?, 2. Variant B, the new price, 3. Your budget, and 4. Duration. You don’t need to come up with a name along the way. The test is automatically named after the shipping method you’re testing—for example, PostNord—and you can rename it afterward under Settings on the test’s own page.
That’s why we measure contribution margin rather than conversion rate
This is the most important difference between a shipping test and a regular A/B test. Free shipping almost always wins in terms of conversion rate. It just almost always costs money because the store pays for shipping itself. If you measure revenue instead, you count the shipping costs you pay yourself as income.
That’s why a shipping test in Shoporama is determined by the contribution margin per visitor. We subtract your shipping expenses from your shipping revenue, add the contribution margin on the products, and divide by the number of visitors. The conversion rate and average order size are still displayed, but they do not determine the result.
Only paid orders are counted. Canceled and credited orders are subtracted, so you see what you’re actually left with.
Be honest about what your traffic can handle
Most testing tools promise you a definitive answer. This one doesn’t—and that’s by design.
As soon as you set up the test, the screen shows how many visitors you’ve had and what your conversion rate has been over the past 30 days, as well as how much of a difference your traffic can realistically make after 2, 4, and 8 weeks. If you have fewer than 15,000 visitors per month, it’s clear as day: shipping rates typically vary by 5 to 15%, so you should expect the test to end in a tie.
A tie is also a useful result. It means the difference is so small that you can confidently choose the solution that best suits your store, rather than spending more time on the question.
Once the test is complete, you won’t get a statistical formula, but a specific amount: “The maximum loss from choosing this option is 0.01 DKK per visitor.” This isn’t a guaranteed upper limit, but rather the average loss you can expect per visitor if you choose this option and it turns out not to be the best one. If the result isn’t certain yet, the report indicates how far along the test is, for example: “The test has run for 12 of at least 14 days. The smallest group has 63 orders (target: 100).”
A winner is declared only after the test has run for at least 14 days, has at least 100 orders in each variant, and the expected loss from choosing that variant is no more than 10 øre per visitor. The report also warns you against the classic pitfall: if you check the numbers every day and stop when things look good, you’ll find a winner almost every time—even when there isn’t one.
What You Can Test
- Increase or decrease the shipping price by a certain amount, for example, 20 kr less.
- Try a flat rate regardless of country and weight, for example, 0 kr.
- Change the price by a percentage, for example, 15% lower.
- Test your shipping price per country and your weight ranges all at once. The adjustment applies to all prices for the selected method.
- Adjust your free shipping threshold, or try eliminating free shipping entirely, for the shipping methods you select. You can also test the threshold alone and leave the price unchanged: enter 0 in the “Change in DKK” field.
While you’re setting up the test, a live preview shows what a typical cart costs in both scenarios. Simply enter a cart total and a weight, and you’ll immediately see “Denmark, as of today: 39.00 DKK” compared to “Option B: 19.00 DKK.” All prices are listed without tax, just like on the shipping rates page.
You decide what happens next
The checkbox “Stop automatically when the result is reliable enough” is enabled by default. This means the test stops on its own, either when the result is reliable enough or when the end date is reached. Prices are never changed when the test stops. You’ll receive a notification in the admin panel to open the test and view the result, and then you can decide for yourself. If you’d rather let the test run for the entire period no matter what, simply uncheck the box.
Shoporama never changes your prices on its own—not even when the test ends, nor when there’s a clear winner. Prices are only changed when you click “Activate Permanently.” The shipping price is changed only for the tested method, and a new free shipping threshold is applied to all the methods you’ve selected. Your old prices are saved, so a “Cancel” button can revert everything: the price per country, the weight ranges, the free shipping threshold for the selected methods, and the country-specific limits.
Once you’ve enabled the new prices, the test cannot be restarted or deleted. This is intentional: the test is the only place where your old prices are saved, so if it could be deleted, the undo option would disappear along with it.
You can stop a test at any time, and stopping a test never changes the prices. If the test is stopped without a definitive result, the “Choose Yourself” box will appear, so you can still activate the new price if it best suits your store.
What You Need to Know First
- If the shipping price is listed as free-text on your shipping page or in your terms of service, it must be corrected. Otherwise, the store will contradict itself while the test is running.
- Only one shipping test can run at a time. The shipping price is tested for a single shipping method, while the free shipping threshold may apply to multiple methods. You can prepare the next test in the meantime.
- Variant B uses the same free shipping threshold for all countries. After activation, the selected methods each have their own threshold, so a later change to the country’s threshold won’t affect them.
- While the test is running, the static page cache is disabled, so pages load slightly slower than usual. It will return to normal once the test is stopped.
- Search engines and bots always see your current prices and are not included in the statistics.
- The report will alert you if traffic is unevenly distributed between the two variants, so you don’t draw a conclusion based on skewed data.
- If the test has been running for a long time without reaching a conclusion, the report will state clearly that the difference is small enough for you to safely make the choice yourself.
- The pricing configuration is locked once the test has started: shipping method, variant B, free shipping threshold and the methods it applies to, cost prices, and coverage rate. This is because the numbers are specific to that particular configuration. You can still change the name, duration, and the “auto-stop” checkbox along the way, and a running test can be extended.
You must specify your cost per shipment and your average contribution margin on the items for the calculations to work out. If the free shipping threshold applies to multiple methods, you can specify the cost for each one. If the field is left blank, we’ll use the cost price for the selected method. If cost prices are missing for more than 20% of revenue, the report will indicate this and use the estimated gross margin for the remainder.
Get Started
You’ll find this feature under Delivery > Split Test Shipping Rates, and there’s a shortcut from the “Split Test Shipping Rates” button under Delivery > Shipping Methods. A test is always created as a draft, so you can set it up at your own pace and only press “Start” when you’re ready.
See the step-by-step guide to A/B testing shipping rates. If you need to set up the basics first, read about creating a new shipping option, weight-based shipping rates, shipping rates by country, and countries and currencies. If you want to see the results afterward, you’ll find them in the shipping statistics and in the shipping section of the checkout flow. If you have any questions, please email support@shoporama.dk.