A/B Testing of Shipping Rates: Find the Shipping Rate That Generates the Most Revenue

Most online stores set the shipping rate once, when the store was first set up, and then it stays that way: 39 kroner. Or free shipping on orders over 499. The amount is usually an educated guess, perhaps influenced by a look at what the competition is doing, and afterward, no one ever measures whether it was actually the right choice.

That’s a shame, because the shipping price is one of the few figures in an online store that affects everything at once. It influences how many people complete their purchase, how large an order they place, and how much you yourself have to pay to ship the package. A wrongly set amount costs money every single day, without ever showing up as an entry in the books.

With A/B testing of shipping rates, you can replace guesswork with experimentation. You show two different shipping rates to two separate groups of your visitors and let the numbers determine which one generates the most revenue. This guide shows you how to set up a test, how to interpret the results, and what you should test first.

What a shipping rate split test does

A split test, also known as an A/B test, divides your visitors into two equal groups. One half sees the store’s current shipping rates. The other half sees the new price you want to test. Everything else in the store is exactly the same for both groups, so any difference in the results can only be due to the shipping price.

Once the test has run for a while, Shoporama compares the two groups and tells you which one generated the most revenue per visitor. Not which one sold the most, nor which one had the best conversion rate, but which one generated the highest contribution margin—that is, what’s left after the cost of goods and your shipping expenses have been deducted. This is an important distinction, and we’ll come back to why later on.

Here’s how to set up a test

Go to Delivery > A/B Testing of Shipping Rates, and click New Shipping Test (Create your first shipping test if you don’t have one yet). You must have at least one shipping method in your store before you can get started. If you don’t have any, the page will prompt you to create a shipping method first. The screen has four steps:

1. Which shipping method do you want to test? The new shipping price applies only to the selected method. All your other shipping methods will retain their prices while the test is running.
2. Variant B: the new price. You have three options: increase or decrease the price by a specific amount (for example, minus 20 DKK), set a flat rate regardless of country and weight (for example, 0 DKK), or change the price by a percentage (for example, minus 15%). Here, you also decide whether to include the free shipping threshold: you can keep the threshold as it is today, test a different threshold, or omit free shipping entirely in Variant B. If you’re testing a different threshold or omitting free shipping, you’ll also select which shipping methods it should apply to in the “Threshold applies to” list. This is explained further down. You can also simply test the threshold and leave the shipping price itself unchanged. The “Change in kroner” field is set to -20 by default, so enter 0 there. Finally , the “Ignore shipping prices” checkbox is located here for each individual product. It is disabled by default, and that’s the right choice for most people. Only check this box if the products’ own shipping prices would otherwise override what you want to test.
3. Your finances. Enter your cost per shipment—that is, what you personally pay the shipping company to send a package, excluding VAT. Shoporama remembers this amount for the shipping method and automatically suggests it next time. Then enter your average gross margin on items. The gross margin is the portion of the selling price that remains after the item’s cost price has been deducted. If you sell an item for 100 kroner that you paid 60 kroner for, the gross margin is 40%. The field is set to 40% by default, and the number is saved in the store, so you only need to enter it once. If the free shipping threshold applies to multiple methods, there is also a field per method under “Cost Price” for the other methods to which the threshold applies. If the field is left blank, we’ll use the cost price for the selected shipping method; if you fill it in, the amount will be saved for that shipping method.
4. Duration. You can choose 2, 4, 6, or 8 weeks. The test also has a checkbox, “Stop automatically when the result is reliable enough,” which is enabled by default.

There is no name field on the creation screen. The test is automatically named after the shipping method you’ve selected. If you want to name it something else, you can edit the name later under “Settings” on the test’s own page.

All amounts are listed excluding sales tax, just as on the page with shipping rates by country. The adjustment applies to all prices for the selected method—that is, all countries and all weight ranges. This is worth keeping an eye on if your heavy packages are already close to your own cost price.

Live preview: see the price before your customers do

As you set up the test, the screen instantly calculates what a typical cart will cost in both variants. You enter a cart total and a weight, and then it shows, for example, Denmark at 39.00 DKK today compared to Variant B at 19.00 DKK. It’s an inexpensive way to catch a typo before half your customers see it.

Honest preliminary calculation: Can your traffic even show a difference?

During setup, the screen shows what your own traffic can realistically demonstrate. For example, it states that over the past 30 days you’ve had 19,280 visitors and a 1.89% conversion rate, and below that is a small table showing the earliest you can expect to see a difference after 2, 4, and 8 weeks, respectively. In the example, the figures are 43%, 30%, and 21%.

These figures should be interpreted as an indication of how sensitive the results might be given your current traffic levels. Shipping rates typically affect conversion rates by around 5 to 15%. If you have fewer than approximately 15,000 visitors per month, you should therefore expect the test to end in a tie. This isn’t a waste of time, because a tie is also an answer: the difference between the two prices is so small that you can confidently choose the solution that best suits your store.

Which shipping methods should the free shipping threshold apply to?

The shipping price is tested only for the method you selected in Step 1. With the free shipping threshold, it’s different, because customers can work around it. If you raise the threshold for one method, customers will simply switch to another method with the old threshold and still get free shipping. So the test mainly measures how many customers switch shipping methods, rather than whether the new threshold generates the most revenue.

That’s why the screen displays the “Threshold Applies To” list when you select “Test a Different Threshold” or “No Free Shipping” in Variant B. Next to each method is its current threshold, and you check the boxes for all the methods the customer could otherwise switch to. If you select “Leave it as it is today,” the list won’t appear, and the test will only affect the price.

• The method from Step 1 is always included and cannot be deselected.
• Methods with a checkmark next to “Never Free Shipping ” cannot be selected and are labeled “Never Free Shipping.” They will never qualify for free shipping, regardless of the test.
• In a new test, all shipping methods are selected by default, except for those with their own threshold that is higher than the one you enter. Otherwise, a carrier’s shipping method offering free shipping only above 5,000 kr. would become free much earlier than it does today. The default selection is reset when you manually adjust the list.
• If you deselect a method with a lower threshold than the one you’re testing, or a method with a threshold when you’re testing “no free shipping,” the screen will warn you. For example, it might say: “GLS Parcel Shop still offers free shipping on orders over 348.00 DKK.” Customers in variant B can choose that option instead, and the test will then primarily measure how many people switch shipping methods.

Afterward, you can view your selection on the test’s own page, where the methods are listed under “Free Shipping Threshold Applies.”

Start the test and keep an eye on it in the overview

When you click “Create Shipping Test,” the test is created as a draft. It does nothing until you click “Start” yourself. After creation, you’ll be taken to the test’s own page, and the “Start” button there launches the test immediately. If you click the “Start” icon in the overview instead, you’ll be asked one more time: “Start the test?” Half of your visitors will see the new shipping price from now on.

The overview shows your tests with their status (Draft, Running, or Completed), which shipping method the test applies to, when it started, and a series of action icons: Start or Stop, View Results (which only appears once the test has started), Settings, and Delete. If a test is already running in the store, the start button on your other drafts will be grayed out. This means you can still prepare the next test in the meantime; you just can’t run two at the same time.

The checkbox “Stop automatically when the result is reliable enough” is enabled by default. This means the test will stop on its own when a reliable result is obtained or when the end date is reached. Prices do not change when the test stops. You’ll receive a notification in the admin panel to open the test and make a decision yourself.

You can stop the test at any time. Stopping the test never changes your prices.

What the Customer Experiences

• Each visitor is assigned a single variant and sees the same price throughout the entire process: on the product page, in the cart, at checkout, and on return visits. The assignment typically lasts up to 30 days.
• The price of the order is exactly what the customer saw. There are no surprises in the final step.
• The free shipping bar in the cart in the Montana and Delaware themes—the one that says “Spend 120 kr. more and get free shipping”—follows the variant’s threshold. If the threshold is 599 kr. for all shipping methods, and there are items totaling 400 kr. in the cart, all shipping methods will incur a fee, and the banner will indicate how much more is needed. Above 599 kr., all shipping methods are free. Customers in variant A will see no change.
• The same applies to the free shipping text [FREESHIPPING], which you can use on pages, categories, products, and blog posts (see dynamic tags). If you’re not offering free shipping on any shipping methods, everyone pays for shipping in variant B, and the text is blank.
• A shipping method you’ve deselected from the list may still be free at its old threshold in variant B. If it uses the country’s threshold of, for example, 348 kr., the free shipping text and the bar will show 348 kr. instead of 599 kr., because that threshold still applies to that shipping method. A deselected shipping method with its own lower threshold is not included in the text, just as it would be without a test. Once the customer has selected a shipping method, the bar displays the threshold for that specific method.
• Subscription renewals and upsell orders always use the store’s normal prices, never the test’s.
• Search engines and bots are not assigned a variant. They always see your current prices, and they are not included in the statistics.
• While the test is running, the store’s static page cache is disabled, so pages load slightly slower than usual. This returns to normal once the test is stopped.

How to Read the Results

The results page shows a conclusion at the top and a table with the numbers below. The table has one row per variant and shows visitors, orders, conversion rate, average order size, shipping revenue, shipping costs, product contribution margin, contribution margin per 100 visitors, and the probability that this particular variant is the best. The leading variant is marked with a “Leader” badge, even before the test is finalized. The badge indicates which variant is currently in the lead; it is not a recommendation to stop the test.

The example in the image is a test of free shipping on orders over 599 kr. via Postnord Parcel Shop. The test concluded after 27 days, and the report recommends switching to the new shipping rate. The smallest group has 240 orders, and the maximum loss from selecting it is 0.01 DKK per visitor. The numbers illustrate the whole point of the feature:

• The “As Is” variant had a 2.88% conversion rate, an average order value of 228.33 DKK, and a contribution margin of 194.84 DKK per 100 visitors.
• The “New Shipping Price” variant had a 2.87% conversion rate—essentially the same—but an average order value of 242.03 DKK and a contribution margin of 228.31 DKK per 100 visitors.
• The probability of being the best is 8% for the current price and 92% for the new one.

So the new price sold to just as many customers and still made more money, primarily because the orders were larger. If you had only looked at the conversion rate, you would have thought the change was insignificant and left the money where it was. That’s exactly the trap this feature is designed to help you avoid.

Below the numbers is the “Orders by Shipping Method” table. It shows how many orders each shipping method received in each variant and what percentage of that variant’s total orders that represents. If customers switch shipping methods between variants, that’s part of the explanation for the result. In the example, the distribution shifts by only a few percentage points: Postnord Delivery accounts for 64% of orders in “As Is” and 62% in “New Shipping Price,” while Postnord Package Shop accounts for 7% and 10%.

If the test changes the free shipping threshold, the shipping cost is calculated using each method’s own cost price. If the cost price for a method is missing, the cost price of the selected method is used. If you’re only testing the shipping price, all orders are calculated using the cost price of the selected method.

That’s why we measure contribution margin rather than conversion rate

Contribution margin is what remains from a sale after the cost of goods and shipping costs have been deducted. It’s the money used to pay salaries, rent, marketing, and ultimately generate a profit.

The reason this test is based on contribution margin per visitor is simple: free shipping almost always wins in terms of conversion but almost always loses money. If you lower the shipping cost, more people will buy. That’s no surprise. The question is whether those extra orders cover the shipping costs you’re now left to pay yourself, and the answer lies solely in the contribution margin.

If you measure revenue instead, you’re fooling yourself in a different way: you’re counting the shipping fee the customer pays as revenue, even though the money goes directly to the shipping company. Conversion rate and average order size are still shown in the table because they’re interesting to look at, but they don’t determine the outcome of the test.

When is the test considered complete, and when is it not?

The feature never promises you a statistical certainty. Instead, it provides a result in Danish kroner, with a statement like this: “The maximum loss from choosing this option is 0.01 DKK per visitor.” Here’s how to interpret that number: if you choose this option and it turns out not to be the best one, it will cost you about 1 øre per visitor. This is the average expected loss from making the wrong choice—not a guaranteed upper limit—and it’s a figure you can multiply by your own traffic.

A winner is not declared until all three requirements are met:

1. The test has run for at least 14 days—that is, two full weeks—so both weekdays and weekends are included.
2. There are at least 100 orders in each variant.
3. The expected loss from making the wrong choice is no more than 10 øre per visitor.

If the test is not yet concluded, you’ll be informed of its progress, for example: The test has run for 12 of at least 14 days. The smallest group has 63 orders (target: 100). Then you’ll know whether you should just wait or if the traffic isn’t sufficient.

If the test has run for more than 56 days without reaching a definitive conclusion, the report states it plainly: with the current traffic, the test is unlikely to reach a definitive conclusion, and the difference between the two prices is therefore small enough that you can safely choose the option that best suits your store.

The report also warns you if traffic is unevenly distributed between the two variants. If that happens, the numbers can’t be used as is, and the explanation is usually caching or bot traffic. Email support@shoporama.dk if you see that warning.

Activate a winner or cancel

The system never changes your prices on its own—not even when the test ends automatically, nor when a clear winner has been identified. Prices will only change the moment you click “Activate Permanently” on the results page.

If the test ended without a clear result, you can still choose the new price. The “Choose Yourself” box will then appear on the results page with an “Activate” button, so you can make a decision based on the information you have now—for example, because the new price is easier to communicate. The box only appears in this specific situation—that is, when the test has ended without a definitive result.

When you activate it, the shipping price changes only for the method you tested. If you tested a different free shipping threshold, it will be applied to all the methods you selected under “Threshold Applies To.” If you tested “No free shipping,” the selected shipping methods will instead be checked under “Never free shipping.” This also disables free shipping from discount codes and the loyalty program for those methods. During the test itself, these features still work, so in this case, activation has a slightly broader effect than what the test measured.

The country-specific threshold is only adjusted in countries where all shipping methods that use that threshold are selected. This ensures that the free shipping text in the store remains accurate afterward. If you’ve tested “No Free Shipping,” the country’s threshold is removed in those same countries. If you’ve deselected a shipping method that still uses the country’s threshold, the threshold remains in place.

The old prices are saved when you activate the settings. There is therefore a “Cancel” button that reverts everything: the price per country, the prices for the weight ranges, the free shipping threshold for all selected shipping methods, and the country-specific thresholds.

Once you’ve activated the prices, the test cannot be restarted or deleted. It cannot be restarted because the adjustments would then be applied on top of the new prices. And it cannot be deleted because the test is the only place where the old prices are saved, so the undo option would disappear along with it. Shoporama explains both of these points when you encounter this situation: the “Start” button is grayed out with an explanation, and any attempt to delete the test is rejected with a message explaining why.

What can be changed along the way?

As long as the test has never run, you can change everything in it. Once it’s started, the pricing setup is locked—that is, the shipping method, the new price, the free shipping threshold, the list of methods to which the threshold applies, the cost prices, and the coverage rate. The collected data is specific to that particular setup, so it must not be changed in the middle of the test.

You can still change the name, duration, and the “auto-stop” checkbox while the test is running. If you want to give the test more time, you can extend it by 2, 4, 6, or 8 weeks from now.

What You Should Test First

Start with what makes the biggest difference, and stick to one thing at a time:

• The free shipping threshold. This is usually the most insightful test. For example, try changing the free shipping threshold from 499 to 399 kr., or the other way around. You can test the threshold on its own and leave the shipping price as is: enter 0 in the “Change in DKK” field. Apply the threshold to all the shipping methods the customer could switch to. If you raise it for only one method, the customer will simply switch to another one with the old threshold, and then the test will mainly measure how many people switch shipping methods.
• Your most commonly used shipping method. Choose the method that most customers actually select—typically a package pickup location. That way, you’ll accumulate orders quickly enough to get an answer. If you’re unsure which one it is, you can check the shipping statistics.
• A moderate change rather than an extreme one. A reduction of 20 kr. or 15% is a good start. Free shipping for everyone is rarely the first thing you should try, because it often costs more than it’s worth.
• A price increase. This is the test most people skip, and it’s often the most profitable. Many online stores can raise shipping by a few kroner without anyone even noticing.

If you have multiple stores, test the largest one first. If the smaller store is roughly comparable in terms of product selection and customer type, it’s more worthwhile to apply the results from there than to spend two months on a test that will almost certainly end in a draw.

Limitations and things to know before you start

• Only one shipping test can run at a time per store. You can, however, prepare the next one as a draft in the meantime.
• The shipping price is tested for a single shipping method. Your other shipping methods retain their prices. The free shipping threshold can apply to multiple methods if you choose to do so.
• The free shipping threshold in Variant B is the same in all countries. The test does not have country-specific thresholds, and a country without a free shipping threshold will also be assigned the test’s threshold in Variant B. If your countries have different thresholds, the setup will warn you: “Your countries have different free shipping thresholds.” Variant B uses the same threshold in all countries, so some countries will have a different threshold than they do today.
• After activation, the selected shipping methods will each have their own threshold. If you later change a country’s threshold under Shipping > Countries, the change will not affect these shipping methods.
• A test always has two variants, “As Today” and “New Shipping Price,” with half of the visitors assigned to each.
• The price adjustment affects all prices for the selected shipping method—that is, all countries and all weight ranges. See also Countries and Currencies if you sell to multiple countries.
• All amounts are listed excluding sales tax.
• If you delete the tested shipping method in the middle of a test, the prices revert to the store’s normal rates, and the overview shows the shipping method as deleted. The test cannot be restarted.
• The cost prices per shipment are locked in the moment the test begins. If you adjust them later, it will not change the figures in a test that has already run.

Frequently Asked Questions

Where can I find this feature, and is it the same as the split test under Design?

No, these are two different features. The split test under Design tests appearance and layout. The shipping test is located under Delivery > Split Test Shipping Rates, and there is also a button labeled “Split Test Shipping Rates” on the Delivery > Shipping Methods page. If you want to test a price, use the one under Delivery.

I already have a shipping test running. What happens to it?

It will continue to run exactly as before. A test set up before the “Threshold Applies To” list was introduced will only have the threshold applied to the selected shipping method, and the shipping cost will be calculated using a single cost figure. You cannot add methods once the test has started, so if you want to test the limit on multiple methods, you’ll need to create a new one. You can still open and edit a draft that has never been started. The “Orders by Shipping Method” table is also displayed for old tests, so you can see if customers have switched methods.

What if a customer browses on their phone and then makes a purchase on a computer?

Attribution follows the browser the customer uses, so if she switches devices, she could, in principle, end up in the other variant. This is quite rare in practice, but it can happen. If that happens and the customer asks, give her the lower of the two prices. It costs just a few kroner and is far cheaper than having a dissatisfied customer.

Does the price adjustment also apply to my heavy weight ranges?

Yes. If you choose a reduction of 20 kroner, that 20 kroner will be deducted from all prices for that shipping method, including heavy packages. If you’re already close to your own cost price for heavy shipments, use a percentage change instead, or test a shipping method that’s only used for light packages. You can’t select individual weight ranges within the test.

What happens to products that have their own shipping price?

They retain their own shipping price as long as you leave the “Ignore shipping prices set on individual products” checkbox unchecked, which is the default setting. If you have pallet-sized items or large, bulky items with a fixed shipping rate listed on the product itself, you should not check that box. It’s only intended for stores where the products’ shipping rates would otherwise override what you want to measure.

I have a shipping method that’s free only for orders over 5,000 kr. What should I do with it?

Leave it unchecked in the list, and it will retain its own threshold in both variants. If you’re testing a lower threshold, it’s unchecked by default. If you’re testing no free shipping, all methods are checked by default, so you’ll need to uncheck it yourself. The screen will then display a warning about that method. If the method should never be free, check the “Never free shipping” box for that method under Delivery > Shipping Methods. Then it won’t be available as an option at all.

Should the cost per shipment be specified with or without sales tax?

Excluding VAT, just like all other amounts in the test and on the shipping rates page. Take the amount from your shipping invoice before tax. The report’s shipping revenue and shipping expenses are calculated in the same unit—that is, excluding tax—so the figures can be compared with your pre-tax accounting records. If you have any questions when reconciling your records, please contact us.

What cost price should I use when my shipping agreement varies?

Use the average amount you actually pay per shipment for each shipping method, including any surcharges that are typically applied. Since both scenarios use the same cost prices, a small inaccuracy rarely affects which option comes out on top, but it does shift the figures by a few kroner. It’s better to take ten minutes to calculate the average based on a full month’s invoices than to guess.

My cost prices aren’t up to date for all items, and I have quite a few returns. Can I trust the result?

Returns are taken into account as long as you credit them in Shoporama, since credited and canceled orders are deducted. Note that an order with a credit memo is completely excluded from the figures, even if you’ve only credited a single item on it. If you have many partial returns, fewer orders will be included in the statistics. If the cost of goods sold is missing for more than 20% of your revenue, you’ll receive a warning in the report, and your estimated gross margin will be applied to the remainder. It’s a good idea to have the cost of goods sold for your best-selling items set up before you start.

Will the results be adjusted retroactively if I correct cost prices or issue a credit memo for an order?

Yes, for the items. The report calculates using today’s cost prices and the orders currently marked as paid, so both a price increase from your supplier and a credit issued after the test ends can shift the numbers slightly. Cost prices per shipment, however, are fixed as of the day the test was started. Make a note of or save the figures on the day you make the decision.

I’ve activated the test and want to adjust the free shipping threshold again. Where do I do that?

The applicable threshold is listed under the shipping method. Go to Delivery > Shipping Methods, open the method, and edit the Free Shipping field. If you clear the field, the method will revert to the country’s default threshold. Also adjust the threshold under Delivery > Countries, as the free shipping text and bar display the country’s threshold until the customer has selected a shipping method. If you’ve enabled “No Free Shipping,” you must uncheck “Never Free Shipping” and enter a limit in “Free Shipping Over,” as the country’s limit may also have been removed. If you want to revert everything to how it was before activation, clicking “Cancel” on the results page is the easiest option.

Could the test cause problems with my Google Shopping feed?

Bots and crawlers aren’t assigned a variant, so Google always sees your current prices—the same ones listed in the feed. If you want to be completely safe, test a lower shipping price rather than a higher one. In that case, half of your customers will experience lower shipping costs than expected, and that’s the right way to err on the safe side.

Will the store run slower while the test is running?

A little. The static page cache is disabled while the test is running because the pages need to show each visitor the correct price. As a result, the pages are rebuilt for each visit instead of being served pre-rendered. Things will return to normal as soon as the test is stopped. If page speed is a major concern for you, choose a shorter test period or run the test outside of your busiest season.

Good luck!

If you haven’t set up your shipping methods yet, start by creating a shipping option and setting shipping rates by country. If shipping costs should depend on the package’s weight, you can read about weight-based rates, and if you want to see which delivery options your customers are already choosing, check the shipping statistics.

If you have any questions or would like someone to review your setup before you hit “Start,” please email us at support@shoporama.dk. We’re happy to help.