How to create and run a split test in Shoporama
Step-by-step guide to create and run a split test (A/B test) in Shoporama. You will learn how to clone a page setup, stylesheet, or split test entire themes, create the test, read the statistics and let the system find the winner automatically.
Split testing allows you to compare two or more variants of the same page, stylesheet, or entire theme and see which one actually sells better. Here is the step-by-step guide to creating and running your first split test. Read the feature description if you want to understand how the system works behind the scenes.
Prerequisite: You must have two variants
A split test compares two or more variants of the same element. Before you can create the test, the variants must exist. There are three types:
- Page layouts - created in Page designer under Design → Page designer
- Stylesheets (CSS variants) - created in Design → Stylesheet
- Entire themes - exist as folders under the shop's private themes. Requires pro subscription and Smarty 4. You don't create the themes themselves via the Split test page - they must already exist in the shop (uploaded via FTP, copied from an existing theme, or built by a developer).
Step 1: Clone the page or stylesheet you want to test
Go to Design → Page designer (or Stylesheet if you want to test CSS). Find the page you want to test in the list and click the copy icon (the two sheets of paper) on the right side of the row.
You will now get a copy that starts offline with the name "(original name) (copy)". The clone is automatically given lower priority than the original, so if you put it online without testing, the original will still appear.
Step 2: Customize the copy
Click the edit (pencil) icon next to the copy to open it. Make the changes you want to test:
- Change the hero image
- Correct headline or text
- Add or remove sections
- Rearrange the order
- Fix CSS (if stylesheet variant)
Important: Only test ONE thing at a time. If you change five things and B wins, you won't know what made the difference. Start with just the header, for example, or just the button color.
Step 3: Put the copy online
When you're done with the changes, go back to the overview and click the toggle icon next to the copy to put it online. It will change color to green when it is active.
You now have two online versions of the same page. Without a split test, the one with the highest priority would automatically win and be shown to everyone. This is where the test comes in.
Theme test: slightly different flow
If you want to split test entire themes against each other, steps 1-3 don't apply. Instead, you need to have two private themes on the shop already - for example, MyTheme and MyTheme-v2. On the create page, select Theme in the dropdown and on the edit page, add the two (or more) theme folders you want to test. The rest of the flow (weight, stats, winner) is identical to the page layouts and stylesheets.
Theme option only appears if the shop is on the pro plan + running Smarty 4 + has at least 2 private themes. The pro requirement is because private themes are a pro feature in themselves. The Smarty 4 requirement is technical: only Smarty 4 can render two different themes side-by-side on the same shop without their template cache crashing.
When a theme test is selected as the winner, the winning theme is automatically set as the shop's new default theme (similar to how you selected it yourself under Design → Theme). The other theme folders are not touched - they are still there if you want to test them again later.
Step 4: Create the split test
Go to Design → Split test and click New split test.
- Give the test a name so you can find it again later. For example, "Home Hero A vs B" or "Button color red vs green".
- Select what you want to test in the dropdown. You only see the target types where there are at least 2 variants.
- Click Create split test. You are taken to the test edit page.
Step 5: Add variants to the test
On the test edit page, you will see a "Variants" section. Use the dropdown to select a variant and enter a weight (typically 50). Click Add. Repeat for the second variant.
The weight determines what percentage of visitors see each variant:
- 50/50 - recommended. Fastest way to a conclusion because data is distributed evenly.
- 90/10 - used if you are cautious and only want to expose a new variant to a small portion of traffic. Takes longer before there is enough data.
Step 6: Configure settings and start
On the right side of the edit page you can set:
- Start and end date - optional. Leave the end date blank to let the test run until you stop it yourself.
- Stop automatically when there is a clear winner - recommended. The test stops when there is ≥95% probability that a variant is best and the minimum requirements are met (5,000 impressions per variant, 100 orders in total, 14 days duration).
- Use the winner as the new default when the test stops - recommended. When the test ends with a valid winner, the winning variant is put online with high priority and the losers offline. You can always roll back manually.
Click Start test when you are ready. The button is disabled until you have at least 2 variants added.
Step 7: Wait and watch the statistics
While the test is running, you can return to the test edit page and view the statistics at any time. You will see:
- Views per variant
- Orders and revenue
- Conversion rate in percent
- Revenue per visitor (RPV) - the most important KPI in e-commerce
- Win probability as colored bar (grey=uncertain, blue=leading, green=winning)
The leading variant is highlighted with a green background and gets a "In front" badge. When a winner is declared with 95%+ probability and the minimum requirements are met, they get a trophy badge and a green alert at the top.
Be patient. The statistics are updated when visitors hit one of the variants. On a small shop, it can take 4-8 weeks before there is enough data for a conclusion.
Step 8: When the test stops
If you have enabled "Use winner as new default", it happens automatically: the winning variant is activated as the new default and the losers are taken offline. You will get a notice in admin telling you what happened.
If you do NOT have auto-apply enabled, you can click into the test, see which variant won and manually update your shop by going to Page Designer/Stylesheet and put the winning variant online (and possibly the losers offline).
The statistics are preserved no matter what. You can always go back and view historical tests and their results.
How to stop a test manually
If you want to stop a test early, you can do so in two ways:
- From the split test list: Click the toggle icon (green = running) to stop.
- From the test edit page: Click the Stop test button on the right side.
Collected data is retained as history. If auto-apply is enabled AND there is a valid winner with enough data, the winner is set by default. Otherwise, the test just stops without any changes.
How to restart a test
A stopped test can be restarted by clicking the toggle icon in the split test list or from the edit page. The probabilities will be reset and the test will collect new statistics from that moment.
How to delete a test
Click the trash can icon next to the test in the list. You will get a confirmation dialog. The statistics on tagged orders are kept as history on the orders themselves, but the test cannot be continued.
Time management without split testing
You don't need to run a split test to use the variant system. You can also just create timed pages:
- Clone an existing page (e.g. the front page) and customize it for the Black Friday look.
- Open the clone and go to Settings.
- Leave the Online checkbox turned off.
- Set Display from and including to the start date (e.g. 28-11-2026) and Display up to and including to the end date (e.g. 01-12-2026).
- Save.
The clone will automatically appear in the interval (from midnight start date to 23:59 end date) and disappear again when the period is over.
Cookie and GDPR
When a split test runs, Shoporama sets a functional cookie called pb_v on your visitors' browser. The cookie is HttpOnly, SameSite=Lax and expires after 30 days. It only contains a random visitor ID and which variant the user is viewing. It is not shared with third parties.
You should add the cookie to your cookie policy as a functional/preference cookie. If visitors reject cookies (via accept_cookies=reject), a session-based fallback is used.
Frequently asked questions
Can I change the weights while the test is running?
No, you can't. The weight fields are locked when the test is active. To change the weights, stop the test, adjust the weights and start it again. Note that changing weights in the middle of a test can contaminate the result.
Can I add a new variant to a running test?
No, you can't. You can only add variants when the test is stopped or in draft status. This is to ensure data quality - adding a new variant in the middle of a test would skew the sample size.
What happens if I delete a variant that is in the test?
The variant row is removed from the test automatically. If the test falls below 2 variants, it will stop completely. The statistics on already tagged orders are retained.
Where can I find historical tests?
All tests (drafts, stopped, completed) are shown in the list under Design → Split test. The Status column shows whether the test is running, a draft, stopped or completed with a winner.
Can I split test two different themes against each other?
Yes, you can. If you have at least 2 private themes on the shop + pro plan + Smarty 4, "Theme" will appear as a third target type in the dropdown when you create a new test. You select two (or more) theme folders and run the test like any other split test. When a winner is chosen, it automatically becomes the new default theme for the shop.
What do I do if the Theme option is not there?
Three reasons may prevent it from appearing: (1) the shop is not on pro subscription - upgrade under Account → Shops; (2) the shop is running Smarty 2 - switch to Smarty 4 under Design → Theme; (3) you have less than 2 private themes - upload or copy an extra theme first.
How do I know if a variant is online?
In the Page Designer list and Stylesheet list, the status column shows whether the variant is Active now, Standby, Offline, Scheduled from X or Expired. The toggle icon changes color (green = online, default = offline).
Can the AI Assistant create split tests for me?
No, it can't. The split test orchestration itself (create test, select variants, start/stop) is intentionally not automated via AI. These are choices you have to make yourself so that no one accidentally starts a test. On the other hand, the AI can create and schedule page setups via natural language, e.g. "Create a Black Friday front page that automatically goes online on November 28 and shuts down on December 1".
Will I get a warning if the test fails to pick a winner?
Yes, you will. If the traffic is clearly skewed (e.g. 70/30 instead of 50/50), you will get a warning in admin. This is typically due to a cache configuration or bot pattern. We send a maximum of one warning per week per test to avoid spamming you.
What happens to my eCommerce cache?
The cache is automatically disabled for pages that are part of an active test. This ensures that each user gets their correct variant. When the test stops, the cache is reactivated for new visitors. However, users who have participated in the test will have an uncached experience for up to 30 days (lifetime of the cookie), so we don't serve them a cached version from another variant.
Where can I find more info on how it works?
Read the feature description of split testing to understand the statistics, GDPR handling and how we measure the winner behind the scenes.