Split Tests

Split testing is a method for doing controlled experiments with for instance content or layout, to see which versions performs the best. In other words, it is a way to test ideas and optimize value generation, without taking unnecessary risks.

With split test, you can:

• Test different versions of your content and design
• Improve conversion rates by testing different calls to action
• Test on 10 different conversion metrics
• Compare conversions of original and variation
• Pick the winner manually – or based on statistical significance.

In short, split tests are a useful way for you to improve your conversion rate with a minimum of risk.

You can conduct split tests on newsletters and marketing emails as well - read more about how to do that under the documentation for Email Marketing.

With split tests, you create variation on a content or layout level, direct a portion of your visitors through the variation, and measure the conversion achieved for each version over time.

Before setting up a split test, you must:

• Identify an area of your website where a split test could be useful
• Define a specific conversion goal (such as ‘more visitors must place an order’ or ‘more customers should sign up for the newsletter’)
• Create a variant of your original content containing a change in content or layout.

Split tests are set up on a page level:

• Navigate to the page where you want to do a split test.
• Click the Marketing tab 
• Click Setup split test (Figure 3.1)

Figure 3.1 The Marketing tab

You will then be presented with the split test wizard (Figure 3.2)

Figure 3.2 The split test wizard

Following the wizard will take you through the basic setup of a split test .

Here's a rundown of your options, and what they mean.

You can choose between content-based and a page-based split test.

A content based split test:

• Tests paragraphs against each other
• Is useful for testing variation on design elements, such as buttons
• Is useful for testing variation on text and images

A page based split test:

• Tests pages against each other
• Is useful for testing different layouts

In summary, choose content based split testing if you want to test content like buttons, images or texts, and page based split testing if you want to test a new layout or design.

To be effective, a split test needs a clearly defined goal.

The conversion goals available to you are:

You should design your split test variant around a single goal, to minimize variables and get better result. For the same reason, you can only choose one type of conversion goal for each split test.

In order to use the 'Highest average order markup'-goal, you must enable "Product Cost Field".

To do so:

• Go to Settings > Ecommerce > Fields
• Enable "Product Cost Field" for Across all languages and Across all variants
• Save

Don't forget to set the Price and Cost field of your products to the correct values.

After choosing your conversion goal, you will be presented with the settings window (Figure 6.1).

Figure 6.1 Specifying the settings for you split test

From here you must:

• Name the split test
• Decide if the split test is active immediately or not
• Choose between registering conversion metrics only on the next step, or through the entire visit
• Specify the amount of traffic directed through the experiment
  • Note: If you choose 10%, 5% of visitors will be shown the variant and 5% the original, and conversion metrics measured based on those 10%. The remaining 90% of the traffic will not be part of the test.

After having made your selection, you will be asked to select a setting for ending the split test.

Your options are:

• Manually
• At given time
• After x views
• When result is significant

If you select one of the last three automatic options, you can define an action to take when the split test has run its course (Figure 6.2)

Figure 6.2 Selecting an automatic action after a split test ends

With these settings, you can:

• Decide if you want to delete visitor and conversion data or not
• Keep all versions or only keep the best performing version
• Notify people of the split test results (and specify an email notification template or create your own)

You can also end a split test manually.

Starting and stopping split tests can be done in a number of ways, depending on the type of split test, and what your settings are.

Page based split tests start automatically when you finish the split test wizard, if you checked the This split test is active checkbox under the split test settings.

If you did not, simply

• Navigate to the page you are testing
• Click the Marketing tab
• Click start (Figure 7.1)

Figure 7.1 Starting a page based split test

Before starting a content based split test, you must select one or more paragraphs on a page and create variants (Figure 7.2).

This will create duplicates, called variations, of the selected paragraphs.

Figure 7.2 Creating variants for split tests

Then simply:

• Open the variant paragraph
• Make the change you want to test

Half the visitors sent through the split test will see the variation paragraph, the other half the original.

Click the start button to activate the test on your site and begin collecting conversion data.

You can preview the frontend differences between the variant and the original with the preview button.

Split tests may be stopped automatically when certain conditions are fulfilled, but you can also pause or stop them manually using the stop and pause buttons.

When stopping a split test manually, you will be prompted for a decision on the what to do with the split test variation.

You must:

• Confirm your choice by checking the Yes, stop this experiment box
• Select what to do with test variations:
  • Keep all versions, with the original published
  • Keep the original and delete the variation
  • Keep the variation and delete the original
  • Keep the best performing version and delete the other

You can modify your template to create variants dynamically, depending on the session.

Use the template tag Global:OMC.Experiment.Variation.SOMEVARIATION to check if it’s an original or variant session, and change what happens on the frontend.

SOMEVARIATION will be replaced with ‘Original’ for the original and ‘2’ for the variant.

For example:

You still need to setup and start a split test on the page or paragraph – the tag merely enables you to create your variations directly in the template.