Creating Split Test Campaigns

Use split testing to send variants of your campaign to a percentage of your mailing list to see which variant achieves the best results. Tests can be as simple as a different subject line or more complex testing of the email content itself.

Note: the split testing feature cannot be used in conjunction with recurring campaigns.

To create a split test campaign:

  1. Start a new campaign as normal, i.e. enter your settings, define your options, and select a target audience.

    Tip: you can filter a list associated with a split test campaign. The filter will be applied at launch.

  2. On the compose step, press the + split campaign button.

  3. Your split test will create two tabs (A and B) and copy any created content into each tab.

    Tip: if you are planing to create variants with minor differences, create the main body of your content before pressing the + split campaign button to create the majority of your variant content when the tabs are generated.

  4. On tab B:

    • Press the variant options button to edit the campaign options specific to the variant. The options you can edit are:

      • The subject line.

      • from address.

      • Reply address.

      • Forward Replies (if you have this on demand feature active).

      Tip: if you do not wish to save the changes made, navigate back to the campaign overview using the arrow button or breadcrumb bar.

    • Press the create HTML for variant button to begin editing the content specific to the variant.

  5. When you have edited all your variants, proceed to launching your split test.

Adding or removing a variant

Press the + icon in the tab bar to add additional variants. You can create up to 26 variants, though a larger list size is recommended the more variants you create.

To delete a variant, hover over a tab and press the X button.

Editing content

Until you create content for a new variant, it will be a duplicate of the 'A' variant.

To edit the HTML content for your variant:

  1. Press the create HTML for variant button to go the editing page.

  2. Open the editor and make your changes.

  3. Note: which editor you use (source, HTML, email) will depend on which editor was used to create the original content.

  4. Save your changes and return to the compose page using the back arrow on the editing page.

To edit the text content for your variant:

  1. Press the create text for this variant button to go to the editing page.

  2. If you have edited the HTML for the variant, press the generate from HTML button to create a text version of your HTML content.

  3. Press the edit button to allow you to type into the text window and manually update your text content.

  4. Save your changes and return to the compose page using the back arrow on the editing page.

Note: if you delete edited content using the delete HTML variant or delete text variant button, it will revert back to showing content of the 'A' variant.

When you have edited all your variants, proceed to launching your split test.

Note: you do not need to change the mailing list for each variant. The variant lists are a random selection from the campaign list.

Testing and launching a split test

Note: your launch options may differ if the campaign is for an automation program.

Testing and launching a split test follows the standard launch process with some additional considerations to account for the variants.

  • Test email - lists the different variants, allowing you to choose which you wish to send. You can send a test email of each variant separately.

  • Spam check - reports each variant separately (giving a score to each), making it clear which variants need editing if any problems are detected.

  • Link check - reviews the links in both your text and HTML content for all variants and reports on each separately so you can go back and edit individual variants to fix any broken links.

  • Setup launch - includes additional split test options for the percentage of list contacts to receive split test variants, criteria to determine best performing variant, launch automation, and launch approvals.

Once the split test has launched, the campaign overview will display an active split test notice, which shows the split test report and close split test buttons.

You will be able to see the individual results for each variant and the full campaign in the split test report.

Setting a mailing list percentage

By default, split test variants are sent to ten percent of your list.

Adjust the percentage of list contacts to receive split test variants by:

  • Adjusting the slider to your preferred percentage (displayed in the text box), or,

  • Entering a value in the text box, or,

  • Incrementing the value in the text box up or down with the arrow keys.

    Note: if 100% is chosen on the slider the system will attempt to split the list as equally as possible, potentially providing a range per variant.

Changing the percentage will change the:

  • Estimated test contacts - the number of contacts receiving variants.

  • Estimated final list - the number of contacts who will receive the chosen variant.
Note: the system may make adjustments to variant counts if exact division of contacts between variants isn't possible. The number in brackets shows the contacts per variant.

Setting performance criteria

Using the drop down menu, set the criteria to determine the best performing (winning) variant.

  • Opens - the number of times an email has been opened. Calculated as a percentage (%) of the total number of delivered launches.
  • Clicks - the number of times an email has been clicked (links/images etc.). Calculated as a percentage (%) of the total number of delivered launches
  • Clicks of opens - number of times someone opened and clicked. Calculated as a percentage (%) of clicks against total number of opens.

Adding launch automation

Use the checkbox under the criteria to determine best performing variant to decide whether to launch the best performing campaign automatically or not.

If this option isn't selected you can choose which campaign to send based on the split test results.

When selected you can add a delay, in case you want to stop the launch.

Note: should variants perform equally, the first variant will be launched.

Choosing to launch automatically will also display the follow-up campaigns option, providing you with a simple way to resend your campaigns to recipients who haven't opened.

Managing split test approvals

If your split test is sent for approval, the user or group assigned to the split test will receive an approval email per variant.

When accepting or rejecting the split test approval, the form will display all variants (from which you can preview HTML and/or Text parts).

Caution: you will be accepting or rejecting the whole split test campaign, not the variant.