Email A/B Testing Now Available in Dynamics 365 for Marketing

A/B Testing for Dynamics 365 for Marketing

Email A/B testing arrives in Dynamics 365 for Marketing with the 2019 Release Wave 2 which is already available for Early Access in non-production instances.

Through A/B testing, marketing teams are better able to optimise email engagement rates by evaluating the performance of key message components. In this release, an A/B test can be run on an email subject or sender name.

SEPTEMBER 2019 UPDATE: A/B Testing Now Includes Body Elements

For example, by evaluating the performance of two email subject lines, marketers are able to collect useful data about the type of messaging their audience are most receptive to. Rather than committing every email send in a campaign to the same subject title, this enables a sample to be tested with the best performing variant automatically applied to the remaining audience. As well as improving open / click rates on the immediate email, test results provide insights to help teams determine future email subjects.

Configuring An Email Template for A/B Testing

Within an email template in Dynamics 365 for Marketing, click the A/B Test button to open this panel and select New to define a split test for either Subject or From Name.

This adds a new test to the panel which can renamed or left with the default 'New Test' title. As we'll need to reference this later in a different screen it'll be helpful to give this a descriptive title.

In this instance, we'll configure a test for the email subject and name it accordingly.

By toggling between the Version A and Version B buttons, email subjects can be set for each.

In Version A for this newsletter template a subject line has been inserted that will dynamically be prefixed by the recipient contact's first name.

Version A Email Test Subject Line

Upon switching to the Version B for this template, an alternative subject has been entered.

Test Email Subject Lines

Only one test can be run at a time but as a template may be used across different audience segments it could be beneficial to configure this for additional split tests which can be set within the context of an individual customer journey. 

In this example, an additional test has been configured for the 'From name' that will apply to this template.

Version A will reflect the name of the sender organisation while Version B test shall be dynamically populated to reflect the Owner / Account Manager of the parent account that a recipient is associated with. 

A/B Test Variants

Applying a Test Within a Customer Journey

Once a test has been configured for a specific template this can be applied within a customer journey sequence. 

In the below example, our template has been connected to a contact segment in a simple customer journey but this could also be applied within a more complex journey.

Customer Journey tile

While A/B settings have been applied for the template, a test has yet to be set-up for this Journey so the A/B icons on the tile are shown in grey.

An A/B test is activated within the properties tab for the marketing email message tile by enabling the slider control and completing the supporting settings.  

Enable A/B testing

Choose A/B Test - select one of tests previously defined for this email template, in this example we'll use the first example shown above to test the email subject

A/B distribution - control what percentage of the segment receiving this message should be included within the test, and how this will be weighted between the A/B variants. In the above example, 20% of the audience will be enrolled in the test with 10% receiving the Version A subject and 10% receiving Version B. If appropriate, tests can be weighted to increase the sample of either option. For example, this would support a scenario where 30% of contacts will be put into the test with 20% receiving Version A and 10% receiving Version B. While this gives marketing teams with some extra flexibility the distribution bands are only set in increments of 10% so this doesn't currently allow 15% or 25% allocations.

Winning metric - define the success criteria for either email opens or email clicks

Test duration - set how long should elapse before Dynamics 365 assesses which version has performed best. In addition to tests running for several hours this can also be extended to days and weeks.

If results are inconclusive - this enables marketers to choose what should happen if the test doesn't produce a clear winner. For this eventuality, users can set the default action to be Version A, Version B or a 50/50 split between both in the event of this outcome.

Evaluating Test Results

A/B progress and results can be checked at any time by clicking the email marketing message tile in the Customer Journey and opening the data panel. 

This reveals the test status including how much time is remaining for an active test, or which version was the winner if the test has been completed.

A/B Test Results

RELATED: How is Dynamics 365 for Marketing Licensed

Learn more about Dynamics 365 Marketing

Get advice from our consultants and arrange a demonstration to understand how Dynamics 365 Marketing can be configured to fit your requirements.

Request Callback >