No amount of experimentation can ever prove me right; a single experiment can prove me wrong.
– Albert Einstein (1879-1955)

If you’ve been following my Marketing Over Coffee (MoC) blog series, you know that we’ve been running a little experiment here at the Factory.  My first blog post introduced the experiment, which involved testing two MoC email templates against each other.  The first template was very text-heavy and light on design.  The second, which Blue Sky Factory designed, involved more HTML design, followed industry best practices, and was more polished and professional looking.

My second blog post revealed these two creatives and compared them against each other.  I explained how, by all email best practices, the second template was “better” than the first.  I then went on to outline how the A/B split test would work.

Previously Used Template

New Template

Let The Testing Begin!

The A/B split test was launched on September 22nd to 40% of the MoC email list.  20% of their list received the old template, while the other 20% received the new template.  Subject line, From Name, email content and date/time of send were all completely identical so that the only factor we were testing was the actual template design.  We allowed the A/B test to run for a sufficient amount of time, then gathered the email metrics to determine the winning email that would be sent to the remaining 60% of the list.

After letting the A/B test run for two hours, the new template had a slight edge on the old template.  The old template had received an open rate of 9.8% and a click-through rate of 0.9%.  The new template, however, had received an open rate of 11.6% and a click-through rate of 2.3%.  Satisfied with these results, we sent the “winning” new template to the remainder of the list.

We continued to follow the results of the A/B test into the next day, and then something interesting happened…

It Turned Into A TIE!

Seriously, the results were almost identical.  The old template finished strong with a 25.7% open rate and a 4.2% click-through rate, whereas the new template received a 27.0% open rate and a 4.7% click-through rate.  The new template’s results were still slightly higher, but it certainly did not blow the old template out of the water.  In addition, the MoC team tracked downloads of an OPML file on their site.  MoC found that unique clicks from each email to the OPML file were almost identical from the two templates, while there were more repeat clicks to the file from the old template than the new template.

What We Learned

What a colossal waste of time, you may be thinking.  Not so! What we learned from this experiment was that industry best practices do not necessarily work for all audiences.  To learn what works for your email recipients, you must Test, Test, Test!  Not all audiences will respond to the same type of subject line, call-to-action, email design, landing page, etc.  Test different types with your own email recipients, and use what works best for your email campaigns.

So What Now?!?

With these inconclusive results in mind, what does Marketing Over Coffee plan to do moving forward?  After speaking with Chris Penn, one of the co-hosts of MoC, about the A/B test results, they plan on using a “hybrid” email template for future campaigns.  They will use the design and look of the new template, and apply that to the “barebones simplicity” of the old template.  This will help MoC to maintain the polished, professional look of the new template, while keeping the email template easy to use for the MoC team.

 

Intrigued by this experiment and want to conduct your own?  Click here to read how to get started with your own A/B test.  Ready to run a creative A/B test of your own?  Check out our creative portfolio to see what our team can do, and contact us if you’re ready to start experimenting!

Joanna Lawson-Matthew
Account Manager, Blue Sky Factory