One of the reasons I ended up choosing a career with a digital marketing agency was my consumer behavior college class. In it, we had to run some focus groups and also learn various ways to analyze what potential consumers thought (sometimes without actually asking them directly.)
Seeing what makes people tick is one of my favorite pastimes and this class spoke to me. Fast forward a few (or a lot of) years later and now we have the digital tools to see what changes consumer’s behavior in real time. We don’t even have to spend time categorizing and adding the results ourselves. With marketing tools such as HubSpot, Google Analytics, and Optimizely, we have the ability to perform A/B test of individual pages and actually see what is working and isn’t working on our websites.
Why Should You A/B Test
The best way to tell if your campaign is working is to measure results. Going a step further involves testing different ways to give the same message to see what resonates better in order to improve those results. A/B testing can be done to test your current results (the control) against a new option, or by using 2 completely new versions and then comparing against your normal (control) baseline. The most important part is to have an accurate baseline to compare against at the start of your test.
What to A/B Test?
In order to successfully test there has to be consistency between the pages being tested. This means testing should be kept to no more than 1 or 2 items. This does not mean you have to keep it super basic; this is an opportunity to be creative and test things you have been curious about. Hate the typography? Test it against that shiny new Google font that everyone is using. Would video improve conversions? Go ahead and test it! Do people really not understand the copy on this CTA as much as the comments say or are they just a loud minority? Put some lipstick on that pig and let visitors tell you for certain.
How Long to Test?
In statistics, you often hear of “statistical significance” and “statistical confidence.” What do those mean? It is a numbers game. The end goal is to have a sample size that is big enough that you can recreate the results later and that is tested long enough to achieve that sample size and also to allow for the “statistical confidence” to reach as high of a level as possible (industry standard is either 90% or 95%.) Many factors can skew your results so it is important to extend the test for as long as possible to prevent variables like seasonality, holidays, weather, elections (don’t get me started…) from not being accounted for in your results. The most commonly used minimum duration is two weeks, and if extended to keep extending by 7 days at a time and no less. This prevents daily fluctuations in behavior from greatly affecting what is seen in the results. Still unsure? Then head over to some online tools like this handy dandy test duration calculator from Visual Website Optimizer…by adding your own result goals it uses some wizard magic (aka formulas) to recommend the number of days you should test.
What to Do with the Results?
Now that you’ve spent time patiently waiting for the visitors to roll by and the results to come in, what do you do? The answer is simple: analyze! While looking at the overall totals is a quick and dirty glance into what your overall conclusion might be you still need to segment your results and do a deep dive.
Any segment could potentially change your view on what originally might seem like the clear winner.
Once you are “confident” in your results, it is time to implement the changes that proved to have the highest conversion. One thing to keep in mind: keep an eye on your conversions. Simply because the A/B test is over does not mean we can open the bottle of champagne and watch the conversions roll in. If you are not seeing what should be expected based on testing there may be additional variables or segmentation that need to be analyzed.