AB Testing

“Why is Digital Marketing so fantastic? Because everything is trackable and since everything is trackable, everything is also testable. Basically, you never have an excuse to have a crappy landing page, or email campaign or banner ad because of A/B testing.”

– Mark Staton

So what is A/B Testing? According to Optimizely A/B Testing: is a simple way to test changes to your page against the current design and determine which ones produce positive results. It is a method to validate that any new design or change to an element on your webpage is improving your conversion rate before you make that change to your site code.” To perform an A/B test, Smashing Magazine’s Ultimate Guide To A/B Testing explains it as” [having] two versions of an element (A and B) and a metric that defines success. To determine which version is better, you subject both versions to experimentation simultaneously. In the end, you measure which version was more successful and select that version for real-world use.” The overall main point of AB testing is to gain insight with the goal of increasing conversion


The A/B Testing Process:

  • Decide your goal – what are you testing?
  • Select a tool for the job – you can set up A/B tests in two ways
    • Replace the element to be tested before the page loads
    • Redirect to another page
  • Set a conversion goal – using code (e.g. javascript) you can program what characterizes a successful test for a visitor, and track the success rates, like reaching a “Thank you” page after making a purchase.

Do’s and Don’s of A/B Testing


  • Make sure that you understand how long you should run your tests – If you end a test prematurely, your test will have been a waste of time and will have potentially lost very insightful and valuable information.
  • Be consistent on which visitors see what. Preventing bundlers (showing a user a different price or different promotional offer) can be done with your tool – using a mechanism or algorithm that keeps track of which visitors have been exposed to which variations.
  • Keep A/B test treatments consistent throughout your whole webpage. If testing the color of a sign-up button, make sure the color is changed not just on your landing page, but across all your pages.
  • Frequently perform A/B tests. The more tests you perform, the better your website will be!
  • Treatment Rampup: instead of splitting up your testing 50/50 between your users, gradually implement your treatment on your test market (e.g. 99.9%/0.1% to 99.5%/.05% to 97.5%/2.5% etc.) This avoids introducing potentially bad problems to a larger audience. 


  • Always test your variation and control at the same time – there can be extraneous factors that happen one week that don’t happen the next, which can affect results.
  • Don’t jump to conclusions – “Statistical confidence” tests determine the significance of test results, telling you if they are important or not.
  • Don’t take regular visitors off guard – test new visitors to avoid confusing your returning visitors.
  • Don’t let emotions overrule test results – sometimes the results aren’t what was expected, but remember that you are testing for improved conversion rates not aesthetics.

So why use A/B testing? A/B testing considers affects on traffic. It shows what works best with your customers with quantifiable data. This allows you to optimize and personalize your website, making it about your customers, making you more likely to succeed.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s