Advice for A B and multivariate testing

By Paul Morris

This post concerns a/b and multivariate testing as I’ve become tired of people using the terms interchangeably and would like to help people get started.

A/B testing (also called split testing) is where a baseline control sample is compared to a variety of single-variable tests (page elements on the one page or the effectiveness of one landing page over another).

Multiviariate testing (MVT) allows you to test multiple page variables at the same time and picks the winning combination of page elements.

I am not going to sell A/B or MVT to you however trust me when I say testing can reap huge £rewards due to conversion rate improvements.

Testing Tips

My two platform recommendations are Google website optimiser (cheap and very good) and (expensive).

Start with A/B and progress to MVT. Walk before you run. Do not polish a turd i.e. your current site and progress to MVT before you have the ‘right’ template to work with in the first place (obtained via A/B testing).

Again I’d like to reinforce the ‘start simple’ statement. Do not put 10’s tests live as it will complicate initial proceedings and could take months to obtain a statistically significant winner(s).

The winning page is often defined by the best converting however you should take other softer KPI’s into consideration when picking the winner e.g. time on site, social media engagement, number of people signing up to the e-newsletter, etc.

Test everything (headlines, images, buttons, call to action messages, content, links, website templates, forms) yet still prioratise the order in which you test.

Be bold with your Conversion Rate Optimisation (CRO) choices.

Do not let the HIPPO (highest paid person’s opinion) determine what to test. Brainstorm ideas with various interest groups (designers, sales, marketing, etc) and develop a testing recipe/ matrix.

Do not be a smart arse! Do not presume something will win/ lose. I would have bet my house on certain variants winning in the past only to be proved wrong!

Use A/B and MVT in collaboration with other useful tools e.g. Crazy Egg (allows you to see the parts of your page that have been clicked on and how far they scroll), Click Tale (Flash movies of your visitors’ browsing sessions), customer surveys (e.g. by Avinash Kaushik) and Live Chat.

People often only test the landing pages and omit the shopping cart = a huge missed CRO opportunity.