Tuesday, October 22, 2013

Where to Start with A/B Testing

I am astonished by how often I run into Internet entrepreneurs who have an Internet business but have yet to build A/B testing into their culture.  By using A/B tests, you can quickly improve how parts of your business operate.  You can A/B test the web site, emails, and many other facets of your business.  Each time you are measuring the effectiveness of one approach versus another. 

The benefits can be huge.  At TeamSnap we made one relatively minor change to our home page that immediately increased trial signups by over 30%.  That test required just a couple hours of a designer's time plus an A/B test tool which cost us about $300/month at the time.  If we tried to buy 30% more trials through customer acquisition methods, it would have cost us a fortune.

The first place to start with A/B testing is to agree on what you want to optimize. Yes, this sounds obvious but sometimes not everyone on the team might agree.  For a typical SaaS business, you might look at things like trial sign-up rate and conversion from trial to paid plans.  For an e-commerce business you probably want to focus on revenue per visit or a similar metric.

Next you need to put in place a testing system.  We use Optimizely because it is super easy to use and allows designers and marketing professionals to launch tests with little to no involvement by development. That allows us to iterate very quickly.

From there you need to decide where to start testing.  There are a couple approaches you might consider:
  • High traffic pages are the most obvious choices.  If you focus on pages that are seen by lots of people, you get two benefits.  First, your tests will complete more quickly because you collect a lot of data in a short period of time.  Second, as a general rule, high traffic pages will impact your key metrics more than low traffic pages.
  • "Broken" pages are another logical place to start.  Everyone has a few pages on their site that they know they would like to blow up and re-do.  A/B testing can help guide you through that process.  Run an A/B test with a new design that you have been itching to roll out and see if it improves performance.
  • High "page value" pages are a more scientific way of approaching things.  Google Analytics computes a page value for each page on your site using a formula that links that page to transactions and goals on your site.  This is a more systematic way of identifying pages that are likely to impact your key metrics (assuming you are using Google Analytics goals and advanced e-commerce capabilities).

One final thing to consider is that you need to have some basic understanding of the math behind A/B testing before you start drawing too many conclusions from them.  Remember what a confidence interval is from your college probability class?  If not, time to brush up.  Here is a very common mistake.  Let's say that your test shows that variant A was better than variant B at a 95% confidence interval.  A on average was 10% better at whatever goal you defined.  The test is saying that you can be 95% certain that A is better than B, but you cannot be 95% certain that the improvement is actually 10%.  Brush up on your math and you'll understand why.

The most important thing with A/B testing is to build it into the culture.  Every time you approach a problem and every time you consider a change to your site, ask whether you can use A/B testing to help guide you.