Can You Become a Slave to Data?

In a recent discussion with my colleagues about data-driven decision making, one of them asked, "Can you become a slave to data?  In other words, can you become so focused on the numbers that you ignore common sense and make bad decisions?"  It was a great question.  After a little reflection, I responded that yes in theory you can be a slave to data.  However, in the real world I struggle to think of companies that truly have become slaves to data.

On a scale of 1-10 (where 1 means totally winging it and 10 means being completely data driven), most companies are much closer to 1 than 10.  When other Internet professionals ask for my advice on issues they are facing, I am often shocked by how little analysis they have done.  I often will say something like, "That is a good issue.  I bet looking at metrics X, Y, and Z will tell you your answer.  I am sure you have looked at X, Y, and Z, right?"  Usually I am greeted by a long pause and a lot of foot shuffling.  If an average company is a 2 or 3 on the data driven scale, they can easily move up a number or two (or more) and not become slaves to data.

Secondly, if you are using a hypothesis driven approach as I talked about in my last blog post, you already are bringing intuition and common sense into the equation.  You are only running analyses on things that you have already thought about.

Lastly, if you really understand the limitations of the common analytical tools, you know that there is a pretty large margin of error built in.  Let's take the free version of Google Analytics (GA).  Try running a report one way and then run the same report a different way.  The results are often a little different, even when GA doesn't say it is sampling the data.  When GA says it is sampling the data, the variance between reports can be pretty large.  Alternately, hook up the GA e-commerce tracking to your website and compare how many transactions GA gets to how many your e-comm platform says you actually received.  You will often see that GA misses about 15% of the transactions.  Drill down on some of the transactions in GA and you will find garbage transactions.

I picked on Google Analytics, but you can find similar limitations in most of the tracking tools.  Let's not even get into attribution modeling.  I can barely explain to you what made me buy product A over product B.  How is a relatively simple attribution algorithm able to describe what drove my buying behavior?  It can't.

If you understand these limitations in today's tools, you realize that our tools are best used as directional indicators.  They cannot give you exact answers, but they sure can point you in the right direction.  Your brain must be fully engaged at all times - you cannot blindly trust the numbers that are spit out.  I often like to triangulate my results using a couple different methods or tools to increase my confidence in my measurements.

The bottom line is that if you have a solid analytics team, the chances are extremely slim that you have to worry about becoming a slave to data.  Whew, you can cross that off your list of things to worry about!

Are You Boiling the Ocean?

You have just been assigned a new project at work, and you think to yourself, "I am going to be more data driven on this project."  The first thing you do is start daydreaming about all the data you want.  Before you know it, you have a list of reports you want that is longer than War and Peace.  Heck, you don't what you are going to do with all the data, but you want it!  You are now officially boiling the ocean. Your analyst will get all the reports done about the same time as you boil the ocean.

A far more time efficient process is to be hypothesis driven.  You have a hunch about what is going on.  Why not leverage that intuition to reduce your decision making time dramatically? By creating a hypothesis about what the problem might be, you do two things.  First, you limit the data you need to collect.  Second, you pull data that you can take actions on.  Both make you go faster.

Let's say that you have been assigned to improve your account creation funnel.  You have a four step funnel and you have a hunch that step #3 is where you lose everyone.  You can postulate a hypothesis that says that you think that you are losing more people from step #3 than any other step in your funnel.  If the hypothesis is true, the implication is that you should focus on improving step #3 before you work on other steps (all other things being equal).  You now have a fairly straightforward analysis that your analyst should be able to bang out in Google Analytics or a similar tool.

Let's say that your analysis comes back the way you expect.  You now can formulate a hypothesis about what is wrong on step #3.  For example, I think that customers are getting confused about what to input into field XYZ.  You can now use a tool like Mouse Stats to see if customers are pausing longer on field XYZ or exiting on that field.  If they are, you know where to focus your actions.

At the end of the day, making rapid improvements to your site or app is the name of the game.  Boiling the ocean is your worst enemy when trying to make rapid improvements.  Let hypotheses guide your actions and forever be more efficient!