Can You Become a Slave to Data?

In a recent discussion with my colleagues about data-driven decision making, one of them asked, "Can you become a slave to data?  In other words, can you become so focused on the numbers that you ignore common sense and make bad decisions?"  It was a great question.  After a little reflection, I responded that yes in theory you can be a slave to data.  However, in the real world I struggle to think of companies that truly have become slaves to data.

On a scale of 1-10 (where 1 means totally winging it and 10 means being completely data driven), most companies are much closer to 1 than 10.  When other Internet professionals ask for my advice on issues they are facing, I am often shocked by how little analysis they have done.  I often will say something like, "That is a good issue.  I bet looking at metrics X, Y, and Z will tell you your answer.  I am sure you have looked at X, Y, and Z, right?"  Usually I am greeted by a long pause and a lot of foot shuffling.  If an average company is a 2 or 3 on the data driven scale, they can easily move up a number or two (or more) and not become slaves to data.

Secondly, if you are using a hypothesis driven approach as I talked about in my last blog post, you already are bringing intuition and common sense into the equation.  You are only running analyses on things that you have already thought about.

Lastly, if you really understand the limitations of the common analytical tools, you know that there is a pretty large margin of error built in.  Let's take the free version of Google Analytics (GA).  Try running a report one way and then run the same report a different way.  The results are often a little different, even when GA doesn't say it is sampling the data.  When GA says it is sampling the data, the variance between reports can be pretty large.  Alternately, hook up the GA e-commerce tracking to your website and compare how many transactions GA gets to how many your e-comm platform says you actually received.  You will often see that GA misses about 15% of the transactions.  Drill down on some of the transactions in GA and you will find garbage transactions.

I picked on Google Analytics, but you can find similar limitations in most of the tracking tools.  Let's not even get into attribution modeling.  I can barely explain to you what made me buy product A over product B.  How is a relatively simple attribution algorithm able to describe what drove my buying behavior?  It can't.

If you understand these limitations in today's tools, you realize that our tools are best used as directional indicators.  They cannot give you exact answers, but they sure can point you in the right direction.  Your brain must be fully engaged at all times - you cannot blindly trust the numbers that are spit out.  I often like to triangulate my results using a couple different methods or tools to increase my confidence in my measurements.

The bottom line is that if you have a solid analytics team, the chances are extremely slim that you have to worry about becoming a slave to data.  Whew, you can cross that off your list of things to worry about!