Marketing professionals and analysts often exude a level of confidence in their data that is misguided. I regularly see people state with confidence what their return on ad spend is on a given campaign down to three decimal points. They will tell you that their analysis is based on Google Analytics or a similar web tracking tool and a certain attribution tool. Yet, they fail to state that these tools have a significant margin of error.
For example, Google Analytics and similar tools have a host
of issues:
- Missing data. Look at the list of transactions as recorded in Google Analytics and compare that to what you see in your transactional system. You will be surprised by how much is missing in GA and at some of the bogus transactions GA records.
- Malformed or bad referral sources and URLs. Look at the sites that supposedly refer traffic to you and will see all kinds of odd sites that don’t exist. Look at the top pages on your site according to GA and you will find pages that don’t exist.
- Self referrals. Ever look in GA and see that one of the main sources of traffic is your own site? If so, you are polluting your data and overwriting valid traffic sources.
Attribution engines are even more fraught with issues
especially depending on your implementation.
For example:
- Missing sources. Are you looking only at click conversions or click and view through conversions? A lot of folks only consider click conversions which leaves out a lot of information. If you are looking at view through conversions, are you tracking all of them or just view throughs from one ad network?
- Overconfidence in a given attribution model. Do you have the analytics to prove that the attribution model you picked is the right one? How do you really know which touch drove the conversion and therefore should get the credit for the sale?
- No consideration of offline. Do you have offline marketing that isn’t factored into your attribution model?
Do yourself a favor and learn how the internals work on
whatever analytics platform you use. The
more you know, the better you will understand how much confidence you can have
in a particular analysis. And ignore the vendor hype – none of analytics tools
work as cleanly as the vendors promote.
Get over it.
Do these issues mean you should give up on analytics? Absolutely not, but stop reporting on data
down to three decimal points and make sure that whoever consumes your analyses
understands that there is a non-trivial margin of error. Said another way, use the data directionally. The data is not dogma so keep your common
sense engaged when thinking about the results.
