As you may have noticed, I haven't posted here in a little while. That is because I was busy releasing How to Acquire Your First Million Million Customers, a book on how to grow your site to a million customers or more. The book is now available on Amazon and we have a site up with a blog on growth hacking, digital marketing, and other customer acquisition tips.
I would encourage you to check out the book and the new blog. The new blog will be more active going forward than this site.
Ken
Thursday, June 22, 2017
Thursday, January 19, 2017
Overconfidence in Analytics – Why You Need to Dig Deeper into How Your Analytics Platforms Work
Marketing professionals and analysts often exude a level of confidence in their data that is misguided. I regularly see people state with confidence what their return on ad spend is on a given campaign down to three decimal points. They will tell you that their analysis is based on Google Analytics or a similar web tracking tool and a certain attribution tool. Yet, they fail to state that these tools have a significant margin of error.
For example, Google Analytics and similar tools have a host
of issues:
- Missing data. Look at the list of transactions as recorded in Google Analytics and compare that to what you see in your transactional system. You will be surprised by how much is missing in GA and at some of the bogus transactions GA records.
- Malformed or bad referral sources and URLs. Look at the sites that supposedly refer traffic to you and will see all kinds of odd sites that don’t exist. Look at the top pages on your site according to GA and you will find pages that don’t exist.
- Self referrals. Ever look in GA and see that one of the main sources of traffic is your own site? If so, you are polluting your data and overwriting valid traffic sources.
Attribution engines are even more fraught with issues
especially depending on your implementation.
For example:
- Missing sources. Are you looking only at click conversions or click and view through conversions? A lot of folks only consider click conversions which leaves out a lot of information. If you are looking at view through conversions, are you tracking all of them or just view throughs from one ad network?
- Overconfidence in a given attribution model. Do you have the analytics to prove that the attribution model you picked is the right one? How do you really know which touch drove the conversion and therefore should get the credit for the sale?
- No consideration of offline. Do you have offline marketing that isn’t factored into your attribution model?
Do yourself a favor and learn how the internals work on
whatever analytics platform you use. The
more you know, the better you will understand how much confidence you can have
in a particular analysis. And ignore the vendor hype – none of analytics tools
work as cleanly as the vendors promote.
Get over it.
Do these issues mean you should give up on analytics? Absolutely not, but stop reporting on data
down to three decimal points and make sure that whoever consumes your analyses
understands that there is a non-trivial margin of error. Said another way, use the data directionally. The data is not dogma so keep your common
sense engaged when thinking about the results.
Friday, July 15, 2016
Bringing Data to the People
One of my big themes lately at TeamSnap has been "bringing data to the people." Our approach has been to push broad swaths of the company to immerse themselves in the data through better tools and a lot more training.
As I have talked about before, we leverage Tableau to push analyses down to marketing, bus dev, and product people. It allows relatively non-technical people to run sophisticated analyses without any knowledge of the underlying data. What is really amazing is that the underlying data is often coming from multiple data sources in the cloud and is joined through some sophisticated data hygiene and migration techniques that these business users don't need to know about.
Over the past two days we pushed harder on a different technique to democratize our data - training. We did an intensive 2 day training session on Google Analytics with Analytics Pros. It was a huge commitment in money but more importantly in time. We had a wide range of employees from across the company attend. It was amazing to see people really get the religion about how much you can improve your product through better data and that you don't need to rely on the data scientist types to get you data. Google Analytics is incredibly powerful, but also very usable by non-technical folks especially if they some training.
One of the key takeaways for a lot of people was how flexible Google Analytics is. I often say that Google Analytics is a platform, not a tool. It can be whatever you want it to be if you know all the ways it can be used.
Upon the conclusion of the training, my main advice to most people was to find one or two ways to integrate what they learned into their daily routines. If you manage social, build a report or a dashboard or a segment that drills down into social. Have that report or dashboard show up in your email box on a regular basis so you can make decisions from it. Then tweak that report or dashboard over time to improve how you manage social. By taking these small, incremental steps to analyze the data yourself, you set yourself on the path toward data empowerment.
Are you investing enough in the analytical skills of your rank and file employees?
As I have talked about before, we leverage Tableau to push analyses down to marketing, bus dev, and product people. It allows relatively non-technical people to run sophisticated analyses without any knowledge of the underlying data. What is really amazing is that the underlying data is often coming from multiple data sources in the cloud and is joined through some sophisticated data hygiene and migration techniques that these business users don't need to know about.
Over the past two days we pushed harder on a different technique to democratize our data - training. We did an intensive 2 day training session on Google Analytics with Analytics Pros. It was a huge commitment in money but more importantly in time. We had a wide range of employees from across the company attend. It was amazing to see people really get the religion about how much you can improve your product through better data and that you don't need to rely on the data scientist types to get you data. Google Analytics is incredibly powerful, but also very usable by non-technical folks especially if they some training.
One of the key takeaways for a lot of people was how flexible Google Analytics is. I often say that Google Analytics is a platform, not a tool. It can be whatever you want it to be if you know all the ways it can be used.
Upon the conclusion of the training, my main advice to most people was to find one or two ways to integrate what they learned into their daily routines. If you manage social, build a report or a dashboard or a segment that drills down into social. Have that report or dashboard show up in your email box on a regular basis so you can make decisions from it. Then tweak that report or dashboard over time to improve how you manage social. By taking these small, incremental steps to analyze the data yourself, you set yourself on the path toward data empowerment.
Are you investing enough in the analytical skills of your rank and file employees?
Saturday, May 7, 2016
Slicing and Dicing Your Way to Paid Advertising Nirvana
Digital marketing professionals who work in paid online marketing often get so focused on campaign and targeting level data that they forget to dig deeper and find the really interesting matches between those campaigns and their customer segments. By doing so, they waste large amounts of their marketing budget.
Let's say that you are running marketing campaigns in AdWords. You almost certainly will be looking at your CPA (cost per action) at a campaign, keyword, and a device level (desktop, tablet or mobile). If you know what you are doing, you hopefully have also thought about whether you want to target men or women and what age groups you want to go after. In addition, with luck you thought about your geo targeting as well as whether there are certain times of the week when your product is going to sell better.
The next step, and it is a big one, is to link AdWords to your back end database so you can track campaigns and keywords all the way through to what customers spend over time. How much do you spend on a given keyword compared to the LTV (lifetime customer value) you get from customers coming in on that keyword? Were these new customers or existing customers coming back again?
Now most folks are thrilled to just get this far (and only about 5-10% of the folks I speak with make it this far). However, the reality is that the fun is just starting at this point. On the back-end you probably have additional information about your customers. For example, their industry vertical, the product they purchased, their role, etc. So you now have the following dimensions to slice and dice your data by:
When you start dicing up your marketing spend at this granular a level, you start to realize that have an enormous amount of fat in your marketing spend. You will find certain combinations of the above dimensions where the ROI is extremely poor and you will find other segments where the ROI is excellent. If you take this approach, you often realize that 10-25% of your marketing budget is completely ineffective. Rebalancing the budget to double down on the most impact programs can have a huge impact.
Analyzing your data at this level definitely isn't easy. This approach takes a tremendous amount of work to set up and process. However, think about it this way. If your manager was to come to you and offer you an increase of 10-25% in your paid marketing budget, you would probably be ecstatic. If you slice up your data enough to find 10-25% fat in your budget, you are essentially getting a lot of extra money to spend each month.
Let's say that you are running marketing campaigns in AdWords. You almost certainly will be looking at your CPA (cost per action) at a campaign, keyword, and a device level (desktop, tablet or mobile). If you know what you are doing, you hopefully have also thought about whether you want to target men or women and what age groups you want to go after. In addition, with luck you thought about your geo targeting as well as whether there are certain times of the week when your product is going to sell better.
The next step, and it is a big one, is to link AdWords to your back end database so you can track campaigns and keywords all the way through to what customers spend over time. How much do you spend on a given keyword compared to the LTV (lifetime customer value) you get from customers coming in on that keyword? Were these new customers or existing customers coming back again?
Now most folks are thrilled to just get this far (and only about 5-10% of the folks I speak with make it this far). However, the reality is that the fun is just starting at this point. On the back-end you probably have additional information about your customers. For example, their industry vertical, the product they purchased, their role, etc. So you now have the following dimensions to slice and dice your data by:
- Campaign
- Keyword
- Ad
- Device type
- Geo
- Gender
- Age
- Time of day / day of week
- New customer vs returning
- Industry vertical
- Product purchased
- Role
When you start dicing up your marketing spend at this granular a level, you start to realize that have an enormous amount of fat in your marketing spend. You will find certain combinations of the above dimensions where the ROI is extremely poor and you will find other segments where the ROI is excellent. If you take this approach, you often realize that 10-25% of your marketing budget is completely ineffective. Rebalancing the budget to double down on the most impact programs can have a huge impact.
Analyzing your data at this level definitely isn't easy. This approach takes a tremendous amount of work to set up and process. However, think about it this way. If your manager was to come to you and offer you an increase of 10-25% in your paid marketing budget, you would probably be ecstatic. If you slice up your data enough to find 10-25% fat in your budget, you are essentially getting a lot of extra money to spend each month.
Wednesday, May 4, 2016
Wrangling the Google Analytics Sampling Beast
After reading my last post, you have determined that you are really struggling with sampling in Google Analytics. Now what do you do?
Your first option is to stick to the main reports on the left hand rail of Google Analytics.
The reports under these main categories rely heavily on roll-up tables in Google Analytics and therefore generally do not have sampling. However, once you start adding secondary dimensions, filters, or custom segments, all bets are off and your sampling is likely to come roaring back. While this can be very limiting, using these main reports is an option to remember.
A second option is to reduce the amount of data you are looking at, particularly by reducing your date range. On the positive side this is a quick and easy way to deal with sampling. The negatives are that this is really time consuming if you want to look at a large date range. Second, you have to reduce your date ranges as your site grows. Lastly, for some queries (e.g., unique users within say a month or a year), you may not be able to use this trick. If you are looking at monthly unique users, you cannot chop the month into two date ranges because you don't know how much overlap you have between the two periods.
A related option is to use Analytics Canvas. This tool basically takes the date range trick and automates it for you. You can tell it to take a Google Analytics report for the past year and split it into 365 separate reports, each one day in length. This is highly effective in dealing with sampling but again doesn't work for metrics like unique users.
The most comprehensive solution for dealing with sampling is upgrading to Google Analytics Premium. There is no doubt about it - it is an expensive product at $150,000 per year (as of the time of this article). In addition, to really exploit the data in GA Premium, you need to learn BigQuery, Google's Big Data tool. If you know SQL, this isn't too hard, but there are some quirks for sure. All these cons aside, the data that you can get out of GA Premium is amazing. GA stores your data at a very granular level behind the scenes. If you have worked with the GA API before, you will have seen a taste of what GA stores although the BigQuery interface to GA Premium exposes a far more extensive set of data that GA is collecting. It can take some time to wrap your head around the data, but it gives you an almost limitless number of analyses you can run about your customers.
Sampling in Google Analytics can be a real drag, but now hopefully you will be able to wrangle that sampling beast into submission.
Your first option is to stick to the main reports on the left hand rail of Google Analytics.
The reports under these main categories rely heavily on roll-up tables in Google Analytics and therefore generally do not have sampling. However, once you start adding secondary dimensions, filters, or custom segments, all bets are off and your sampling is likely to come roaring back. While this can be very limiting, using these main reports is an option to remember.
A second option is to reduce the amount of data you are looking at, particularly by reducing your date range. On the positive side this is a quick and easy way to deal with sampling. The negatives are that this is really time consuming if you want to look at a large date range. Second, you have to reduce your date ranges as your site grows. Lastly, for some queries (e.g., unique users within say a month or a year), you may not be able to use this trick. If you are looking at monthly unique users, you cannot chop the month into two date ranges because you don't know how much overlap you have between the two periods.
A related option is to use Analytics Canvas. This tool basically takes the date range trick and automates it for you. You can tell it to take a Google Analytics report for the past year and split it into 365 separate reports, each one day in length. This is highly effective in dealing with sampling but again doesn't work for metrics like unique users.
The most comprehensive solution for dealing with sampling is upgrading to Google Analytics Premium. There is no doubt about it - it is an expensive product at $150,000 per year (as of the time of this article). In addition, to really exploit the data in GA Premium, you need to learn BigQuery, Google's Big Data tool. If you know SQL, this isn't too hard, but there are some quirks for sure. All these cons aside, the data that you can get out of GA Premium is amazing. GA stores your data at a very granular level behind the scenes. If you have worked with the GA API before, you will have seen a taste of what GA stores although the BigQuery interface to GA Premium exposes a far more extensive set of data that GA is collecting. It can take some time to wrap your head around the data, but it gives you an almost limitless number of analyses you can run about your customers.
Sampling in Google Analytics can be a real drag, but now hopefully you will be able to wrangle that sampling beast into submission.
Thursday, April 14, 2016
Using the Presidential Election to Understand Google Analytics Sampling
When I coach people on how to use Google Analytics, I see one issue come up over and over - not understanding sampling. In many cases Google Analytics will use a sample of your data instead of all your data when you run a report. Is that bad? The answer is not necessarily. However, you need to understand sampling to be aware of the potential impact.
The way I explain sampling to folks is that when you see poll results for who is leading in the presidential election, the pollsters are only calling a sample of the 220 million eligible voters in the U.S. They often only call several thousand people. As long as those people are representative of the overall voter base in the U.S., you can accurately predict election results with a very small sample.
If you have a large site, you have a lot of data in Google Analytics. Some reports would take a very long time to run if Google were to use all your data. Hence the use of sampling.
One of the big issues with sampling is that Google doesn't tell you the statistical significance when it uses sampling. Think back to the election polls. If you read the fine print, you will almost always see something that says, "Margin of error for this poll is +/- x%." Often the margin of error is just a few percent, and it gives you confidence in the results. However, Google Analytics doesn't tell you the margin of error. If you have an event that doesn't happen very often, the margin of error can be huge. One of the first things that a lot of people notice when they have sampling is that their results are very inconsistent. They run a report one day and then the next day and get very different results. Similarly, if they pull the same data using two slightly differently approaches in GA and get very different results, they likely have a lot of sampling going on and a large margin of error in the results.
The first thing to check is how much sampling Google is doing. If you see a message like this in the upper right hand corner of your report, you have sampling going on.
So how much sampling is bad? A lot of people say that below 5-10% and you have a problem. However, it really depends. In the presidential polls, they only call a couple thousand people out of 220 million people. That is ~0.001% sampling, well below the rule of thumb some people call for in Google Analytics. However, in the election polls the statisticians can calculate the margin of error.
The margin of error depends a lot on how often the event you are measuring happens in GA. For example, let's say you are looking at device category under the mobile overview. Because there are only three choices in that report (desktop, mobile, and tablet), each session has a device, and each category shows up fairly often, you can have a low sampling percentage and you can still be confident in your results.
However, let's say that Google Analytics is using 10,000 data points in your report, and you are trying to estimate the frequency of a rare error message on your site. Let's say your error only happens 1 in every million times although you don't know that yet. The problem is that in this case GA is almost always going tell you that this event happens 0% of the time because the odds of the event happening in your 10,000 data points are very low (10,000 / 1,000,000). When GA does see the event in your sampled data, it is usually going to say the event happens 1 in every 10,000 times because it has 10,000 data points and it sees one occurrence. However, both answers, 0 and 1 in 10,000, are way off from the truth. Sampling is a huge problem in this case.
In the next post we will talk about your options if you suspect sampling is impacting your GA results. Until then watch for the yellow sampling message in the upper right hand corner of your GA reports to see how much sampling you are experiencing.
The way I explain sampling to folks is that when you see poll results for who is leading in the presidential election, the pollsters are only calling a sample of the 220 million eligible voters in the U.S. They often only call several thousand people. As long as those people are representative of the overall voter base in the U.S., you can accurately predict election results with a very small sample.
If you have a large site, you have a lot of data in Google Analytics. Some reports would take a very long time to run if Google were to use all your data. Hence the use of sampling.
One of the big issues with sampling is that Google doesn't tell you the statistical significance when it uses sampling. Think back to the election polls. If you read the fine print, you will almost always see something that says, "Margin of error for this poll is +/- x%." Often the margin of error is just a few percent, and it gives you confidence in the results. However, Google Analytics doesn't tell you the margin of error. If you have an event that doesn't happen very often, the margin of error can be huge. One of the first things that a lot of people notice when they have sampling is that their results are very inconsistent. They run a report one day and then the next day and get very different results. Similarly, if they pull the same data using two slightly differently approaches in GA and get very different results, they likely have a lot of sampling going on and a large margin of error in the results.
The first thing to check is how much sampling Google is doing. If you see a message like this in the upper right hand corner of your report, you have sampling going on.
So how much sampling is bad? A lot of people say that below 5-10% and you have a problem. However, it really depends. In the presidential polls, they only call a couple thousand people out of 220 million people. That is ~0.001% sampling, well below the rule of thumb some people call for in Google Analytics. However, in the election polls the statisticians can calculate the margin of error.
The margin of error depends a lot on how often the event you are measuring happens in GA. For example, let's say you are looking at device category under the mobile overview. Because there are only three choices in that report (desktop, mobile, and tablet), each session has a device, and each category shows up fairly often, you can have a low sampling percentage and you can still be confident in your results.
However, let's say that Google Analytics is using 10,000 data points in your report, and you are trying to estimate the frequency of a rare error message on your site. Let's say your error only happens 1 in every million times although you don't know that yet. The problem is that in this case GA is almost always going tell you that this event happens 0% of the time because the odds of the event happening in your 10,000 data points are very low (10,000 / 1,000,000). When GA does see the event in your sampled data, it is usually going to say the event happens 1 in every 10,000 times because it has 10,000 data points and it sees one occurrence. However, both answers, 0 and 1 in 10,000, are way off from the truth. Sampling is a huge problem in this case.
In the next post we will talk about your options if you suspect sampling is impacting your GA results. Until then watch for the yellow sampling message in the upper right hand corner of your GA reports to see how much sampling you are experiencing.
Sunday, February 7, 2016
How Big Is Your Analytics Team?
As Chief Growth Officer at TeamSnap, I am often asked how big our analytics team is. TeamSnap has about 10 million customers. Those customers are highly engaged in the app so we have a lot of information about what is happening in recreational and competitive sports, especially youth sports. We have 75 employees and are quite analytical as a company. Yet, we only have one full time data analyst. Instead we have taken a bit more of a distributed approach that I will describe.
Our basic philosophy is to get different people around the company engaged with the data. We do that by creating simple interactive dashboards that show our progress against targets. For example, in business development we have a dashboard that shows how we are doing in working with other partners to drive business to TeamSnap. The dashboards are so closely tied to comp plans that we use the same data to figure out quarterly bonuses. At the same time, the purpose of the dashboards is to allow people in those functions to interact and drill down on the data.
What we typically do is allow the end users to filter the dashboards by a variety of factors including geo, sport, type of customer, etc. If our business development folks are working with soccer partners in Arizona, they can isolate our performance in soccer in Arizona. Most of the analyses are actually much more complex and deeper than that example, but the interesting thing is that they are being run by business folks who are totally abstracted from the underlying data. They are just using a simple point-and-click dashboard.
By empowering the team with these interactive dashboards, we tend to pull out far more conclusions from the same sets of data. We have a lot of eyes on the data and different folks approach the data different ways and ask different questions. I often tell the team, "With these dashboards you can slice the data almost an infinite number of ways so your findings may vary." Sure enough the team often teases out all kinds of insights that we never would have seen with a more top-down driven approach where the analytics team spoon fed the data to the business team.
Several of our most analytical people in the company are ones who had little to no prior experience in digital analytics. Many of them came out of sports backgrounds or creative professions. It always puts a smile on my face when someone who does not have a formal data analytics background incorporates some great analyses into their work.
The vast majority of our dashboards are created in Tableau. We love the way it dynamically syncs to various data sources and presents the data to users in a way that is comfortable for anyone who has used Excel before. While Tableau isn't cheap, it was a much better fit for us than the half a dozen tools we tried before.
So I circle back to my question of how big our analytics team is. Our goal is to think of our analytics team not as our one data analyst, but rather as our entire employee base. We want every employee teasing out great insights from the data and integrating data into what they do.
Our basic philosophy is to get different people around the company engaged with the data. We do that by creating simple interactive dashboards that show our progress against targets. For example, in business development we have a dashboard that shows how we are doing in working with other partners to drive business to TeamSnap. The dashboards are so closely tied to comp plans that we use the same data to figure out quarterly bonuses. At the same time, the purpose of the dashboards is to allow people in those functions to interact and drill down on the data.
What we typically do is allow the end users to filter the dashboards by a variety of factors including geo, sport, type of customer, etc. If our business development folks are working with soccer partners in Arizona, they can isolate our performance in soccer in Arizona. Most of the analyses are actually much more complex and deeper than that example, but the interesting thing is that they are being run by business folks who are totally abstracted from the underlying data. They are just using a simple point-and-click dashboard.
By empowering the team with these interactive dashboards, we tend to pull out far more conclusions from the same sets of data. We have a lot of eyes on the data and different folks approach the data different ways and ask different questions. I often tell the team, "With these dashboards you can slice the data almost an infinite number of ways so your findings may vary." Sure enough the team often teases out all kinds of insights that we never would have seen with a more top-down driven approach where the analytics team spoon fed the data to the business team.
Several of our most analytical people in the company are ones who had little to no prior experience in digital analytics. Many of them came out of sports backgrounds or creative professions. It always puts a smile on my face when someone who does not have a formal data analytics background incorporates some great analyses into their work.
The vast majority of our dashboards are created in Tableau. We love the way it dynamically syncs to various data sources and presents the data to users in a way that is comfortable for anyone who has used Excel before. While Tableau isn't cheap, it was a much better fit for us than the half a dozen tools we tried before.
So I circle back to my question of how big our analytics team is. Our goal is to think of our analytics team not as our one data analyst, but rather as our entire employee base. We want every employee teasing out great insights from the data and integrating data into what they do.
Subscribe to:
Posts (Atom)