Using the Presidential Election to Understand Google Analytics Sampling

When I coach people on how to use Google Analytics, I see one issue come up over and over - not understanding sampling.  In many cases Google Analytics will use a sample of your data instead of all your data when you run a report.  Is that bad?  The answer is not necessarily.  However, you need to understand sampling to be aware of the potential impact.

The way I explain sampling to folks is that when you see poll results for who is leading in the presidential election, the pollsters are only calling a sample of the 220 million eligible voters in the U.S.  They often only call several thousand people.  As long as those people are representative of the overall voter base in the U.S., you can accurately predict election results with a very small sample.

If you have a large site, you have a lot of data in Google Analytics.  Some reports would take a very long time to run if Google were to use all your data.  Hence the use of sampling.

One of the big issues with sampling is that Google doesn't tell you the statistical significance when it uses sampling.  Think back to the election polls.  If you read the fine print, you will almost always see something that says, "Margin of error for this poll is +/- x%."  Often the margin of error is just a few percent, and it gives you confidence in the results.  However, Google Analytics doesn't tell you the margin of error.  If you have an event that doesn't happen very often, the margin of error can be huge.  One of the first things that a lot of people notice when they have sampling is that their results are very inconsistent.  They run a report one day and then the next day and get very different results.  Similarly, if they pull the same data using two slightly differently approaches in GA and get very different results, they likely have a lot of sampling going on and a large margin of error in the results.

The first thing to check is how much sampling Google is doing.  If you see a message like this in the upper right hand corner of your report, you have sampling going on.



So how much sampling is bad?  A lot of people say that below 5-10% and you have a problem.  However, it really depends.  In the presidential polls, they only call a couple thousand people out of 220 million people.  That is ~0.001% sampling, well below the rule of thumb some people call for in Google Analytics.  However, in the election polls the statisticians can calculate the margin of error.

The margin of error depends a lot on how often the event you are measuring happens in GA.  For example, let's say you are looking at device category under the mobile overview.  Because there are only three choices in that report (desktop, mobile, and tablet), each session has a device, and each category shows up fairly often, you can have a low sampling percentage and you can still be confident in your results.

However, let's say that Google Analytics is using 10,000 data points in your report, and you are trying to estimate the frequency of a rare error message on your site.  Let's say your error only happens 1 in every million times although you don't know that yet.  The problem is that in this case GA is almost always going tell you that this event happens 0% of the time because the odds of the event happening in your 10,000 data points are very low (10,000 / 1,000,000).  When GA does see the event in your sampled data, it is usually going to say the event happens 1 in every 10,000 times because it has 10,000 data points and it sees one occurrence. However, both answers, 0 and 1 in 10,000, are way off from the truth.  Sampling is a huge problem in this case.

In the next post we will talk about your options if you suspect sampling is impacting your GA results.  Until then watch for the yellow sampling message in the upper right hand corner of your GA reports to see how much sampling you are experiencing.

How Big Is Your Analytics Team?

As Chief Growth Officer at TeamSnap, I am often asked how big our analytics team is.  TeamSnap has about 10 million customers.  Those customers are highly engaged in the app so we have a lot of information about what is happening in recreational and competitive sports, especially youth sports.  We have 75 employees and are quite analytical as a company.  Yet, we only have one full time data analyst.  Instead we have taken a bit more of a distributed approach that I will describe.

Our basic philosophy is to get different people around the company engaged with the data.  We do that by creating simple interactive dashboards that show our progress against targets.  For example, in business development we have a dashboard that shows how we are doing in working with other partners to drive business to TeamSnap.  The dashboards are so closely tied to comp plans that we use the same data to figure out quarterly bonuses.  At the same time, the purpose of the dashboards is to allow people in those functions to interact and drill down on the data.

What we typically do is allow the end users to filter the dashboards by a variety of factors including geo, sport, type of customer, etc.  If our business development folks are working with soccer partners in Arizona, they can isolate our performance in soccer in Arizona.  Most of the analyses are actually much more complex and deeper than that example, but the interesting thing is that they are being run by business folks who are totally abstracted from the underlying data.  They are just using a simple point-and-click dashboard.

By empowering the team with these interactive dashboards, we tend to pull out far more conclusions from the same sets of data.  We have a lot of eyes on the data and different folks approach the data different ways and ask different questions.  I often tell the team, "With these dashboards you can slice the data almost an infinite number of ways so your findings may vary." Sure enough the team often teases out all kinds of insights that we never would have seen with a more top-down driven approach where the analytics team spoon fed the data to the business team.

Several of our most analytical people in the company are ones who had little to no prior experience in digital analytics.  Many of them came out of sports backgrounds or creative professions.  It always puts a smile on my face when someone who does not have a formal data analytics background incorporates some great analyses into their work.

The vast majority of our dashboards are created in Tableau.  We love the way it dynamically syncs to various data sources and presents the data to users in a way that is comfortable for anyone who has used Excel before.  While Tableau isn't cheap, it was a much better fit for us than the half a dozen tools we tried before.

So I circle back to my question of how big our analytics team is.  Our goal is to think of our analytics team not as our one data analyst, but rather as our entire employee base. We want every employee teasing out great insights from the data and integrating data into what they do.

Want to Improve Your Paid Ads? Get out of Your Ivory Castle!

Last blog post I argued that it was pretty hard to become a slave to data.  While I strongly believe that, I also believe that sometimes you have to put your analytical tools aside and get out into the real world.  Leave your ivory castle!

Paid advertising teams often are very analytical folks.  They speak in the language of CTRs (click through rates), CPAs (cost per acquisitions), view through conversions (conversions that came from folks who saw ads but didn't click on them), etc.  Many spend their days analyzing numbers, moving money across keywords, ads, campaigns, etc.  I too spend a tremendous amount of time looking at these metrics and often say that my job is akin to being a portfolio manager.  I look at numbers much of the day and move money around between projects to optimize our spend.

However, I feel strongly that marketing folks, and in particular paid advertising types, need to go out into the real world and mix it up with customers.  One of my favorite ways to do that is to go to tradeshows.  I try to go to roughly only show a month.  People (especially my wife) often ask me, "Why are you going to so many shows?  Don't you have people who can run the shows for you?"  Of course I have very competent team members who can run the shows.  However, I go to listen to our customers.

I go to the booth and ask questions.  I often don't do a lot of talking myself.  Here are some of the questions I ask:

To Existing Customers:

• What do you like best about the product?
• How did you find out about us?
• What other solutions did you consider?

To Potential Customers:

• What are the biggest challenges you face today?
• What options are you considering to address those challenges?
• How are you researching those options?

My favorite scenario is to have one or more existing customers in the booth try to sell some potential customers.  This happens a lot with TeamSnap because customers generally love the product and are pretty vocal about singing our praises.  I just sit back and listen to how the existing customers sell the potential customers.  They often can really crisply describe the benefits of the product.  Best of all, they use language that potential customers understand. They don't have the problem of being too close to the product.

During these types of interactions, I just sit there and take notes.  After the show I huddle with the folks working with me on our paid ad programs.  We almost always come up with new campaigns for AdWords, Facebook, etc.  As well, we typically come up with alternate versions of existing ads that more succinctly describe the product benefits.  These new campaigns and creatives generate lots of new analyses and learnings.

The more varied your customer base, the more you need to get out and meet folks.  TeamSnap is used in over 100 different sports.  As well, we have customers in 196 countries.  As such, there are a ton of ways customers use the product.  An adult hockey team in Vancouver has very different needs than say a youth travel soccer team in L.A. or a high school rugby club in London.   You cannot pick up those sort of differences from behind a desk - sometimes there is just no substitute for leaving the ivory tower.

Can You Become a Slave to Data?

In a recent discussion with my colleagues about data-driven decision making, one of them asked, "Can you become a slave to data?  In other words, can you become so focused on the numbers that you ignore common sense and make bad decisions?"  It was a great question.  After a little reflection, I responded that yes in theory you can be a slave to data.  However, in the real world I struggle to think of companies that truly have become slaves to data.

On a scale of 1-10 (where 1 means totally winging it and 10 means being completely data driven), most companies are much closer to 1 than 10.  When other Internet professionals ask for my advice on issues they are facing, I am often shocked by how little analysis they have done.  I often will say something like, "That is a good issue.  I bet looking at metrics X, Y, and Z will tell you your answer.  I am sure you have looked at X, Y, and Z, right?"  Usually I am greeted by a long pause and a lot of foot shuffling.  If an average company is a 2 or 3 on the data driven scale, they can easily move up a number or two (or more) and not become slaves to data.

Secondly, if you are using a hypothesis driven approach as I talked about in my last blog post, you already are bringing intuition and common sense into the equation.  You are only running analyses on things that you have already thought about.

Lastly, if you really understand the limitations of the common analytical tools, you know that there is a pretty large margin of error built in.  Let's take the free version of Google Analytics (GA).  Try running a report one way and then run the same report a different way.  The results are often a little different, even when GA doesn't say it is sampling the data.  When GA says it is sampling the data, the variance between reports can be pretty large.  Alternately, hook up the GA e-commerce tracking to your website and compare how many transactions GA gets to how many your e-comm platform says you actually received.  You will often see that GA misses about 15% of the transactions.  Drill down on some of the transactions in GA and you will find garbage transactions.

I picked on Google Analytics, but you can find similar limitations in most of the tracking tools.  Let's not even get into attribution modeling.  I can barely explain to you what made me buy product A over product B.  How is a relatively simple attribution algorithm able to describe what drove my buying behavior?  It can't.

If you understand these limitations in today's tools, you realize that our tools are best used as directional indicators.  They cannot give you exact answers, but they sure can point you in the right direction.  Your brain must be fully engaged at all times - you cannot blindly trust the numbers that are spit out.  I often like to triangulate my results using a couple different methods or tools to increase my confidence in my measurements.

The bottom line is that if you have a solid analytics team, the chances are extremely slim that you have to worry about becoming a slave to data.  Whew, you can cross that off your list of things to worry about!

Are You Boiling the Ocean?

You have just been assigned a new project at work, and you think to yourself, "I am going to be more data driven on this project."  The first thing you do is start daydreaming about all the data you want.  Before you know it, you have a list of reports you want that is longer than War and Peace.  Heck, you don't what you are going to do with all the data, but you want it!  You are now officially boiling the ocean. Your analyst will get all the reports done about the same time as you boil the ocean.

A far more time efficient process is to be hypothesis driven.  You have a hunch about what is going on.  Why not leverage that intuition to reduce your decision making time dramatically? By creating a hypothesis about what the problem might be, you do two things.  First, you limit the data you need to collect.  Second, you pull data that you can take actions on.  Both make you go faster.

Let's say that you have been assigned to improve your account creation funnel.  You have a four step funnel and you have a hunch that step #3 is where you lose everyone.  You can postulate a hypothesis that says that you think that you are losing more people from step #3 than any other step in your funnel.  If the hypothesis is true, the implication is that you should focus on improving step #3 before you work on other steps (all other things being equal).  You now have a fairly straightforward analysis that your analyst should be able to bang out in Google Analytics or a similar tool.

Let's say that your analysis comes back the way you expect.  You now can formulate a hypothesis about what is wrong on step #3.  For example, I think that customers are getting confused about what to input into field XYZ.  You can now use a tool like Mouse Stats to see if customers are pausing longer on field XYZ or exiting on that field.  If they are, you know where to focus your actions.

At the end of the day, making rapid improvements to your site or app is the name of the game.  Boiling the ocean is your worst enemy when trying to make rapid improvements.  Let hypotheses guide your actions and forever be more efficient!

How Grow Can You Go?

More than ever, a company's growth rate drives its ability to raise money, at least when the company is an early to mid-stage tech company.  In the SaaS world, a growth rate below 50% means it will be virtually impossible to raise money.  At above 150% investors will practically fall over themselves to give you money. 

Because growth is so important, the ability to forecast a company's growth is critical.  A good growth model sets expectations and drives the company's strategy. Yet, this forecasting is often done with a wing and a prayer and perhaps a little bit of historical data (which often is barely relevant when looking forward). A better way to forecast growth is to look at your baseline growth rate and then think about the major things that could impact that growth.  The main drivers of the model typically are:

• Baseline growth rate.  If you shut off all your paid marketing and left everything status quo, what would happen to your business?  This usually is a good measure of your virality minus your churn.  In other words, how many customers do you get from word of mouth minus how many do you naturally lose through attrition each year?  If you have ever had a period where you didn't spend a lot on marketing, you may have come close to this baseline growth rate.
•  Change in revenue in per user (aka ARPU).  This can include raising prices, selling customers a higher quantity if applicable, up-selling customers on higher cost products, and cross-selling customers on other product lines.  Raising prices is always a little difficult for early to mid-stage companies, but up-selling and cross-selling are often very viable strategies for this stage company, especially if you can regularly create new products.
• Improving the conversion / close rate.  In a B2C environment you should be regularly running A/B experiments to improve your conversion rates.  In a B2B environment that involves sales reps, you can make tweaks to your sales process to improve your close rate.  Your success in these areas will often depend a lot on how far you are down the road.  For example, at TeamSnap we have run thousands of A/B tests.  As such, improvements in conversion rates are much harder to find at this point.
• Increase your paid marketing / sales team.  If you know your cost to acquire a new customer (CAC), you can always see how many extra customers you can acquire if you up your marketing budget or increase your sales team.  However, there are diminishing returns to this strategy.  CAC can grow as you ramp up spend, so it is best to plan for CAC to rise with spend.
• Increase virality.  If you can increase your virality through enhancements to your product, this can be one of the most powerful ways to jumpstart growth at minimal cost.  For a lot of companies there are a lot of things they can do to increase virality.  However, forecasting how much virality will increase is not an easy task.
• Reduce churn.  Growing your business with a high churn rate is like putting out a fire with a leaky bucket.  In the early days this is an area that can bear a lot of fruit.
• M&A.  Do you plan to acquire any companies this year?  Acquisitions can increase traffic, your customer base and/or revenues.

While some of these drivers of growth are harder to forecast than others, the most important part is to take a stab.  Having growth targets for each of the above drivers will help you set your strategic priorities. If most of your growth is going to come from one or two drivers above, you need to make sure your team is focused on those items and that you are closely measuring your success in those areas.

Do You Really Understand Virality?

Everyone loves to say that their product is viral, but is it?  I used to work in the online photo space where everyone said that their product was "incredibly viral" because they let customers publish photos to Facebook and share photos via email.  However, most of the online photo sites didn't do the analytics to prove if there was any virality from this photo sharing.  Heck, many of the sites weren't even checking to see if anyone was sharing their photos.

Let's look at how you test the hypothesis that your product is viral.  We'll stay with the online photo example.  In this case to confirm your product is viral, you would need to:

1. Measure how often people are sharing via email and social media.  You might do this by building tracking into your platform.  In other words, record in your database when a photo is shared.
2. Ensure that customers can easily order photos that have been shared.  This is basic usability blocking and tackling.
3. Track all customers who click on those shared photos and come back to your site.  I personally like Google's UTC tracking links in Universal Analytics.
4. Set up a mechanism to track how many of those visits turned into orders.  The e-commerce tracking module in Universal Analytics is a powerful tool for doing this.
5. Track the lifetime value of these customers.  Do these customers just order the shared photos or do they actually place new orders with their own photos?  To do this, you will probably need to merge together data from your web analytics platform with your transactional back-end database.

Once you have completed these tasks, you should have a decent picture of whether anyone is sharing, and if so, whether it is helping at all.  The next step is to build a customer funnel. Where are customers dropping off?  There are a likely one or two places where you are losing a lot of folks.  Work to optimize those places.

You also can build a virality model from this data.  I often refer people to "Lessons Learned - Viral Marketing" by David Skok as a starting point.  Not only do you want to compute your viral coefficient, but more importantly, you want to find out the time delay.  How long does it take from sharing to the time people place their orders?  The time delay has a far more powerful effect than the viral coefficient.  That means that optimizing your funnel to increase your conversion rates isn't as important as making sure that those people who convert do so quickly.

The next time you start to say that your product is viral, challenge yourself to walk through these steps.  See if you have done the homework to really prove that you have a viral product.