Friday, July 15, 2016

Bringing Data to the People

One of my big themes lately at TeamSnap has been "bringing data to the people."  Our approach has been to push broad swaths of the company to immerse themselves in the data through better tools and a lot more training.

As I have talked about before, we leverage Tableau to push analyses down to marketing, bus dev, and product people.  It allows relatively non-technical people to run sophisticated analyses without any knowledge of the underlying data.  What is really amazing is that the underlying data is often coming from multiple data sources in the cloud and is joined through some sophisticated data hygiene and migration techniques that these business users don't need to know about.

Over the past two days we pushed harder on a different technique to democratize our data - training.  We did an intensive 2 day training session on Google Analytics with Analytics Pros.  It was a huge commitment in money but more importantly in time.  We had a wide range of employees from across the company attend.  It was amazing to see people really get the religion about how much you can improve your product through better data and that you don't need to rely on the data scientist types to get you data.  Google Analytics is incredibly powerful, but also very usable by non-technical folks especially if they some training.

One of the key takeaways for a lot of people was how flexible Google Analytics is.  I often say that Google Analytics is a platform, not a tool.  It can be whatever you want it to be if you know all the ways it can be used. 

Upon the conclusion of the training, my main advice to most people was to find one or two ways to integrate what they learned into their daily routines.  If you manage social, build a report or a dashboard or a segment that drills down into social.  Have that report or dashboard show up in your email box on a regular basis so you can make decisions from it.  Then tweak that report or dashboard over time to improve how you manage social.  By taking these small, incremental steps to analyze the data yourself, you set yourself on the path toward data empowerment.

Are you investing enough in the analytical skills of your rank and file employees?

Saturday, May 7, 2016

Slicing and Dicing Your Way to Paid Advertising Nirvana

Digital marketing professionals who work in paid online marketing often get so focused on campaign and targeting level data that they forget to dig deeper and find the really interesting matches between those campaigns and their customer segments.  By doing so, they waste large amounts of their marketing budget. 

Let's say that you are running marketing campaigns in AdWords.  You almost certainly will be looking at your CPA (cost per action) at a campaign, keyword, and a device level (desktop, tablet or mobile).  If you know what you are doing, you hopefully have also thought about whether you want to target men or women and what age groups you want to go after.  In addition, with luck you thought about your geo targeting as well as whether there are certain times of the week when your product is going to sell better.

The next step, and it is a big one, is to link AdWords to your back end database so you can track campaigns and keywords all the way through to what customers spend over time.  How much do you spend on a given keyword compared to the LTV (lifetime customer value) you get from customers coming in on that keyword?  Were these new customers or existing customers coming back again?

Now most folks are thrilled to just get this far (and only about 5-10% of the folks I speak with make it this far).  However, the reality is that the fun is just starting at this point.  On the back-end you probably have additional information about your customers.  For example, their industry vertical, the product they purchased, their role, etc.  So you now have the following dimensions to slice and dice your data by:
  • Campaign
  • Keyword
  • Ad
  • Device type
  • Geo
  • Gender
  • Age
  • Time of day / day of week
  • New customer vs returning
  • Industry vertical
  • Product purchased
  • Role
Here is where the fun begins.  Most people look at the dimensions one at a time.  They optimize the ROI for a given keyword or they optimize the ROI for a given campaign.  However, the real efficiency gains can be had from analyzing multiple variables at once.  Maybe certain keywords only work with certain devices at certain times of the week for certain age groups.  Maybe certain keywords play better for existing customers who are interested in a specific product whereas other keywords work well with new customers in a given industry.  If you haven't looked at the data at this level, you are missing key insights.

When you start dicing up your marketing spend at this granular a level, you start to realize that have an enormous amount of fat in your marketing spend.  You will find certain combinations of the above dimensions where the ROI is extremely poor and you will find other segments where the ROI is excellent.  If you take this approach, you often realize that 10-25% of your marketing budget is completely ineffective.  Rebalancing the budget to double down on the most impact programs can have a huge impact.

Analyzing your data at this level definitely isn't easy.  This approach takes a tremendous amount of work to set up and process.  However, think about it this way.  If your manager was to come to you and offer you an increase of 10-25% in your paid marketing budget, you would probably be ecstatic.  If you slice up your data enough to find 10-25% fat in your budget, you are essentially getting a lot of extra money to spend each month.

Wednesday, May 4, 2016

Wrangling the Google Analytics Sampling Beast

After reading my last post, you have determined that you are really struggling with sampling in Google Analytics.  Now what do you do?

Your first option is to stick to the main reports on the left hand rail of Google Analytics.

The reports under these main categories rely heavily on roll-up tables in Google Analytics and therefore generally do not have sampling.  However, once you start adding secondary dimensions, filters, or custom segments, all bets are off and your sampling is likely to come roaring back.  While this can be very limiting, using these main reports is an option to remember.

A second option is to reduce the amount of data you are looking at, particularly by reducing your date range.  On the positive side this is a quick and easy way to deal with sampling.  The negatives are that this is really time consuming if you want to look at a large date range.  Second, you have to reduce your date ranges as your site grows.  Lastly, for some queries (e.g., unique users within say a month or a year), you may not be able to use this trick.  If you are looking at monthly unique users, you cannot chop the month into two date ranges because you don't know how much overlap you have between the two periods.

A related option is to use Analytics Canvas.  This tool basically takes the date range trick and automates it for you.  You can tell it to take a Google Analytics report for the past year and split it into 365 separate reports, each one day in length.  This is highly effective in dealing with sampling but again doesn't work for metrics like unique users.

The most comprehensive solution for dealing with sampling is upgrading to Google Analytics Premium.  There is no doubt about it - it is an expensive product at $150,000 per year (as of the time of this article).  In addition, to really exploit the data in GA Premium, you need to learn BigQuery, Google's Big Data tool.  If you know SQL, this isn't too hard, but there are some quirks for sure.  All these cons aside, the data that you can get out of GA Premium is amazing.  GA stores your data at a very granular level behind the scenes.  If you have worked with the GA API before, you will have seen a taste of what GA stores although the BigQuery interface to GA Premium exposes a far more extensive set of data that GA is collecting.  It can take some time to wrap your head around the data, but it gives you an almost limitless number of analyses you can run about your customers.

Sampling in Google Analytics can be a real drag, but now hopefully you will be able to wrangle that sampling beast into submission.

Thursday, April 14, 2016

Using the Presidential Election to Understand Google Analytics Sampling

When I coach people on how to use Google Analytics, I see one issue come up over and over - not understanding sampling.  In many cases Google Analytics will use a sample of your data instead of all your data when you run a report.  Is that bad?  The answer is not necessarily.  However, you need to understand sampling to be aware of the potential impact.

The way I explain sampling to folks is that when you see poll results for who is leading in the presidential election, the pollsters are only calling a sample of the 220 million eligible voters in the U.S.  They often only call several thousand people.  As long as those people are representative of the overall voter base in the U.S., you can accurately predict election results with a very small sample.

If you have a large site, you have a lot of data in Google Analytics.  Some reports would take a very long time to run if Google were to use all your data.  Hence the use of sampling.

One of the big issues with sampling is that Google doesn't tell you the statistical significance when it uses sampling.  Think back to the election polls.  If you read the fine print, you will almost always see something that says, "Margin of error for this poll is +/- x%."  Often the margin of error is just a few percent, and it gives you confidence in the results.  However, Google Analytics doesn't tell you the margin of error.  If you have an event that doesn't happen very often, the margin of error can be huge.  One of the first things that a lot of people notice when they have sampling is that their results are very inconsistent.  They run a report one day and then the next day and get very different results.  Similarly, if they pull the same data using two slightly differently approaches in GA and get very different results, they likely have a lot of sampling going on and a large margin of error in the results.

The first thing to check is how much sampling Google is doing.  If you see a message like this in the upper right hand corner of your report, you have sampling going on.

So how much sampling is bad?  A lot of people say that below 5-10% and you have a problem.  However, it really depends.  In the presidential polls, they only call a couple thousand people out of 220 million people.  That is ~0.001% sampling, well below the rule of thumb some people call for in Google Analytics.  However, in the election polls the statisticians can calculate the margin of error.

The margin of error depends a lot on how often the event you are measuring happens in GA.  For example, let's say you are looking at device category under the mobile overview.  Because there are only three choices in that report (desktop, mobile, and tablet), each session has a device, and each category shows up fairly often, you can have a low sampling percentage and you can still be confident in your results.

However, let's say that Google Analytics is using 10,000 data points in your report, and you are trying to estimate the frequency of a rare error message on your site.  Let's say your error only happens 1 in every million times although you don't know that yet.  The problem is that in this case GA is almost always going tell you that this event happens 0% of the time because the odds of the event happening in your 10,000 data points are very low (10,000 / 1,000,000).  When GA does see the event in your sampled data, it is usually going to say the event happens 1 in every 10,000 times because it has 10,000 data points and it sees one occurrence. However, both answers, 0 and 1 in 10,000, are way off from the truth.  Sampling is a huge problem in this case.

In the next post we will talk about your options if you suspect sampling is impacting your GA results.  Until then watch for the yellow sampling message in the upper right hand corner of your GA reports to see how much sampling you are experiencing.

Sunday, February 7, 2016

How Big Is Your Analytics Team?

As Chief Growth Officer at TeamSnap, I am often asked how big our analytics team is.  TeamSnap has about 10 million customers.  Those customers are highly engaged in the app so we have a lot of information about what is happening in recreational and competitive sports, especially youth sports.  We have 75 employees and are quite analytical as a company.  Yet, we only have one full time data analyst.  Instead we have taken a bit more of a distributed approach that I will describe.

Our basic philosophy is to get different people around the company engaged with the data.  We do that by creating simple interactive dashboards that show our progress against targets.  For example, in business development we have a dashboard that shows how we are doing in working with other partners to drive business to TeamSnap.  The dashboards are so closely tied to comp plans that we use the same data to figure out quarterly bonuses.  At the same time, the purpose of the dashboards is to allow people in those functions to interact and drill down on the data.

What we typically do is allow the end users to filter the dashboards by a variety of factors including geo, sport, type of customer, etc.  If our business development folks are working with soccer partners in Arizona, they can isolate our performance in soccer in Arizona.  Most of the analyses are actually much more complex and deeper than that example, but the interesting thing is that they are being run by business folks who are totally abstracted from the underlying data.  They are just using a simple point-and-click dashboard.

By empowering the team with these interactive dashboards, we tend to pull out far more conclusions from the same sets of data.  We have a lot of eyes on the data and different folks approach the data different ways and ask different questions.  I often tell the team, "With these dashboards you can slice the data almost an infinite number of ways so your findings may vary." Sure enough the team often teases out all kinds of insights that we never would have seen with a more top-down driven approach where the analytics team spoon fed the data to the business team.

Several of our most analytical people in the company are ones who had little to no prior experience in digital analytics.  Many of them came out of sports backgrounds or creative professions.  It always puts a smile on my face when someone who does not have a formal data analytics background incorporates some great analyses into their work.

The vast majority of our dashboards are created in Tableau.  We love the way it dynamically syncs to various data sources and presents the data to users in a way that is comfortable for anyone who has used Excel before.  While Tableau isn't cheap, it was a much better fit for us than the half a dozen tools we tried before.

So I circle back to my question of how big our analytics team is.  Our goal is to think of our analytics team not as our one data analyst, but rather as our entire employee base. We want every employee teasing out great insights from the data and integrating data into what they do.

Tuesday, September 15, 2015

Want to Improve Your Paid Ads? Get out of Your Ivory Castle!

Last blog post I argued that it was pretty hard to become a slave to data.  While I strongly believe that, I also believe that sometimes you have to put your analytical tools aside and get out into the real world.  Leave your ivory castle!

Paid advertising teams often are very analytical folks.  They speak in the language of CTRs (click through rates), CPAs (cost per acquisitions), view through conversions (conversions that came from folks who saw ads but didn't click on them), etc.  Many spend their days analyzing numbers, moving money across keywords, ads, campaigns, etc.  I too spend a tremendous amount of time looking at these metrics and often say that my job is akin to being a portfolio manager.  I look at numbers much of the day and move money around between projects to optimize our spend.

However, I feel strongly that marketing folks, and in particular paid advertising types, need to go out into the real world and mix it up with customers.  One of my favorite ways to do that is to go to tradeshows.  I try to go to roughly only show a month.  People (especially my wife) often ask me, "Why are you going to so many shows?  Don't you have people who can run the shows for you?"  Of course I have very competent team members who can run the shows.  However, I go to listen to our customers.

I go to the booth and ask questions.  I often don't do a lot of talking myself.  Here are some of the questions I ask:

To Existing Customers:
  • What do you like best about the product?
  • How did you find out about us?
  • What other solutions did you consider?
To Potential Customers:
  • What are the biggest challenges you face today?
  • What options are you considering to address those challenges?
  • How are you researching those options?
My favorite scenario is to have one or more existing customers in the booth try to sell some potential customers.  This happens a lot with TeamSnap because customers generally love the product and are pretty vocal about singing our praises.  I just sit back and listen to how the existing customers sell the potential customers.  They often can really crisply describe the benefits of the product.  Best of all, they use language that potential customers understand. They don't have the problem of being too close to the product.

During these types of interactions, I just sit there and take notes.  After the show I huddle with the folks working with me on our paid ad programs.  We almost always come up with new campaigns for AdWords, Facebook, etc.  As well, we typically come up with alternate versions of existing ads that more succinctly describe the product benefits.  These new campaigns and creatives generate lots of new analyses and learnings.

The more varied your customer base, the more you need to get out and meet folks.  TeamSnap is used in over 100 different sports.  As well, we have customers in 196 countries.  As such, there are a ton of ways customers use the product.  An adult hockey team in Vancouver has very different needs than say a youth travel soccer team in L.A. or a high school rugby club in London.   You cannot pick up those sort of differences from behind a desk - sometimes there is just no substitute for leaving the ivory tower.

Monday, August 31, 2015

Can You Become a Slave to Data?

In a recent discussion with my colleagues about data-driven decision making, one of them asked, "Can you become a slave to data?  In other words, can you become so focused on the numbers that you ignore common sense and make bad decisions?"  It was a great question.  After a little reflection, I responded that yes in theory you can be a slave to data.  However, in the real world I struggle to think of companies that truly have become slaves to data.

On a scale of 1-10 (where 1 means totally winging it and 10 means being completely data driven), most companies are much closer to 1 than 10.  When other Internet professionals ask for my advice on issues they are facing, I am often shocked by how little analysis they have done.  I often will say something like, "That is a good issue.  I bet looking at metrics X, Y, and Z will tell you your answer.  I am sure you have looked at X, Y, and Z, right?"  Usually I am greeted by a long pause and a lot of foot shuffling.  If an average company is a 2 or 3 on the data driven scale, they can easily move up a number or two (or more) and not become slaves to data.

Secondly, if you are using a hypothesis driven approach as I talked about in my last blog post, you already are bringing intuition and common sense into the equation.  You are only running analyses on things that you have already thought about.

Lastly, if you really understand the limitations of the common analytical tools, you know that there is a pretty large margin of error built in.  Let's take the free version of Google Analytics (GA).  Try running a report one way and then run the same report a different way.  The results are often a little different, even when GA doesn't say it is sampling the data.  When GA says it is sampling the data, the variance between reports can be pretty large.  Alternately, hook up the GA e-commerce tracking to your website and compare how many transactions GA gets to how many your e-comm platform says you actually received.  You will often see that GA misses about 15% of the transactions.  Drill down on some of the transactions in GA and you will find garbage transactions.

I picked on Google Analytics, but you can find similar limitations in most of the tracking tools.  Let's not even get into attribution modeling.  I can barely explain to you what made me buy product A over product B.  How is a relatively simple attribution algorithm able to describe what drove my buying behavior?  It can't.

If you understand these limitations in today's tools, you realize that our tools are best used as directional indicators.  They cannot give you exact answers, but they sure can point you in the right direction.  Your brain must be fully engaged at all times - you cannot blindly trust the numbers that are spit out.  I often like to triangulate my results using a couple different methods or tools to increase my confidence in my measurements.

The bottom line is that if you have a solid analytics team, the chances are extremely slim that you have to worry about becoming a slave to data.  Whew, you can cross that off your list of things to worry about!