Can You Become a Slave to Data?

In a recent discussion with my colleagues about data-driven decision making, one of them asked, "Can you become a slave to data?  In other words, can you become so focused on the numbers that you ignore common sense and make bad decisions?"  It was a great question.  After a little reflection, I responded that yes in theory you can be a slave to data.  However, in the real world I struggle to think of companies that truly have become slaves to data.

On a scale of 1-10 (where 1 means totally winging it and 10 means being completely data driven), most companies are much closer to 1 than 10.  When other Internet professionals ask for my advice on issues they are facing, I am often shocked by how little analysis they have done.  I often will say something like, "That is a good issue.  I bet looking at metrics X, Y, and Z will tell you your answer.  I am sure you have looked at X, Y, and Z, right?"  Usually I am greeted by a long pause and a lot of foot shuffling.  If an average company is a 2 or 3 on the data driven scale, they can easily move up a number or two (or more) and not become slaves to data.

Secondly, if you are using a hypothesis driven approach as I talked about in my last blog post, you already are bringing intuition and common sense into the equation.  You are only running analyses on things that you have already thought about.

Lastly, if you really understand the limitations of the common analytical tools, you know that there is a pretty large margin of error built in.  Let's take the free version of Google Analytics (GA).  Try running a report one way and then run the same report a different way.  The results are often a little different, even when GA doesn't say it is sampling the data.  When GA says it is sampling the data, the variance between reports can be pretty large.  Alternately, hook up the GA e-commerce tracking to your website and compare how many transactions GA gets to how many your e-comm platform says you actually received.  You will often see that GA misses about 15% of the transactions.  Drill down on some of the transactions in GA and you will find garbage transactions.

I picked on Google Analytics, but you can find similar limitations in most of the tracking tools.  Let's not even get into attribution modeling.  I can barely explain to you what made me buy product A over product B.  How is a relatively simple attribution algorithm able to describe what drove my buying behavior?  It can't.

If you understand these limitations in today's tools, you realize that our tools are best used as directional indicators.  They cannot give you exact answers, but they sure can point you in the right direction.  Your brain must be fully engaged at all times - you cannot blindly trust the numbers that are spit out.  I often like to triangulate my results using a couple different methods or tools to increase my confidence in my measurements.

The bottom line is that if you have a solid analytics team, the chances are extremely slim that you have to worry about becoming a slave to data.  Whew, you can cross that off your list of things to worry about!

Are You Boiling the Ocean?

You have just been assigned a new project at work, and you think to yourself, "I am going to be more data driven on this project."  The first thing you do is start daydreaming about all the data you want.  Before you know it, you have a list of reports you want that is longer than War and Peace.  Heck, you don't what you are going to do with all the data, but you want it!  You are now officially boiling the ocean. Your analyst will get all the reports done about the same time as you boil the ocean.

A far more time efficient process is to be hypothesis driven.  You have a hunch about what is going on.  Why not leverage that intuition to reduce your decision making time dramatically? By creating a hypothesis about what the problem might be, you do two things.  First, you limit the data you need to collect.  Second, you pull data that you can take actions on.  Both make you go faster.

Let's say that you have been assigned to improve your account creation funnel.  You have a four step funnel and you have a hunch that step #3 is where you lose everyone.  You can postulate a hypothesis that says that you think that you are losing more people from step #3 than any other step in your funnel.  If the hypothesis is true, the implication is that you should focus on improving step #3 before you work on other steps (all other things being equal).  You now have a fairly straightforward analysis that your analyst should be able to bang out in Google Analytics or a similar tool.

Let's say that your analysis comes back the way you expect.  You now can formulate a hypothesis about what is wrong on step #3.  For example, I think that customers are getting confused about what to input into field XYZ.  You can now use a tool like Mouse Stats to see if customers are pausing longer on field XYZ or exiting on that field.  If they are, you know where to focus your actions.

At the end of the day, making rapid improvements to your site or app is the name of the game.  Boiling the ocean is your worst enemy when trying to make rapid improvements.  Let hypotheses guide your actions and forever be more efficient!

How Grow Can You Go?

More than ever, a company's growth rate drives its ability to raise money, at least when the company is an early to mid-stage tech company.  In the SaaS world, a growth rate below 50% means it will be virtually impossible to raise money.  At above 150% investors will practically fall over themselves to give you money. 

Because growth is so important, the ability to forecast a company's growth is critical.  A good growth model sets expectations and drives the company's strategy. Yet, this forecasting is often done with a wing and a prayer and perhaps a little bit of historical data (which often is barely relevant when looking forward). A better way to forecast growth is to look at your baseline growth rate and then think about the major things that could impact that growth.  The main drivers of the model typically are:

• Baseline growth rate.  If you shut off all your paid marketing and left everything status quo, what would happen to your business?  This usually is a good measure of your virality minus your churn.  In other words, how many customers do you get from word of mouth minus how many do you naturally lose through attrition each year?  If you have ever had a period where you didn't spend a lot on marketing, you may have come close to this baseline growth rate.
•  Change in revenue in per user (aka ARPU).  This can include raising prices, selling customers a higher quantity if applicable, up-selling customers on higher cost products, and cross-selling customers on other product lines.  Raising prices is always a little difficult for early to mid-stage companies, but up-selling and cross-selling are often very viable strategies for this stage company, especially if you can regularly create new products.
• Improving the conversion / close rate.  In a B2C environment you should be regularly running A/B experiments to improve your conversion rates.  In a B2B environment that involves sales reps, you can make tweaks to your sales process to improve your close rate.  Your success in these areas will often depend a lot on how far you are down the road.  For example, at TeamSnap we have run thousands of A/B tests.  As such, improvements in conversion rates are much harder to find at this point.
• Increase your paid marketing / sales team.  If you know your cost to acquire a new customer (CAC), you can always see how many extra customers you can acquire if you up your marketing budget or increase your sales team.  However, there are diminishing returns to this strategy.  CAC can grow as you ramp up spend, so it is best to plan for CAC to rise with spend.
• Increase virality.  If you can increase your virality through enhancements to your product, this can be one of the most powerful ways to jumpstart growth at minimal cost.  For a lot of companies there are a lot of things they can do to increase virality.  However, forecasting how much virality will increase is not an easy task.
• Reduce churn.  Growing your business with a high churn rate is like putting out a fire with a leaky bucket.  In the early days this is an area that can bear a lot of fruit.
• M&A.  Do you plan to acquire any companies this year?  Acquisitions can increase traffic, your customer base and/or revenues.

While some of these drivers of growth are harder to forecast than others, the most important part is to take a stab.  Having growth targets for each of the above drivers will help you set your strategic priorities. If most of your growth is going to come from one or two drivers above, you need to make sure your team is focused on those items and that you are closely measuring your success in those areas.

Do You Really Understand Virality?

Everyone loves to say that their product is viral, but is it?  I used to work in the online photo space where everyone said that their product was "incredibly viral" because they let customers publish photos to Facebook and share photos via email.  However, most of the online photo sites didn't do the analytics to prove if there was any virality from this photo sharing.  Heck, many of the sites weren't even checking to see if anyone was sharing their photos.

Let's look at how you test the hypothesis that your product is viral.  We'll stay with the online photo example.  In this case to confirm your product is viral, you would need to:

1. Measure how often people are sharing via email and social media.  You might do this by building tracking into your platform.  In other words, record in your database when a photo is shared.
2. Ensure that customers can easily order photos that have been shared.  This is basic usability blocking and tackling.
3. Track all customers who click on those shared photos and come back to your site.  I personally like Google's UTC tracking links in Universal Analytics.
4. Set up a mechanism to track how many of those visits turned into orders.  The e-commerce tracking module in Universal Analytics is a powerful tool for doing this.
5. Track the lifetime value of these customers.  Do these customers just order the shared photos or do they actually place new orders with their own photos?  To do this, you will probably need to merge together data from your web analytics platform with your transactional back-end database.

Once you have completed these tasks, you should have a decent picture of whether anyone is sharing, and if so, whether it is helping at all.  The next step is to build a customer funnel. Where are customers dropping off?  There are a likely one or two places where you are losing a lot of folks.  Work to optimize those places.

You also can build a virality model from this data.  I often refer people to "Lessons Learned - Viral Marketing" by David Skok as a starting point.  Not only do you want to compute your viral coefficient, but more importantly, you want to find out the time delay.  How long does it take from sharing to the time people place their orders?  The time delay has a far more powerful effect than the viral coefficient.  That means that optimizing your funnel to increase your conversion rates isn't as important as making sure that those people who convert do so quickly.

The next time you start to say that your product is viral, challenge yourself to walk through these steps.  See if you have done the homework to really prove that you have a viral product.

Top Customer Analysis - You Probably Are Doing It Wrong

When marketing professionals think about their best customers, they often do the same boring analysis. They pull a list of their customers who have spent the most over some time period and call the folks at the top of the list their best customers. They then look at the demographics and a few other traits of those customers and declare victory. However, especially in an online world, this misses a lot of really interesting insights. If you stop with your analysis at this point, you are only about 5% done.

A natural extension of the analysis mentioned above is to factor in customer acquisition costs (CAC) and marketing source. You already calculated how much the customers spent - extend that to calculate Lifetime Value (LTV) of those customers. (You will need to know your churn to compute LTV.) If you have the analytical tools to pull the CAC for these top spenders, you can generate a ratio of LTV to CAC and segment the data by marketing source. What marketing sources have the best LTV to CAC ratio for these top spenders? What does that tell you about marketing activities that were really effective at generating top customers?

The next step is to broaden the analysis beyond just the top spenders. Look at LTV to CAC for all customers acquired by marketing segment. Which marketing segments are most effective using this lens? Until now, the analysis has assumed no virality. But most products, especially online properties, have at least some virality built into them. If you haven't looked at virality before, here is a great primer by David Skok.

Now instead of just tallying up your best customers in terms of spend, look at those customers who shared your product the most. Better yet, see if you can link those shares back to actual new customers acquired. If you can, keeping going - you aren't done yet! Look at those downstream customers acquired by the sharing and see how much they spent. Let's take those dollars and attribute them back to the customers who did the sharing. In other words, we are going to look at which customers spent the most AND how much money you got from their sharing activity. Once again take those top customers and compare the value they brought you to their CAC and segment the data by marketing program.

Whew! If those analyses seem fairly complex, you are right. These are going to take you a while. Even with the best of tools and people, this isn't going to be a quick process. However, if you can complete these analyses, you will likely have a totally new perspective on where you should be focusing your marketing efforts.

Don't Forget about Demographic Data in Universal Analytics

Google has slowly added demographic information to Universal Analytics (formerly Google Analytics).  Because of this phased rollout by Google, many marketers have forgotten to enable this capability.  It is a simple yet highly important setting in Universal Analytics.
Setting up demographic tracking is very straightforward.  In Universal Analytics you navigate to Admin and then Property Settings.  Then enable the slider for "Enable Demographics and Interest Reports." 
 

Universal Analytics Property Setting for Demographics Data

From there you will need to make a change to your Universal Analytics code deployment.  I am a big fan of deploying the Universal Analytics code (and any other tags you have) via Google Tag Manager (GTM).  In GTM to enable demographic data collection for your Universal Analytics account you just need to check a box under the main settings page for Universal Analytics.

Demographics Setting in Google Tag Manager

As in the case of most Universal Analytics changes, you need to exercise a little patience. It typically takes a day or two for the data to start to populate.

Now that you have the data capture working, head over to the audience reports in Universal Analytics.  

Demographics Reports in Universal Analytics

You can drill down by age and gender.  Assuming you enabled the E-Commerce Capabilities (shame on you if you didn't), you should be able to see how your conversion rate compares for men versus women and by age group.  

It is important to realize there is likely a healthy margin of error in these reports.  One of the more common issues is that people share devices which can wreak havoc on these types of reports.  As such, it is good to double check the data against information you capture directly from your users.

Once you dig into the data, it is critical to feed the results back into your paid advertising efforts.  Let's say you discover that certain demographic segments are more inclined to purchase from you than others.  You will then want to adjust your bidding strategy on paid search, Facebook and wherever else you advertise.  If you have segments that perform very poorly you can shut them off entirely in paid advertising.  Alternately you can take a strategy that says you want to bid for all segments, but you are willing to bid much more for your best segments.

This is a very quick win so stop what you are doing and implement demographic reporting now.

Good Marketing People vs Bad Marketing People

I have been reading Ben Horowitz's book, "The Hard Thing about Hard Things" and have been inspired by a section where he describes what a good product manager does versus what a bad product manager does.  As such, I am taking my crack at what a good online marketing professional does versus what a bad online marketing professional does.

A good online marketing professional:

• Constantly looks for ways to do his or her job more efficiently without anyone ever telling him to do so.  S/he aggressively tests out new tools and implements them as they make sense.  S/he has a strong bias toward automating rote tasks.
• Presents conclusions and recommendations instead of just presenting a mountain of data, thereby making the receiver interpret all the data.
• Is fluent in analytics and can back up his/her conclusions and recommendations.  As the saying goes, "In God we trust - all others bring data."
• Is passionate about technology - s/he is always reading about new technologies that could impact his/her job, his/her product, and the world around him/her.  S/he never waits for someone to tell him to read up on a subject.
• Recognizes that the online marketing world doesn't fit neatly in a 9-5 world.  Sometimes the most interesting thing of the week will happen on Sunday morning, whereas things can be quiet  during normal business hours.  When it is quiet, go for the long bike ride you have wanted to go on.  When you strike gold on a Sunday morning, sometimes you have to buckle down and start digging right then.
• A/B tests everything in life down to how s/he makes his/her coffee in the morning. 
• Is intellectually curious.  When s/he spots some interesting data, s/he always digs into it to see if it could improve his/her performance.

A bad online marketing professional:

• Blindly follows what has been done in the past.  S/he never asks why it was done that way before.
• Acts like math is like cooties.
• Doesn't know how to lead people to conclusions.  S/he usually presents no data or enough data to overwhelm a mathematician.
• Is afraid to take risks.  S/he doesn't understand that what you do in online marketing this year cannot be what you did last year.
• Is scared of technology.  Let's face it.  Some parts of this job are technical - deal with it.