Measuring product usage is a challenge which stumps many companies. I've experienced first hand the frustrations of product managers who had no idea how their features were performing.
In this guide I'm going to cover everything you'll need to know to accurately measure product usage. It doesn't matter if you're a one-man shop or part of a growing startup, this guide will help you implement a framework that will help you answer all your product usage questions.
What are our goals?
Before we dive into the details it's a good idea to understand what we want to achieve.
As I see it we have two main goals.
Usage tracking
The first goal is to implement the usage tracking framework. This framework will allow the company to accurately track product usage as it scales.
The product, R&D and operations teams will have joint responsibility on implementing, optimizing and maintaining this framework.
The usage tracking framework should be implemented as soon as possible in the company's history to help prevent data gaps as you scale and optimize your product. Each data gap is a major barrier to answering product usage questions and will add a lot of frustration to both your product and analyst teams.
Analysis and reporting
Once the usage tracking framework is put into place we can start setting goals related to turning raw data into business insights. This responsibility falls on the operations teams and involves the following:
- Answering mission critical questions (this is a high priority goal)
- Continuously discovering useful insights
- Providing self-service reporting to key stakeholders in the organization
- Help to monitor usage of the product
The Usage Tracking Framework
Online products are consistently changing. New features are being added and existing features are evolving frequently. In order to guarantee a consistent flow of quality, "BI-friendly" data you'll need to implement the usage tracking framework.
To implement this framework, follow the steps below.
Step #1 - Run a data audit
I recommend that you go and read my detailed guide on how to run a data audit for your startup. The audit will help you map out the gaps that you have in your tracking infrastructure, and the amount of work needed to start filling in these gaps.
Create a Google Spreadsheet which lists all the gaps you have and work with R&D to help you enrich your spreadsheet with extra info that will be helpful when you present the gaps to the leadership. The more info you have the easier it will be to estimate the work needed to fill in the gaps.
Running the data audit will also help you better understand R&D's current process for implementing product usage tracking.
Step #2 - Understand the data that you need
In order to accurately measure product usage you will need to get your hands on 2 types of data, dimensional and event-based.
What is dimensional data?
Dimensional data is information about our users and other objects that are important to the business. This information is constantly changing and always represents the most up to date version of itself.
Below is an example of 3 rows in a typical dimension table. In this example it happens to be a "users" table.
The biggest challenge with dimensional data is that it's only a snap shot in time, it is not historical.
Let's say we save the date our users turn on feature X. We would see something like this:
activated_feature_X_at
23/10/2018 01:03:15 AM
At first glance there doesn't seem to be any issues with this but remember, this is the last time the user activated this feature. What if the user turned this feature on and off 5 times over the last 6 months? We would have no way to identify when the user activated the feature for the first time.
What if we save both the first and last time this action happened? This is better than the example above but it still prevents us from answering questions which we may want to answer later. An example would be, how long does it take a user to preform this action 10 times, or how many times does a user activate and deactivate features within the first 6 months after creation?
The best kind of data you can save for the purpose of measuring usage is event-based data.
What is event-based data and how do I collect it?
Event-based data is information that is recorded for specific actions that we deem importnat. Event data has a few key components which are important to understand.
- Event name - Simply the name of the event. This may be "Signed Up" or "Page Viewed"
- Timestamp - When the event happened. This timestamp should be reflected in UTC time which is a standard.
- Event properties - Additional information sent with each event to help us understand what exactly took place. Examples of properties which may be sent with the "Page Viewed" event include "pageTitle", "url", "pageCategory". Event properties can be either custom or provided out-of-the-box by whichever service you are using for your tracking. Custom properties are properties which you've decided to send with your events. Custom properties will make up the majority of the properties which are being recorded.
- Trigger - Clearly defined circumstance, or set of circumstances which triggered the event.
Most companies first get a taste of event-based data when they start using tools like Mixpanel, or Kissmetrics. These tools help companies record events which are then used to populate out-of-the-box reports. Another service I highly recommend is Segment. There are a few nuanced issues with these tools but the biggest issue I've found is cost. Both Segment and Mixpanel have raised their prices in the last 12 months and even though they offer great free plans, it is a big ask for most early stage companies to move to a paid tier.
Let's now look at a few examples of events.
Event Example #1 - Signed Up
- Event name - Signed Up
- Timestamp - X (X is equal to when the event is triggered)
- Event properties
- businessType = X (the value the user selected in the signup form for the field, "Type of business", this is an example of a custom property)
- signupEmployeeCount = X (the value the user selected in the signup form for the field, "Number of employees")
- signupPagePath = X (the path of the url of the page where the user signed up. This would be useful to send if the business had multiple signup pages)
- Trigger - Sent once per user. Sent when the user finishes the signup process.
The "Signed Up" event is a classic example of an event which most businesses send.
One obvious question behind this event is why would we send it when we are already saving this information in our database for when a user is created.
There are a few reasons why I'd recommend sending this event. The first is because typically the first reporting a business will do on its' events will be through a tool like Mixpanel. For my startup, Feedio, I use Mixpanel and can easily track the number of signups to the service through their segmentation report. The alternative would be for me to run queries on our database which is more time consuming and carries unnecessary risk. A second reason is to remove the need to join our event data with another data source to get a key user dimension, signup date. Since I'll have the milestone among our event data, I can use it for cohort analyses or funnels within Mixpanel or whichever tool I'm using to record events.
Event Example #2 - Feature Enabled
- Event name - Feature Enabled
- Timestamp - X (X is equal to when the event is triggered)
- Event properties
- feature = X (The name of the feature)
- url = X (The url of the page where the event was triggered. This is important if users can enable the same feature on numerous pages.)
- path = X (The path of the page where the event was triggered. This is important because it allows us to group by the page, instead of having to deal with numerous versions of the url. In short, this is a cleaner version of the url.)
- featureId = X (The id of the feature in the database. This is optional but could be useful at a later stage once the number of features grows and the infrastructure is more ideal for complex analyses.)
- Trigger - Sent each time a user enables a feature. The event should be sent only when a user enables a feature. There will be a separate event for when a user disables a feature.
Feature enabled is a critical event and should be released along with any new feature. The event's twin is "Feature Disabled", which will be sent when a user disables a feature.
How to use event data to understand product usage
Good job, you've got a list of events being recorded and now it's time to start analyzing product usage. At this stage you have 3 options.
Option #1 - Out-of-the-box reporting
If you're using a tool like Mixpanel you'll be able to answer a wide range of questions with their built in reporting. My favorite reports are the segmentation, funnel and cohort reports. There are a few nuanced challenges with these reports but all-in-all they are great.
Option #2 - Export and Excel / Tableau
The next option you have is to export your event data via one of the built-in reports, the tools API, or through a support ticket, and then use Excel, Tableau or some other data visualization tool to run the analysis.
This method is very manual, time consuming and frustrating but it will allow you to do some more advanced analyses which won't be possible with the built-in reports.
Note that you may need to run numerous exports and blend the data before you'll be able to start your analysis.
Option #3 - BI Infra
The third option is to invest in some BI infrastructure and move the event data into your own database. You have a few options here including building your own custom ETLs, using a tool like Stitch to move the data, or to use Segment which offers data pipelines as part of its' services.
This approach is the most costly but in the long-run you will be able to get a far higher return from your data infrastructure than the two other approaches.
Step #3 - Introducing the usage tracking process
The usage tracking process is an internal process for guaranteeing that no product change goes live without tracking being a part of the change.
The funnel has a few steps.
- Step #1 - A new section called "analytics" is added to your product requirement doc. The product manager or a product analyst will need to fill in this section (or indicate that no analytics are required) before the PRD is locked and passed to R&D. If you are using a tool like Jira or Trello then consider adding a step in your kanban board for writing up the analytics requirements for the task. Leverage rules or other UI tricks to make it tougher for tasks to move between product and R&D without analytics being added as a part of the task.
- Step #2 - R&D develop the feature and it is pushed to QA.
- Step #3 - The QA engineer checks that the data that we want to track is actually being sent to the relevant data silo. A stricter policy is to have the product manager or analyst sit with the QA engineer and QA the data flow together. The PM or analyst would need to sign off on the data flow before the change can be pushed live.
The idea behind this process is to put the responsibility of collecting data in the hands of multiple individuals. This makes it much less likely that changes are made to the product and relevant data is not collected.
Step #4 - Bring everything together and communicate the plan to the leadership
At this stage you should have a good idea of the data you need to record, the data gaps you have, and the company's approach to data collection.
At this point you want to organize all this information so it can be communicated to key decision makers in the organization. Once you're done organizing everything, schedule a meeting and share the following:
- The goal - To implement certain processes and infrastructure to allow for tracking and analysis of the product.
- The gaps and your plan to tackle them.
- The usage tracking process.
Infrastructural challenges and selling the dream
From my experience most early stage companies have dedicated very little resources and focus on collecting the data needed to accurately measure product usage. You will most likely find a long list of gaps that will need to be filled before an analyst can start answering even the most basic questions.
It will be your job to convince the leadership that scarce resources should be dedicated to this need. The best way to convince them is to explain the risk to the business in not being able to answer key questions on product usage. Also explain how frustrating it is that the analysts can't support the VP of product and his team with product-related insights.
Make sure you have a well thought out plan on the exact gaps you want to fill and their order of importance. Start with gaps which prevent you from answering mission critical questions (more on that below).
Convince the VP of Product to fight on your behalf and to make sacrifices so that everyone can get more insightful quicker.
Product Analytics 101
Let's assume that you've been smart with your data collection and invested in your BI infrastructure from the early days. You have more data on your users than you know what to do with it. Good job, we can now get to the fun part.
How do we prioritize the product questions we want answered?
The next step in our process is to come up with good product usage questions that we'd like to answer. This might sound simple enough but where do we start?
There is no perfect methodology for prioritizing questions but it does help to put your questions into one of the buckets I've listed below.
- Mission Critical - The answers to these questions will fundamentally shape the direction of the company. The business's very survival relies on understanding the answers to these questions.
- Medium Impact - These questions will help move the needle on key KPIs and help with strategic decision making. An example may be "what is common among users which subscribe to our most expensive plan?".
- Low Impact - These questions are "nice-to-knows" and may help shape the business for the better but the changes will be incremental.
Another way to prioritize questions is to ask yourself, "how many people will be impacted by the answer to the question"? The more people that are directly affected the higher the priority.
The best tip I can give you is to use your common sense and focus first on mission critical and medium impact questions before spending time on low impact questions.
How to answer a mission critical product usage question
Let's assume we were working as a product analyst at Zapier and our goal was to measure product usage in order to direct company strategy.
If you've never heard of Zapier, they are a B2B company that makes it easy for businesses to connect different tools to each other to create awesome automation flows. I'm a big fan and you should definitely check them out.
Let's look at an example of a mission critical product usage question that Zapier would want to answer.
Since Zapier relies on businesses converting from their free tier to paid, and then remaining as a paid user, I'd start with the "wow" funnel.
I define the "wow" funnel as the funnel that the user goes through in order to get to that "wow" moment. This is the bare minimum we expect from the product. From that point users should continue to see value and then upgrade.
I've included a visual below of how I see Zapier's "wow" funnel.
The first step in Zapier's "wow" funnel involves the user signing up. This is our starting point. The next step is for the user to create their first zap. A zap in Zapier is a connection between two tools, like for example Gmail and Google Sheets. Once a zap is created Zapier will automatically run the zap every 15 minutes (on the free plan, less on the paid plans) and if relevant, a task is completed.
Once that first task is completed the user experiences the power of the service and will be pleasantly surprised, hence the name, "wow" moment.
So how would I track this funnel as an analyst?
I'd focus on 3 key events which I've listed below.
- Event #1 - Signed Up
- Event #2 - Zap Created
- Event #3 - Task Completed
Once I've identified these events I can build out a funnel. I personally would use Tableau to do this work, assuming I could get my hands on the raw event data in a standard relational database. If not then I'd build out a funnel report in Mixpanel or look at goal conversions in Google Analytics.
Once I had access to the data I'd aim to build cohort views in either table or bar chart form.
The first view would include the raw numbers. This is important in order to determine if we have enough data to work with. Since Zapier is a mature company I'd imagine they have more than enough data to work with.
The second view would show the conversion rate along the funnel in relation to the first step. We can see from my example that in February 2018 56% of users reached the "wow" moment.
The last view would show the conversion rate between the different steps. This is critical to identify which step of the funnel is the most problematic. In our example we see that the big drop off happens between the first and second steps. If our focus was on optimizing this funnel, step 2 is where we'd spend our time.
The most accurate way to run such an analysis would be with a time constraint. You'd first need to determine the time constraint by running a separate analysis. In this analysis you'd take users which got to the end of the funnel and then group them by how long they took to reach each step in the funnel. I'd then recommend doing a running sum of the percentage of total to determine the 80% percentile.
You want to end up with an insight like "80% of users which reach the end of the funnel, reach that point within 120 hours (5 days)". A good tip is to use hours to normalize the data (some users signup later in the day).
So now that you have your time constraint you can plug that into your analysis. This makes the analysis more accurate since you're now comparing apples to apples. Before the time constraint users that are on the fringes in terms of time to convert will skew your numbers in earlier cohorts.
The biggest issue with time constraints is that it results in delays in determining the "health" of recent cohorts since it can take time for certain things to happen. This is just the nature of analytics and something everyone needs to live with.
Tools of the trade and self-service BI
There are a number of tools which are relevant when talking about measuring product usage.
I could write an entire post dedicated to this subject so I'm just going to list the tools I'd recommend exploring.
- Mixpanel
- Type: Event tracking & self service analytics
- Cost: Very generous free plan and then it starts at $1,000 a year
- Thoughts: Mixpanel is an amazing piece of software which should be seriously considered, especially for very early stage companies. Mixpanel not only helps with the collection of event-based data but also provides out-of-the box reporting.
- Segment
- Type: Event tracking
- Cost: Generous free plan, then starts at $120 per month.
- Thoughts: Segment is one of my favorite companies. Segment brings a massive amount of value to companies by providing event tracking libraries, out-of-the-box data pipelines to hundreds of tools and databases, and more.
- PowerBI
- Type: Data Visualization
- Cost: The viz builder is free, online hosting starts at $9.90 per user.
- Thoughts: The more I use PowerBI the more I like it. It is very affordable and in terms of feature set, they are continuously improving. PowerBI also works great with Redash.
- Tableau
- Type: Data Visualization
- Cost: Starts at $35 a month for the viz builder, and $42 a month for the online hosting.
- Thoughts: Tableau is my favorite BI tool and if you've got the budget you should seriously consider adopting it as the dashboarding piece of your BI stack.
- Other services to consider:
- Google Analytics and Google Data Studio
- Heap Analytics
- Amplitude
Frequently asked questions on measuring product usage
I have so much going on in my business, do I really need to spend valuable resources on measuring product usage?
Well think about it this way. What would the cost be to the business if you can't make an informed product decision and end up making the wrong choice? Is that a risk you can live with?
My advice would be to complete step #1 in the usage tracking framework, complete a data audit. Once the audit is complete you'll have a better idea where you stand and the amount of work that is needed to get a minimal set up in place.
Don't I need a dedicated analyst to help me implement the usage tracking methodology?
No you don't but I'd recommend having a dedicated individual who is responsible for this process.
For very early stage companies this may be the CTO or a senior developer. For later stage companies it could be the responsibility of a product manager or a business analyst. Go with the analyst if you have one.
I have unlimited capital I can invest in my product analytics, which BI stack should I set up?
I'd start by hiring a very strong senior business analyst. This individual will act as your project manager and team leader. Once his onboarding is complete let him read this blog post and get to work on the data audit.
If possible, shift some R&D resources around to start filling in the gaps the analyst has mapped out.
Next I'd signup with Tableau and get your analyst to start learning the tool. I'd look at Segment for my tracking and either Redshift or BigQuery as the tech for my data warehouse.
I'd bring on an outsourcing firm which can build my ETLs, hire an in-house data engineer or consider using a tool like Stitch. If you can do it in-house from day 1.
Depending on the size and type of business I'd start hiring strong business analysts.
If setting up your BI infra is a priority this is where you should be within 6 - 8 months:
- Team of 2 - 4 business analysts, 1 team leader.
- Core data silos sitting in your BI data warehouse (think product database/s, CRM, and event-tracking)
- First set of Tableau dashboards have been built and are being used by middle management and C-level. One of your analysts has the responsibility of Tableau server admin.
- Major data gaps have been filled and you have the usage tracking methodology in place.
Can I set up a fully supported self-service BI for my product usage needs?
Yes and no. In the early stages of the company there aren't any issues with using a service like Mixpanel or Google Analytics which will provide self-service reporting.
The issue with this approach is that it's very limiting and you won't be able to answer more complex questions.
Your managers will demand all kinds of things which you won't be able to deliver without having direct access to the raw data. You want complete freedom so that you can run on demand ad hock analyses and build custom data visualizations.
What are some of the product usage questions I'm not considering?
Time is a dimension which is often overlooked. Dedicate time to understand the time component in your business, especially when it comes to your core funnels.
Some examples to consider:
- How long does it take for a user which ends up on a paid plan, to subscribe to the paid plan?
- An extra tip is to map out how long it takes uses so that you can understand the distribution. The majority might take 7 days but the tail might take more than 90 days. This will result in an average which is misleading. Look at the curve and identify the 80% percentile.
- How long does it take a user to become "active"? What is special about the users which become "active" significantly sooner than the rest of the cohort?
- How long does it take a user to invite a colleague / activate feature X / reach milestone X?
The list goes on and on.
Retention and usage behavior are also critical product metrics you should investigate.
How can I do a better job at communicating my product usage insights to the rest of my company?
There are a number of ways you can improve on this front.
Below are a few ideas:
- Have the CEO or VP Product share some interesting numbers at company-wide updates.
- Have your analyst share their findings with a wider audience. If the analysis was requested by a product manager to run an analysis, ask if the insights can be shared with the entire team.
- Share findings on internal wikis and/or in Slack.
- Send out a weekly internal product updates email and include some insights from recent analyses.
Do I need to measure the usage of every feature I develop? What about "sub-features"?
I believe a company should try and know as much as it can about the inner workings of it's operations. This of course comes with a price. Common sense should be your guide and I'd recommend prioritizing your knowledge gaps and work according to that list. If you're in a fast moving company you may never get to the low priority items on that list which is perfectly fine.
If I can only afford to focus on one of your recommendations, which one should I choose?
I honestly believe that a company which does not adopt the usage tracking framework is putting itself at serious risk. If I had to pick one section of this post it would be to implement that framework.
If that is too much see what needs to happen so you can hire a senior business analyst who can take ownership of this project.
Summary
There are many aspects involved in measuring product usage. I hope this guide has helped you better understand the steps that are needed to implement a solid framework that will support the business.
Let the usage tracking framework be your guide and do your best to complete this project sooner rather than later. The longer you go without measuring your product usage, the tougher it will be to dig yourself out of your hole.
Feel free to comment below with any suggestions on how I can improve this guide or if you have any questions on any of the advice that I've shared.
Thanks for reading.
