Data has become a cargo cult. Collect more data, calculate more metrics, hire more analysts, let them figure out what this is all for – and you’re considered to be data driven. I’ve had it up to here while consulting startups over the past three years and helping them to define business metrics.

In this article, I’ll try to summarize my experience and address both technical and process aspects, useful both for data specialists and business users. We’ll talk about each step of the analytical process of how to define metrics.

The analytical process

From a business user standpoint, data analyst work looks like calculating this or that and plugging the result into some dashboard for every question that pops up. The result is a myriad of dashboards overloaded with metrics that don’t make much sense and which no one is looking at.

To achieve meaningful results, a company needs to adopt a more comprehensive metric definition process that consists of multiple, equally important steps.

The steps of a healthy metric definition process are the following:

1. Getting context
2. Data health check
3. Exploratory analysis
4. Metric selection
5. Implementation of the definition
6. Implementation of the visualization
7. Getting feedback

Business users sometimes tend to prescribe the solution, jumping to step number 5. They think steps 5 and 6 are everything the data analyst does. In fact, these steps represent only a small fraction of an analyst’s routine, so let’s take a closer look at the underestimated steps 1 to 4.

Getting context

When a business analyst knows the real life context of the business process, the underlying data makes more sense. In this case, the result will be more meaningful for both sides.

Users who request data have to provide more information about what they are trying to achieve in terms of business, not in terms of numbers: their goals, their issues, or some hypothesis that has to be tested.

A requester shouldn’t prescribe the solution. The best solution is created during collaboration between the business user and the analyst.

Data health check

This step is about checking how well data captures the business process, if at all. Data collection and cleaning may require at least 80% of time. So if you want to check metrics on a regular basis, you need to invest some time in proper updates of the data pipeline.

Sometimes data doesn’t capture the process at all and requires software engineering and/or data engineering efforts. In the next parts of this series, I’ll tell you how to build a good interaction between the data team, business users, and the software engineering.

Exploratory analysis

This step is just as important as it is underestimated. People like averages because they think averages are good indicators of the overall trend. Meanwhile, the same average value can come out of completely different distributions. Further in this post, I’ll show a good use case on how to handle this problem.

Only if analysts understand the business process, look at the data distribution, and investigate its causality with some business target, they define business metrics for startups the right way.  

Metric selection

One can select a good metric based on the results of the exploratory analysis and the checklist below:

• Business impact. How critical is this metric in defining the success of the business? If you don’t get the answer, how painful would that be? Or if you do, how much benefit would the company get, or you get as a manager?
  
• Actionability. If you see a metric value that doesn’t satisfy you is there anything you can do about it? If you are a manager and you have averages on your dashboard you’re in a position of an overseer that can only order slaves to work harder.

   

  You could do the same thing without the dashboard if you wanted, there’s no extra value that the dashboard brings. A data driven company is not calculating averages, it’s linking clear and reasonable benchmarks to people and processes and monitors their reach.

   

• Transparency. The metric shouldn’t be too obscure and complex and it should give you an idea of what is going on with real numbers behind it. The better the mathematical background of the audience of this metric, the more complex the metric can be, and vise versa.

   

  It’s not recommended to define complex metrics in the low-qualification employee performance review even if they are very indicative and make a lot of sense for a manager. They should make sense for employees first and foremost.
  

• Sensitivity. Would the metric adequately reflect the changes in the business process in a way that not only the direction, but the magnitude of change, is well understood?

Quantiles and % of data points that meet the benchmark work really well, much better than averages.

Example of quantile based metric: the bottom 10% of the worst wait times in your conversion pipeline. Example of the % of data points: the % of tickets with a response time of less than 6 hours.

Metric implementation and visualization

Once the metric is well defined it’s time to implement and visualize. We won’t focus on this part here, as it depends on the selected tech stack and the amount of data, which is out of this article’s scope.

Getting feedback

The process flow for analytics iterations seems quite obvious:

The feedback step is a very important last step of the process in every scenario. The data team needs a short confirmation from a business user if the job is done well. If they don’t receive feedback, they think their work wasn’t useful or was ignored. Next time a data analyst might give tasks from the same business owner a lower priority.

Sometimes the job is done well from a technical standpoint but the results are counterintuitive. In this case, you need to decide how important the task is and whether it’s worth investing more time in further research. This again requires feedback from a business owner.

Marry your definitions

After you define the metric on a local level, make sure the definitions are the same across the entire company. There is a wide variety of angles from which you can look at business data. As the organization grows, along with its analytical demand, people often run into a situation where there is no common understanding of the analytical vocabulary in the company.

Depending on how “lead” and “purchase” are defined and linked in exact technical terms, conversion value can float quite a bit.

Also, if the same metric is defined in different ways on different levels, lower levels do not add up to the upper level value. For example, if every service region of the company is tracking revenue by service date and the company-level revenue − by billing date, there is no way they will add up.

This causes a lot of misunderstanding and, what’s more important, lowers the trust in data and becomes a serious impediment on the way to making the organization data driven. The way to fix this is roughly the following:

1. Discuss thoroughly all implications of every possible definition of the fact-level data model and every metric, and agree on specific definitions that will be used going forward.

   If a set of angles is important for a given metric (like service vs. billing revenue) make it two different metrics with the explicit statement of their nature in the name (“revenue by service date” and “revenue by billing date,” not just “revenue”).

2. Document the approved definitions as much as possible and, if technically possible, attach the link to the documentation every time the metric pops up in a dashboard. The metadata should be in front of the users, so they know what they are looking at.
3. There should be an absolute consensus between the members of the data team about the definitions. If it’s not consistent across the data team, you can’t expect it to be consistent on the company level.

It’s quite similar to marriage. You take your definitions with all their possible flaws, ideally for life, you make sure everyone knows it is your choice, and everyone from your closest circle should accept your decision. Only a serious issue can make you change the choice. Marry your definitions!

Ideally, the definitions should be technically stored in the analytics software.

It acts as a black box, taking a set of dimensions and filters as parameters and returning the series of metric values, with the consistent definition being used for any input. This is how more old-fashioned BI products work, but they lack flexibility and are pretty expensive.

This is why smaller organizations chose ad-hoc culture when there is an analyst responding to the questions and calculating business metrics using raw SQL. This brings us back to the issue with definition consistency.

Keeping all that in mind, new data analytics platforms like Statsbot offer data modeling and metric definition languages and tools. They allow for reaching the same goal, which is a common data definition repository that feeds self-service dashboarding tools, in a very flexible and cost-effective way.

My only advice to them would be that the data definition languages should support more metadata options, like descriptions and links to metric owners. Then, such “metadata-rich” models can be used to autogenerate nicely formatted documentation that is further exposed to non-technical personnel (or, ideally, popups right in the dashboards).

Metric ownership

In addition to a unified metric definition, you need to have clear ownership over these metrics. Ideally, every metric should have three owners:

• Execution owner, who is the closest person responsible for the business process behind the metric.
• Management owner, who is the supervisor of the execution owner.
• Technical owner, the analyst.

This way you can build a chain of metrics from customer-facing employees to the CEO. If you look at the sales process, you can have the following metrics:

Metrics	Execution owner(s)	Management owner(s)
individual performance metrics	sales representative(s)	sales team manager(s)
individual performance metrics	sales team manager(s)	sales/marketing executive
company-level sales metrics	sales/marketing executive	CEO

This approach scales perfectly to any number of sales representatives and teams and allows for propagating specific targets from the CEO to the front desk level, and measuring their reach. This is what makes the company data driven.

Also, it’s a great practice to have weekly or biweekly “business review” meetings where all managers responsible for the main business functions meet with the CEO and discuss the dashboard with main KPIs for every function.

This provides a great space for cross-functional discussion, because often one manager doesn’t know what is going on in the other parts of the business.

For example, imagine a scenario where high-growth targets are set and the marketing and sales teams are doing their best to achieve them, while operations don’t scale that well. Or, in more general terms, demand is created but there is not enough supply to satisfy it.

In this case, the best solution would be to relax demand until the supply is figured out. This can be done by the CEO as a person that connects all the dots, but it makes much more sense for people to actually see what’s going on in other teams on the dashboard versus taking instructions.

The opposite of data driven is not non-data driven, but authority-driven.

I’ll show you a bonus example (it’s a long read) based on my own practice to illustrate how a startup can benefit from building right business metrics.

Sales use case

Two sales representatives, Rob and Ben, make calls to leads that appear in CRM. Ben’s number of closed deals is higher than Rob’s. They have a manager, Ellie, that thinks they can close more deals, especially Rob, and she asks the analyst, Tim, to help. Ellie is a strong believer in warm calls and her hypothesis is that if leads are called right away they are more likely to close.

If the metric definition process is not followed:

Ellie asks Tim to calculate Rob and Ben’s average time between the lead arrival and the call. It turns out the time is 4 hours for Rob versus 3 hours for Ben, so the insight that Rob is calling leads later than Ben is on the surface. However, this is the only insight.

Ellie needs to find why this happens, so she talks to the guys or shadows them, and finds out their calling patterns. It turns out that Rob doesn’t call the leads right after they appear. He picks them up using his own logic, like his idea about the best target audience or just a gut feeling, and sometimes makes big pauses in a search of inspiration for another conversation.

Ben is monitoring CRM very closely and tries to call the leads as soon as possible after they appear. Ellie tells Rob to pay closer attention to CRM for incoming leads to make more warm calls. She also asks the engineering team to create a Slack channel with notifications for every incoming lead to make it easier for sales people to not drop the ball.

After a few weeks Rob’s average time goes down from 4 hours to 3 hours and the conversion increases a bit, bringing some degree of satisfaction to the team. In the following week, the number of leads increases a lot due to an increased marketing budget, and the averages naturally go up because there is not enough capacity. Ellie knows the number of leads increased and she is smart enough not to put pressure on the guys for the worsened KPI.

This means the KPI is useless and the choice is wrong. Even if you take it out of this story, pretty much nothing changes. It’s not the data that has driven the improvement process, it was Ellie with her thorough approach. Data just reflected what’s happening. Also, imagine if it were not two, but ten or fifty sales people, and talking to each and shadowing would be much more difficult for Ellie.

If the process is followed:

Ellie describes the sales process to Tim, shows the sales script and how leads appear in the CRM, and explains the hypothesis about the warm calls. Tim looks at the distribution of times between lead arrival and the call and finds out they are quite different for Rob and Ben.

Rob has nearly random distribution and Ben has bimodal distribution with one peak near zero, because he calls many leads immediately, and another peak on the next day when he cleans the backlog.

Tim divides data points by one-hour buckets, and checks what the conversion for every bucket is. He finds out that conversion for the first three buckets is the highest and it declines steeply starting from the four-hour bucket, so the warm call hypothesis is right. It is clear that Ben closes more deals because he has a higher share of warm calls.

Then Tim checks the average duration of the call that ends up with a closed deal and finds out it’s 3 minutes. Checking the number of leads that come and their distribution over daytime, he finds out that a good sales person can theoretically make around 100 calls in a day. 8 hours by 60 minutes by 50-75% to give some slack for various overheads, divided by 3 minutes.

Also, it’s not possible to make timely calls to leads that arrived outside business hours, so these leads should either be excluded from the metric calculation, or their call times should be converted to business hours. For example, if the lead arrives at 9:00 p.m. and the sales day starts at 9:00 a.m., a call at 9:10 a.m. the next day will have 10 minutes of lag versus 12 hours and 10 minutes.

After all, the suggested metric is the % of leads that were called within 3 hours, where the denominator (the N of leads) is capped to 100 per salesperson, and outside business hours leads are excluded for the sake of simplicity at the first iteration.

It doesn’t put unnecessary pressure on sales people because the call doesn’t have to be made ASAP; 2 or 3 hours is still good per research. The metric is robust enough to lead influx because the denominator is capped. Another metric (just the number of leads) will signal the requirement to extend the team or automate the sales process, but does not affect the performance KPI of the current team members in any way.

It is very transparent and descriptive because when you see that Rob called just 28% of leads within 3 hours and Ben called 65%, you immediately understand what’s going on, while 5 and 3 hour averages don’t make much practical sense.

It is actionable both for the sales people (put in more effort to call leads faster) and the manager (build policies and tools to facilitate timely calls, give clear instructions to sales people, and part ways with those who systematically violate them). Ellie finds it very useful and provides Tim great feedback.

Conclusion

We have reviewed the main steps of how to define business metrics for startup and looked at how differently things can work out when the process is and is not followed. In the next part of this series, I’ll tell you how to build strong relationships between analysts and business users. Stay tuned!

This post was originally published here.