Product Metrics

Previous articles in this blog have looked at the usage of metrics for measuring software developer productivity and software delivery performance. This article dives deeper into the topic of measuring software success with product metrics.

• Product metrics measure software success, i.e. the value / acceptance / usage of a product / service by its customers / users.
  
• DORA metrics measure the software delivery performance of the company / staff providing the service.
  
• SPACE is a framework to assess software developer productivity. 
  [A future article will look at the concept of Developer Experience as another avenue to assess developer productivity.]

A recent LinkedIn Post by Aatir Abdul Rauf provides a good overview by summarizing eight resources on the topic:

1. A Guide to Product Metrics (by Mixpanel)
2. North Star & Guardrail Metrics (Daniel Lee)
3. Defining Product Metrics (Hani Azam)
4. 11 Metrics for Product-led Growth (Katryna Balboni)
5. SaaS Product Metrics (by Userpilot)
6. Ecommerce Metrics (by SendInBlue)
7. Marketplace Metrics (Juho Makkonen)
8. AARRR, GAME, Pirate Metrics (Anthony Murphy)

According to Hani Azam, product metrics help companies

• have clarity, alignment and prioritization in what to build,
• decide how to build the product once they’ve prioritized what to build, and
• determine how successful they are and hold them accountable to an outcome

He distinguishes between four types of product metrics:

1. North Star
• This is the single most important measure of success for a company. 
• A North Star Metric (NSM) should be:
•  a direct reflection of a company’s mission,
• an indicator of how a company brings value to its customers,
• the only one of its kind,
• the answer to the following question:
  what is the one metric that best represents the desired outcome of your company?
2. Primary/Goal Metric
• Primary metrics depict the desired outcome of a particular product, team or initiative. 
• This is unlike the NSM, which represents the desired outcome of the company as a whole.
3. Supporting/Tracking/Input Metrics
• Indicators that the NSM (or primary metric) is moving in the right direction. 
• Particularly useful as leading indicators to your NSM.
• Tell you where your efforts to move your NSM may be falling short.
4. Counter Metrics/Guardrails
• Counter metrics represent other outcomes that the business cares about, which may be negatively affected by a positive change in the primary metric (or NSM). 
• Serve as guardrails.
  They exist to make sure that in the pursuit of your primary metric, you are not doing harm to another aspect of the business.

Hani provides the following examples for North Star Metrics (NSM) of various companies:

He also comments on the nature of good metrics:

Good metrics should be meaningful, measurable, and moveable / adaptable.

The last part of his article describes how to come up with supporting and counter metrics based on the observation that 

the best way to come up with these metrics is by tracing a users journey with your product.

The author refers to a common approach for breaking down the user journey:

AARRR (or Pirate Metric) framework - introduced by Dave McClure - breaks down the user journey by Acquisition, Activation, Retention, Referral and Revenue. 

Hani expanded the framework from AARRR to AAAERRR as follows: 

1. Awareness: How many people are aware your brand exists?
2. Acquisition: How many people are interacting with your product?
3. Activation: How many people are realizing the value of your product?
4. Engagement: What is the breadth and frequency of user engagement?
5. Revenue: How many people are paying for your product?
6. Retention/Renewal: How often are your people coming back?
7. Referral: How many customers are becoming advocates?

Please consult the original article for a more detailed description of the framework with concrete examples of applicable metrics.

Creativity Myths

The 2021 study "Creativity myths: Prevalence and correlates of misconceptions on creativity" tries to separate facts from fiction on what creativity is and what it is not.

Psychology Today summarizes the 15 Myths About Creativity covered in that study.

This study examined the prevalence of known creativity myths across six countries from diverse cultural backgrounds and explored why some people believe in them more than others. Results revealed persistent, widespread biases in the public conception of creativity, such as attributing creative achievements to spontaneity and chance rather than persistence and expertise.

The researchers looked through the existing scientific literature to identify 15 creativity falsehoods, which they divided into four categories:

Creative Definition Myths

• Creativity cannot be measured
• Creativity is essentially the same as art
• Creative ideas are naturally a good thing
• Most people would not be able to distinguish abstract art from abstract children's drawings

Creative Process Myths

• Creative accomplishments are usually the result of a sudden inspiration
• Creative thinking mostly happens in the right hemisphere of the brain
• Creativity tends to be a solitary activity

Creative Person Myths

• Creativity is a rare gift
• People have a certain amount of creativity and cannot do much to change it
• Children are more creative than adults
• Mental health disorders usually accompany exceptional creativity

Creative Stimulation Myths

• People get more creative ideas under the influence of alcohol or marijuana
• Long-term schooling harms the creativity of children
• Brainstorming in a group generates more ideas than if people were thinking by themselves
• One is most creative when with total freedom in one's actions

“A ‘naivety’ conceptualization of creativity is problematic for two reasons,” say the authors.

First, relating creativity to childlike behavior and chance implies low appreciation for the hard work behind creative achievements. Second, it externalizes relevant factors in the development of creativity. Emphasizing the role of inspiration rather than active engagement may undermine creativity by suggesting we need to wait until creativity hits us with a ‘Eureka’-experience.

 

The authors contrast the myths with the following

Creativity Facts

1. To be considered creative, something has to be both novel and useful or appropriate
2. Teachers appreciate the idea of creativity but not necessarily creative pupils
3. Whether or not something is viewed as creative depends on zeitgeist and social norms
4. Creativity is an important part of mathematical thinking
5. Creative ideas are typically based on remembered information that is combined in new ways
6. The first idea someone has is often not the best one
7. Alpha activity (10Hz) in the brain plays an important role in creative thought
8. Creative people are usually more open to new experiences
9. Creative people are usually more intelligent
10. Achieving a creative breakthrough in a domain (i.e. publishing a successful novel) typically requires at least 10 years of deliberate practice and work
11. Men and women generally do not differ in their creativity
12. A man's creativity increases his attractiveness to potential partners
13. When stuck on a problem, it is helpful to continue working on it after taking a break
14. Positive moods help people get creative ideas
15. Getting rewarded for creative performance at work increases one’s creativity

Source:   Creativity myths: Prevalence and correlates of misconceptions on creativity - ScienceDirect
              Appendix B (.xls) references the research papers backing their claims.

The article Top Ten Myths About Creativity (futurefocusedlearning.net) lists 10 Creativity Myths mostly in line with the study's findings:

1. Creativity belongs to the geniuses
2. Creativity is making something from nothing
3. Creativity can’t be forced
4. Mental illness causes creativity
5. Drugs make you more creative
6. To be creative you need to be free
7. Creativity belongs to the arts
8. Creativity is a solitary activity
9. Extrinsic motivation is detrimental to creativity
10. To explain creativity is to damage it

How to Misuse and Abuse DORA DevOps Metrics

In the How To Measure Software Delivery Using DORA Metrics (YouTube) presentation, Dave Farley, author of "Continuous Delivery" and "Modern Software Engineering" describes how one can apply DORA measurements to drive software development to deliver on this state-of-the-art approach, but also explores a few of the common mistakes that can trip us up along the way.

I found the reference to Bryan Finster's October 2021 presentation How to Misuse DORA DevOps Metrics especially useful.

Bryan contrasts common pitfalls & fallacies with pragmatic and realistic advice.

He also points out that the 4 prominent DORA metrics constitute only the tip of the iceberg.

My earlier blog article on Software Productivity Metrics provides further details on these additional metrics.

Slide #29 in Bryan's deck puts these metrics into perspective ("To improve flow, we must improve CI.") and makes the case for a set of balanced metrics (#34):

Summary ("Closing Thoughts")

• The 4 outcome metrics are only the tip of the iceberg.

• Product development is a complex interaction of people, process, and products. There are no simple metrics.

• Measures require guardrails to avoid perverse incentives.

• Metrics are a critical part of the improvement toolbox, but…

• We cannot measure our way to improvement.

• We use them to monitor and inform the next improvement experiment.

• Don’t measure people, invest in them. They are our most valuable asset.

[July 26, 2022 -- Update:

Abi Noda discusses Finster's recent article in the The DevOps Enterprise Journal | Spring 2022 (itrevolution.com) edition on the same topic:

Common misuses of the DORA metrics

• Focusing too much on speed.
• “Measuring deployment frequency without using quality metrics as guardrails will result in poor outcomes.”
• Setting goals around DORA metrics. 
• “The goal isn’t better DORA metrics… OKRs should be focused on desirable business outcomes.”
• Choose goals, then choose metrics that align with those goals. 
• Mistaking measuring DORA metrics as a way to improve. 
• “[DORA metrics] don’t fix things.
  If we simply get a dashboard and do not buy into using it to identify improvement items, then nothing will get better.” 
• Using DORA metrics as vanity metrics. 
• “[DORA dashboards] are often used as ‘vanity radiators’ instead of information we can use to help us improve.”
• Not including other signals in addition to the four key DORA metrics.
• “The four key metrics DORA used to correlate behaviors of high-performing organizations are a small subset of the metrics recommended in the book Accelerate. They also represent only one aspect of the health of a system…”

]

[January 25, 2023 -- Update:

In his LinkedIn article, Abi Noda summarizes 

Common pitfalls of the DORA metrics, according to Nathen Harvey who helps lead DORA at Google:

1. Comparing teams to each other based on the four key metrics. Different projects have different needs, so we can think more critically about whether a team's metrics should fall in the low, medium, or high performance category given that context.

2. Setting goals for improving the DORA metrics, and in turn creating the wrong incentives. Instead set goals to improve the capabilities or factors that drive the DORA metrics.

3. Spending more effort on pulling data into dashboards than on actually improving. 

4. Not using the metrics to guide improvement at the team level. When the teams doing the work aren’t using the metrics to improve, this defeats the purpose of the metrics.

5. Using "industry" as an excuse for not improving. Even companies in well-regulated industries can focus on improvement.

6. Assuming you’re already world-class, so your organization doesn’t need to focus on improving. If software delivery is no longer the constraint, then what is? Identify what is preventing teams from making progress and focus on that.

7. Fixating on the four DORA metrics (which are outcomes) and forgetting about the capabilities. “We don’t get better at those outcomes by focusing on the outcomes. We have to focus on the capabilities that drive those outcomes.”

The big takeaways:

• the DORA metrics are outcomes not goals,
• context matters, and
• a team must look to understand and improve the factors that drive the DORA outcomes.

P.S. I like the "You might also deliver wrong things 10x faster" statement in the "Fantastic Facts and How to Use Them" presentation referenced in one of the comments.

]

Software Productivity Metrics

"When a measure becomes a target, 
it ceases to be a good measure." 
― Goodhart's Law 

Traditional research presentations such as Software Productivity Decoded by Thomas Zimmermann (co-editor of the book Rethinking Productivity in Software Engineering discussed in the previous blog entry) have focused on productivity measures for the development of software delivered on premises, e.g.

• Modification requests and added lines of code per year
• Tasks per month
• Function points per month
• Source lines of code per hour
• Lines of code per person month of coding effort
• Amount of work completed per reported hour of effort for each technology
• Ratio of produced logical code lines and spent effort
• Average number of logical source statements output per month over the product development cycle
• Total equivalent lines of code per person-month
• Resolution time defined as the time, in days, it took to resolve a particular modification request
• Number of editing events to number of selection and navigation events needed to find where to edit code

The Accelerate: State of DevOps Report 2019 has identified four metrics - commonly referred to as DevOps Research and Assessment (DORA) Metrics - that capture the effectiveness of the development and delivery process summarized in terms of throughput and stability. Their research has consistently shown that speed and stability are outcomes that enable each other.

They measure the throughput of the software delivery process using lead time of code changes from check-in to release along with deployment frequency. Stability is measured using time to restore— the time it takes from detecting a user-impacting incident to having it remediated— and change fail rate, a measure of the quality of the release process. 

In addition to speed and stability, availability is important for operational performance. Availability is about ensuring a product or service is available to and can be accessed by your end users.

• Deployment Frequency—
  How often an organization successfully releases to production.
• Lead Time for Changes—
  The amount of time it takes a code commit to get into production.
• Change Failure Rate—
  The percentage of deployments causing a failure in production.
• Time to Restore Service—
  How long it takes an organization to recover from a failure in production.

I found the webinar by Jez Humble, CTO of DORA / Google Cloud Develeoper Advocate, to provide a great overview of the content of the DevOps Report 2019:

Performance metrics

Improving performance

Improving productivity 

Culture 

[Update 06/23/2022: See my new blog entry How to Misuse and Abuse DORA DevOps Metrics to avoid common pitfalls when applying DORA metrics in practice.]

Julian Colina endorses these process metrics and warns against flawed output metrics in his summary of the Top 5 Commonly Misused Metrics:

1. Lines of Code
2. Commit Frequency
3. Pull Request Count
4. Velocity or Story Points
5. "Impact“

Dan Lines even makes the point that Velocity is the Most Dangerous Metric for Dev Teams. 

Velocity is a measure of predictability, not productivity. Never use velocity to measure performance and never share velocity outside of individual teams.

He proposes the following alternative measures:

• If speed to value is your main goal, consider Cycle Time.
• If predictability is your main goal, look at Iteration Churn.
• If quality is your priority, Change Failure Rate and Mean Time to Restore are good.

Be aware of the kind of culture you want to create by applying these measures as measuring the wrong things for the wrong reasons can backfire. 

Patrick Anderson acknowledges that DORA Metrics help to deliver more quickly but notes their limited focus on product development & delivery leaving out the product discovery phase. He advocates end-to-end Flow Metrics as part of Value Stream Management to deliver the right things more quickly at the right quality and cost and with the necessary team engagement.

• Flow Time measures the whole system from ideation to production—starting from when work is accepted by the value stream and ending when the value is delivered to the customer. 
• Flow Velocity—How much customer value is delivered over time.
• Flow Efficiency—What are the delays and wait times slowing you down?
• Flow Load—Are demand vs. capacity balanced to ensure future productivity?
• Flow Distribution—What are the trade-offs between value creation and protection work?

Check out the Linearb.io White Paper 17 Metrics for Modern Dev Leaders if you are looking for even more metrics clustered into three categories of KPIs (work quality, delivery pipeline, investment profile) across two dimensions (iterations, teams).

Somewhat similar Gitential has broken up its Value Drivers & Objective Metrics to Improve Your Software Development into four buckets (speed, quality, efficiency, collaboration):

The SPACE Framework (The SPACE of Developer Productivity) features five different dimensions of looking at productivity; hence the acronym SPACE:

• Satisfaction is how fulfilled developers feel with their work, team, tools, or culture; Well-being is how healthy and happy they are, and how their work impacts it.
• Performance is the outcome of a system or process.
• Activity is a count of actions or outputs completed in the course of performing work.
• Communication and collaboration capture how people and teams communicate and work together.
• Efficiency and flow capture the ability to complete work or make progress on it with minimal interruptions or delays, whether individually or through a system.

To measure developer productivity, teams and leaders (and even individuals) should capture several metrics across multiple dimensions of the framework—at least three are recommended.

Another recommendation is that at least one of the metrics include perceptual measures such as survey data. […]. Many times, perceptual data may provide more accurate and complete information than what can be observed from instrumenting system behavior alone.

Including metrics from multiple dimensions and types of measurements often creates metrics in tension; this is by design, because a balanced view provides a truer picture of what is happening in your work and systems. 

This leads to an important point about metrics and their effect on teams and organizations: 
They signal what is important. 

One way to see indirectly what is important in an organization is to see what is measured, because that often communicates what is valued and influences the way people behave and react. […]. As a corollary, adding to or removing metrics can nudge behavior, because that also communicates what is important.

Listen to the Tech Lead Journal podcast #43 - The SPACE of Developer Productivity and New Future of Work - Dr. Jenna Butler (Microsoft) for an overview of the framework and other research initiatives related to the "New Future of Work". See link above for transcript, mentions and noteworthy links to related research articles.

In this context you may want to take note of Steven A. Lowe's six heuristics for effective use of metrics:

1. Metrics cannot tell you the story; only the team can do that.
2. Comparing snowflakes is waste.
3. You can measure almost anything, but you can't pay attention to everything.
4. Business success metrics drive software improvements, not the other way round.
5. Every feature adds value; either measure it or don't do it.
6. Measure only what matters now.

And if you want to read more about the topic then this blog provides further references. 

Productivity in Software Engineering

Here are my key takeaways from the 2019 book Rethinking Productivity in Software Engineering (edited by Caitlin Sadowski and Thomas Zimmermann). 

This open access book collects the wisdom of the 2017 "Dagstuhl" seminar on productivity in software engineering, a meeting of community leaders, who came together with the goal of rethinking traditional definitions and measures of productivity.

The results of their work includes chapters covering definitions and core concepts related to productivity, guidelines for measuring productivity in specific contexts, best practices and pitfalls, and theories and open questions on productivity. 

Key Findings

• There is no single metric to measure software development productivity
  and attempting to find one is counterproductive
• Many productivity factors (technical, social, cultural) need to be considered (> Context)
• Relationship between developer job satisfaction and productivity
• Different stakeholders may have varied goals and interpretations of any sort of productivity measurement (> Alignment)
• Individual developers, teams, organizations and market have different perceptions of productivity (> Level)
• Productivity goals may be in tension across these different groups
• Developers do not like metrics focused on identifying the productivity of individual engineers
• Productivity perceptions vary greatly according to the period of time that is considered (> Time)
  

What to do instead: 

• Design a set of metrics tailored for answering a specific goal
• Invest in finding and growing managers who can observe productivity 

BTW, the myth of the "10x programmer" also gets debunked.

Most of the key points outlined above are covered in Chapter 2 No Single Metric Captures Productivity (Ciera Jaspan, Caitlin Sadowski; Google) which quotes Bill Gates:

“Measuring software productivity by lines of code is like measuring progress on an airplane by how much it weighs.”

A variety of alternative metrics are discussed in my next blog article. 

Innovation Metrics

It should not come as a surprise since there is no single definition of the term "innovation" to start with: 

There is no established measurement framework in the software industry to measure innovation yet – according to this 2013 study*

Edison, H., Ali, N.B., & Torkar, R. (2013). Towards innovation measurement in the software industry. Journal of Systems and Software 86(5), 1390-1407. Available at: http://www.torkar.se/resources/jss-edisonNT13.pdf 

The article lists some metrics, though, that we can consider in this context (see Table 5 on pg. 1397) if we find a simple and reliable way to track them – highlighting is mine:

Category     Example 
Determinants The existence of a project champion, knowledge sharing, government regulation              effect
Inputs       R&D expenditure, R&D intensity (ratio of R&D expenditure to total assets),
             percentage of workforce time that is currently dedicated to innovation                      projects
Outputs      Patent density, new organisational programs, number of new processes              and significant enhancement per year

Performance  Percentage of sales that is generated by new products, citation ratio, impact              of brand
Activities   Percentage of ideas funded, quality of adaptation, managers survey

*Excerpt from 8. Conclusions on pg. 1405

The purpose of this study was to establish the current practices, mechanisms and challenges of innovation measurement in the software industry. […] The study found that among major challenges is a lack of a consistent perspective of innovation. This difference in views affects how innovation measurement initiatives are conceived (what is considered key aspect of innovation) and executed (which metrics are required to capture a particular aspect). […]

[…] there are several shortcomings in the state of practice. In the software industry, there is a lack of defined innovation process and measurement programs. Similarly, none of the well-known measurement frameworks are used to measure innovation. […]

The outcome of this review contributed to the existing body of knowledge in the form of an innovation measurement model, enumeration of metrics and their classification based on what aspect of innovation they are used to measure. […]

The metrics listed above are in line with the traditional innovation metrics documented in " The Complete Guide to Innovation Metrics – How to Measure Innovation for Business Growth" by Soren Kaplan, but he goes on to define more modern metrics that better reflect today's "Open Innovation" methods.

How do you measure innovation? One of the reasons that only about 1/3 of all Fortune 1000 companies have formal innovation metrics is because this simple question does not have a simple answer. [...]

The heart of the problem is that today’s competitive environment is radically different from the industrial environment in which traditional innovation metrics were born. Because most metrics programs begin with benchmarks of established companies that have been successful with new products (like 3M or Google), metrics tend to revert back to traditional measures of R&D or technology investment and effectiveness. Across the Fortune 1000 that do possess innovation metrics, for example, the most prevalent metrics include:

• Annual R&D budget as a percentage of annual sales
• Number of patents filed in the past year 
• Total R&D headcount or budget as a percentage of sales
• Number of active projects 
• Number of ideas submitted by employees
• Percentage of sales from products introduced in the past X year(s)

While some of these metrics are valuable for driving investment in innovation and evaluating results, they provide a limited view. In today’s environment in which “open innovation” (sourcing ideas and technology from outside the company) can create differentiation and competitive advantage, for example, some of these metrics actually inhibit strategic innovation. And in an environment in which disruptive innovation and cannibalization must be wholeheartedly embraced as a core strategy, fundamentally new types of behaviors are required, and subsequently new structures and related metrics to drive these behaviors. [...]

A Framework for Innovation Metrics

The best solutions create simplicity from complexity. Assuming that successful innovation results from the synergies between complementary success factors, it is important to address these by:

• Creating a “family of metrics” for ensuring a well-rounded portfolio of measures
• Including both “input metrics” and “output metrics” to ensure measures that drive resource allocation and capability building, as well as return on investment

A “family of metrics” ensures a portfolio of measures that cover the most important innovation drivers.  The following are the three categories to consider for any metrics portfolio:

Return on Investment Metrics
ROI metrics address two measures: resource investments and financial returns.
ROI metrics give innovation management fiscal discipline and help justify and recognize the value of strategic initiatives, programs and the overall investment in innovation.

Organizational Capability Metrics
Organizational capability metrics focus on the infrastructure and process of innovation.
Capability measures provide focus for initiatives geared toward building repeatable and sustainable approaches to invention and re-invention.

Leadership Metrics
Leadership metrics address the behaviors that senior managers and leaders must exhibit to support a culture of innovation within the organization, including the support of specific growth initiatives.

[...]

Please consult “Soren Kaplan’s article” for the details of the proposed metrics.