How to Measure the Quality of Your Product

We all want to create a quality product. Every team–from Executives to Sales, Marketing, Support, and Development–talk about the quality of the product. But the truth is, no matter how much teams speak about quality, chances are your company doesn’t have a clear or unified definition of what quality is.

Even worse, companies usually don’t have a clear approach to measure and continuously improve the quality of their product. If you ask ten people in the organization to define quality, you’ll end up with ten different answers. And that’s one reason why quality is so elusive and hard to implement.

For example, the Development team might define quality relative to the number of bugs or stability. Sales might describe it as how easy the product is to sell. And the Executive team might define it as how well the product supports company objectives.

There’s also the view of the customer/user. For them, a quality product might be simply something that meets their needs. All different definitions, all very valid, and as a business-savvy Product Manager, you need to make sense of them all.

As a Product Manager, it is your responsibility to create a quality product. The first step is to agree on a company-wide definition and then agree on the metrics to enforce that quality.

The key here is to focus on creating metrics. Having metrics and a baseline to measure against gives you the confidence to factually say you have a quality product based on your company’s definition. And having clear definitions and targets makes everything easier for you because now you know how to measure success.

Also, keep in mind that it is more efficient to define and agree on these metrics early on during the innovation journey. 

Now you can focus on writing detailed requirements that incorporate quality as a critical element of every new feature. Detailed doesn’t mean you should go back to writing long Product Requirement Documents (PRD). It means that when writing requirements (or stories or epics or whatever you use), these requirements should have clear acceptance criteria and should include metrics you can evaluate before launch.

Related post: Internet of Things: A Primer for Product Managers.

To measure the quality of your product, you need to answer these questions first.

Failure to meet any of these areas would imply that your product does not have the right level of quality (as defined by you and the company), and therefore, it is not ready to be launched to market. This evaluation is not a one-time thing, though. It should be incorporated into your sprints and releases. It needs to become the way you approach building software.

How will you measure the quality of the proposed solution?

Here you are trying to answer the question: will this solution meet users’ definition of quality and your company’s UX definitions of quality? Before the development team builds the new features, these validations should be done using mockups, other research, or “Lean” techniques.

• You have run user testing to validate that the proposed features solve the user’s problem(s).
• User testing shows that your proposed solution is intuitive and easy to use.
• The proposed user design adheres to the company’s interaction design and visual design patterns.
• Onboarding functionality has been designed for all new features.

How will you measure the quality of implementation?

These items answer the question: did the development team build what was defined in the requirements and design specs? Metrics revolve around functional tests and manual demonstrations showing that every story/task was implemented according to specification.

• The functionality of all features matches the requirements (front-end and back-end).
• The implementation of the UX design in all form factors (web, tablet, phone, etc.) matches what was defined by the design team and approved by the Product Manager.

How will you measure performance and stability?

These items answer the typical questions that the development team considers “QA.”

• No severity 1 bugs are present (or whatever categorization you use).
• The product performs according to the established metrics (page load time, number of concurrent sessions, etc.).
• The product is stable and doesn’t crash or hang.
• New features didn’t break any of the existing functionality.

Related post: How to Protect Your IoT Product from Hackers.

How will you measure the quality of your whole product offering?

• Pre-sales material and collateral correctly describe the problem your product solves and the functionality defined by the Product Manager.
• The new features are included in a demo for the Sales team to use.
• The new features include the proper hooks/functionality required by the support team.

This partial list gives you an idea of the quality areas you need to consider. But you and your team will need to determine which quality areas make sense for your company. You’ll need to agree on:

1. A definition of quality
2. The metrics you will use
3. The process to evaluate quality before launching every new release

The Bottom Line

This conversation should revolve around outcomes and consequences of missing the metrics instead of tactical details and tools. It is straightforward to jump into the minutia and focus on bugs or unit testing. As you start this process, make sure you focus on the big picture first. Start from the business goals, and then implement the metrics that make sense for your company in its current state. No more and no less.

Most companies, especially smaller ones, probably haven’t thought about their definition of quality in that much detail. So if you bring this discussion to the table, you’ll probably be ahead of many companies out there.

So, how do you measure the quality of your product? Leave a comment below to keep the conversation going.