As a Product Manager, quality should be top of mind. But how do you measure the quality of your product? In this post, I talk about the importance of having a clear definition of quality, and some of the metrics you need to ensure success.
We all want to create a quality product. Every team–from Executives to Sales, Marketing, Support, and Development–talk about the quality of the product. But the truth is, no matter how much teams talk about quality, chances are your company doesn’t have a clear or unified definition of what quality is.
Even worse, companies usually don’t have a clear approach to measure and continuously improve the quality of their product. If you ask 10 people in the organization to define quality, you’ll end up with 10 different answers. And that’s one reason why quality is so elusive and so hard to implement.
For example, the Development team might define quality relative to the number of bugs or stability. Sales might define it as how easy the product is to sell. And the Executive team might define it as how well the product supports company objectives.
There’s also the view of the customer/user. For them, a quality product might be simply something that meets their needs. All different definitions, all very valid, and as a business-savvy Product Manager, you need to make sense of them all.
The first step is to agree on a company-wide definition, and then agree on the metrics to enforce that quality.
The key here is to focus on creating metrics. Having metrics and a baseline to measure against gives you the confidence to factually say you have a quality product, based on your company’s definition. And having clear definitions and targets makes everything easier for you because now you know how to measure success.
Now you can focus on writing detailed requirements that incorporate quality as a key element of every new feature. Detailed doesn’t mean you should go back to writing long Product Requirement Documents (PRD). It means that when writing requirements (or stories or epics or whatever you use), these requirements should have clear acceptance criteria, and should include metrics that align with all the different areas of quality that your product will be evaluated against before launch.
Related post: Internet of Things: A Primer for Product Managers.
To measure the quality of your product, you need to answer these questions first
Failure to meet any of these areas would imply that your product does not have the right level of quality (as defined by you and the company) and therefore, it is not ready to be launched to market. This evaluation is not a one-time thing though. It should be incorporated into your sprints and releases. It needs to become the way you approach building software.
How will you measure the quality of the proposed solution?
Here you are trying to answer the question: will this solution meet users’ definition of quality, and your company’s UX definitions of quality? These validations should be done using mockups, other research, or “Lean” techniques before the development team builds the new features.
- You have run user testing to validate that the proposed features solve the user’s problem(s).
- User testing shows that your proposed solution is intuitive and easy-to-use.
- The proposed user design adheres to the company’s interaction design and visual design patterns.
- Onboarding functionality has been designed for all new features.
How will you measure the quality of implementation?
These items answer the question: did the development team build what was defined in the requirements and design specs? Metrics revolve around functional tests of the application and manual demonstrations by the development team that every story/task in the sprint has been developed according to specification.
- The functionality of all features matches what was defined in the requirements (front-end and back-end).
- The implementation of the UX design in all form factors (web, tablet, phone, etc) matches exactly what was defined by the design team and approved by the Product Manager.
How will you measure performance and stability?
These items answer the typical questions that the development team considers “QA”.
- No severity 1 bugs are present (or whatever categorization you use).
- Product performs according to the established metrics (page load time, number of concurrent sessions, etc).
- The product is stable and doesn’t crash or hang.
- New features didn’t break any of the existing functionality.
Related post: How to Protect Your IoT Product from Hackers.
How will you measure the quality of your whole product offering?
- Pre-sales material and collateral correctly describe the problem being solved and the functionality defined by the Product Manager.
- The new features are included in a demo for the Sales team to use.
- The new features include the right hooks/functionality required by the support team.
This partial list gives you an idea of the quality areas that you need to consider. But you and your team will need to determine which quality areas make sense for your company. You’ll need to agree on:
- A definition of quality
- The metrics you will use
- The process to evaluate quality before launching every new release
The Bottom Line
This conversation should revolve around outcomes and consequences of missing the metrics, as opposed to tactical details and tools. It is very easy to jump into the minutia and focus on bugs or unit testing, for example. As you start this process, make sure you focus on the big picture first. Start from the business goals, and then implement the metrics that make sense for your company in its current state. No more and no less.
Most companies, especially smaller ones, probably haven’t thought about their definition of quality in that much detail. So if you bring this discussion to the table, you’ll probably be ahead of many companies out there.
So, how do you measure the quality of your product? Leave me a note to keep the conversation going.