Big Data: 6 Key Areas Every IoT Product Manager Should Address

Big Data: 6 Key Areas Every IoT Product Manager Should Address
Daniel Elizalde

A few years ago, when I was working as Product Manager of an Enterprise Data Management Product, a company approached us to help them solve a major challenge.  They had just invested in a state-of-the-art automation system capable of generating tons of data (to the rate of gigabytes/second).  A few months later, they realized that their new investment wasn’t paying off because they had just replaced one problem with another: they were able to produce quality data, but they found themselves inundated in it.  They weren’t able to analyze it, store it, or manage it.

This situation is faced today by many companies in all industries: our ability to produce data is growing at unprecedented rates, which creates a new challenge commonly known as Big Data.  And as technology evolves, this challenge will only get worse.  With trends like The Internet of Things, more and more devices are becoming data producers.  In short, Big Data is only getting started..

As Software Product Managers, we need to make sure that our product is ready to handle large amounts of data while providing a good user experience.  There are many considerations in this area, but here are 6 key areas you should definitely put some thought to in order to ensure your Product is able to keep up with the times.

The 6 areas every Product Manager should consider are:

  1. Personas
  2. Data storage
  3. Analytics
  4. Visualization
  5. Data access tools
  6. Security

#1 – Personas: Who are you building this for?

An important first step is to understand the different ‘data personas’ that will use your Product.  For example, here are some high-level groupings of possible persona types:

  • Data producers
  • Data consumers
  • Personas focused on report creation
  • Personas focused on maintenance or data integrity
  • Etc

And within each of these groupings, you’ll have sub-categories based on different user roles and contexts.  For example, data producers might be broken into:

  • Engineer:
    • At the lab (desktop use, investigatory by nature)
    • On the go (tablet use, quick access to specific data)
  • Executive:
    • At his desk (web-based use, reports and drill-down capabilities)
    • On the go (phone use, quick notifications and dashboards)
  • Etc

Think carefully about all of your personas and map out each of their scenarios, so you can have a good picture of all of your use cases.  These personas will have an impact on the decisions you make in each of the following 5 areas.

#2 – Storage: Think carefully about your strategy

The two main considerations regarding storage are: how to store and where to store.  How to store your data depends on your overall use case.  The type of data you produce will determine the type of database you will require.  If you have structured data, then a relational database such as SQL Server or MySQL are your best bet.  On the other hand, if you have unstructured data such as images, videos, or tweets, then you probably need a schema-less database such as Hadoop or MongoDB.  Or maybe, like some systems I’ve worked on, you need both.

I’m not suggesting that Product Managers dictate the type of DB or the architecture of the data tier.  That’s the role of your Architecture and IT team.  However, it IS our job as Product Managers to define clear use cases and convey those to our technical team, so they can implement the right infrastructure for your product.

As far as where to store, well, that’s more of a business question.  Depending on your business model and the understanding of your customers, you need to decide if your product should be installed on-premise or in the Cloud.

If your Product is Cloud-based, you’ll need to ask yourself some questions about your servers, such as:

  • Should my company build and maintain that infrastructure ourselves?
  • Should we leverage third-party infrastructure providers such as Public or Private Clouds?
  • Should we go for a Managed Hosting approach?

Again, it is IT’s role to define the infrastructure, but as Product Managers, we should define the usage requirements, the scalability, elasticity, and ultimately the business model.  Whichever direction we go with, we need to make sure it’s backed by some good ROI analysis.

On the other hand, if your  approach is to be on-premise, then on top of storage, you’ll need to decide how your solution will be deployed.  For example:

  • Should you only sell the software, and require your customer’s IT to be responsible for providing the hardware?
  • Should you provide and deploy the hardware in the form of servers that can be expanded as needed?  Will you offer a hardware maintenance contract?
  • Will you offer your product as an appliance, meaning they get a system in a box and therefore, the storage characteristics are pre-defined?

Regardless of the choices you make, it’s important to consider the amount of data your system will produce and project the storage needs for 1 year, 5 years, and so on.  That way you can calculate the elasticity of your Product and even consider using storage as a parameter in your pricing model.

#3 – Number Crunching and Analytics

The goal (and challenge) of Big Data is to get actionable information out of raw data.  Therefore, providing good analytics is very important.  Many companies focus only on storing large amounts of data, but their analytic tools often fall short.  Remember, storage is just half the battle.  Some questions to consider regarding analytics include:

  • Will you serve your users with raw data, or will you massage the data before consumption (i.e. reports)?
  • Will the data processing occur on-demand, or will you pre-process the data to increase performance (i.e. cubes)?
  • Is all data coming from the same place, or do you need to aggregate data from multiple locations?
  • What are your performance requirements for retrieving data?

When building your Product, keep in mind that building analytics tools is a huge endeavor.  The Business Intelligence (BI) industry has come a long way, and they often provide very powerful tools for you to white-label.  Unless your product is, in fact, an analytics tool, I strongly recommend outsourcing this part of your solution.  That way, you can focus on building your core competency while leveraging best of breed data analytics and visualization tools. Gartner-BI-Analytics-Quadrant-2014 There are many BI players that offer robust OEM solutions.  Gartner’s BI Magic Quadrant shows some of the top ones.  Now, keep in mind that these solutions are not cheap.  But think of it this way, for less than the yearly salary of one single engineer, you can include a top-of-the-line BI solution into your product.  The price often includes updates, support, plus assurance that you’ll be protected against obsolescence.  It’s easy to see the ROI of this investment if you compare that cost with how much progress one engineer can make in a year.

#4 – Provide Great Tools for Data Visualization

Number crunching is important, but in my opinion, it means nothing if the data is not presented in a user-friendly and useful way.  When creating a roadmap for data visualization, it’s very important to understand your user personas and demographics.  It’s also very important to understand the user state of mind when accessing data, including form factor.  The same persona might be interested in different visualizations when looking at data at his desk, on a tablet, or on a phone.

Also, try not to be prescriptive.  One of the benefits of Big Data is the opportunity to explore and discover trends in our same ‘ol data.  As Product Managers, we should understand the use case and provide some basic reports and views.  But we can’t assume to know everything that the user will want to know or do.  Instead, we should provide tools that enable the user to explore the data at will.  I remember when a customer told me:

“With your Product, I need to be able to see my data in any way I can think of today or any day in the future.”

At the time, I was frustrated with his comment, but now I understand the need for this flexibility.  I apply his wisdom every time I’m working with my team on data visualization.  By the way, visualization is another area where BI platforms shine.  They already provide dashboards and tools that give the user this exploratory ability out-of-the-box.  Something to keep in mind.

#5 – Provide Multiple Ways to Access the Data

As much as we’d like to anticipate all the possible user scenarios and build them into our product, we know that’s not realistic.  Let’s face it, it doesn’t matter how good of a tool you provide, your users will always want to use other tools–maybe because they are already familiar with them or maybe because they are superior.  For example, business users will continue to love Excel for a simple reason: it’s an amazing tool!

So, instead of fighting it, embrace it.  As part of your roadmap, be sure to include ways to get the data in and out of the system through various methods, not only your UIs.  Good examples include Import/Export utilities and migration tools.

Of course, the most flexible approach is to provide an open API to access your data.  That way you can build additional tools yourself, or you can enable your customer or the community to write whatever tools they require.  In fact, this approach has been very successful for companies like Box and Dropbox which generate a large portion of their revenue from API usage.  Having an open API is a big topic that has huge business and technical implications…a topic for a future post.

#6 – Focus on Security

Security is always a big concern, and it has been getting a lot more attention given the recent security breaches of companies like Target.  For Cloud-based systems, there’s an added concern since the data is stored outside of the customer’s firewalls.  Plus, if you are building a multi-tenant system, then there’s the added security risk of other tenants getting access to your data.  Yikes!

The topic of security is a very big one, and it really needs to be the focus of any Software Product Manager.  It’s our job to define the right criteria and to ensure that you have the right quality controls to ensure your system is bulletproof.  Security cannot be an after-thought.  It must be baked into every feature and should go across the full stack, from databases to API to User Interface.  It should also be part of your QA testing and acceptance plans.  And it should be tested in every sprint, and bugs related to security should always be prioritized high before any release.

I highly recommend hiring security companies to act like “hackers” to break into your system and show you where the weaknesses are.  Financial institutions use them all the time, but they are becoming common practice for any Cloud-based company.

The Bottom Line

Big Data might be a buzz word, but it’s a very real challenge, and it’s going to hit your Product sooner or later.  Will you be ready?  The considerations above are certainly not all-encompassing, but hopefully, they will provide food for thought and somewhere to start.

I’d love to hear from you about your approach and experiences managing Big Data, and any additional important areas you consider in your product plans.  Please leave a comment below, and if you liked this article, please share it around!


  1. Vinay Kumar 6 years ago

    Hey Daniel,

    Thanks for sharing the knowledge in very simple way. I really loved the way you categorized the stuff.
    Keep up the good work!


  2. Seema 7 years ago

    Loved it! I was glad to see the PM lens was intact even with otherwise what people like to see in a nebulous way-big data, yes; sealoads of data, yes; but in context of problem and persona!

  3. Anjali 7 years ago

    Hi Daniel
    Great article, extremely informative! I am currently working in the Data Warehousing domain and looking to jump into the big ocean of Big Data. Do you suggest learning from a professional training provider or self-learning? Also, can any one provide any pointers on a good training provider or other online learning sources?

    • Daniel Elizalde 7 years ago

      Hi Anjali,
      I’m glad you enjoyed the article. Big data is a big, umbrella term. Trying to learn “big data” is like saying I want to learn “engineering”. You need a more narrow focus so you can target your learning. I suggest identifying the type of roles and companies you’d like to work in and then connecting with those people to get advice. That way, you can understand what type of training you’ll need and what is the best way to get it.

      Best of lucks!

  4. Chary 7 years ago

    Hi Daniel,
    Good and insightful article. While preparing the proposition for a data- based service model, i was faced with the challenge of answering the basic question of – whose needs are we addressing, are we addressing the right needs and and how is our product going to satisfy those needs. Unless these are clear, simply exhibiting our skills and expertise in big data handling and presenting nice looking visuals will not get us anywhere.

    • Daniel Elizalde 7 years ago

      Thank you for your comment. You make a very good point. Many companies focus simply on displaying random data, instead of researching the needs of their users and focusing on adding real value.


  5. Ross Hall 7 years ago

    #7 – Regulation. Regulatory constraints around what is / is not acceptable should be factored into Product Design and ongoing Management. Missing this at the outset creates the potential for products to be shut down, companies forced to pay compensation and individuals to be removed from post.

    • Daniel Elizalde 7 years ago

      Hi Ross,
      Great comment! I agree 100%. In fact, Regulation is already baked into my IoT Decision Framework as an area that can make or break your product. You can read that article here:

      Let me know what you think,

  6. Luiz Marcelo Santos 7 years ago

    Hi Daniel. I like the concept “Here”or “To go” for data producing based on users perspective and context. Actually, users needs, data storage and security are still walking together. Thanks for that.

  7. Margarita Hernandez 8 years ago

    Hi Daniel:

    The pricing and the costs structure of an IoT product is similar to any other “real world”/handcrafted product? or there are any others considerations involved in the cost calculation?

    Thanks in advance,

    Margarita Hernandez

    • Daniel Elizalde 8 years ago

      Hi Margarita,
      Good question. Cost calculation for IoT products needs to consider the cost of the “real world” device plus any ongoing digital costs required to offer your service. This includes the cost of sending data to the cloud, cloud storage and processing, etc.

      The price will depend on your business model, but as opposed to “real world” products, IoT products often include a recurring (i.e. subscription) component. You’ll need to decide how much of the device the customer pays for in advance or is amortized throughout the life of your product.

      I plan to write a full post on calculating costs the IoT technology stack. I’ll email a notice to my newsletter when the post is out.

      I hope this helps. Thanks for reading!

  8. Jocelyn Byrne Houle 9 years ago

    Very helpful article! I am teaching a class on product management for technology products at GA. I would like to include some thoughts on how a product manager can successfully interact with Analytic teams. Data Scientists are in their ascendancy and, as a result, it is not always clear who is setting the agenda for the product or data sources. Thoughts? experiences? strategies?

    • Daniel Elizalde 9 years ago

      Hi Jocelyn,
      I’m glad you enjoyed the article. Interesting question about data scientists and PMs. I’ve run into this before and I can see where there can be confusion.

      I look at this in two parts:
      1) Data scientists respond to a business need: From talking to customers/market/competitors/strategy, etc, the PM determines there is an opportunity to provide more value to users by performance additional analysis on the data. In this case, the PM would define the business goal and task data scientists to come up with the best possible solution by looking at the existing data. Once the solution from an analytics perspective is done (or meets the business criteria), then the PM works with analysts and the rest of the team to productize it.

      2) Data scientists work in an “R&D” fashion: By virtue of working with the data, data scientists might come across new trends or insights that would provide value to the customer. This is communicated to the PM who then determines how/if to productize this new finding.

      The key is that PMs are the ones who are responsible for the product. Even if data scientists discover some new insight, that doesn’t mean it will be immediately included in the product. PMs need to prioritize it, make the supporting business case and drive the creation of a customer-ready product out of this. There is much more to a product than just the insights from data analysts. There is how your users will interact with it (UX), pricing, ultimate value to customers, strategic alignment, cost of development, opportunity cost, etc. That’s where the PM adds the most value.

      What do you think about this approach? What has been your experience?


  9. Ryan 10 years ago

    Reading a Product Manager who understands the user centered approach gives me hope for the world, thank you Daniel.

    I would add that, prior to implementing a Big Data capability, one should know why they want it and what decisions they want to be informed by it. It sounds obvious, but this is one of those things a lot of companies will do because everyone else is doing it.

    • Daniel Elizalde 10 years ago

      Ryan, thanks for the kind words! You bring a great point. It’s important to understand why you need all that data and what will be your approach to managing it. They also need to consider the total cost of ownership which probably includes dedicated staff to generate reports, data mining, maintenance, etc.

      Thanks for reading.

  10. Olga Nash 10 years ago

    When does the Records Management get involved? Do you ever consider retention periods for the data stored in the system? What about purge functionality? How will the data be destroyed, once it has passed retention?

    • Daniel Elizalde 10 years ago

      Olga, good question. From a Product Management perspective, Records Management should be included as part of your requirements. The need will be different for every company, specially if they belong to a regulated industry like Health Care or the Military.

      The requirements and then the solution design should capture all the places within the system that need to comply with the records management policy. The database will need to have special design considerations, your business layer will need to account for security, tighter permissions, etc. Your UIs will need to expose only the info that is needed and prevent the user from violating the policies established by the company (i.e. your can’t print a document or nobody can delete a record).

      At the end of the day, it will be a joint effort between the Product Manager and the Engineering team to make sure all requirements are accounted for, validated, and fully tested. What do you think?

      Hope this helps. Thanks for reading,

Leave a reply

Your email address will not be published. Required fields are marked *