Building the right products

I've been doing 'product development' in various guises for a while now.  Recently, I even have the word 'product' in my job title. It basically involves helping companies develop and build new services, offerings and products, that they sell to their customers. It requires understanding the market, what customers want, what they need, what they pay, what competitors offer. My specific responsibility in this development cycle is the technical architecture of the product.
All in all, product development comes down to two things:

1. Build the correct products
2. Build products correctly

The 2nd one is we I/we normally focus. It includes the Technical Architecture, and focusses on building systems that are scalable, extendable, flexible and performant. We know how to do these things.

But what about the 1st one: making sure you actually building what the market and customers want and need? This usually involves a business case - that shows the need for the product, and what its returns will be. But this step (from my experience) is mostly superficial - its glossed over, and does not contain enough market and customer research. Usually this is because this product was promised by someone high up, or its a me-too product, and come what may, it will be developed and launched.

I always hear people complain about the lack of customer and market research. In some cases, if the company is really trying to be innovative, there might not be any research around. But in most cases, there should be data that can be gathered to shape the new product. I believe only a few market leaders  (e.g. Apple) exist that can invent new products, and the market will follow. I recently read about MicroStrategy, that was chosen to be amongst the top 50 companies in their ability to read the market. This leads to really cutting edge product development, and involves quite a large amount of risk I would imagine.

When companies are building new products, they have a budget, and they want the returns to be maximised. So they try to choose a product that will lead to the greatest return, based on the investment. It might not be the most snazzy or flashy product, but as long as it can promise good returns, it will be a good investment. Its like a small store or shop owner: if he is thinking of what additional merchandise to stock in this shop, he will choose something that is likely to sell well, and that is within his budget.

So the question is how do companies gather sufficient data and requirements to build new products:

• Where does requirements for products come from?
• How do companies know which products to launch?

These might be done in using different ways, sometimes employing many techniques at the same time, in different departments:

• Research and Development departments - they might consider risky and future based technologies and products
• Incubation divisions - these might be used to venture into new markets, and building completely new capabilities that the company does not have.

I imagine that even with these there is quite a high hit-and-miss ratio.  And because of changing markets and economics, things that made sense last year all of a sudden dont make any sense this year. Think of Mxit not being able to deal with the coming of smartphones. Heck - atleast Duke Nukem Forever got released.

I am still quite struck by an experience a few years ago. Two large mobile operators, at roughly the same time, launched a feature phone IM application. They both chose the same vendor. Less than 3 years later, both products were decommissioned as failures.  That means both sets of operators got the research wrong, both went down the same path, and both failed. Clever people in both companies wrote business cases, that got approved by even more clever people. And they all got it wrong. Wow....product development is hard!