Big Data relevance

I never really understood the Big Data hype. Its data, but lots of it, so you need special ways to deal with it. Big deal!

But this article about how twitter processes large amounts of data really got me excited. This post too explains all the moving parts in big data, which I had no idea about before this (I heard about Hadoop once or twice)

What made it more relevant for me was this article about how big data and this fancy-named software is actually used.  This is the best part:

The data science team embeds itself with the product team and they work together to either prove out product managers’ hunches or build products around data scientists’ findings. In 2013, Garten said, developers should expect infrastructure that lets them prototype applications and test ideas in near real time. And even business managers need to see analytics as close to real time as possible so they can monitor how new applications are performing.

Now for the last few weeks I have been prattling along about relevance: may be even re-inventing yourself and learning new skills. So perhaps Big Data is a good thing to learn and get into. And its not only just whitepapers and vapourware - the industry has a real need for these skills.  The question is how long will a person with typical current skills, SQL in this case, be relevant in tech industry if these new Big Data systems are becoming more prevalent. Will a typical RDBMS and SQL become the next COBOL very soon?

There is real world training as well from Udacity(Cloudera), Coursera and EDX to help a person build these skills while still in his old job.

Relevance

IT was an exciting field to get into. It was young, and even for the layman, fairly easy to get into. You could easily build up skills, and make a name for your self. Even without a degree, there are many courses, and guys (mostly) got jobs. Our generation replaced the 'main-frame' guys - we used our Java to mock their COBOL,  as we showed them Linux running on our laptops that blew their mind.

But after 10 years now, I can see the 'gotchas'.  There is no "science" to IT. It changes far too fast to mature. And thats the double-edged sword of IT - low barrier to entry as supposed to medicine or engineering, but because it changes so fast, we are becoming quickly out-dated. Not because we don't have the desire and capability to learn and up-skill, but because we will miss the next big mind-shift, and we won't be fast enough to catch up. The next generation will take our combined learnings as innate and obvious and basic, and will define new architectures that we just won't 'GET'.

They say that Computer Science is about learning fundamentals, that can be applied in other areas. Degrees (Engineering, Computer Science, etc) is more about learning how to learn, then learning any language or system in particular. Its about the basic building blocks of Operating Systems, and databases, and other stuff. They should be able to re-learn and re-invent themselves as systems and tools change. But I argue that in 20 years or less, most of those building blocks (data structures, CPUs, networks) will be so vastly changed, that they wont apply any more. They will be so far abstracted away by new languages and frameworks, that you wont even have to think about DNS and IP any more, it will just be assumed. So all those things that we are so particular about when designing systems, like High Availability, Modularity and so forth, will be 'built-in' to the new building blocks of the future, rendering our past experiences and knowledge as null and void.

The people of a few hundred years ago had to primarily worry about food and heat. Those were their basic building blocks of their life. Nowadays, food and heat are just assumed to be there. We can get it with no effort - its included in everything we know and see. So if a person of the past appeared in this day, his complete life of experiences built around learning how to hunt, and how to generate fire, is completely useless. In the same way, as the building blocks of IT change, at some point, it will render past generations (us) irrelevant.

I think thats the key thought: relevance! Will the skills that we have built now, still be relevant in 20 years time. For a doctor and engineer, as he ages, he just gets better at what he does. For guys in IT, change and time is the killer.

It would be interesting to track how a doctor who qualified ten years, would continue to earn over the next fifty years, compared to an IT-skilled person.


Some interesting posts I came across:

• http://www.quora.com/What-jobs-will-exist-in-10-years-that-dont-exist-now-and-how-do-I-prepare-to-qualify-for-them
• http://www.quora.com/Will-a-degree-in-computer-science-still-be-in-demand-50+-years-from-today