Big Data Analytics Relies on Big Storage

While everybody and their uncle seems to be drooling over the potential of Big Data and the data analytics it will drive, the key enabling technology may be something less exotic and much more familiar—storage.  That is the message from a study of the Big Data marketplace released earlier this year by the market research firm IDC.  The Big Data market should enjoy a Compound Annual Growth Rate (CAGR) of 40 percent from 2010 to 2015, or seven times the rate of the overall Information and Communications Technology sector, IDC reported.  But the storage sector supporting Big Data applications should grow at a CAGR of a whopping 61 percent. Now that’s a lot of growth.

Interestingly, storage currently plays two roles in the growth of Big Data.  First, the more data that companies create, capture and store, the greater the pressure to do something with that data.  Simply storing data costs money and in many ways it is a wasted resource.  Efficient data analytics can help companies make money.

At the same time, Big Data analytics demands a high performance infrastructure.  To seriously perform analytics using terabytes of structured and unstructured information in a useful way, all aspects of the hardware infrastructure have to be upgraded.  And while moving Big Data analytics to the cloud or to third party providers may be a solution of sorts for smaller companies, that data still has to be stored somewhere.  Everybody who uses Gmail and watches the storage capacity of their account climb knows this reality.   It is not a question of the amount of storage needed, just who buys it.

But how much storage capacity a company can support is not the only question.  The desire to implement Big Data applications and reap their benefits is putting pressure on companies to make their storage infrastructure more intelligent.  According to a white paper from the Taneja Group, intelligence storage infrastructures have certain specific characteristics: 1. They are easy to scale out both in terms of capacity and performance.  2. They have resiliency and data protection built in. 3. Management of the storage infrastructure is radically simplified.

Looked at through the gauzy haze of romance, Big Data and storage need each other and each is a driving force in making the other improve.  How are you handling your Big Data initiatives?  What is the role of your storage?


Image contributed by:  Photokanok