By Brad Anderson, Liaison Technologies
In my previous blog post, I introduced some basic concepts of big data: what it is, the forces that are driving it and why everyone is buzzing about it. In a nutshell, big data is the ability to store unlimited amounts of disparate data in order to perform endless analysis in batch and real time.
The promise of big data is very real (more data + new ways to analyze data = better business intelligence) so every business should be rushing out to see what big data can do for them, right? Not so fast.
Although big data has the capability to provide unprecedented insight into your operations, it is not for everyone. At least not yet. Big data tools are still immature enough, even eight years after we’ve started down this road, that you really have to be highly motivated to take on the task.
In my experience, this motivation typically comes from pain. For example, it is the pain of dropping valuable data on the floor because long-term storage isn’t feasible within the current infrastructure. Or possibly it is the pain of not being able to monetize collected data due to technical hurdles with your existing data systems. This type of pain—and the knowledge that the only way to escape it is to embark on big data—provides the fortitude necessary to push through on what can be quite a challenging project. In fact, it was this kind of pain, as experienced by Internet giants who needed to find a way to store, analyze and, ultimately, monetize the massive amounts of customer data to which they had access, which birthed the big data movement in the first place.
In addition to being highly motivated to take on big data, an organization should also be data driven. Organizations that treat data as currency are better equipped to tackle the challenges of big data because they inherently understand its value and will nurture the initiative, no matter how unruly it becomes. Without this level of buy-in from top-level management, the obstacles of big data may prove to be too large to overcome.
Although I present these qualities separately (i.e. an organization should be highly motivated by the pain of unexploited data and should have a data-driven culture), you will typically not find one without the other. In other words, you need to be data driven in order to understand its value and lament its absence.
Other related big data reads you might like: