An organization needs to be data driven in order to understand the data’s value and lament its absence
Big Data is an umbrella term for a multitude of new capabilities that are being used for storage and computing operations at scale. These capabilities allow organizations to store massive amounts of data, in disparate formats, and perform both batch and real-time analyses upon them.
The forces driving Big Data into the mainstream are the ever-decreasing cost of storage and processing, coupled with the open source enhancements of distributed systems techniques and software. Companies have realized that data storage is on the verge of being limitless, and they no longer need to be as judicious about what kinds of data they store. This realization has led to the storage of all manner of data, in addition to the traditional structured data found in relational databases.
Sometimes unstructured or semi-structured, this type of data encompasses emails, social media feeds, clickstreams, sensor data, videos and more. Further, the questions companies can ask of their data to realize value have become more complex. But the time window for analysis completion has remained the same or shrunk due to the massively parallel computation Big Data systems provide.
To organize all this limitless data – structured and unstructured – new tools have emerged. We no longer have just one hammer in our toolkit – the relational database – with which to fashion data. There are now a myriad of systems, thanks to big organizations, many of them Internet giants (Google, Yahoo!, Amazon, Facebook, and LinkedIn to name a few). Out of a need to scale storage and compute tasks confronting them, they made new data systems to satisfy these specific use cases. The most common characteristics of these systems are that they are not row based or relational databases. The systems have horizontal linear scalability (just add more nodes), and they expect components and nodes to fail so they are fault tolerant.
Most of these tools have been open sourced and, years later, are being adopted by other businesses that recognize their value. Examples of these non-relational databases are key-value pair databases (e.g., Riak), document databases (e.g., CouchDB, MongoDB), columnar databases (e.g., HBase, Cassandra), graph databases (e.g., Titan, Neo4J), distributed queues (e.g., Kafka, Kestrel) and spatial databases. There are also new computing tools such as Hadoop and Spark that allow immense amounts of data to be processed, and Storm, Samza and Spark Streaming which analyze data in near real-time, something that was previously only possible with supercomputers.
The ability to store unlimited amounts of disparate data in order to perform endless analysis in batch and real time is the allure of Big Data. But is it for everyone?
Are you ready for the Big Data bandwagon?
The promise of Big Data is very real – more data + new ways to analyze data = better business intelligence. So every business should be rushing out to see what Big Data can do for them, right? Not so fast.
Although Big Data has the capability to provide unprecedented insight into business operations, it is not for everyone. At least not yet. Big Data tools are still immature enough, even eight years after we’ve started down this road, that companies really have to be highly motivated to take on the task.
In my experience, this motivation typically comes from pain. For example, it is the pain of dropping valuable data on the floor because long-term storage isn’t feasible within the current infrastructure. Or possibly it’s the pain of not being able to monetize collected data due to technical hurdles with existing data systems. This type of pain – and the knowledge that the only way to escape it is to embark on Big Data – provides the fortitude necessary to push through on what can be quite a challenging project. In fact, it was this kind of pain, as experienced by Internet giants who needed to find a way to store, analyze and, ultimately, monetize the massive amounts of customer data to which they had access, that birthed the Big Data movement in the first place.
In addition to being highly motivated to take on Big Data, an organization must also be data driven. Organizations that treat data as currency are better equipped to tackle the challenges of Big Data, because they inherently understand its value and will nurture the initiative, no matter how unruly it becomes. Without this level of buy-in from top-level management, the obstacles of Big Data may prove to be too large to overcome.
The combination of the pain of unexploited data with a data-driven culture goes hand in hand. You typically won’t find one without the other. It is when these two factors come together that organizations can then successfully tackle the challenge of Big Data. Without the right motivation or executive support, Big Data endeavors often die before they even begin. Ultimately, an organization needs to be data driven in order to understand the data’s value and lament its absence.
Liaison Technologies is a global data management and integration company. It provides innovative solutions to integrate, transform, harmonize, manage and secure critical business data on-premise or in the cloud. With a comprehensive array of business-to-business and application-to-application integration and data transformation services, as well as on-premise and cloud-based data security solutions, Liaison’s practitioners implement data management infrastructures adapted to each client’s specific business requirements. Headquartered in Atlanta, Liaison has offices in the Netherlands, Finland, Sweden and the United Kingdom. For more information, visit www.liaison.com.
Liaison and the Liaison logo are trademarks of Liaison Technologies, Inc. All other names or product names mentioned in this release are trademarks or registered trademarks of their respective companies.