Keeping up with Big Data
Big Data is at the forefront of the challenges that organizations face today. But the first question is: What is big data?
Big data is a term for data sets that are so large or complex that traditional data processing application software is inadequate to deal with them. Big data challenges include capturing the data, its storage, analysis, and information privacy.
Frankly, these challenges are not new. The disciplines required to look at big data have been with us for many years. Data is a precious asset whether it is personnel, partner, or customer data, and needs securing from both internal and external access.
The good news is that products that deal with data have consistently improved over the years from the perspective of storage, security, and user access. The latter has seen massive strides with a wide range of products dealing with basic reporting right through to state-of-the-art visualization techniques. The analysis of such data, with the use of appropriate analytical tools, can improve the way organizations work and provide real benefits to all.
Examples of the use of big data cut across private, public, voluntary, and government sectors. For example, measuring the success of a new drug or technique in the health sector can be analyzed by measuring its worth right down to the finest detail.
Previously, analyzing millions of records was a genuine challenge. Now, not just millions, but billions of rows of data can be processed and analyzed almost instantaneously because of improvements in analytical products and software. The cost of information technology has also fallen drastically, making data analysis cheaper than before.
Another key driver of big data is The Internet of things (IoT). Put simply, IoT is all the devices that dominate the modern world: Any device that can connect or be connected to a network and can store data that can be used for other purposes in real time or for later analysis.
The key point here is the data these devices collect; this data will drive the IoT market place. The add-on here is the analysis of the data: Without analytics the data would be redundant as it cannot be used.
Key challenges
One trap we fall into is the concentration on “product.” There are many good products to store and analyze data. All of them do a fine job — admittedly with varying degrees of success.
As well as focusing on the product that is used, we need to look at the processes that follow the data journey to ensure what we are analyzing is correct. We also need to concentrate on the cultural issues.
Some companies have made multimillion dollar investments to deliver insight to them. The major constraint has been that less than 40% of employees have sufficiently mature processes and skills to do so.
The data journey is critical to success. Firstly, the data captured must be validated and stored in a secure way. Any analytical solution built on bad data is by definition not fit for purpose as it leads to misleading insights on the data.
In our health example this can be as simple as the misreporting of gender or date of birth.
A set of processes and procedures need to be put in place that ensure accuracy in the data foundations, and this needs to be followed through on the data journey. The journey is the life of the data right through to its analysis.
The analysis of the data is more complex. Organizations need to invest in their employees to ensure they are true knowledge workers; by this we mean the people analyzing the data need to understand the fundamentals of the data and the reason why they are using it.
One discipline that is driving this is the new concept of a data scientist. Data scientists are the key to the success of being able to control and use data throughout an organization. They will help to make sure that the data journey conforms to the right principles. This includes the capture, storage, and security of the data and finally the analysis of the data itself.
The data scientist’s key role is around the management of the data — the data journey, and the selection of products. They will need to ensure that the products selected are fit for the purpose and deliver the real needs and insights that are required.
This does not mean that we all need to be data scientists. But what we do need is a new breed of employees: knowledge workers.
However, all of us need to understand the data and the underlying knowledge that the data will contain. This will enable organizations to leverage the investment that they have made and thus make progress.
Way Forward
We should not be afraid of data: Big or small! Data will be an enabler in the coming years and will be a key driver in the progress that we make. The important message is that data, whether traditional or big data, should be treated with respect.
All of us need to understand its importance and meaning to the organization and use it to answer the important questions that will make lives better for us all.
Paul Titley is a graduate of the London School of Economics and is now a Data Scientist delivering solutions through AlphasoftAnalytics.com in Bangladesh, Egypt, the UAE, Malaysia, and Europe.
