Big data is an evolving topic. The new technology is not suitable for standard simple relational databases, but it is used for collecting, storing and processing large amounts of data. Correct implementation of Big Data technologies can benefit various organizations to analyze their data and improve business decisions.

This article will help you know more about Big Data and what it means and how insider threat detection tools can be used to safeguard your data online. Learn about the need of Big Data and data collection techniques. Get to know more about the details and characteristics of Big Data. Touch upon Big Data processing techniques and technologies.

What is Big Data?

Data, as we understand are any facts, figures and characters; anything that are collated or put together for further analysis. Data can be collected from anywhere – it can be the number of time a person breathes, the distance between two solar systems, sales and profits of companies. Basically, any kind of statistics collected is called data. Data becomes information once it is analyzed and transformed into useful data. Advanced technologies, use of the internet for transactions, eCommerce, Internet of Things (IoT), all contribute to this growing data.

In the last few years, with advancing technologies, the digital universe is increasing every minute. It is said that by 2020, the digital bits generated will compete with the stars in the universe. It is doubling in size every two years and will reach 44 Zettabytes. Enterprises will manage 50 times more data and files will grow 75 times in the next decade. Middle East and Africa are leading the pack when it comes to increase in Internet Traffic, closely followed by Asia Pacific.

When data becomes voluminous and cannot be analyzed by simple traditional data analysis methods and simple relational databases; but requires special processing, it is called Big Data.

So Big Data is-

• Large collection of data and datasets
• Strategies and technologies to process this data

Big Data as a term is being used since 1990s and John Mashey, a computer scientist and an entrepreneur is considered to be one who made the term popular.

In 2012, Gartner came up with a modified definition of Big Data – “Big data is high volume, high velocity, and/or high variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization.”

Where does all this data come from?

With increasing digitization and evolution there are more than 5 billion people who use mobile devices of some form or the other. Transactions over the internet have increased; social media interactions have increased, use of electronic systems is on the rise; basically more and more people are using the Internet, thus generating voluminous data.

Data can be generated from either of the following, but not limited to these source –

• Social Networking Sites – Facebook, Twitter, YouTube, Instagram, and many more such sites generate data by people adding “Likes”, “reactions”, comments; taking surveys. All these events add to the data flowing on the Internet. This data is analyzed later to give an insight into what the customer needs or likes to see and use.
• Business Transactions over the Internet, or as it is more popularly called – Cloud Computing – More and more people are carrying out business over the Internet. All this generates data. Orders are placed and Invoices are paid over the Internet. A business similar to say Amazon, has data about Orders, Wish lists, Delivery records, Inventory, etc. All this is on the Cloud and contributes to e-data being generated.
• Data from electronic machines – Digital sensors, weight balances, global positioning systems, aircraft, ships, closed circuit cameras (CCTV), smart-phones and instant messaging services, all generate electronic data. As usage of electronic devices increases, so does the data being generated.

As can be understood, volume of data being generated is growing exponentially.

Big Data and Internet of Things

Internet of Things (IoT) is connection of different computing devices over the Internet and can be used for everyday living or for business. Using data generated by IoT, companies and media can target the correct users opening up new avenues of growth. As usage of IoT grows; so does the data being generated and collected.

Characteristics of Big Data

Different organizations and consultancies have different nomenclatures, but here we will be looking at the most common characteristics, called the “Vs” of Big Data. The following diagram developed and created by IBM shows those “Vs”  

• Volume – The data is always large in volume, generated from multiple sources. Voluminous data is a prime characteristic of Big Data. If the volumes are small, they can be easily analysed by traditional computer systems. Though there is no specific nomenclature used, volume of Big Data is often described in terabytes, petabytes, exabytes, etc. This data may be raw or processed; maybe structured or unstructured; maybe simple data or maybe media files, so there is no fixed format for the data.
• Velocity – Data that moves quickly, or is generated quickly, e.g. likes and comments on twitter change at a rapid pace. A primary characteristic of Big Data is its ability to change rapidly, as more and more data pores in. Based on the data and analysis requirements, data maybe processed using either real time processing or batch processing.
• Variety – Variety, as the word suggests is different types of data, structured, unstructured, images, videos, etc. Considering the different sources of data, there is a lot of variety in the data that is collected. While analysing the data, the variety of data collected and its segregation plays a major role.
• Veracity – Veracity is uncertainty and quality of data that is collected. For accurate analysis, accurate data collection is a necessity. Based on the sources of data, some data may not be authentic, or honestly collected. Also, the context in which the data is collected is also important. For the same question, under different context, a person may give a different response. So, knowing all this is paramount to get useful data.
• Validity – The data that is collected should be valid. Just generating and then analysing vast amount of data may not give the correct or accurate results. If we are studying market trends, then these trends are valid for a certain period, a certain season; so, validity of the data that is being studied needs to be ensured.
• Variability – Big data is generated from varied sources. This leads to peaks and troughs of extremes. These extremes can give skewed results, so data that is collected can be inconsistent and trends need to be analysed or extremes need to be vetted out before analysis. 

Importance of Big Data

As we read earlier, all this data generated by each and every one of us will be wasted, if it is not put to good use.

A few ways of using Big Data is by studying it with a specific focus, to achieve –

• Reductions in Costs
• Reductions in excesses, leading to savings on required time.
• Developing new products and offerings
• Better, calculated and decision making
• Root cause analysis techniques help us to find and correct failures and defects
• Find irregularities and frauds, if any

 Use of Big Data

Big Data can be used by different types of organizations, whether profit-making, non-profit or Government. Below are some of the different types and how they can use it. The list and details are not exhaustive, as that will be a separate topic of study all together.

 Government

Government organizations can use Big Data with appropriate analytical techniques to know more about the kind of response that is being garnered from the population on its schemes. It can be used for launching government schemes, control crime, improving public utilities, etc.

Banking

According to the information by Fully-Verified and other services, Banks generate large amounts of data. This data when analyzed can be effectively used by banks to increase customer satisfaction, minimize risks, ensure frauds are controlled and effective schemes can be launched.

 Education

Education systems can be improved manifold by using Big Data results effectively. Deciding new courses, student’s progress, improvements in teaching techniques, can all be decided by using Big Data findings.

Health Care

Using Big Data effectively, can make wonders in the health care industry. Accuracy of findings and the ability for quick, in-depth findings is giving a boost to the health care industry.

 Manufacturing

In 2013, TCS disclosed results of a Global Trend Study and in the study it was found that supply chain management and boost in quality of products is a major take-away from usage of Big Data Analysis techniques. Big Data technologies are providing a more predictable environment for the manufacturing industry with optimization in the manufacturing cycle by reducing waste.

Retail

Customer relationships form the foundation of the Retail industry. So, what a customer wants is the driving force in the Retail sector and Big Data is the perfect fit in this case to give you all the details. It forms the core of the sector and helps in driving decisions related to market requirements and strategy.

Other sectors where Big Data is useful are

• Media
• Information technology
• Science
• Sports

And many more 

Big data lifecycle

Now, that we have looked at what Big Data is and its characteristics and uses, let us read a bit more of what is the actual process that the data goes through and what does the system look like.

The lifecycle depicted here is a generic lifecycle and it may be modified based on requirements.

Collecting Relevant Data

As we have read earlier, data is available from multiple sources but all of this data cannot be added to the system for analysis. So, based on the business of the Organization using the data, relevant data is collected and added to the Big Data System. The complexity of this data to be added depends on the type of data collected, how it is stored initially and the processing that it is to be subjected to.

At times, before data is added to the system, it undergoes some initial processing involving basic analysis and sorting. This is at times called “Extract”, “Transform” and “Load” or ETL for short.

But, there should not be too much of processing, because it may then not give desired results.

Storing Data

As can be seen, the volume of data is huge and cannot be stored using simple, traditional storage solutions. Technologies that can handle high volumes as well as access data at faster speeds need to be used.

As a primary characteristic of Big Data is Velocity, the computer power required is large and single servers will not help, so a strong distributed system maybe required.

With technological advancements, it is no longer necessary to hold on to all that data by investing in storage technologies. There are low-cost options that enable an organization to store data on the Cloud, based on the organization’s needs.

A key consideration while choosing storage technologies, whether in-house or outsourced is the privacy and integrity of data as it is of paramount importance.

Analyzing Data

Now that we have all the required data collected and stored appropriately, it is time to analyze this data.

Some organizations that can afford to have high-performance machines and technologies can analyze all the data, while some have to analyze partial data.

Data analysis can be done using either Batch processing or Real Time processing.

Batch processing –

This type of processing involves breaking the entire task into sub-tasks and then scheduling them one after the other. During the intermediate steps, data is re-shuffled based on findings and then processed again. This is usually over a large dataset that requires more computations.

Real-time processing –

As the name suggests, processing of data is done in real-time. This means that the computation timings are smaller and the information is immediately available to the user. Most of such computations are in-memory and the data is not written back to the disk every time. 

Studying Results

It has to be understood that studying Big Data requires proper knowledge of trends and interpreting the findings. Big Data results are not simple results, but more of trends and forecasts, so visualization is a key while studying results.

Graphs, charts, tables are used to study the results for appropriate information.

One most important aspect is that, if we feed junk, we will get junk, so the data that is the input needs to be validated. After that come the processing algorithms. If these are incorrect, the entire exercise will be futile. So, every step of Big Data Analytics is important.

Critiques and Concerns of Big Data

There are some concerns that academics and researches have raised on Big Data. The concerns are on two different fronts –

1. Privacy of Data – As the data is analyzed and accessed by multiple people and agencies, private data can fall into the wrong hands and used for incorrect, fraudulent activities. Collection and processing of personal data needs to fit into the legal framework of the countries where it is used and at it may not be feasible to regulate data collection, especially over the Internet
2. The techniques used – Academicians, and scientists have raised concerns on sampling techniques that are being used. They at times are not confident of the models being used for analysis, but this is a point open to debate and the debate still rages.
3. Human involvement – The final results, no matter the technology used are dependent on the humans operating the systems and analyzing the data using queries. Tools do exist, but if the question asked is incorrect, the answer you get will be incorrect as well. Specialized data scientists are the requirement of the day, along with good technologies like Hadoop, etc.

As of now, Big Data is still being used with these issues being addressed and is effective.

Hadoop

Hadoop is an open source software framework, which is part of the Apache project, sponsored by Apache Software Foundation.

Hadoop is used to store large volumes of data and running analytics software on clusters. It has a distributed file system that helps in faster data transfer and processing. It also provides thousands of commodity hardware nodes that enable handling large amount of data.

Due to its distributed nature, complete system failure and loss of data is a rarity, which makes it an attractive option for people who want to invest in Big data.

Hadoop allows for distributed processing of large volumes of data using the Hadoop Distributed File System (HDFS). The HDFS allows data to be stored across multiple storage devices, linked together allowing large volumes of data to be accessed at fast speeds.

Hand-in hand with the HDFS goes the MapReduce system or module of Hadoop. As the name suggests, mapping is the mapping of data such that it is easier for analysis. “Reduce” stands for the operations carried out on the data that help reduce data sets.

Conclusion

In everyday life organizations, as per design, or inadvertently collect data. If this data is not processed and translated into information that helps business, it is a futile exercise.

Big Data is a technology where large volumes of data are stored and processed and analyzed to improve business.

Big data is resource intensive and so smaller organizations, as well as some big ones cannot afford the investment required. That is when cloud computing and distributed systems help.

So, big data is an evolving technology that is helping organizations reach business decisions faster, based on facts and figures.

References

•  https://www.sas.com/en_us/insights/big-data/what-is-big-data.html
• https://www.sas.com/en_us/insights/big-data/what-is-big-data.html#modal1
• http://www.zettaset.com/index.php/info-center/what-is-big-data/
• http://searchcloudcomputing.techtarget.com/definition/big-data-Big-Data
• https://www.ibm.com/big-data/us/en/
• http://iveybusinessjournal.com/publication/four-strategies-to-capture-and-create-value-from-big-data/
• https://www.emc.com/leadership/digital-universe/2014iview/executive-summary.htm
• https://www.sas.com/en_us/insights/big-data/hadoop.html
• http://searchcloudcomputing.techtarget.com/definition/Hadoop