While looking at articles about technology trends, business productivity, and resource management, you must have encountered the topic of big data.

Big data has penetrated almost all fields and industries.

In an article, Hanah Anderson and Matt Daniels, have used data to study the film industry in Hollywood’s alleged sexism and racism.

Big Data and Internet of Things (IOT) has enabled our everyday grocery labels to become “smart” further enabling the collection and analysis of consumer purchase behavior. 

For more ways big data can be applied, Bernard Marr’s 10 Use Cases Everyone Must Read is a quick and insightful read.

—

But what is big data and how does it affect you? More importantly, is big data an opportunity for you to capitalize on or will it spell the end for your business and occupation?

Early this year, Fukoku Mutual Life Insurance fired 34 employees and replaced them with IBM’s Watson Explorer AI. According to industry watchers, this is likely going to be a trend in the financial and E-commerce industries as automation becomes necessary for survival (demand-driven) and machine intelligence becomes mainstream, cheaper and easier to adopt (supply-pushed). Both demand and supply factors are closely linked to the big data revolution.

This is what my article explores. I’m going to give you big data, in a nutshell, enough information to get you started and show you how you should invest your time, money and human resources.

What is Big Data?

You’ve might have heard of terms such as:

• Internet-of-Things (IOT)

• IFTTT

• Data Analytics

• Hadoop and HDFS

but if you aren’t sure what they are about or how they are relevant to your lives, fret not.

I’ve researched and compiled information that is relevant for entrepreneurs, employees, and business owners who want to gain an insight into big data. The aim of this article is to help you understand how big data can be relevant to your company, and help you make informed business decisions that will value-add to your organization.

_________

Disclaimer: I’m not profiting from the marketing or sales of any commercial big data platforms or IOT products at the point of writing this article. The ideas stated here are for the purpose of your education.

________

Big Data in a nutshell:

Data refers to any ‘bit’ of information. Companies can collect and monetize data by selling them or using them to optimize their own sales processes, marketing campaigns, product development, human resource and other organizational operations. Insights from analyzing data play a critical role in substantiating pivotal business decisions.

“Big data is a term that describes the large volume of data – both structured and unstructured – that inundates a business on a day-to-day basis. But it’s not the amount of data that’s important. It’s what organizations do with the data that matters. Big data can be analyzed for insights that lead to better decisions and strategic business moves.” – a definition by from an article by SAS.

Who is generating these data?

“ Data now stream from daily life: from phones and credit cards and televisions and computers; from the infrastructure of cities; from sensor-equipped buildings, trains, buses, planes, bridges, and factories. The data flow so fast that the total accumulation of the past two years —a zettabyte—dwarfs the prior record of human civilization. ”

– Jonathan Shaw, in Harvard Magazine “Why ‘Big Data’, is a Big Deal” 

Everyone and (almost) everything is generating data. It is possible to categorize them into these two types of sources:

• User (people) generated data: Personal details uploaded through registration forms, Facebook posts, user photos, videos, articles, tweets

• Machines: Modern appliances in smart home systems, sensors, healthcare equipment, traffic lights, etc. Your personal computer stores log files about processes and software use, and this information can be sent as diagnostic reports for computer maintenance.

These data takes two forms:

1. Multi-structured / Structured data: refers to data that have a high degree of organization, such that inclusion in a relational database is seamless and readily searchable by simple straightforward search engine algorithms or other search operations

2. Unstructured data: is essentially the opposite – complex algorithms are required to sift out information.

Since modern states, with a proliferation of commercialised personal digital products, churn out colossal quantities of data, individuals and companies face the challenge of finding ways to sort out important data, then store and process them so that they can be found and utilised in the future (or better yet update them with real-time changes).

How did the Big Data Revolution come about?

One way to understand big data is through the 3 principles, as articulated by Doug Laney:

Volume. Organizations collect data from a variety of sources, including business transactions, social media, and information from sensor or machine-to-machine data. In the past, storing it would’ve been a problem – but new technologies (such as Hadoop) have eased the burden.

Velocity. Data streams in at an unprecedented speed and must be dealt with in a timely manner. RFID tags, sensors, and smart metering are driving the need to deal with torrents of data in near-real time.

Variety. Data comes in all types of formats – from structured, numeric data in traditional databases to unstructured text documents, email, video, audio, stock ticker data and financial transactions. 

To call it a revolution is a bit of a misnomer. This is because the rate of adoption and application of new IT technologies varies across industries and geography. Nonetheless, Big Data came about because of the general improvements in technologies that enabled a greater volume of data storage, higher transfer speeds (velocity) and a proliferation of data formats, hence involving all three principles. With the increase of each principle, new innovations in programming like Hadoop and HDFS has enabled more efficient storage, retrieval, transfers and online communication.

Hadoop is an open source, a Java-based programming framework that supports the processing and storage of extremely large data sets in a distributed computing environment. It is part of the Apache project sponsored by the Apache Software Foundation. “

– Margaret Rose in an article Hadoop

The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware.

– Dhruba Borthakur in an article HDFS Architecture Guide

Today, companies can leverage on commercialized products like IBM Analytics, Google Analytics, SAS  to process data and gain (almost immediate) insights. InformationWeek wrote an article about popular platforms in 2014, and by 2016, the amount of data analysis software, APIs, predictive or visualization tools have exploded. See PCMag 2016 products and Forbes for a comprehensive list of big data and business analytics tools to work with.

Commercial products are commonly developed to tackle challenges faced incorporate functions that are standardized throughout different industries, such as human resource, sales, and marketing reporting, scheduling and updating. This happens because business analytic software and platform developers follow where the market is; Only larger companies can afford to invest and work with customized service providers to analyze their data, develop analytic tools that are specific to the type of data that they require and address their challenges and objectives. This is too costly an option for small firms, especially without government grants. However, there is still a niche for small data where data is still developed in small volumes that make it comprehensible, assessable and actionable.

How is it relevant to you or your business?

• So what if I do nothing? Is it going to be ‘business as usual?’ Perhaps not, and here is why.

Failing to develop the capacity to put data to good use is wastage and incurs high opportunity costs. According to Forbes report, at least half of small businesses fail to adopt modern technology, and Jack Ma, in CeBIT 2015, said that companies that don’t capitalize on IT technologies will find it challenging to stay afloat. Investing in technology to process data is necessary for all businesses that are in it for the long haul. Ma adds that when modern internet companies don’t typically last more than 3 years, it is not possible for them to ever become mainstream.

“ Once regarded as hype, big data and advanced analytics are now busy transforming the enterprise. Organizations, determined to gain a competitive edge — or simply remain competitive — invested heavily in services, technology and people in 2016. The trend doesn’t show any signs of abating soon. “

– Thor Olasrud in an article published in December 2016

Olasrud highlights these main points:

1. The market for data analytics is growing.

2. Careers, staffing, and training remain top-of-mind for most companies: Organisations and institutions in 2016 expanded their efforts to train, hire and retain data professionals.

3. Data analytics going into production: IT operationalising data; Executives are slowly adapting to data approaches

4. Organizations share their successes: companies are open to shared learning and discovery

Big names like General Electrics (GE) are investing a great deal into processing data. Since December 2016, GE has purchased over USD$1.4 billion worth of equity and ownership in such software companies.

“ GE is in a hurry to digitize heavy industry. It’s not worried about borders or worker uprisings or politics. Its laser focus is productivity and profits. “

– Jon Markman

My research reveals that many sources allude to how big data may be the key to longevity because it enables timely decisions to be made in an environment where rapidly changing consumer demands greatly influences the survival of firms. A company that has just 5 key products can be phased out in a matter of days given the rate of venture capital funding, the emergence of disruptive technologies, and ease of access to these new technologies by consumers.

Entrepreneurs and business owners should be concerned if they have not factored big data into account.

Maximizing company potential with big data to stay ahead:

1. How to Harness big data

1. Recognize that big data is a challenge that must be addressed.
Regardless of your industry, review your IT infrastructure (if any at all) according to the 3 principles of big data. Can your IT infrastructure collect and store a large volume of data? Do you have the tools to process these data? Is your IT strategy designed to produce insights that would help your business objectives?

2. Big Data should be part of a wider information management strategy
Mobile data technology is one of the ways a key business process can be automated. With any inclusion of technology, there must be supporting infrastructure to monitor and assess usage. Therefore, IT’s task is to work closely with business strategists and look critically at their database management.

3. A move to convergence between traditional relational databases and new technologies involving big data technologies has brought about different forms of such convergence. The industry is at a stage where it is awaiting for a dominant model to appear. This will directly affect how future databases are developed, and what skill bases that you currently possess can still be relevant. Big data won’t render your current SQL databases obsolete but they will run parallel (especially since they are here to stay).

2. Data exist beyond your own internet entities (company website, social media, blogs, etc).

1. Public archive and data collection companies are huge repositories of data.
Since this is a topic that is industry specific, I shall not delve into the details. MarketingWeek has a helpful article on “How to find the perfect market research partner“ that you should check out.

2. Partnership with research companies and institutes may be able to help you leverage on new data, external processing capabilities, and learn about new emergent technologies before they are available to the mainstream audiences.

3. Invest in the mobile advertising space. It may seem that the mobile device (and app) trend isn’t a new one, but mobile device adoption is still on the rise (especially in developing countries).

____

Additional notes:

1. Other Data Analysis Tools for Businesses
http://www.kdnuggets.com/2014/06/top-10-data-analysis-tools-business.html