Time for a history lesson – the Big Data and Analytics world is changing all the time but casting an eye back at what went before can help us understand what is to come. Today, we are all swimming in a sea of data. Businesses have realized the immense potential that data-driven insights can provide and are leveraging all means to derive business benefits from it. The importance of data in today’s economy simply cannot be overstated. However, the tide of data has been rising right from the early 1970’s when ACNielsen and IRI used ‘dimensional data marts’ to increase retail sales. That was the time when the foundation of the data warehouse was being laid. However, the genesis of the modern day data warehouse as we know it happened in the 1980’s when the term “business data warehouse,” was mentioned in the 1988 IBM Systems Journal article, ‘An architecture for a business information system’. This article threw light on the importance of data and highlighted the need for an architecture “to draw together the various strands of informational system activity within the company.” In fact, it was Bill Inmon, the Father of Data Warehousing, and one of the “Ten IT People Who Mattered in the Last 40 Years” according to Computerworld, who coined the term “Data Warehousing” in the 1970’s.
Within a short period of time, data warehouses emerged as an important contributor to the success of a company. Effective storage of large amounts of data, building large reports after mining this data and bulk updates helped companies stay ahead of the curve, avoid challenges and provide strategic value by helping them take smart decisions in a smart manner.
As does everything else, with time data warehousing evolved too. As business intelligence data began piling up owing to the growth of computing power and the rise of the internet in the mid-1990’s, data management faced newer challenges. Data became harder to maintain and this gave rise to digital storage to vast amounts of data which drove the rise of BI solutions. Since the volume of data was huge, the term ‘Big Data’ was used as a broad term to define data sets that were so large and complex they were a problem for the existing computer systems and databases to process. This is when Big Data became a part of the modern day data warehousing practice. Adoption of new technologies such as Cloud and the need for real-time, high-velocity data analysis also contributed to this evolution of the data warehouses.
Initially, businesses focused on generating huge volumes of data. The idea was that the more data you had, the more helpful it was. Soon industry experts realized that along with the three V’s of Big Data (volume, velocity, variety) data also needed analysis. After all, raw information had to be converted into actionable insights and this could only be done with the help of data analysis.
Big Data analytics came into the picture to examine and decode hidden data patterns, identify data correlations by going through millions of data combinations in multiple data stores and formats that conventional analytics and business intelligence solutions could not comprehend. Big Data analytics and data visualization provided organizations a proactive way to identify weaknesses, spot trends and make informed business decisions.
As the data avalanche continued to grow, virtually every sector of the economy began to accumulate more data to extract value from it. Now this data could be structured and/or unstructured could be of uncertain quality and provenance and could often need to be combined with other working data to be useful. Data Science emerged as the interdisciplinary field which was a continuum of data analytics and employed techniques and theories drawn from several fields such as statistics, mathematics, information science, computer programming, predictive analysis, data mining, high performance computing, data warehousing and many others, to investigate problems, identify trends and find solutions for businesses and business functions.
In order to deal with the data deluge, we saw Machine Learning gain prominence. Today, Data Scientists can leverage machine learning to apply complex mathematical calculations to big data over and over again and gain insights that are far more intuitive and intelligent. Machine learning helps make predictive analytics more accurate and hence facilitates smarter decisions since it can help analyze more complex data sets and deliver results that are faster and more accurate that help in making high-value predictions without the real-time human interaction. According to Thomas H. Davenport “… you need fast-moving modeling streams to keep up” with the fast changing and growing data volumes and this can be achieved with Machine Learning. Additionally, according to Davenport, “Humans can typically create one or two good models a week; machine learning can create thousands of models a week.”
Along with Machine Learning today there is a lot of talk on Artificial Intelligence. Though the conversation around AI is mostly on the lines of the rise of the machines, AI algorithms use computational resources that use step by step reasoning much like the humans to make logical deductions and solve puzzles. Given that the use of data is increasing in our lives there has been a rapid expansion of the Internet of Things. Connected cars, smarter and intelligent devices, connected homes and appliances etc are just a few of the possibilities of the Internet of Things. As the data end points begin to grow, gaining insights into this with Machine Learning and Artificial Intelligence becomes more common as well as more important. So we witness AI permeating into almost everyday aspect of our daily lives…Siri, Google’s Voice Search, wearable devices, Netflix, IBM’s Watson, autonomous cars, fraud detection all are existing examples of Artificial Intelligence applications. While AI and Machine Learning are not interchangeable, they are effectively applied together in several areas such as Bioinformatics, Cheminformatics, Computational finance, Game playing, Speech and handwriting recognition to name a few. From where we are standing right now, Artificial Intelligence certainly seems like the final frontier for data. Until the next big thing in Big Data, that is.
The post Evolution from Data Warehousing to AI appeared first on Lemoxo.