Data Ecosystem

Every enterprise maintains the data about sales, profit, and the cost incurred periodically. While these aspects fall into the transactional category, the other one is the analytical data used for strategic decisions. For example, in a retail business, data about the products sold and revenues from each of the branches across different locations in a country is transactional. Subjecting the transactional data for further analysis to identify a unit is profitable or not and deciding on the branches' continuity is Analytical data. The process of capturing, storing, and analyzing the data has been happening for a long time. 

At the beginning of the emergence of modern-day computers, data was getting stored in text files. As time passed, people realized that retrieving and analyzing the data from text files is challenging and resorted to spreadsheets. Thanks to Edgar F Codd's relational model during the 1970s, that revolutionized storage and processing. Several relational database management systems like MySQL, Oracle, Postgresql are still relevant in the current period.

The need for Analytical processing led to the emergence of Data Warehouse technologies as RDBMS systems tackle only transactional data. Many tools like Informatica, Abinitio, and in recent times, Amazon Redshift are aiding in many enterprises' analytical data processing. Meta Data Management tools like IBM InfoSphere appeared to manage the metadata of big organizations.

At some point in time, the amount of data generation exploded and added to the different varieties of data added to the complexity of storing and processing the data. Big data technologies emerged as a winner to tackle the complexities of large volumes of data in different varieties generated at a faster pace. We are in an era of Big Data that handles the data with distributed storage and distributed processing.

Though we are in a Big data era, the data ecosystems of any mid or large enterprises do not bank on only Big data technologies or any one of the technologies mentioned above. Modern-day data ecosystems of mid-sized or large-sized organizations would have a combination of components of RDBMS, Data warehouse, Meta Data Management tools, and Big Data technologies.

Wed Mar 10, 2021