Each week we find a new topic for our readers to learn about in our AI Education column.
Data is the digital stuff of business, and everybody needs a place to put their stuff.
George Carlin had a great routine, back in the bygone days of great standup comedy, about the way people accumulate, move and store stuff—from homes and attics and basements and storage units, to hotel rooms, luggage and wallets, Carlin, using more words than I care to spend here, pointed out that we, as humans, invest a lot of time, thought and other resources into carrying, moving, protecting and storing… stuff.
“You’re really spread out now, you’ve got stuff all over the WORLD! You’ve got stuff at home, stuff in storage, stuff in Honolulu, stuff in Maui, stuff in your pockets…supply lines are getting longer and harder to maintain…” – George Carlin.
Carlin’s thoughts on stuff, and our opening statement, bring us to the topic of this week’s AI Education, a data warehouse, one of the several places and storage arrangements businesses can use to store their digital stuff—data. In computer science, data is just information contained in strings of code. This information sometimes has physical-world representation, it could stand for account balances or financial transactions, for example, or it could refer to purely digital objects that are being manipulated by or moved between computers.
What Is a Data Warehouse?
Before we jump into a formal definition of data warehouse, we’d like to recall an AI Education column from earlier this year about data lakes, another option for data storage, organization and management. A data lake is a centralized data storage system designed to contain data in any format and in any state or structure. What’s important about a data lake is that the data is there—often, some infrastructure around a data lake makes the information searchable and findable—but the storage is kind of the point.
A data warehouse, on the other hand, is also a centralized data storage system, but one intended to take information from disparate sources and organize it into a consistent system where it can be searched and analyzed with greater ease. Data warehouses enable institutions to organize their data in various ways, rather than just leaving it there in a lake. Software within and working along-side the data warehouse sanitizes and transforms the data into one format. A data warehouse enables stable, long-term storage and use of processed data.
In a data warehouse, information flows from sources—usually other databases used in an institution’s operations—into an integration mechanism which processes, cleanses and formats the data. Data architecture moves the processed information from the integration mechanism into storage, where it can be accessed by users.
How Did We Get to Data Warehouses?
The technological egg came before this digital chicken. Data warehouses were born out of necessity. In the 1980s, but even more so in the 1990s and 2000s, we—meaning people in general—started collecting heaps of data from cars, business networks, colleges, sports teams, personal computers, devices, the internet, social media, you know, what have you. It went everywhere, but mostly, into data lakes, if organizations decided to store that data over the long term at all.
Then, in the 1990s and 2000s, we realized that all that data might actually be useful to us. Not only could we use it to power deeper and more useful applications of our technology (eventually leading us to today’s AI and machine learning tools), but we could also potentially use to make better decisions ourselves. However, we also discovered that it was difficult to use large amounts of information in unstructured, disorganized form—in fact, the more information we collected, and the more data sources we used, the harder it became to actually use that data and the more we realized that we needed to process and format the information we were collecting.
Why Data Warehouses Are Important
Apart from its function as a searchable data storage mechanism, similar to a data lake, a data warehouse is useful because it can integrate data from multiple upstream applications into a single place, state and format, for use by other applications downstream or by any entity that might need to view or monitor a holistic view of operations. A data warehouse design facilitates the use of data analytics.
Data within a warehouse is much easier to query than data within a data lake by virtue of its organization, and it can be queried as it arrives in the data warehouse in real time—so any applications of AI that need to use data in real time, like autonomous vehicles and smart factories, are probably also making use of a data warehouse or a related form of data management and infrastructure.
A data warehouse, then, is in many ways a leap forward in utility from a data lake. Furthermore, data warehouses enable organizations to store consistent, high-quality data, vastly simplifying any future forays into machine learning or artificial intelligence.
Businesses can use data warehouses to keep historical records of their transactions, operations and processes when data sources do not. As a theoretical example, if a store’s connected checkout register is out of tape, a data warehouse can still record and report any transactions being taken place without the need to access the register itself. Here’s where that holistic view of operations comes into play—leaders of businesses and institutions, and the software they rely on, can use the information collected and organized within data warehouses to make better-informed decisions.
What Is a Data Lakehouse?
There is a problem with data warehouses, a serious drawback: because data warehouses are designed so that information can be used in real time, as it is being taken in, they can’t handle extremely large amounts of unstructured data. The time it takes to move, process and structure very large amounts information can confound the real-time demands placed on a data warehouse.
One strategy has been to build both a data lake and a data warehouse—a data lake to take in information first, and then a data warehouse to process and store information that is going to be needed for downstream applications and other purposes. This, of course, requires the development of two different sets of data infrastructure and does nothing to solve issues for applications that need to use data in real time.
So technologists decided to combine elements of a data lake with a data warehouse, creating a new form of storage, organization and management: The data lakehouse. A data lakehouse takes the lower cost, higher volume capacity and flexibility of a data lake and combines it with some of the data structuring, governance and business intelligence capabilities of a data warehouse.
