The Complexity of Modern Data Environments


Most enterprises today lock away data behind multiple silos. When most people think of these silos, data marts and other old school data architecture approaches usually come to mind. But the modern cloud environment has made things much more complex. 

Fractured, siloed data environments are not beneficial to any business looking to actually drive value from their data and use it to improve decision-making across the board. In order to empower employees, data must be clean, updated and accessible at all times. For some organizations – especially those with a history of data being locked away by specific departments – getting data to a useful state can be a monumental task. 

While there are two common approaches to overcoming these data silos – data lakehouses and data warehouses – there has long been a debate about which is better (and why). 

To investigate further, we need to start by looking at the traditional definition of each.


Data Lakehouse

According to industry publication TechTarget, a data lakehouse is a data management architecture that combines the benefits of a traditional data warehouse and a data lake. It seeks to merge the ease of access and support for enterprise analytics capabilities found in data warehouses with the flexibility and relatively low cost of the data lake. 

The major attribute of a data lakehouse is that it’s usually made up of unstructured data, stored in its native format, without there being a specific purpose in mind when it was stored.


Data Warehouse

On the other hand, a data warehouse is a database which is optimized for analytics, scale and ease of use. Data warehouses often contain a large amount of historical data, intended for queries and analysis.

The major difference between a data warehouse and a data lakehouse is that the data warehouse is made up of structured data; i.e., data that has already undergone a transformation process to get where it is today.


Complementary Technologies

This leads us to the question of which is better to power your organization’s decision-making, but a better question is: are there certain situations where one should be used instead of the other? And how can these approaches help solve the problem of siloed data within my organization?

When it comes right down to it, data lakehouses and data warehouses actually complement each other. Data lakehouses are great for working with data stored in the flat architecture of a data lake, where data is left in its native format. Data warehouses, on the other hand, are great for large analysis workloads, due to the data being structured and ready to be worked with. Very few organizations will be able to claim their data is all optimized in a single format, with no additional work needed for employees to utilize it for decision making. 

For this reason, we often see organizations deciding that the only real answer to the “which is better” question is “both.” A company’s finance team typically will want their data to be structured, clean data from a warehouse, while teams such as those in marketing would be more than happy to review unstructured, immediate data as it is added to their data lake.

Having both types in play within their organizations enables those looking to work with data to be able to simply use the best tool for the job.


Solving the Complexity Issue

Now that we understand the answer to be “both,” what remains is our data complexity problem, where there is siloed data in a fractured environment that employees are looking to use. Putting a company’s data in the cloud is often seen as the answer here – but the internet is littered with stories of organizations attempting a migration from data lakehouses and/or data warehouses to the cloud and only finding failure. 

For many, their data migrations grind to a halt because success depends on pushing users such as business analysts and data scientists to change their habits around how they pull, access and utilize data. No small task indeed. 

The amount of data an organization captures and looks to make use of will only continue to grow. There will also be an increasing amount of potential uses for that data. New business models, new insights, new ways to improve operations or reach customers – and all reliant on a reliable, real-time analysis of data. Complexity will increase as time goes on – that’s a fact.

What organizations need to solve the complexity problem – and set themselves up for future data use (and success) is one interface to data that all consumers can access. This is where the idea of a universal semantic layer – a representation of data that helps users access and consume it using common business terms – makes sense. By creating a central, consolidated location for all your company’s data, end-users – be they business users or data analysts – have access to the same source, and can choose the tools they want to use with said data.

With a universal semantic layer, organizations can provide access to both the warehouse and the data lake, and not care about the data’s location or level of complexity. Providing access to both the raw and prepared data means both approaches are supported, giving different business functions the ability to use the tools they feel are best suited to them – and no one has to worry about the complexity or accuracy of the data being used.

The post The Complexity of Modern Data Environments appeared first on Datafloq.