There is no Data Warehouse in a Digital World
Over the past couple of years I’ve notice a consistent problem with data architectures:
People keep trying to build a better data warehouse
The thought process is pretty clear, and in one sense its hard to fault:
- Data used to mean Data Warehouses
- Now we have Data Lakes/Fabrics/Meshes
- Therefore we need to build a better data warehouse
This however misses the fundamental shift in what data means, and what Data Warehouses were about.
Data Warehouses were about the post transactional reporting of what had already happened, often primarily for financial or compliance purposes.
What was the number one BI tool in this data warehousing world? Excel. Where was most Excel used? In operations, folks engaging in data rustling, pulling together data for specific purposes, sometimes in local data stores, but often just straight into a chain of spreadsheets. Even at the strategic level we’d see spreadsheets dominate, often emailed around as one person built on top of another spreadsheet. Not that the data warehouse had no value, but it was just one cog, one of the most expensive, in the organizations data landscape.
This was also a world where data didn’t have value, people would say it would, but the lack of investment in data quality, the sheer size of the data latency, and the skewed IT budget between data and transactional systems really showed that the data warehouse was a clipped tail on the enterprise dog. In other words:
A data warehouse was the minimum acceptable product to achieve the basics of data consistency
I’m saying here “acceptable” rather than viable, as I’d argue it wasn’t an MVP because of just how much data work occurred well beyond the scope of an “Enterprise” data warehouse. The warehouse was what people could cope with and importantly what IT could actually implement. This is why we ended up with “Enterprise” Data Models, as its the easiest way to get to that minimum acceptable product. It is literally the least that we can get people to agree upon.
Data Warehouses were just that: Warehouses for data, places where we left data until somebody wanted it, and hoped we had in stock what they needed or they’d get it from a different supplier. They weren’t the front of house for data, they weren’t the equivalent of the retail stores or customer engagement, they were the repositories of what we hoped the business needed.
Today however that isn’t the challenge. Data actually has business value and data matters at operational speed. This is why Gartner have said to abandon the customer 360. Our world now is about data collaboration in data ecosystems, its about how data drives business outcomes.
The age old phrase of “build a better mouse trap and the world will beat a path to your door”, we aren’t trying to catch mice any more, data isn’t about post transactional reporting, its about data driving outcomes.
Data Warehouses are the Mini-Computers of the data age, they had their time, they had their place, but that time and place is over. We need to think differently, not replicate the mistakes of the past at scale.
