The Data Warehouse is Dead, Long Live the Data Warehouse! – A Paradigm Shift in Modern Data Management

Author: Adrian Woolmore, practice lead data and analytics at Trustmarque

The once-reigning king, the traditional data warehouse, is witnessing an evolution and transformation in the face of unprecedented technological advancements. The traditional notion of the data warehouse is declared “dead,” but make no mistake, this death cry marks the inception of a more powerful, dynamic, and versatile data warehouse. They are far from dead.

At Trustmarque we support NHS organisations in building data warehouses in the Microsoft Azure Cloud using our Bedrock data warehouse. We are often asked whether a data warehouse is still needed in a modern data stack… my answer is always an emphatic “Yes!” While we often support many health organisations in making huge steps forward quickly using such tools as Microsoft Power BI – which grows more powerful by the month – the reality is that it is never a replacement for a well-designed and well-served data warehouse.

Implementing a well-curated, high-quality Azure data warehouse in the Cloud into a modern analytics function opens the door to a vast range of other possibilities which can support the organisation. It can evolve into game-changing insight which can support in delivering better care, reducing organisation risk, improving productivity and reducing costs.

Bedrock provides many benefits that might be expected such as automating the cleansing and curation of data, automating national and statutory reports, and freeing up analysts to do more. However, there are other Use Cases which are not so obvious and provide examples of how Bedrock demonstrates the versatility of a modern data warehouse in the Cloud.

1. Saving lives when the chips are down: When Cerner, a major hospital data system, went down across the UK last year, some of our Bedrock hospitals rapidly gained access to the full data set in our data warehouse solution using our Patient Viewer. This ability to provide rapid access at scale to the real-time data that had been collated allowed those hospitals to continue to provide services for those days to a far greater degree than others. This protected services at a time when waiting lists are long and patients need the care the most. Quick thinking, an infinitely scalable data warehouse in the cloud, and a 100% accurate dataset may have saved lives and reduced harm.
2. Releasing you from the shackles: Many organisations will be all too familiar with ‘locked-in’ syndrome. However, with all critical data from systems in the Bedrock data warehouse, curated and high quality, these shackles become more fragile. When the need to upgrade arrives – you have resilience when you come to change – you have high-quality data available for migration, and when it comes to analytics and AI, you have a clean source of history and depth.
3. Polishing the quality into AI: You can’t consider a modern data warehouse without mentioning AI. However, AI does not replace foundational data principles. Building AI and Machine Learning solutions on low-quality data will yield poor, or worse, inaccurate solutions. High-quality data, with excellent governance, lineage, and integration will yield better results. Data held in Bedrock can be accessed instantly with in theory infinite history, providing substantive data mining capability.
4. Tapping into data: Modern Cloud data solutions such as Datalakehouses provide unparalleled access to data. More, though, now, the ability to Federate data from and into Microsoft Azure and new Mesh solutions brings with them the ability to interrogate and use data without moving it from the governed safe stores it resides in. This is best done in a highly auditable, tightly controlled manner. In health, this also is the means for getting data out of the local systems that may, in the future, service data to national data aggregation and research platforms.
5. Building trust and confidence: It sounds a little far-fetched to suggest a data warehouse can build confidence. But numerous surveys are identifying that far from trust and literacy improving in data, it is in fact reducing. Perhaps that’s the sheer volume of new technology and data, perhaps a realisation that quality has not kept up with technology. It’s no surprise that well-curated, clean data from a single source of data drives trust. Surfacing this data through Bedrock and Azure, and providing consistency in how it is consumed can build this confidence. Nothing keeps that conformance in check like a dependable data warehouse.
6. Rationalising tech debt: As technology has changed rapidly, data has grown vastly, and data organisations have organically grown around these challenges; many of you will recognise the scenario of technical debt build-up: many databases, historical data retrieval systems littering the environments, uncontrolled sources. Absorbing this into an infinitely scalable data warehouse environment in the Cloud can dramatically reduce this technical littering, but far more, reduce the cost of the historical estate, and release this data through integration for wider, more in-depth analysis.

So far from data warehouses being sidelined, it’s now more important than ever that the data warehouse, and the quality on which it is built, forms a core part of your data and analytics strategy. The demise of the data warehouse may come, but it is more likely that it will become a central part of the future data strategy providing resilience, quality, and consistency in the ever-expanding data universe.