Experience with the data vault

Experience with the data vault

The data vault is a data warehouse design structure which varies from traditional data warehouse architecture in several ways including the practice of separating the “thing” from how this thing changes over time. For example, there is something about a Customer that could never change over time, even if this is just its candidate keys. Everything else about a customer that can change exists in entities (called “satellites”) attached to this customer entity (called the “hub”). (More about the data vault at http://danlinstedt.com/.)

Has your organization considered moving from a more traditional data warehouse design, such as a relational hub-and-spoke or a dimensional bus architecture to a data vault? What was the thought process is considering the data vault? Did you make the move to the data vault? If yes, was it a good move? Why or why not?

1 Comment

  1. John Giles 5 years ago

    G’day Steve. Data Vault seems to be really gaining traction, and I would suggest this is a reflection of well-deserved recognition.

    You asked for feedback on Data Vault experiences from within “your organization”. As a freelance consultant, I am not in a position to talk about what’s happened in “my” organization, but am more than happy to share observations across a few enterprises.

    The first is a national regulation agency for medical practitioners (nurses, chemists, doctors etc.) in my home country. Anyone who has been involved in the merger of two companies will realise the challenges. This single organization was created from the merger of 83 separate organisations, each with multiple IT systems. To cut a long story short, a Data Vault was constructed with the goal of initially capturing today’s data going forward, but more importantly, facilitating the back-loading of decades of practitioner history from 83 disparate organizations, each with multiple IT systems. Due to funding issues, the back-loading is, I believe, still to occur, but they are delighted with the functionality already delivered.

    Story number two is from a water utility. Two products – deliver clean water, and take away less-than clean water. At face value, pretty simple stuff for a data warehouse. However, their goal was to deliver an “Operational Data Vault”, with near-real-time feeds from multiple operational systems to facilitate a 360-degree view of customers, assets, properties and more. A proof-of-concept was built in an “agile” manner. The “product owners” declared the project a success, but the wider roll-out of the Data Vault is yet to occur.

    Finally, just yesterday I chatted with a potential client. I can’t say much for business reasons, but sufficient to say this organization is a cooperative collection of hundreds of independent businesses that benefit from shared purchasing, shared branding, and the like. My guess is that without a Data Vault approach, the integration of data will be unnecessarily difficult.

    In summary, the good news is that I have a growing confidence in Data Vaults to handle pretty well anything that is thrown at their way, but for Data Vault projects to reach their full potential, you need more than technical enthusiasm and skill – you need business backing. But isn’t that true elsewhere?

    The not-so-good news? While the Data Vault architecture is mature, proven and robust, project time can be spent on evaluation and purchase (or construction) of the tools to get data into, and out of, a Data Vault. Thankfully there are a growing number of choices of quality tools, and their features are expanding to meet the market’s demands.

Leave a reply

Your email address will not be published. Required fields are marked *