Data Contracts
Data Mesh Radio was probably the first entity to really identify data contracts as a crucial topic in late 2021 before interest around the topic exploded in mid 2022. Scott did a series of dives into data contracts in early 2022 but quite honestly, the topic is still not really well defined. This is one where there needs to be a complete overhaul on how we think about things which is why, even before getting into specifics, we have an episode with Zhamak from a Zhamak’s Corner recording about Postel’s Law and the TolerantReader pattern so we know where we need to head even though we are starting with overly rigid solutions, much like we did in the API revolution. Alas, we appear doomed to repeat the mistakes of the past 😅
Part of a longer series of episodes (ZC20-25) where we covered a lot about what we can learn from microservices and be smarter about in data – basically, as stated above, let’s not make the same mistakes 😎 you can listen to the episode and see the brief show notes here.
A Mesh Musings episode where I discuss the complicated aspects of data contracts as we’ve seen them evolve and where we need to go. Basically, we are nowhere near good yet!
To listen to the episode and/or see the full show notes, please go here.
Abe Gong and team were actually the first people to use the phrase “data contracts” and talk about it that I could find (I did some work in early 2022). Great Expectations, the open source projects Abe helped create (and then create a company around) is one of the best known tools in the space. The conversation is much more about needs than the tool but it should be very enlightening.
To listen to the episode and/or see the full show notes, please go here. For the episode transcript, please see here.
Very quickly after speaking with Olivier I started asking around and Abhi was one of the two people I found (Abe Gong took longer to chase down) that had a really developed framework for data contracts, why you need them, how to think about the different potential patterns to leverage them, etc. Really probably the most complete data contract vision to date and it was from December of 2021!
To listen to the episode and/or see the full show notes, please go here.
(Releasing July 10, 2023. Ping Scott for early access). Sofia did an amazing presentation in mid 2021 about testing in data mesh and how a robust testing framework can make it far easier to have a working data contract model/ecosystem. Unfortunately, lots of it is theoretical and we still need to develop the OSS and the practices to do it well. But it’s a fantastic primer on where we need to go!
To listen to the episode and/or see the full show notes, please go here. For the episode transcript, please see here.
This is the episode I discovered the topic of data contracts. I had heard vaguely of schema contracts but as soon as I heard what they were, this became an need to know topic! Olivier does a great job explaining data contracts and why they are so necessary!
To listen to the episode and/or see the full show notes, please go here.
Continuing on the theme of data contracts and testing, while this one isn’t specific to data contracts, it really helps in the same vein. In the life sciences space, especially the drug trial space, there are really exacting standards around data quality and this will help you to develop your standards.
To listen to the episode and/or see the full show notes, please go here.
A great conversation with one of the people leading the charge when it comes to data contracts, this was his first podcast and first bit of content other than an early blog post about what they were trying to achieve.
To listen to the episode and/or see the full show notes, please go here.
A Mesh Musings episode that is pretty relevant where I call on lines of business/domains to register their consumption patterns to prevent the contract being the only thing that matters – humans can prevent issues with a bit of focus and forethought!
To listen to the episode and/or see the full show notes, please go here.
For any interview or panel episode, there are show notes – that start to get quite extensive after episode 100 – as well as transcripts for any interview or panel episode past #46 and most past #32. You can see a list of all episodes and available transcripts here.
