As we released meshFaro (Data Mesh Lighthouse) I will be promoting it at lot in the future 🥳 But I will also share some experience with the technologies we are working with and what we have learned to help others. Some of the technologies we use are Google DATAPLEX and the Snowflake Internal Marketplace.
Today I can bring some info on Google DATAPLEX.
Google is deprecating their Data Catalog (link in comments) and replacing it with DATAPLEX to consolidate their offering. The current solution is a (temporary) mess and for the next 10 months you will experience a UI and data that is confusing. You will be switching between the old Data Catalog and the new DATAPLEX Catalog non-stop. There will be different UI options and menus items only working with the Data Catalog and some working with the DATAPLEX Catalog.
DATAPLEX Catalog is working with 'Aspect Types' replacing the traditional Data Catalog TAGs and is a better and more structured way of working with meta data. You can define regional or global Aspect Types depending on where the Aspect Types should be available and you can ensure mandatory data is added as well. You also have different data types you can add from Boolean, Text, Date and many more options.
We have been using DATAPLEX and Aspect Types for a period of time and it works great, available via API and it's fast and stable.
If you are just starting with GCP start right away with DATAPLEX Data Catalog. If your solution is currently based on Data Catalog and TAGs you should start a migration project (I know! YAMP? Yet Another Migration Project 🤓 )
If you are migrating? Maybe it's the time to look into your future Meta Data strategy and not just doing a 1:1 mapping of Tags to Aspect Types?
Google is cleaning up! Consolidating their solution is great, but for a limited period of time we will have to endure the confusion 😵💫
Book a demo to learn more!