I have been part of the journey of Business Intelligence and data warehousing from the classic EDW times, to Big data and Hadoop, hyperscalers, self-service and now the Data Mesh idea. In some sense all these phases are still valid and alive depending on the scenario and which company you look at.
In this blog post you will learn the high level concept of Data mesh and why SAP Data Warehouse Cloud is well suited for implementing a Data mesh architecture and how companies can benefit from it.
Introduction
Organizations today are quite diverse when it comes to their maturity in data and analytics. While some are still struggling to get a data warehouse in place and are still very much using Excel extract from disconnected sources, some have already adopted self-service and have data democratization top of mind.
In the light of self-service, the next step of the journey is to take it further, not only doing self-service reporting and analysis. Now also self-service data modeling and data pipelining is a hot topic.
Many companies have already adopted the Data as a service concept, where an IT organizations primary goal is to deliver prepared, validated and controlled data sets for self-service reporting and analysis from the enterprise data warehouse. This is all good, but it creates some challenges. The data pipelining and development teams becomes bottlenecks in the process of producing good quality and actionable data and these teams sometimes lack business understanding of the data.
What is Data Mesh?
Data mesh is addressing these challenges where a Data mesh architecture is considering a domain view, giving the domains the freedom to take ownership of their own data. A business domain knows their data best. Hence it is logical in a data democratization organization to let them be accountable and responsible for it. The product a domain “sells” or produces is data, which is why “data as a product” is a key concept.
![](/legacyfs/online/storage/blog_attachments/2020/10/DM_architecure.jpg)
Data mesh architecture
Data mesh conceptual architecture. Source:
https://towardsdatascience.com/what-is-a-data-mesh-and-how-not-to-mesh-it-up-210710bb41e0
The data infrastructure supports the domains by providing solutions for processing of data. The domains are responsible for managing data pipelines, data ingestion, cleansing and aggregations, creating a data asset that can be used by BI tools or business applications.
The data that a domain produces are often needed and valuable for other domains, which is why sharing of data assets and collaboration is crucial. In additional, standard formats, business glossary, meta data catalogs and similar are important capabilities to make this work.
Interoperability between domains also requires well defined SLA´s and data quality measures that can be monitored.
Why customers should consider Data mesh architecture?
Companies who wants to go for data democratization and see their data as true revenue generating assets should consider data mesh or at least see beyond Self-service reporting and look at self-service data pipelining and self-service data modeling. Those organizations should consider involving business in their data creation process and acknowledge that domains have unique knowledge and understanding their own data and can be accountable for the data product.
This will also allow companies to minimize shadow IT and Excel extracts for producing local data assets and self-service modeling.
If the company is at this maturity level of their data democratization journey, then data mesh is the next step.
What is SAP Data warehouse cloud?
SAP Data Warehouse Cloud is SAP´s cloud Data warehouse offering. It is built on SAP HANA cloud and offers new features every quarter.
The product is built for the cloud and is not an SAP BW/4HANA in the cloud. The main theme of the product is Self-service Data modeling & analytics, Enabling data democratization and End-to-End data warehousing capabilities.
SAP Data Warehouse Cloud is shipped with SAP Analytics Cloud integrated into the product for creating visualizations and stories.
![](/legacyfs/online/storage/blog_attachments/2020/10/DWC_use_case.jpg)
SAP Data Warehouse Cloud use cases
SAP Data Warehouse Cloud Architecture
![](/legacyfs/online/storage/blog_attachments/2020/10/dwc_architecture.jpg)
Key capabilities of SAP Data Warehouse Cloud
- Spaces – Metadata isolation for tables, views, connections, Data flows, users, cost control, access control and more
- Self-service Data modeling and self-service data flow capabilities and user-friendly UI
- In-memory performance using the power of SAP HANA cloud
- Data ingestion methods – You can integrate to any source and use any ingestion method; batch, real-time replication, streaming and virtual access (federation scenarios)
- Openness – Any frontend tool can access SAP data warehouse cloud
- Standard integration and pre-built content for SAP sources
Why is SAP Data warehouse cloud and Data mesh a perfect match?
SAP Data Warehouse Cloud support all key characteristics of a Data mesh architecture. A key component is the SPACE concept which enables metadata isolation, self-service data modeling and self-service data pipelines. In short, a SPACE can represent a domain in the Data Mesh framework.
![](/legacyfs/online/storage/blog_attachments/2020/10/spacesJPG.jpg)
SAP Data Warehouse Cloud - Spaces
A central space and cross space sharing capabilities facilitate common data structures and central common data to be shared and used in the domain specific models. This is well suited for common master data such as profit center, organizational units, customers, product and similar.
SAP Data Warehouse Cloud is positioned as a self-service data modeling tool which enables business to create tables, views, data flows with an intuitive user interface, having graphical modeling and scripting. If you have your favorite SQL tool that is also fine.
The platform offers in-built collaboration features and an easy to use business catalog to support metadata tagging and sharing.
A Data mesh architecture using SAP Data Warehouse Cloud artifacts could look like this;
![](/legacyfs/online/storage/blog_attachments/2020/10/dwc_architecturefinal.jpg)
Applied Data mesh architecture with SAP Data Warehouse Cloud
SAP Data Warehouse Cloud allows for a variety of data ingestion methods and excellent support for federations scenarios of in practice any source. The Smart Data Integration framework (SDI) gives you batch, real-time replication, streaming and virtual access for federation scenarios. This allows entire data flows to be fully virtual and data stays at source. SAP Data warehouse cloud would in those cases only act as modeling layer for your data product. More often solutions are a hybrid of persisted and virtual solutions.
SAP Sources and pre-built content
Standard integration to SAP sources such as SAP BW/4HANA, SAP S/4HANA and other SAP sources ensure high data quality and low total cost of ownership.
The pre-built content will further speed up the creation of objects and allowing SAP and the global ECO system co-develop artifacts.
Summary and conclusion
In short, Data Mesh Architecture and framework is about empowering and enabling business domains to be involve, responsible and accountable in the data creation process. A business domain is responsible for the definition of data, the quality of data and the creation of data. The product a business domains produces and sells is actionable data. In order to achieve the goal of Data Mesh and get the outcome from it, the IT platforms are enablers and needs to have certain capabilities and characteristics.
Data Mesh is a very interesting approach to modern and digital organizations. It requires a certain degree of maturity in the organization and skills of the employees involved in the creation of data. I see Data Mesh as a natural next step after Self-service reporting and self-service data modeling.
It will be interesting to follow Data Mesh and see how the adoption among companies will be.
Conclusion of why SAP Data warehouse cloud and data mesh is a good fit
- Spaces – Allowing metadata isolation where one domain is one space and can be fully accountable for its data product and allowing cross-space sharing of metadata objects enabling collaboration between domains
- Self-service Data modeling and self-service data flow capabilities and user-friendly user interface
- In-memory performance of domain Data marts
- Data ingestion methods – You can integrate to any source and use any ingestion method; batch, real-time replication, streaming and virtual access (federation scenarios)
- Openness – Any frontend tool can access SAP Data Warehouse Cloud
- Standard integration to SAP sources- Standard integration to SAP sources such as SAP BW/4HANA, SAP S/4HANA and other SAP source ensures high data quality and low TCO.