Create Data Products with SAP Datasphere Marketplace

[ViktorS]

Hello readers! We're excited to share this blog post with you today. It was a collaborative effort between my colleague @eduard_spiess and myself.

Data Mesh - a new way of thinking BI

In the ever-evolving world of data management and analysis, one term has emerged into the spotlight that is redefining our approach: Data Mesh. Proposed by Zhamak Dehghani, a Data Scientist at ThoughtWorks, this paradigm shift in data integration seeks to solve complexities that arise from processing an unprecedented volume of data.

So, what exactly is Data Mesh?

Data Mesh, in essence, is an architectural and organizational framework. It proposes decentralizing data ownership and administration to promote cross-functional use and autonomous governance. This concept transforms the former monolithic data platforms into a network of interconnected, yet independent data sources owned by individual teams.

The four principles of Data Mesh

Data Mesh consists of four principles that aim to revolutionize data management within organizations. These principles are:

1. Domain-driven data ownership
2. Data as a product
3. Self-serve infrastructure platform
4. Federated computational governance

By embracing these principles, organizations can foster a more collaborative and scalable data management approach, enabling domain-driven decision making and maximizing the value of their data assets.

The Need for Data Mesh

The idea of a data mesh is rooted in addressing the limitations of traditional data architectures in handling the enormous quantities and variety of data produced by modern digital businesses. Centralized data systems often struggle with scalability, agility, and the constraints of a single point of management, leading to costly and ineffective data operations.

But how can data mesh help us when facing these challenges?

In decentralized data mesh architecture, each team owns its data product, following the key principle of "domain-oriented decentralized data ownership". Thus, separate teams have the responsibility for data quality, governance, and operations. This ensures not only independent functioning but also encourages innovation within the teams.

Moreover, each of these data products is treated as an independent product, with its life cycle, from design to operations, which radically changes how data is handled, making it more manageable and efficient.

Benefits of Data Mesh

Adopting a data mesh architecture provides various advantages - agility in decision-making, improved scalability, and streamlined operations, to name a few. As data sources are treated as independent products, it promotes reliability, as the failure of one does not directly impact the others.

Data Mesh’s decentralization also encourages a culture of shared responsibility. Each team is accountable for its data product, enhancing the overall data quality and ownership.

Data Mesh with SAP Datasphere

With this paradigm shift, however, comes an equally challenging question—how can companies successfully adopt and effectively implement Data Mesh? Many are on the lookout for efficient tools that can assist in navigating this innovative path. Through this blog post, we aim to demonstrate how SAP Datasphere, particularly its marketplace, can be instrumental in guiding businesses towards becoming Data Mesh-equipped and ready.

So, what could a high-level data mesh architecture with SAP Datasphere look like?

1. Operating systems from which the data is collected.
2. Use of SAP Datasphere as a self-serve infrastructure platform for data storage, security, and data networking.
3. Creation of domain-specific spaces within Datasphere for each respective domain.
4. Generation and provision of domain specific data products via SAP Data Marketplace.
5. Implementation of federated governance to establish standards.

Although the focus of this blog post will be on creating data products with SAP Datasphere, we have decided to give a short “introduction” to data mesh to show the importance of data products for modern organizations. If you are interested in learning more about Data Mesh, you can find more information at the following link: https://datameshlearning.com/.

What opportunities does “Data as a Product” offer and how does Datasphere's Data Marketplace support companies in offering their data as a valuable product?

Use Case

A central idea of Data Mesh is to view data as a product and not as an asset. To facilitate the provision and exploration of data products, SAP Datasphere offers the Data Marketplace, with which data products can be searched for and explored or even published. This applies to both internal and external data products. Further details on the Data Marketplace can be found in the following blog post: The SAP Datasphere Data Marketplace - New Home. Ne... - SAP Community.

Publishing a Data Product on the internal Data Product

To successfully publish a data product on the internal data marketplace, certain prerequisites must be met.

Prerequisites:

1. Creation of a Data Provider Profile
   For more detailed instructions on how to complete the fields and explore the available options for drag and drop fields, please refer to the following link: Maintaining your Data Provider Profile | SAP Help Portal.
2. Creation of a Context
   We will not go into detail regarding the options of the context, however if more information is needed, please check the following link: Context Management App | SAP Help Portal.

Creation of Data Product:

After making sure that the necessary prerequisites are met, the data product can be created as follows:

1. Open the Data Sharing Cockpit
2. Choose "My Data Products"
3. Click on "Create Product"

1. Fill out the name, description, and if necessary, an image representing your data product
2. It is helpful to upload some sample data (Note: Pay attention to the correct format of the sample data, when uploading it.)
3. Fill out the product details such as data category, replication mode etc.
4. In the context, choose the previously created context

1. Next, determine the pricing model. For our example, we will select "One Time" as the pricing model. Additionally, in the description, we will specify that the license key can be requested from the data product owner.

1. If necessary, define the terms and conditions of the data product.
2. Additionally, you have the option to attach additional documents. These documents can provide further information about the data product, such as metadata, specific calculations, or any other information that might be relevant for consumers of the data product.
3. Finally, you will need to select the space and Datasphere artifact that includes the data. These choices are crucial for organizing and storing the data within the appropriate environment.

1. As soon as the data product is created it can be listed on the internal data marketplace.

1. Depending on the chosen delivery mode it might be necessary to create releases for your data product. In our case we used Live Access and, therefore, do not have to create a release.

Finally, the data product is available on the market. The general information can be viewed by everyone that has access to the internal marketplace. If the respective person is interested, access can be requested directly via the data marketplace.

Conclusion

In conclusion, Data Mesh is a groundbreaking approach to data management and analysis that tackles the challenges posed by the ever-increasing volume of data. By decentralizing data ownership and administration, organizations can promote cross-functional collaboration and autonomous governance. The four principles of Data Mesh provide a framework for more scalable and collaborative data management. Implementing Data Mesh architecture offers benefits such as agility, scalability, reliability, and improved data quality.

Companies can leverage SAP Datasphere, particularly its marketplace, to navigate and effectively adopt Data Mesh principles. By viewing data as a product and utilizing the Data Marketplace, organizations can facilitate the availability and exploration of data products. Through the creation of data provider profiles, contexts, and data products, businesses can publish their data on the internal marketplace and enable access for interested parties. Overall, Data Mesh and tools like SAP Datasphere offer a new way of thinking about business intelligence and data management, revolutionizing how organizations handle and leverage their data assets.