Technology Blog Posts by SAP
cancel
Showing results for 
Search instead for 
Did you mean: 
CameronK
Product and Topic Expert
Product and Topic Expert
33,101

Introduction 

SAP Datasphere has introduced a new feature, 'Replication Flows.' This new capability (now available with Azure Data Lake) allows for the process of copying multiple tables from one source to another, offering a fast and seamless experience in data management. For detailed insights into replication flows and their functionalities, please refer to our comprehensive guide. 

In this blog, we’ll provide a step-by-step tutorial on replicating data from SAP S/4HANA to Azure Data Lake, showcasing the practical application and efficiency of this new feature in real-world scenarios. 

Now, let's dive in. We'll walk you through each step necessary to effectively utilize 'Replication Flows' for transferring data from SAP S/4HANA to Azure Data Lake. 

Steps :

1. To start, you will need to create a connection in your SAP Datasphere instance to Azure Data Lake. 

CameronK_0-1706549376274.png

2. Please ensure you have a Dataset in your Azure Data Lake that you would like to replicate the tables into.

3. Make sure you have a source connection. In this case, we will be using a S4/HANA On-Premise connection. You will need to create this connection in the ‘Connections’ tab in SAP Datasphere. 

CameronK_1-1706549376279.png

4. Navigate to SAP Datasphere and click on ‘Data Builder’ option on the left panel. Find and click the ‘New Replication Flow’ option. 

CameronK_2-1706549376283.png

5. Click on ‘Select Source Connection’. 

CameronK_3-1706549376286.png

6. Choose the source connection you want. We will be choosing SAP S/4 HANA On-Premise. 

7. Click select Source Container. 

CameronK_4-1706549376290.png

8. Choose ‘CDS Views’ then click Select. 

CameronK_5-1706549376292.png

9. Click “Add Source Objects” and choose the views you want to replicate. You can choose multiple if needed. Once you finalize the objects, click add selection. 

CameronK_6-1706549376295.png

CameronK_7-1706549376299.png

CameronK_8-1706549376301.png

10. Now, we select our target connection. We will be choosing Azure Data Lake as our target. If you experience any errors during this step, please refer to the note at the end of this blog. 

CameronK_9-1706549376304.png

CameronK_10-1706549376306.png

 11. Next, we choose the target container. Recall the dataset you created in Azure Data Lake earlier in step 2. This is the container you will choose here. 

CameronK_11-1706549376309.png

CameronK_12-1706549376312.png

12. In the middle selector, click ‘Settings’ and set your load type. ‘Initial Only’ means to load all selected data once. ‘Initial and Delta’ means that after the initial load, you want the system to check every 60 minutes for any changes (delta), and copy the changes to the target. 

CameronK_13-1706549376315.png

13. Once done, click on the ‘Edit’ projections icon on the top toolbar to set any filters and mapping. For more information on filters and mapping, please refer here, and here. 

CameronK_14-1706549376317.png

14. You also can change the write settings to your target through the settings icon next to the target connection name and container. 

15. Finally, rename the replication flow to the name of your choosing in the right details panel. Then, click on ‘Save’, ‘Deploy’ and ‘Run’ the replication flow through the top toolbar icons. You can monitor the run in the ‘Data integration monitor’ tab on the left panel in SAP Datasphere. 

CameronK_15-1706549376320.png

CameronK_16-1706549376323.png

CameronK_17-1706549376327.png

CameronK_18-1706549376331.png

16. When the replication flow is done, you should see the target tables in Azure Data Lake as such. It should be noted that every table will have 3 columns added from the replication flow to allow for delta capturing. These columns are ‘operation_flag’, ‘recordstamp’, and ‘is_deleted’. 

CameronK_19-1706549376334.png

CameronK_20-1706549376335.png

Note: You may have to include Premium Outbound Integration block in your tenant to deploy the replication flow. 

CameronK_21-1706549376337.jpeg

Conclusion 

Congratulations on successfully setting up a replication flow from SAP S/4HANA to Azure Data Lake! 
This integration exemplifies the power and efficiency of using SAP Datasphere's ‘Replication Flow’ feature for streamlined data management. Should you have any inquiries or need further assistance, feel free to leave a comment below. Your feedback and questions are invaluable to us.  
Thank you for following this guide and stay tuned for more insights on leveraging SAP Datasphere for your data integration needs! 

 

21 Comments
kpsauer
Product and Topic Expert
Product and Topic Expert

Great blog. Thanks for sharing.

VolhaS
Explorer
0 Kudos

Very useful blog. Thank you.

krishanusaha
Discoverer
0 Kudos

This is great, very helpful but What is the advantage with Datasphere? If I am not using Datasphere otherwise in my landscape. 

The same replication can be done by Aecorsoft data integrator, which is SAP certified.

TuncayKaraca
Active Contributor
0 Kudos

Hello @CameronK

Thank you for sharing a good example of using Microsoft Azure Data Lake Gen2 as target. 

Regards,
Tuncay

TuncayKaraca
Active Contributor
0 Kudos

Hello @krishanusaha,

It's a good question.  Once it comes with ETL / Data Integration tools, the options are many. 🙂

One advantage would be with SAP Datasphere you would use/replicate not data directly from source --just @CameronK's example of using S/4HANA CDS View -- you would also use/replicate your modeled data from Datasphere to Microsoft Azure Data Lake. 

The moving data here and there is just technical piece of the work but designing your flows & modeling your data is the core of the work.

BTW I didn't about  Aecorsoft Data Integrator, thanks for the reference.

Regards,
Tuncay

ganadha203
Explorer
0 Kudos

@TuncayKaraca@CameronK 

Can you please share me the some lights extracting data from SAP datasphere models data into Azure Data lake?  I tried creating replication flow and selected sap datasphere as source but I am unable to select the datasphere models or space local tables.

Thanks

Gana

__PRESENT

TuncayKaraca
Active Contributor

@ganadha203 ,

That's how I see it below. @CameronK  please feel free to chime in.

If you select SAP Datasphere as source in the replication flow make sure that you have local tables in the space. You can use local tables with only primary keys, it seems to be that way, from Datasphere. Therefore remote tables, views can be used as a source. 

After selecting SAP Datasphere as Source Connection.

TuncayKaraca_1-1709824814107.png

TuncayKaraca_0-1709824756857.png

Reference Creating a Replication Flow :

TuncayKaraca_2-1709824932528.png

Regards,
Tuncay

 

ganadha203
Explorer

Thanks @TuncayKaraca

for quick response actually I was looking that if we can use datasphere models as source like analytical model, SQL view, Graphical view etc.. I think this is not possible with replication flow.

Thanks

Gana

TuncayKaraca
Active Contributor

@ganadha203 You are welcome.

Yes, it seems we cannot use the views, remote tables, etc. only local tables.  But of course, just thinking aloud,  you could write results of views into a local table with a data flow, then you can use the local table for the replication flow.  It's matter of designing your views, models, data integrations. 

Regards,
Tuncay

ganadha203
Explorer

@TuncayKaraca  yes thats possible integration option.. 

__PRESENT

Srikanth
Product and Topic Expert
Product and Topic Expert
0 Kudos

@CameronK @TuncayKaraca , Thanks for sharing the knowledge. 

Does the same process works for S/4HANA Cloud, Public Edition and Azure Data Lake? Thanks for clarifying. 

Best Regards,
Sri 

TuncayKaraca
Active Contributor
0 Kudos

@Srikanth Yes, it's the same process works for SAP S/4HANA Cloud Public Edition and Azure Data Lake as a target. You can create a replication flow and use as a source either CDS Views or local tables with primary keys from SAP Datasphere . Of course you need to load data from CDS Views from S/4HANA Cloud, Public Edition to the local tables in case if you'd like to use the local tables.

Regards,
Tuncay

Srikanth
Product and Topic Expert
Product and Topic Expert

@TuncayKaraca , Thanks for clarifying. 

atharvapangerkar
Explorer
0 Kudos

@TuncayKaraca@CameronK 

I need some help . While selecting the datasphere as source we can not select load type as initial and delta. Is there any workaround for this?

Thanks,

Atharva

RuslanX
Associate
Associate
0 Kudos

Hello! 

Is there a way to get data from Azure storage to Datasphere? Is it possible to integrate Azure file storage with DS? 

Thanks for any kind of information!

Thanks,

Ruslan

 

 

Shailender
Newcomer
0 Kudos

Dear @CameronK ,

Do we have any way to integrate Data from S4HANA --> Azure Data Lake directly without going to Datasphere Route??

We have similar Need to get all data for Sales and Inventory process from S4 to Azure Datalake.

Kindly Advise if any direct way or Std. Connector we have to get this served.

Hope for positive response!

Regards,

SD 

0 Kudos

Dear @CameronK ,

I'm currently using replication flows to do the same and it's working quite good.

The only concern I have is about the connection parameters between SAP DataSphere and the Azure Datalake. We would like to avoid using Shared Key or Shared Access Signature to connect to the Azure DataLake.
The other solution I can see is using OAuth 2.0, but this means, if we have a password reset policy we would have to re-enter the password every time it's changing if I'm not mistaken.
Is there another possibility to connect from DataSphere to the Azure DataLake? maybe using a VNet Peering?

Regards,

Ben

S0012570412
Discoverer
0 Kudos

Great learning. I have a question. Is it possible to build  ECC6 replication flow with source as ODP enabled extractor for say 2LIS_02_VCITM?

My requirement is to load the data from ECC to Azure datalake using replication flow. 

 

Kindly advise

Uday

putzsch
Explorer
0 Kudos

Hey community,

I'd read several times this great blog and do not understand the following outcome.

1) Just following this case trying to replicate one single cds view (in fact a CDS view on table QMEL)

2) After few minutes, I got the following error message: 

Transferring initial data failed due to the following error:
data could not be converted (keys: values for key fields, columns: names of failing columns):
keys=[000090001141]
columns=[/sapsmoss/erdat /sapsmoss/erzeit /isdfps/mhioAddate /isdfps/mhioAdtime]
reason=[/sapsmoss/erdat:unexpected value: string(8) has invalid length: 10]
[/sapsmoss/erzeit:unexpected value: string(6) has invalid length: 12]
[/isdfps/mhioAddate:unexpected value: string(8) has invalid length: 10]
[/isdfps/mhioAdtime:unexpected value: string(6) has invalid length: 12].

3) I'd explicitly also ask to copy columns from source object. And I played with different settings, also creating a projection and forcing the fields length at the mentioned target, No chances 😞 always the same errors appears. 

Now, I'd tried to first copy the cds view to a local table into the DSP. Then replicate thise local table to Azure and it works perfectly !? 

What could be the reason of the initial error?
How can I fix it?

Thank you folks for any advice !

 

 

sapdeveloper01
Newcomer
0 Kudos

Dear @putzsch ,

We had a similar problem, this solved it: https://me.sap.com/notes/3254953/E

Hope it helps

flischewski
Newcomer
0 Kudos

@sapdeveloper01 In the sap notes link that you send, I can't find the solution. What is a way to solve the DATE to String issue when replicating to Azure Data Lake Gen2 ?