cancel
Showing results for 
Search instead for 
Did you mean: 

Extracting Huge volume of data from SAP BW4HANA to CSV via SAP Data Intelligence.

abhimanyu_sharma
Contributor
0 Kudos

Hi All,

we have a requirement to send data from ACDOCA DSO from BW4HANA to Azure Data lake via SAP Data intelligence in CSV files.

The initial volume is around 115 million.

We have an inbuilt operator in SAP DI: "Data Transfer" which can connect to BW DSO as ODP Source and in the target, we can mention CSV file format.

we have 3 options: Append, Create and Create based on Data packages. The append option will create one file and data will be appended in the same file.

Create and Create based on data packages will generate multiple files of 5-6 MB. Hence for such a big data volume, it will generate around 1000+data files which could be nightmare for Azure team to combine it in a single file.

Hence I am looking for a strategy how can we load such a big volume as initial load in CSV file via SAP DI w/o getting the job failed due to volume of data. And once it is done, delta will work based on subscription ID.



RolandKramer
Product and Topic Expert
Product and Topic Expert
0 Kudos

Hello abhimanyu24366

Why not using SAP IQ Database for that purpose, so that the sensitive Data is not leaving your premises?See the Blog - Q – the easy Installer for SAP IQBest Regards RolandPlease follow the Rules of Engagement

Accepted Solutions (0)

Answers (5)

Answers (5)

jimgiffin
Product and Topic Expert
Product and Topic Expert

The issue with a single file (using Join Batches on the Write File operator) is that there is no native append to blob storage. Therefore DI will write the first block to the file, but the second block of data requires DI to read the first block back into memory, combine it, then write it back. This is a non-issue for smaller datasets, but as the data gets larger, the entire block will cause an OOM.

abhimanyu_sharma
Contributor
0 Kudos

so should we use Create Option to generate the multiple files? In this option, DATA TRANSFER operator create multiple files of 3-4 MB..

abhimanyu_sharma
Contributor
0 Kudos

so should we go ahead with create option? Only issue in this we can see that it will generate huge number of smaller files of 2 MB size each.

S0004647611
Active Participant
0 Kudos

SNP (former DataVard) Glue [https://www.snpgroup.com/snp-glue] is a very good SAP centric (S4 and BW) DI tool for exporting to Azure, Snowflake etc.

0 Kudos

Hello Abhimanyu,

Could you suggest if any solution found for this issue?

Thanks,

Sirisha

former_member802271
Discoverer
0 Kudos

We have the same issue,. Anybody found a solution?

abhimanyu_sharma
Contributor
0 Kudos

Hi Roland,

Thanks for your inputs.But the strategy by the client is to use SAP DI for any data extraction out of SAP to data lake. Also , we are in the stage of the project where we can't install SAP IQ server.

Hence I am looking for a strategy how can we load such a big volume as initial load in CSV file via SAP DI w/o getting the job failed due to the volume of data. And once it is done, delta will work based on subscription ID. There is a standard operator : Data Transfer which can be connected to BW DSO and extract the data but would like to if the graph failed in between, is there a way we can get the previous delta instead of running the full load again for such huge volume ?

abhimanyu_sharma
Contributor
0 Kudos

hi Roland, do you have any suggestions on this?

RolandKramer
Product and Topic Expert
Product and Topic Expert
0 Kudos

Hello abhimanyu24366

using SAP IQ is less complex as the Data doesn't leave the Customers premises

the SAP IQ Database Installation is described here - Q – the easy Installer for SAP IQ

recently, I found a 3rd Party ELT tool, AercoSoft which obviously can deal with these kind of challenges

Best Regards Roland

abhimanyu_sharma
Contributor

so are we saying SAP DI is not good or best choice for these kind of ETL?