How to run parallel processing with "background" work process instead of "dialog"

[Former Member]

Hello, I need to process 200k records in a business process that we have developed.

When processing the data regularly, it takes a very long time so i've decided to implement parallel processing according to this tutorial:

https://help.sap.com/saphelp_nw70/helpdata/en/fa/096e92543b11d1898e0000e8322d00/content.htm

I've decided to divide the 200k records into 15 packets, and each packet is sent to be procecced using the "call function starting new task" command.

The problem is that the call to the function creates a dialog work process!

This means that after about 30-60 minutes in my system I receive a time out dump and the processing fails.

In other words, I'm extremely limited with the amounts of data that I want to process in each packet, thus leading to the following simple formula:

Large quantity of packets + Small amount of data to process in each packet = bad performance.

Why are the work process made of type dialog instead of type background, why did SAP choose to implement it this way?

Anyway, I've been thinking about the following alternative method:

1) Call the function module to open a job

2) Use the "SUBMIT PROGRAM VIA JOB" syntax which will run the program in background mode and assign it a background workprocess (no time-out limit)

3) Call function module to close the job

Will this method work for me?

Thanks!

[Former Member]

Hello, I need to process 200k records in a business process that we have developed.

When processing the data regularly, it takes a very long time so i've decided to implement parallel processing according to this tutorial:

https://help.sap.com/saphelp_nw70/helpdata/en/fa/096e92543b11d1898e0000e8322d00/content.htm

I've decided to divide the 200k records into 15 packets, and each packet is sent to be procecced using the "call function starting new task" command.

The problem is that the call to the function creates a dialog work process!

This means that after about 30-60 minutes in my system I receive a time out dump and the processing fails.

In other words, I'm extremely limited with the amounts of data that I want to process in each packet, thus leading to the following simple formula:

Large quantity of packets + Small amount of data to process in each packet = bad performance.

Why are the work process made of type dialog instead of type background, why did SAP choose to implement it this way?

Anyway, I've been thinking about the following alternative method:

1) Call the function module to open a job

2) Use the "SUBMIT PROGRAM VIA JOB" syntax which will run the program in background mode and assign it a background workprocess (no time-out limit)

3) Call function module to close the job

Will this method work for me?

Thanks!

[Former Member]

That method will work for you, but I think your conclusion that the parallel processing will lead to bad performance might not be correct. I have had some experience with data loads for which we used the parallel processing you refer to and the performance scaled nicely, i.e. there was little overhead from the parallel processing mechanics. Of course this will only work if you can separate your work in distinct parts that do not need to redo any work which I assume the 200K records are. Advantage of this approach is that you can control the amount of work processes used through some basis settings and tweak it to your hardware resources without having to change to the program.

[Former Member]

Hi,

Go through below blog.

Regards

Prithviraj

[Former Member]

This is exactly what I was looking for, thank you very much!

[Former Member]

Thank you for your answer but my approach is to send big chunks of data in each packet, so background parallel processing is more relevant here.

[Former Member]

Hi

Please mark the appropriate response and correct answer and set the thread to answered status.

Thanks.

Regards

Prithviraj