Parallel processing program switches to seq. processing after some time

Former Member — Wed, 26 Sep 2007 13:55:28 GMT

Hello,

I am facing a peculiar issue in one of my || processing program. The program, when executed with a data-set of 10,000 records takes 65 minutes to complete. One would expect it to take 650 minutes (or even lesser) to process a data-set of app. 100,000 records.

However, when I run the program for a file with app. 100,000 records the program runs OK initially (i.e; I could see multiple dialog processes getting invoked in SM50) but, after a while it starts running on ONLY ONE dialog process. I am not quite sure where, when and why this PARALLEL to SEQUENTIAL switch is happening. Due to this, the program drags on and on and on. I would highly appreciate your suggestions/tips to put this bug to sleep.

Here is a summary of the logic used...

w_group = 'BATCH_PARALLEL'.

w_task = w_task + 1.

CALL FUNCTION 'SPBT_INITIALIZE'

EXPORTING

group_name = w_group

IMPORTING

max_pbt_wps = w_pr_total "Total processes

free_pbt_wps = w_pr_avl "Avail processes

EXCEPTIONS

invalid_group_name = 1

internal_error = 2

pbt_env_already_initialized = 3

currently_no_resources_avail = 4

no_pbt_resources_found = 5

cant_init_different_pbt_groups = 6

OTHERS = 7.

IF sy-subrc <> 0.

Raise error mesage and quit

w_wait = c_x.

If everything went well, continue processing

ELSE.

CLEAR: w_wait.

The subroutine that receives results from the parallel FMs will reduce
this counter and set the flag W_WAIT once the value is equal to ZERO

w_count = LINES( data ).

Refresh the temporary table that will be populated for every partner

REFRESH: t_data.

LOOP AT data.

Keep appending data to the temporary table

APPEND data TO t_data.

AT END OF partner.

CLEAR: w_subrc.

CALL FUNCTION 'Z_PARALLEL_FUNCTION'

STARTING NEW TASK w_task

DESTINATION IN GROUP w_group

PERFORMING process_return ON END OF TASK

TABLES

data = t_data

EXCEPTIONS

communication_failure = 1 "Mandatory for || processing

system_failure = 2 "Mandatory for || processing

RESOURCE_FAILURE = 3 "Mandatory for || processing

OTHERS = 4.

w_subrc = sy-subrc.

Check if everything went well...

CLEAR: w_rfcdest.

CASE w_subrc.

WHEN 0.

This variable keeps track of the number of threads initiated. In case
all the processes are busy, we should compare this with the variable
w_recd (set later in the subroutine 'PROCESS_RETURN'), and wait till
w_sent >= w_recd.

w_sent = w_sent + 1.

Track all the tasks initiated.

CLEAR: wa_tasklist.

wa_tasklist-taskname = w_task.

APPEND wa_tasklist TO t_tasklist.

WHEN 1 OR 2.

Populate the error log table and continue to process the rest.

WHEN OTHERS.

There might be a lack of resources. Wait till some processes
are freed again. Populate the records back to the main table

CLEAR: wa_data.

LOOP AT t_data INTO wa_data.

APPEND wa_data TO data.

ENDLOOP.

WAIT UNTIL w_recd >= w_sent. "IS THIS THE CULPRIT?

ENDCASE.

Increment the task number

w_task = w_task + 1.

Refresh the temporary table

REFRESH t_data.

ENDAT.

ENDLOOP.

ENDIF.

Wait till all the records are returned.

WAIT UNTIL w_wait = c_x UP TO '120' SECONDS.

FORM process_return USING p_taskname. "#EC CALLED

REFRESH: t_data_tmp.

CLEAR : w_subrc.

Check the task for which this subroutine is processed!!!

CLEAR: wa_tasklist.

READ TABLE t_tasklist INTO wa_tasklist WITH KEY taskname = p_taskname.

If the task wasn't already processed...

IF sy-subrc eq 0.

Delete the task from the table T_TASKLIST

DELETE TABLE t_tasklist FROM wa_tasklist.

Receive the results back from the function module

RECEIVE RESULTS FROM FUNCTION 'Z_PARALLEL_FUNCTION'

TABLES

address_data = t_data_tmp

EXCEPTIONS

communication_failure = 1 "Mandatory for || processing

system_failure = 2 "Mandatory for || processing

RESOURCE_FAILURE = 3 "Mandatory for || processing

OTHERS = 4.

Store sy-subrc in a temporary variable.

w_subrc = sy-subrc.

Update the counter (Number of tasks/jobs/threads received)

w_recd = w_recd + 1.

Check the returned values

IF w_subrc EQ 0.

Do necessary processing!!!

ENDIF.

Subtract the number of records that were returned back from the
total number of records to be processed

w_count = w_count - LINES( t_data_tmp ).

If the counter is ZERO, set W_WAIT.

IF w_count = 0.

w_wait = c_x.

ENDIF.

ENDFORM. " process_return

Thanks,

Muthu

Re: Parallel processing program switches to seq. processing after some time

Former Member — Wed, 26 Sep 2007 14:02:07 GMT

I have not gone through your code, but this is what i think:

The number of dialog work processes available is limited. Once you reach the limit all the processing happens only in the last work process, as no new work processes can be created . (Not too sure about this).

topic Parallel processing program switches to seq. processing after some time in Application Development and Automation Discussions

Parallel processing program switches to seq. processing after some time

Re: Parallel processing program switches to seq. processing after some time