topic Re: problem with work processes in parallel processing in Application Development and Automation Discussions

problem with work processes in parallel processing

Former Member — Wed, 06 Oct 2010 05:52:03 GMT

Hi,

We are processing large amount of data around 25K records(fetching some 10 infotypes data for each employee) so I am implementing parallel processing to avoid TIME_OUT issue and here is the code


call function 'SPBT_INITIALIZE'
    importing
      free_pbt_wps                   = lv_free_threads
    exceptions
      invalid_group_name             = 1
      internal_error                 = 2
      pbt_env_already_initialized    = 3
      currently_no_resources_avail   = 4
      no_pbt_resources_found         = 5
      cant_init_different_pbt_groups = 6
      others                         = 7.

  if sy-subrc <> 0.                                         "#EC NEEDED
*** Issue error message
*    LEAVE LIST-PROCESSING.
  endif.
* Calculate how many sessions we'd like to run in parallel
*** Determine no of Free threads to be used. Based on % of available
  lv_thread_count =  lv_free_threads * lc_per / 100 .
  lv_thread         = 0.
  gv_active_threads = 0.

loop at pernr_tab into ls_pernr_tab .
    loop_index = sy-tabix.
    move ls_pernr_tab to ls_pernr_data.
    append ls_pernr_data to lt_pernr_data.
    clear ls_pernr_data.
    modulo = loop_index mod 4000 .
    delta_change_count = total_count - loop_index.
    if modulo eq 0 or delta_change_count eq 0.
      add 1 to gv_active_threads.
      if gv_active_threads <= lv_thread_count.
        add 1 to lv_thread.
        call function 'ZDATA_PI_MDM' starting new task lv_thread
          destination in group default
          performing set_session_done on end of task
          exporting
            last_run_date = last_run_date
          tables
            lt_pernr_data = lt_pernr_data.
*        wait until session = 'X'.
        if sy-subrc <> 0.                                   "#EC *
          subtract 1 from gv_active_threads.
        else.
          wait until session = 'X'.
          append lines of emp_data to lt_emp_data.
          total_deleted = total_deleted + deleted_records.
****** collect the data
*          FREE MEMORY ID 'HRDATA'.
          clear: lt_pernr_data[],emp_data[],session,deleted_records.
        endif.
      endif.
    endif.
    clear ls_pernr_tab.
  endloop.

I am calculating the number of free threads and within IF condition if gv_active_threads <= lv_thread_count. I am splitting the records into 4K each and calling the FM but the problem is if number of free threads are let's say 6 and if have 25k records only the first 24k records are processed and skipping the last 1k records(6*4k).

So how do I handle this situation.If some records are skipped how do I make sure I wait for some work process to become free and process the remaining data

Thanks

Bala Duvvuri

Re: problem with work processes in parallel processing

Former Member — Wed, 06 Oct 2010 19:02:19 GMT

Hi,

You should use a dynamic calculation to know how many threads your process each time.

Lets see 2 examples.

FIRST:

25000 threads / 4000 = 6,25. You look if the division has a "," "." in the result if yes then:

Then 0,25 * 4000 = 1000. So you will split the records 6 times each one of 4K and 1 time of 1k.

SECOND:

11233 threads / 4000 = 2,80825

0,80825 * 4000 = 3233.

So you will split it in 3 times 2 of each of 4k and 1 of the rest.

My advice is to calculate everything before start the loop and keep all the records in a table, so every loop you know how many threads you have to execute because will be on the table.

Regards, Hope it helps

Re: problem with work processes in parallel processing

Former Member — Thu, 07 Oct 2010 04:06:57 GMT

Jorge,

My problem is initially i am getting number of free workprocesses and based on the count,for each work process i am processing 4k but by the time I execute some 16k records number of free work processes are becoming 0 but still I need to process some more records,how do i procees the remaining records.

Once number of free work processes becomes zero after executing some records,can I again get the number of free work processes at that time and process the remaning records like as mentioned below



loop at pernr_tab into ls_pernr_tab .
    loop_index = sy-tabix.
    move ls_pernr_tab to ls_pernr_data.
    append ls_pernr_data to lt_pernr_data.
    clear ls_pernr_data.
    modulo = loop_index mod 4000 .
    delta_change_count = total_count - loop_index.
    if modulo eq 0 or delta_change_count eq 0.
      add 1 to gv_active_threads.
      if gv_active_threads <= lv_thread_count.
        add 1 to lv_thread.
        call function 'ZDATA_PI_MDM' starting new task lv_thread
          destination in group default
          performing set_session_done on end of task
          exporting
            last_run_date = last_run_date
          tables
            lt_pernr_data = lt_pernr_data.
*        wait until session = 'X'.
        if sy-subrc <> 0.                                   "#EC *
          subtract 1 from gv_active_threads.
        else.
          wait until session = 'X'.
          append lines of emp_data to lt_emp_data.
          total_deleted = total_deleted + deleted_records.
****** collect the data
*          FREE MEMORY ID 'HRDATA'.
          clear: lt_pernr_data[],emp_data[],session,deleted_records.
        endif.
else. " free work processes become zero so again get the free work processes and process the remaining records
     
 endif.
    endif.
    clear ls_pernr_tab.
  endloop.

Thanks

Bala Duvvuri

Re: problem with work processes in parallel processing

Former Member — Thu, 07 Oct 2010 04:18:57 GMT

Hi Bala,

The last remaining records were not executed because it does not satisfy the if condition if modulo eq 0 or delta_change_count eq 0. You can call one last parallel process before clear ls_pernr_tab. but be sure to verify that the table has entries in it.

Re: problem with work processes in parallel processing

Former Member — Thu, 07 Oct 2010 04:48:06 GMT

I am not able to understand



The last remaining records were not executed because it does not satisfy the if condition if modulo eq 0 or delta_change_count eq 0.

I am getting the loop index and finding the modulo and delta_change_count.for each 4000 records modulo will be 0 and if i have 10000 records and for the 10000th iteration delta_change_count will be 0 so it will go inside the if loop right


loop at pernr_tab into ls_pernr_tab .
    loop_index = sy-tabix.
    move ls_pernr_tab to ls_pernr_data.
    append ls_pernr_data to lt_pernr_data.
    clear ls_pernr_data.
    modulo = loop_index mod 4000 .
    delta_change_count = total_count - loop_index.
    if modulo eq 0 or delta_change_count eq 0.

Thanks

Bala Duvvuri