Parallel processing of a server file ???

[ealimeta]

Hello guys .

Acctually I am trying to fulfill a task like this .
Read a file in the server and split it into several files on the server depending on  the  first 4 digits and the prelast one of each line that was read,
and all this must be done inside ONE report only .
But it takes too much time because the file I am trying to READ/SPLIT contains 2-3 millions of records.

Is there any way to make this thing run i parallel mode ,
so do in parallel 5 reads of the same  MEGA FILE , one  read for each split. ???

Actually the code is like this : 
Waiting for your suggestions , thnx in advance .

OPEN DATASET path_mega_file FOR INPUT IN TEXT MODE ENCODING DEFAULT.
   DO.

     READ DATASET path_mega_file INTO ls_file.
     IF sy-subrc EQ 0.
       length = strlen( ls_file ).
       length = length - 2 .
***************************************************************************
       IF ls_file+0(4) EQ 'IT02' AND  ls_file+length(1) EQ 'B' .

         OPEN DATASET path_azc_b FOR APPENDING IN TEXT MODE ENCODING DEFAULT.
         TRANSFER ls_file TO path_azc_b .
         CLOSE DATASET path_azc_b.

   ELSEIF ls_file+0(4) EQ 'IT02' AND  ls_file+length(1) EQ 'P'.

  OPEN DATASET path_azc_p FOR APPENDING IN TEXT MODE ENCODING DEFAULT.
         TRANSFER ls_file TO path_azc_p .
         CLOSE DATASET path_azc_p.

       ELSEIF ls_file+0(4) EQ 'IT08' AND  ls_file+length(1) EQ 'B'.

         OPEN DATASET path_azi_b FOR APPENDING IN TEXT MODE ENCODING DEFAULT.
         TRANSFER ls_file TO path_azi_b .

         CLOSE DATASET path_azi_b.

       ELSEIF ls_file+0(4) EQ 'IT08' AND  ls_file+length(1) EQ 'P'.

         OPEN DATASET path_azi_p FOR APPENDING IN TEXT MODE ENCODING DEFAULT.
         TRANSFER ls_file TO path_azi_p .
         CLOSE DATASET path_azi_p.

       ELSEIF ls_file+0(4) EQ 'IT09' AND  ls_file+length(1) EQ 'P'.

         OPEN DATASET path_apo_p FOR APPENDING IN TEXT MODE ENCODING DEFAULT.
         TRANSFER ls_file TO path_apo_p .

         CLOSE DATASET path_apo_p.

       ENDIF.
*****************************************************************************
     ENDIF.

   ENDDO.
   CLOSE DATASET path_mega_file.

[ealimeta]

Hi,

If your application server is on UNIX: you can use its split command (execute the split command as an external command). This will create multiple files that you can now process in parallel.

Regards, Gabriel

[Juwin]

Instead of too many READs, can you read the contents of the file into a XSTRING, with a single read statement and then process that XSTRING variable in ABAP to do the splits?

Thanks,

Juwin

[SuhaSaha]

then process that XSTRING variable in ABAP to do the splits?

And how would one do that?

[Juwin]

XString to String using: LXE_COMMON_XSTRING_TO_STRING

Split String at CL_ABAP_CHAR_UTILITIES=>NEWLINE into table.

Thus, you will have all the lines of the file in one single internal table.

Will this help? May or may not, because he still has to do the same number of loops on this table, to do the splits.

Thanks,

Juwin

[ealimeta]

Hello Juwin .
Firstly thnx for taking time to reply to my post .
Do you think it is possible to read with a single READ a file with 2.5 million records ??
I think the Internal table would go in SHORT_DUMP out of memory , or exeded memory .

Have you tried it by yourself ?

[Juwin]

That's correct. There is a risk of OUT_OF_MEMORY error. But, whether 2.5 million records will cause the dump or not, is also dependent on the record length, isn't it?

Total memory used = Rows x Memory used per row.

Thanks.

[gabriel_pill-kahan]

Hi,

If your application server is on UNIX: you can use its split command (execute the split command as an external command). This will create multiple files that you can now process in parallel.

Regards, Gabriel

[ealimeta]

Hello Gabriel.
Thnx for replying .
The problem was at the DO , ENDO. 
It needed to become   DO  n TIMES .   ......................    ENDO.

Where the n  was the  number of lines the MEGA FILE had inside . 
I found the nr of lines executing an external comand on my OS LINUX  creating a new comand
z_test  in SM69    setting the attributes  
Operating System Command  :  wc
Parameters for Operating System : -l

And in the ABAP report    call this comand  like this :

CALL FUNCTION 'SXPG_COMMAND_EXECUTE'
   EXPORTING
     commandname                         = 'Z_TEST'
    ADDITIONAL_PARAMETERS               = 'Server path of the file for witch i run the command'
  TABLES
    EXEC_PROTOCOL                       = EXEC_PROTOCOL
  EXCEPTIONS
    NO_PERMISSION                       = 1
    COMMAND_NOT_FOUND                   = 2
    PARAMETERS_TOO_LONG                 = 3
    SECURITY_RISK                       = 4
    WRONG_CHECK_CALL_INTERFACE          = 5
    PROGRAM_START_ERROR                 = 6
    PROGRAM_TERMINATION_ERROR           = 7
    X_ERROR                             = 8
    PARAMETER_EXPECTED                  = 9
    TOO_MANY_PARAMETERS                 = 10
    ILLEGAL_COMMAND                     = 11
    WRONG_ASYNCHRONOUS_PARAMETERS       = 12
    CANT_ENQ_TBTCO_ENTRY                = 13
    JOBCOUNT_GENERATION_ERROR           = 14
    OTHERS                              = 15
           .
Read TABLE exec_protocol into st_exec_protocOl index 1 .

DATA : LV_NR_REC(10),
        LV_PATH(128),
        NR_REC TYPE I .
SPLIT st_exec_protocol-message at '/'  into  lv_nr_rec lv_path.
CONDENSE LV_NR_REC.
Move LV_NR_REC TO NR_REC.

That worked for me .
Thnx again for your time on replying posts .
Keep rocking guys     

[ThangaPrakash]

Hello Elton,

Try like below, split files into different internal tables and using those internal tables run the fucntion module where you have the split logic with STARTING NEW TASK.

OPEN DATASET path_mega_file FOR INPUT IN TEXT MODE ENCODING DEFAULT.

  DO.

     READ DATASET path_mega_file INTO ls_file.

   IF sy-index LT 10000.

     APPEND ls_file TO itab1.

     CALL FUCNTION 'Z_CUSTOM_SPLIT_FILES' STARTING NEW TASK

      EXPORTING

         it_tab = itab1.

   ELSEIF sy-index GT 10000 AND SY-INDEX LT 20000.

     APPEND ls_file TO itab2.

     CALL FUCNTION 'Z_CUSTOM_SPLIT_FILES' STARTING NEW TASK

      EXPORTING

         it_tab = itab2.

   ENDIF.

  ENDDO.

CLOSE DATASET path_mega_file.

*********************************************************************88

FUNCTION Z_CUSTOM_SPLIT_FILES.

LOOP AT it_tab INTO ls_file.

       length = strlen( ls_file ).

       length = length - 2 .

       IF ls_file+0(4) EQ 'IT02' AND  ls_file+length(1) EQ 'B' .

         OPEN DATASET path_azc_b FOR APPENDING IN TEXT MODE ENCODING DEFAULT.

         TRANSFER ls_file TO path_azc_b .

         CLOSE DATASET path_azc_b.

   ELSEIF ls_file+0(4) EQ 'IT02' AND  ls_file+length(1) EQ 'P'.

  OPEN DATASET path_azc_p FOR APPENDING IN TEXT MODE ENCODING DEFAULT.

         TRANSFER ls_file TO path_azc_p .

         CLOSE DATASET path_azc_p.

       ELSEIF ls_file+0(4) EQ 'IT08' AND  ls_file+length(1) EQ 'B'.

         OPEN DATASET path_azi_b FOR APPENDING IN TEXT MODE ENCODING DEFAULT.

         TRANSFER ls_file TO path_azi_b .

         CLOSE DATASET path_azi_b.

       ELSEIF ls_file+0(4) EQ 'IT08' AND  ls_file+length(1) EQ 'P'.

         OPEN DATASET path_azi_p FOR APPENDING IN TEXT MODE ENCODING DEFAULT.

         TRANSFER ls_file TO path_azi_p .

         CLOSE DATASET path_azi_p.

       ELSEIF ls_file+0(4) EQ 'IT09' AND  ls_file+length(1) EQ 'P'.

         OPEN DATASET path_apo_p FOR APPENDING IN TEXT MODE ENCODING DEFAULT.

         TRANSFER ls_file TO path_apo_p .

         CLOSE DATASET path_apo_p.

       ENDIF.

ENDLOOP.

ENDFUCNTION.