topic Re: Parallel processing of a server file ??? in Application Development and Automation Discussions

Parallel processing of a server file ???

ealimeta — Mon, 22 Feb 2016 11:49:20 GMT

Hello guys .

Acctually I am trying to fulfill a task like this .
Read a file in the server and split it into several files on the server depending on the first 4 digits and the prelast one of each line that was read,
and all this must be done inside ONE report only .
But it takes too much time because the file I am trying to READ/SPLIT contains 2-3 millions of records.

Is there any way to make this thing run i parallel mode ,
so do in parallel 5 reads of the same MEGA FILE , one read for each split. ???

Actually the code is like this :
Waiting for your suggestions , thnx in advance .

OPEN DATASET path_mega_file FOR INPUT IN TEXT MODE ENCODING DEFAULT.
   DO.

     READ DATASET path_mega_file INTO ls_file.
     IF sy-subrc EQ 0.
       length = strlen( ls_file ).
       length = length - 2 .
***************************************************************************
       IF ls_file+0(4) EQ 'IT02' AND ls_file+length(1) EQ 'B' .

         OPEN DATASET path_azc_b FOR APPENDING IN TEXT MODE ENCODING DEFAULT.
         TRANSFER ls_file TO path_azc_b .
         CLOSE DATASET path_azc_b.

ELSEIF ls_file+0(4) EQ 'IT02' AND ls_file+length(1) EQ 'P'.

OPEN DATASET path_azc_p FOR APPENDING IN TEXT MODE ENCODING DEFAULT.
TRANSFER ls_file TO path_azc_p .
CLOSE DATASET path_azc_p.

ELSEIF ls_file+0(4) EQ 'IT08' AND ls_file+length(1) EQ 'B'.

OPEN DATASET path_azi_b FOR APPENDING IN TEXT MODE ENCODING DEFAULT.
TRANSFER ls_file TO path_azi_b .

CLOSE DATASET path_azi_b.

ELSEIF ls_file+0(4) EQ 'IT08' AND ls_file+length(1) EQ 'P'.

         OPEN DATASET path_azi_p FOR APPENDING IN TEXT MODE ENCODING DEFAULT.
         TRANSFER ls_file TO path_azi_p .
         CLOSE DATASET path_azi_p.

ELSEIF ls_file+0(4) EQ 'IT09' AND ls_file+length(1) EQ 'P'.

OPEN DATASET path_apo_p FOR APPENDING IN TEXT MODE ENCODING DEFAULT.
TRANSFER ls_file TO path_apo_p .

CLOSE DATASET path_apo_p.

ENDIF.
*****************************************************************************
ENDIF.

ENDDO.
CLOSE DATASET path_mega_file.

Re: Parallel processing of a server file ???

Juwin — Mon, 22 Feb 2016 13:57:52 GMT

Instead of too many READs, can you read the contents of the file into a XSTRING, with a single read statement and then process that XSTRING variable in ABAP to do the splits?

Thanks,

Juwin

Re: Parallel processing of a server file ???

SuhaSaha — Mon, 22 Feb 2016 14:41:34 GMT


then process that XSTRING variable in ABAP to do the splits?

And how would one do that?

Re: Parallel processing of a server file ???

Juwin — Mon, 22 Feb 2016 16:06:14 GMT

XString to String using: LXE_COMMON_XSTRING_TO_STRING

Split String at CL_ABAP_CHAR_UTILITIES=>NEWLINE into table.

Thus, you will have all the lines of the file in one single internal table.

Will this help? May or may not, because he still has to do the same number of loops on this table, to do the splits.

Thanks,

Juwin

Re: Parallel processing of a server file ???

ealimeta — Mon, 22 Feb 2016 16:20:45 GMT

Hello Juwin .
Firstly thnx for taking time to reply to my post .
Do you think it is possible to read with a single READ a file with 2.5 million records ??
I think the Internal table would go in SHORT_DUMP out of memory , or exeded memory .

Have you tried it by yourself ?

Re: Parallel processing of a server file ???

Juwin — Mon, 22 Feb 2016 17:00:18 GMT

That's correct. There is a risk of OUT_OF_MEMORY error. But, whether 2.5 million records will cause the dump or not, is also dependent on the record length, isn't it?

Total memory used = Rows x Memory used per row.

Thanks.

Re: Parallel processing of a server file ???

gabriel_pill-kahan — Mon, 22 Feb 2016 21:53:30 GMT

Hi,

If your application server is on UNIX: you can use its split command (execute the split command as an external command). This will create multiple files that you can now process in parallel.

Regards, Gabriel

Re: Parallel processing of a server file ???

ThangaPrakash — Tue, 23 Feb 2016 07:25:20 GMT

Hello Elton,

Try like below, split files into different internal tables and using those internal tables run the fucntion module where you have the split logic with STARTING NEW TASK.

OPEN DATASET path_mega_file FOR INPUT IN TEXT MODE ENCODING DEFAULT.

DO.

READ DATASET path_mega_file INTO ls_file.

IF sy-index LT 10000.

APPEND ls_file TO itab1.

CALL FUCNTION 'Z_CUSTOM_SPLIT_FILES' STARTING NEW TASK

EXPORTING

it_tab = itab1.

ELSEIF sy-index GT 10000 AND SY-INDEX LT 20000.

APPEND ls_file TO itab2.

CALL FUCNTION 'Z_CUSTOM_SPLIT_FILES' STARTING NEW TASK

EXPORTING

it_tab = itab2.

ENDIF.

ENDDO.

CLOSE DATASET path_mega_file.

*********************************************************************88

FUNCTION Z_CUSTOM_SPLIT_FILES.

LOOP AT it_tab INTO ls_file.

length = strlen( ls_file ).

length = length - 2 .

IF ls_file+0(4) EQ 'IT02' AND ls_file+length(1) EQ 'B' .

OPEN DATASET path_azc_b FOR APPENDING IN TEXT MODE ENCODING DEFAULT.

TRANSFER ls_file TO path_azc_b .

CLOSE DATASET path_azc_b.

ELSEIF ls_file+0(4) EQ 'IT02' AND ls_file+length(1) EQ 'P'.

OPEN DATASET path_azc_p FOR APPENDING IN TEXT MODE ENCODING DEFAULT.

TRANSFER ls_file TO path_azc_p .

CLOSE DATASET path_azc_p.

ELSEIF ls_file+0(4) EQ 'IT08' AND ls_file+length(1) EQ 'B'.

OPEN DATASET path_azi_b FOR APPENDING IN TEXT MODE ENCODING DEFAULT.

TRANSFER ls_file TO path_azi_b .

CLOSE DATASET path_azi_b.

ELSEIF ls_file+0(4) EQ 'IT08' AND ls_file+length(1) EQ 'P'.

OPEN DATASET path_azi_p FOR APPENDING IN TEXT MODE ENCODING DEFAULT.

TRANSFER ls_file TO path_azi_p .

CLOSE DATASET path_azi_p.

ELSEIF ls_file+0(4) EQ 'IT09' AND ls_file+length(1) EQ 'P'.

OPEN DATASET path_apo_p FOR APPENDING IN TEXT MODE ENCODING DEFAULT.

TRANSFER ls_file TO path_apo_p .

CLOSE DATASET path_apo_p.

ENDIF.

ENDLOOP.

ENDFUCNTION.

Re: Parallel processing of a server file ???

ealimeta — Tue, 23 Feb 2016 17:41:29 GMT

Hello Gabriel.
Thnx for replying .
The problem was at the DO , ENDO.
It needed to become DO n TIMES . ...................... ENDO.

Where the n was the number of lines the MEGA FILE had inside .
I found the nr of lines executing an external comand on my OS LINUX creating a new comand
z_test in SM69 setting the attributes
Operating System Command : wc
Parameters for Operating System : -l

And in the ABAP report call this comand like this :

CALL FUNCTION 'SXPG_COMMAND_EXECUTE'
   EXPORTING
     commandname                         = 'Z_TEST'
    ADDITIONAL_PARAMETERS               = 'Server path of the file for witch i run the command'
TABLES
    EXEC_PROTOCOL                       = EXEC_PROTOCOL
EXCEPTIONS
    NO_PERMISSION                       = 1
    COMMAND_NOT_FOUND                   = 2
    PARAMETERS_TOO_LONG                 = 3
    SECURITY_RISK                       = 4
    WRONG_CHECK_CALL_INTERFACE          = 5
    PROGRAM_START_ERROR                 = 6
    PROGRAM_TERMINATION_ERROR           = 7
    X_ERROR                             = 8
    PARAMETER_EXPECTED                  = 9
    TOO_MANY_PARAMETERS                 = 10
    ILLEGAL_COMMAND                     = 11
    WRONG_ASYNCHRONOUS_PARAMETERS       = 12
    CANT_ENQ_TBTCO_ENTRY                = 13
    JOBCOUNT_GENERATION_ERROR           = 14
    OTHERS                              = 15
           .
Read TABLE exec_protocol into st_exec_protocOl index 1 .

DATA : LV_NR_REC(10),
        LV_PATH(128),
        NR_REC TYPE I .
SPLIT st_exec_protocol-message at '/' into lv_nr_rec lv_path.
CONDENSE LV_NR_REC.
Move LV_NR_REC TO NR_REC.

That worked for me .
Thnx again for your time on replying posts .
Keep rocking guys