Memory and performance when copying a sorted table to a standard table

[dhorions]

Hello,

As you all probably know, it's not possible to use a sorted table as a tables parameter of a function module, but sometimes you want to use a sorted table in your function module for performance reasons, and at the end of the function module, you just copy it to a standard table to return to the calling program.

The problem with this is that at that moment, the contents of the table is in memory twice, which could result in the well known STORAGE_PARAMETERS_WRONG_SET runtime exception.

I've been looking for ways to do this without using an excessive amount of memory and still being performant. I tried four methods, all have their advantages and disadvantages, so I was hoping someone here could help me come up with the best way to do this. Both memory and performance are an issue.

Requirements :

- Memory usage must be as low as possible

- Performance must be as high as possible

- Method must work on all SAP versions from 4.6c and up

So far I have tried 3 methods.

I included a test report to this message, the output of this on my dev system is :

Test report for memory usage of copying tables

table1[] = table2[]

Memory : 192,751 Kb

Runtime: 436,842

Loop using workarea (with delete from original table)

Memory : 196,797 Kb

Runtime: 1,312,839

Loop using field symbol (with delete from original table)

Memory : 196,766 Kb

Runtime: 1,295,009

The code of the program :

I had some problems pasting the code here, so it can be found at [http://pastebin.com/f5e2848b5|http://pastebin.com/f5e2848b5]

Thanks in advance for the help.

Edited by: Dries Horions on Jun 19, 2009 1:23 PM

Edited by: Dries Horions on Jun 19, 2009 1:39 PM

Edited by: Dries Horions on Jun 19, 2009 1:40 PM

Edited by: Dries Horions on Jun 19, 2009 1:40 PM

[former_member192616]

Hi,

>

> As you all probably know, it's not possible to use a sorted table as a tables parameter of a function module,

yes. But you can use them as importing / exporting parameters with DDIC types.

Have you tried it?

>...

> The problem with this is that at that moment, the contents of the table is in memory twice, which could result in the well known STORAGE_PARAMETERS_WRONG_SET runtime exception.

>...

> Requirements :

> - Memory usage must be as low as possible

> - Performance must be as high as possible

> - Method must work on all SAP versions from 4.6c and up

The DELETE does not release any memory at all.

Up to the current release:

If you copy a standard table into a sorted table you need two times the memory the standard table occupies (memory aspect) and the table has to be sorted implicitly (run time aspect). The Array or Block operations (itab_sorted = itab_standard, INSERT LINES OF ...) processing multiple lines will be more efficient then the single line operations. They have less overhead.

Kind regards,

Hermann

[dhorions]

yes, it's true that you can use tham as importing or exporting parameters, unfortunately that doesn't work over RFC connections.

[dhorions]

Using append lines of or insert lines of is still marginally slower then just table1[] = table2[].

I haven't found any solution yet to reduce the memory usage.

table1[] = table2[]

Memory : 95,896 Kb

Runtime: 189,797

Loop using workarea (with delete from original table)

Memory : 97,888 Kb

Runtime: 533,683

Loop using field symbol (with delete from original table)

Memory : 97,874 Kb

Runtime: 623,627

Using insert Lines of into table

Memory : 95,835 Kb

Runtime: 203,670

Append lines of

Memory : 95,835 Kb

Runtime: 198,588

[former_member192616]

what do you mean by 'it doesn't work'? it should work as well....

[MarcinPciak]

Hi,

You are partly right. You can't pass sorted table types to RFC functions, but only via TABLES parameters. You can however used that one for IMPORTING/EXPORTING/CHANGING parameters. CHANGING would be what you need. But one strong disadvange is that you have to pass it via VALUE , which for large tables can drastiacally affect performance. I think, considering this, still better solution is the one you already chose. As for keeping good performance and fewer memory usage use this statement:

table1[] = table2[]

because the task of copying data is passed directly to kernel. It is the most recommended method for this purpose by SAP.

Regards

Marcin

[Former Member]

This message was moderated.

[Former Member]

You may also want to try using

APPEND LINES OF table1 TO table2.

or

INSERT LINES OF table1 INTO TABLE table2.

[Former Member]

How about something like this - not tested, but may be working:

EXPORT p1 = table1[] TO MEMORY ID 'ZTEST'.
CLEAR table1[].
IMPORT p1 = table2[] FROM MEMORY ID 'ZTEST'.

[dhorions]

Yes, that works fine and uses almost no extra memory. Problem is that it is very slow.

This is my backup solution, if I can't find any better solution, i'll use this. Thanks.

[former_member194613]

Please rethink you original approach.

*I always recommend sorted tables, but I would not recommend the temporary usage of a sorted table which is then

copied back to a standard table*

Use the sort command together with the binary search! With that you can rebuild all advantages of the sorted table.

The read is simple, the loop more complicated but also possible.

Measurements on internal tables: Reads and Loops:

/people/siegfried.boes/blog/2007/09/12/runtimes-of-reads-and-loops-on-internal-tables

Otherwise if your task really needs the copy, then

itab1 = itab2

is the best choice.

Siegfried

[former_member194613]

> but sometimes you want to use a sorted table in your function module for performance reasons

and then you copy it to memory .... this sound like n.onsense to me.

If you go down to 4.6 then use binray search and your issue is solved.

There is no better performance and memory advantage possible.

[dhorions]

Sorry about this response,i'm having a bad day and I forgot the 10 second rule. I removed my rather unfriendly answer.

Your statement is probably true in most cases, however, in this case the function module in question handles a lot of data, and I think, although I still need to run some tests on this, that the performance lost by exporting the table to memory will be less than the performance gained by using a sorted table.

I will look into it and respond in this thread if you suggestion helped.

Edited by: Dries Horions on Jul 7, 2009 4:49 PM

[former_member194613]

... fortunately I don't read here so often recently.

But still you can not gain performance by using sorted tables against a sorted standard table with BINARY SEARCH.

So in your case, if there are problems with the sorted table, the manual usage of the standard table is the way to go.

And f you go down to releases as old as 4.6, then the sorted tables have not all automatic optimizations which they have in later releases.

And don't take the word 'nonsense' personally, I use it, when I want to get a reply

Siegfried

[Former Member]

I've had another idea:

Create a RFC function like this (replace SOLI_TAB with your table types):

FUNCTION Z_COPY_TABLE .
*"----------------------------------------------------------------------
*"*"Lokale Schnittstelle:
*"  IMPORTING
*"     VALUE(IT_IN) TYPE  SOLI_TAB
*"  EXPORTING
*"     VALUE(ET_OUT) TYPE  SOLI_TAB
*"----------------------------------------------------------------------

et_out[] = it_in[].

ENDFUNCTION.

and then try something like this in your program:

DATA: gd_copy_done TYPE c LENGTH 1.
DATA: gt_one TYPE soli_tab.
DATA: gt_two TYPE soli_tab.

**************************************************************

PERFORM move_tables.

**************************************************************

FORM move_tables.

  CLEAR gd_copy_done.

  CALL FUNCTION 'Z_COPY_TABLE'
    STARTING NEW TASK 'ztest'
    PERFORMING copy_done ON END OF TASK
    EXPORTING
      it_in = gt_one[].

  CLEAR gt_one[].

  WAIT UNTIL gd_copy_done IS NOT INITIAL.

ENDFORM.

**************************************************************

FORM copy_done USING ld_task TYPE clike.
  RECEIVE RESULTS FROM FUNCTION 'Z_COPY_TABLE'
   IMPORTING
     et_out        = gt_two[].

  gd_copy_done = 'X'.
ENDFORM.

Maybe this is a little bit faster than the Memory-Export?

Edited by: Carsten Grafflage on Jul 20, 2009 11:06 AM