Big memory consumption XSLT, why Heap is used? Best with ST.

[Sandra_Rossi]

I've got an XSLT transformation (not ST) which converts a big ABAP internal table into XML (serialization). It did a short dump SYSTEM_NO_ROLL due to something like 4 GB memory needed. 

Here's the memory consumption in the short dump:

I can reproduce the issue in my DEV system with a small test program with dummy data:

I can solve the issue by creating an equivalent ST transformation, which is consuming less memory (below), and is also 10 times faster.

If I reduce the volume, XSLT is successful, but I see that it's consuming the HEAP memory (which I guess is bad),

while ST is not consuming it.

My question is, I'd like to understand why the Heap memory is consumed by XSLT although the total memory is not so much solicited (284 MB + 631 MB) and there's no other activity in the system, so I'm interested by any information.

EDIT: more information added below.

Initial screen of RSMEMORY:

SM50:

Here's an animated GIF of maximum memory consumption by XSLT (+ its source and target data) for around 50 seconds, going from second 52 to second 42, the heap memory is immediately used:

Here's an animated GIF of maximum memory consumption by ST (+ its source and target data) for around 40 seconds, from second 36 to second 16, the heap memory is immediately used too:

Here's the source code of ABAP and transformations: sandraros/test-xslt-memory (installation via abapGit/abapGit: Git client for ABAP).

Thank you.

Sandra

System: ABAP 7.40 SP 23; Kernel 753 PL 1300.

NB: I guess that Heap is bad because the workprocess becomes PRIV, so it cannot be reused by other ABAP tasks, and it's especially a problem if the program is running in a DIA workprocess. I don't know for BGD workprocess (my case).

NB: for generating the XSLT short dump, the internal table contains more than 260.000 lines, 20 fields of total 1032 bytes, including characters (Unicode 2 bytes), packed, NUMC, dates, no references, and two nested internal tables, the first one containing 1 line (always) and having 4 fields of total 186 bytes, the second one containing 1 line (always) having 20 fields of total 616 bytes including one nested internal table containing 1 line (always) having 4 fields of total 36 bytes. Via the ST transformation, the generated XML file is 413 MB.

[HermannGahm]

Hi Sandra, like Ryan i think as well the intermediate format and the higher flexibility of XSLT have their price regarding memory and performance. Batch processes can't go priv, they typically use heap memory before they use EM (see report RSMEMORY for memory allocation sequence differences for DIA and BTC processes. Kind regards, Hermann

[HermannGahm]

Hi Sandra,

> I'd like to understand why Heap is used although EM is only few hundreds of MB and no other task in the system.
can you post the output of RS MEMORY? HEAP Memory for DIALOG processes is used when EM Quota is reached. it is workprocess local memory an not released until the wp is restarted (this is why we call it priv). Measure memory Usage in SAT with not aggregated trace should show you peak memory usage i think.

Kind regards,

Hermann

[Sandra_Rossi]

@HerrmanGahm The initial screen of RSMEMORY is added to my question.

[HermannGahm]

Hi Sandra,

thank you.

In the output you can see the difference in the allocation sequence.

Dialog processes use up to 2000000000 EM and after that is used 2000000000 HEAP.
Background processes do it the other way round (first HEAP then EM).

So if you see a HEAP usage in dialog processes it means either the 2000000000 EM have been used up (or were not available anymore at that point in time)
and afterwards HEAP memory had to be used. HEAP is work process local memory which will be released after wp restart (we call that priv see
Private Memory, PRIV Memory (HEAP) | SAP Help Portal,

I would assume that during your XSLT transformation the available EM for dialog process is used up and HEAP is used. Maybe memory might be released 
after peak consumption (EM can go down) but HEAP will stay. I'm not sure if you could verify that with un unaggregated trace with memory activated in SAT
or with memory snapshots in memory inspector right after the transformation call. STATS as well should show you maximum extended memory usage during
execution (e.g.).

and now i'm mot sure anymore if the above is helpful for you... 😉

Kind regards,

Hermann

[Sandra_Rossi]

@HermannGahm Your answers help me a lot to better understand memory consumption (e.g. RSMEMORY to show the difference between DIA and BGD memory strategy). I have added two animated GIF of MAXIMUM (if more source data is added, it would dump) memory consumption of XSLT versus ST via SM50, running in DIA processes. In this test, I realize that ST can consume Heap too, but I couldn't see it in my first test because ST consumes so little memory compared to XSLT. I'll check SAT, STATS and Memory Inspector tomorrow. My original question concerns the big difference between XSLT and ST, so I guess that @Ryan-Crosby answer points to the right direction (respectively random access versus serial access, i.e. maybe the source XML is parsed entirely and stored in memory before running XSLT).

[Ryan-Crosby]

Hi Sandra,

I think it has to do with the following as described in the help for simple transformation - 

• ST programs only have serial access to the XML data and are therefore very efficient even with large data volumes. The nodes in XML data, however, cannot be accessed in an arbitrary order.

In the help reference to XSLT in the same link it displays a diagram showing the asXML intermediate format when XSLT is used along with a basic description of the processing, which I would assume would account for additional memory usage (both links included below).

ST and XSLT links

Regards,

Ryan Crosby

[Sandra_Rossi]

Thanks. Yes, ST is about 10 times faster than XSLT (rule of thumb). XSLT should be used only for stuff which cannot be done via ST. My question seems to be the first one giving information about the memory used by XSLT versus ST. But my question is more about Heap and memory in general, I'd like to understand why Heap is used although EM is only few hundreds of MB and no other task in the system.

[jmodaal]

Hello,

did you try this?

https://blogs.sap.com/2011/01/18/next-generation-abap-runtime-analysis-sat-how-to-analyze-memory-con...

Kind regards

Jan

[Sandra_Rossi]

No. Will it trace inside XSLT? (basic transformation, no method call) Do you think it will help me understand why the Heap memory is consumed by XSLT although the total memory is not so much solicited and there's no other activity in the system?

[jmodaal]

Hello,

I do not know whether it may help you. It is just the first approach coming to my mind when it is about memory problems.

Kind regards

Jan