Memory overflow problems when processing huge XML files

[Former Member]

Hi All,

We are in need of processing very large XML file.(more than 100MB)..

We ran this job in background and it resulted in runtime errors.

Is there any way of processing this file as a whole?

Edited by: Thomas Zloch on Nov 17, 2010 4:16 PM - subject adjusted

[ThomasZloch]

What are these runtime errors exactly? please quote title and section "error analysis".

Which technique are you using for XML processing? DOM classes, XSLT, simple transformations?

Thomas

[Former Member]

We are using DOM method..

The error text is "Warning: EM-Memory exhausted: Workprocess gets PRIV "

The termination occurred in the ABAP program "CL_IXML=======================CP"

in "IF_IXML_NODE~GET_NAME".

[ThomasZloch]

Normally such memory problems can be avoided by using block processing and clearing tempory data inbetween the blocks. However all XML techniques that I know (DOM, XSLT, ST) require all data to reside in an internal table at once. I will be facing a similar problem soon, so I'm quite interested in a solution.

One way would be to upgrade the hardware and allow more memory to be allocated to the workprocess (system administration). Some background information:

http://help.sap.com/saphelp_nw70ehp1/helpdata/en/49/32eb1fe92e3504e10000000a421937/frameset.htm

I wonder if there are other workarounds, let's see if there will be additional replies.

Thomas

[former_member229034]

Hello Majualex,

As you said you are processing a 100 mb file, could you please let us know whether you are downloading or uploading a 100 mb file(from the application server). Else, let us know what kind of processing you are performing on the file.

Best Regards,

Chaitanya

[Former Member]

Many Thanks for your replies.........

Time being, I have split the file into chunks and then uploaded.

Our requirement is to upload the file from application server after comparing many fields in the XML file with R/3 data.

[ThomasZloch]

How did you split it? Normally there are outer brackets that only close at the very end of the file, how did you keep the XML integrity while splitting? Are you using a preprocessor progam?

Thomas

[Former Member]

Yes Thomas, You are very correct.....

Here, I split the XML file manually....

Its a good idea to develop a preprocessor program which will split up the file keeping the integrity of XML file.