topic Re: Memory overflow problems when processing huge XML files in Application Development and Automation Discussions

Memory overflow problems when processing huge XML files

Former Member — Tue, 16 Nov 2010 18:09:35 GMT

Hi All,

We are in need of processing very large XML file.(more than 100MB)..

We ran this job in background and it resulted in runtime errors.

Is there any way of processing this file as a whole?

Edited by: Thomas Zloch on Nov 17, 2010 4:16 PM - subject adjusted

Re: Memory overflow problems when processing huge XML files

ThomasZloch — Tue, 16 Nov 2010 19:42:08 GMT

What are these runtime errors exactly? please quote title and section "error analysis".

Which technique are you using for XML processing? DOM classes, XSLT, simple transformations?

Thomas

Re: Memory overflow problems when processing huge XML files

Former Member — Tue, 16 Nov 2010 21:00:14 GMT

We are using DOM method..

The error text is "Warning: EM-Memory exhausted: Workprocess gets PRIV "

The termination occurred in the ABAP program "CL_IXML=======================CP"

in "IF_IXML_NODE~GET_NAME".

Re: Memory overflow problems when processing huge XML files

ThomasZloch — Wed, 17 Nov 2010 15:16:38 GMT

Normally such memory problems can be avoided by using block processing and clearing tempory data inbetween the blocks. However all XML techniques that I know (DOM, XSLT, ST) require all data to reside in an internal table at once. I will be facing a similar problem soon, so I'm quite interested in a solution.

One way would be to upgrade the hardware and allow more memory to be allocated to the workprocess (system administration). Some background information:

http://help.sap.com/saphelp_nw70ehp1/helpdata/en/49/32eb1fe92e3504e10000000a421937/frameset.htm

I wonder if there are other workarounds, let's see if there will be additional replies.

Thomas

Re: Memory overflow problems when processing huge XML files

former_member229034 — Wed, 17 Nov 2010 16:21:18 GMT

Hello Majualex,

As you said you are processing a 100 mb file, could you please let us know whether you are downloading or uploading a 100 mb file(from the application server). Else, let us know what kind of processing you are performing on the file.

Best Regards,

Chaitanya

Re: Memory overflow problems when processing huge XML files

Former Member — Wed, 17 Nov 2010 17:06:37 GMT

Many Thanks for your replies.........

Time being, I have split the file into chunks and then uploaded.

Our requirement is to upload the file from application server after comparing many fields in the XML file with R/3 data.

Re: Memory overflow problems when processing huge XML files

ThomasZloch — Wed, 17 Nov 2010 17:17:26 GMT

How did you split it? Normally there are outer brackets that only close at the very end of the file, how did you keep the XML integrity while splitting? Are you using a preprocessor progam?

Thomas

Re: Memory overflow problems when processing huge XML files

Former Member — Wed, 17 Nov 2010 17:37:07 GMT

Yes Thomas, You are very correct.....

Here, I split the XML file manually....

Its a good idea to develop a preprocessor program which will split up the file keeping the integrity of XML file.