topic Re: XML parser and memory in Application Development and Automation Discussions

XML parser and memory

eddy_declercq — Thu, 15 Mar 2012 07:58:11 GMT

Hi,

I need to parse several (rather large) XML files and I'm doing this with the classic:

DATA:     pixml          TYPE REF TO if_ixml,
             pdocument      TYPE REF TO if_ixml_document,
             pstreamfactory TYPE REF TO if_ixml_stream_factory,
             pistream       TYPE REF TO if_ixml_istream.

*-- create the main factory
   pixml = cl_ixml=>create( ).
*-- create the initial document
   pdocument = pixml->create_document( ).
*-- create the stream factory
   pstreamfactory = pixml->create_stream_factory( ).

*-- create an input stream for the table
   pistream = pstreamfactory->create_istream_itable( table = xml_table2
                                                     size = totalsize ).

*-- create the parser
   pparser = pixml->create_parser( stream_factory = pstreamfactory
                                   istream        = pistream
                                   document       = pdocument ).

....

FREE: pixml, pdocument, pstreamfactory, pistream, pparser, pnode.

Everything works fine for the first file, but with the second file (although a bit smaller) I get a mem_alloc_failed dump. It occurs to me that things keep piling up. This is strange since I do a free of all objects. There doesn't seem to be a destructor method.

What can I do in order to solve this memory issue?

Eddy

Re: XML parser and memory

Former Member — Thu, 15 Mar 2012 09:56:27 GMT

Can attach a screenshot of the dump? I'd like to see the exact line in your code where the dump happens.

Re: XML parser and memory

eddy_declercq — Thu, 15 Mar 2012 10:04:04 GMT

Hi,

Not a lot to see, just a dump on SAP code:

Runtime errors

MEM_ALLOC_FAILED

Short text

Internally required memory is not available

What happened?

Resource bottleneck

	The current program "CL_IXML=======================CP" had to be terminated
	because
	a capacity limit has been reached.

....


1653	method if_ixml_parser~has_event_subscription
1654	by kernel module abkm_iXMLParser_HasEventSubscription.
1655	endmethod.

1656

1657	method if_ixml_parser~parse_event
1658	by kernel module abkm_iXMLParser_ParseEvent.
1659	endmethod.

1660

1661	method if_ixml_parser~parse
1662	by kernel module abkm_iXMLParser_Parse.
>>>>	endmethod.

1664

1665	method if_ixml_parser~num_errors
1666	by kernel module abkm_iXMLParser_NumErrors.
1667	endmethod.

....

Eddy

Re: XML parser and memory

Former Member — Thu, 15 Mar 2012 10:08:05 GMT

Just trying to zero in on the problem here... Have you tried running only the second xml file independently? has it worked?

Re: XML parser and memory

eddy_declercq — Thu, 15 Mar 2012 10:12:42 GMT

Parsing files one at the time doesn't give any problem. The problem is when one loops on all files in a dir.

There is no rocket science since it relies on the code from BCCIIXMLT1 (where that code doesn't even free the objects).

Re: XML parser and memory

naimesh_patel — Thu, 15 Mar 2012 18:23:56 GMT

Do you have the method DESTRUCTOR in class CL_IXML? If yes, Try calling this method before freeing the PIXML object.

Re: XML parser and memory

eddy_declercq — Fri, 16 Mar 2012 07:02:27 GMT

Like said in the initial question, there is no destructor method.

Meanwhile I've found a workaround in order to avoid the problem.

Re: XML parser and memory

naimesh_patel — Fri, 16 Mar 2012 16:00:13 GMT

Eddy De Clercq wrote:
Like said in the initial question, there is no destructor method.

What version you are on?

Meanwhile I've found a workaround in order to avoid the problem.

Do you wish to share with us?

Re: XML parser and memory

eddy_declercq — Tue, 20 Mar 2012 08:18:53 GMT

Version of what?

Clearing the XML table before the actual parsing instead of before reading the file.

Re: XML parser and memory

hendrik_brandes — Tue, 20 Mar 2012 10:34:42 GMT

Hi Eddy,

why do you need the CL_IXML classes?

I have made very good experiences with the XSLT-Transformations ( not the Simple-Transformations ). Especially for filtering data, converting data. And: We have at least processed very huge files.

Kind regards,

Hendrik

Re: XML parser and memory

Former Member — Wed, 21 Mar 2012 06:49:07 GMT

Hi Eddy,

Are you using xmlparser->parse( ) or set_dom_generating( ) methods? These methods can result in memory issues. Avoid these methods and use only event based parsing for large files.

Regards,Varun

Re: XML parser and memory

eddy_declercq — Wed, 21 Mar 2012 06:55:45 GMT

Do you have a link to documentation to that?

Re: XML parser and memory

eddy_declercq — Wed, 21 Mar 2012 06:56:54 GMT

See my initial msg for the methods used. Examples of alternatives are welcome.

Re: XML parser and memory

Former Member — Wed, 21 Mar 2012 07:09:44 GMT

I saw the initial message already, but it shows only dots '....' after step pixml->create_parser. So it is not clear if Parse( ) is causing the problem. I saw BCCIIXMLT11 is using both Parse() and set_dom_generating( ) . Solution is also in BCCIIXMLT11, but only follow the code when PA_EVENT = 'X' and PA_DOM = ' '.

Re: XML parser and memory

hendrik_brandes — Wed, 21 Mar 2012 10:01:33 GMT

Sure: http://help.sap.com/saphelp_nw73/helpdata/en/fd/9d734b389211d596a200a0c94260a5/frameset.htm

Another good reference is this book: http://www.amazon.de/ABAP-Fortgeschrittene-Techniken-Tools-PRESS/dp/3836211513/ref=sr_1_1?ie=UTF8&qid=1332323329&sr=8-1 . Sadly, the first volume is not avaiable any more, because the XML-processing has been described very detailed over 2 chapters. (Sorry for the link of the german book).

Within your ABAP-Coding you just have to use the CALL TRANSFORMATION-Statement.

One big advantage of CALL TRANSFORMATION is, that you can pass ABAP-Values to your XSL-program and vice versa. E.g. you do not have to handle with documents at all. If you need to have a internal table as result of your XML-Processing, you can get it.

I think the integrated way for XSL-Transformations in ABAP is very smart. For example look at my blog http://scn.sap.com/community/netweaver-as/blog/2011/12/30/extend-the-sicf-logonscreen-with-xslt -- of course its not the same way, what you want todo, but it should give you a hint, about what you can expect.

Kind regards,

Hendrik