cancel
Showing results for 
Search instead for 
Did you mean: 

Frequent heap dumps and system load average

balajik
Explorer
0 Kudos
353

Dear Experts,

Performance of our development has gone worst. There has been frequent Heap Dumps and Paging, also System Load Average '/Kernel/SAPJVM/SystemProblemReporting/System Load Average' is over 133%.

We are using NW731 with EP core.

JVM parameters arememory are according to SAP recommendations

-Xms4096m

-Xmx4096m

-XX:NewSize=1365m

-XX:MaxNewSize=1365m

-XX:PermSize=1024m

-XX:MaxPermSize=1024m

-XX:NewRatio=6

SCS, Application and Database are on single host.

Number of nodes=2.

Physical memory size is 15GB

Appreciate your assistance in analyzing and fixing the performance issue.

Thanks,

Kumar

Accepted Solutions (0)

Answers (3)

Answers (3)

balajik
Explorer
0 Kudos

Hi Divyanshu,


Either zip or direct thread dump format b00 is not supported for attaching. OOM and java_PID**** files are more than 4 GB in size.....

Is there any other way to send the files....

divyanshu_srivastava3
Active Contributor
0 Kudos

Hi Kumar,

You can share your files using external spaces like Dropbox etc.

Also, as server0 log files(dev_server0) contains thread dumps, you can zip and attach this file.

Please upload these files to SAP as well, they will provide you a container for uploading heap dumps.

Regards,
Divyanshu

balajik
Explorer
0 Kudos

Hi Divyanshu,

I've sent you a mail with thread attached.....please check and assist.

Thanks,

Kumar.

divyanshu_srivastava3
Active Contributor
0 Kudos

Hi Kumar,

I scanned your dumped and there is not much to say until I have the heap dump and some other details.

Regards,

Divyanshu

balajik
Explorer
0 Kudos

Currently SAP has provided latest patch to fine tune the application (add-on on NW) with few indexes. Performance has been increased to a good extent. I'll post once if any further performance issues encounter.

Thank you all

Kumar

Former Member
0 Kudos

Hi Balaji,

Please try to generate heap hprof dump, by adding below parameters in configtool

-XX:+HeapDumpOnOutOfMemoryError

-XX:+HeapDumpOnCtrlBreak

And after generating hprof file you can analyze it using memory analyzer.

Regards,

Jitendra

Former Member
0 Kudos

Hi Balaji,

Please use below doc to know more about using memory analyzer.

http://wiki.scn.sap.com/wiki/display/TechTSG/(JSTTSG)(P4)+Produce+Heap+Dump+in+Java+Serverhttp://

Regards,

Jitendra

divyanshu_srivastava3
Active Contributor
0 Kudos

Hi Kumar,

I would suggest you to take 7-8 thread dumps in interval 10 15-20 sec when the utilization is high. Then find and map the corresponding Kernel ID with Thread ID of the processes causing high utilization. Also, use MAT for HEAP dumps to find problem in heaps, check for leaking loaders and memory leaks.

Regards,

Divyanshu

balajik
Explorer
0 Kudos

Hi Divyanshu,

Thanks for your quick reply. I just downloaded memory analyzer tool, will analyze recent OOM.hprof file. Btw....how to takte thread dumps, please assist. I am using NWA don't have access to configtool.

Thanks,

Kumar

divyanshu_srivastava3
Active Contributor
0 Kudos

Hi,

Simply use sap mmc or sap mc(host:5XX13) and right click on server node and click dump stack. Then u can check corresponding dev_serverX.out file in work directory in which thread dumps are generated.

Other simple way is use jcmon to take thread dumps.

Regards,

Divyanshu

davefitzgibbon
Advisor
Advisor
0 Kudos

Hello Kumar,

Have a look at the following link which shows you how to obtain thread dumps.

http://wiki.sdn.sap.com/wiki/x/VYZ3Cw

Regards,

David

divyanshu_srivastava3
Active Contributor
0 Kudos

Since you don't have access to config tool, use JVVMON to generate HEAP dumps.

Regards,

Divyanshu

divyanshu_srivastava3
Active Contributor
0 Kudos

Hi Balaji,

How is it working ? Any updates ..

Regards,

DV

balajik
Explorer
0 Kudos

Hi Divyanshu,

I was away on to another project for configuring change control tool.

As mentioned earlier I don't have OS access.

But could able to download Thread Dumps via NWA as mentioned by David...., but couldn't able to open them.

Opened OOM.hprof file through MAT to check leak suspects etc. I could hardly understand as it is my first Java AS project ....here is bits and pieces from the dump analyzer tool.

Leap Suspects:

Problem Suspect 1:

The thread com.sap.engine.core.thread.execution.CentralExecutor$SingleThread @ 0x58dcd248 HTTP Worker [@662359980] keeps local variables with total size 3,66,84,54,240 (89.70%) bytes.

The memory is accumulated in one instance of "java.lang.Object[]" loaded by "<system class loader>".

The stacktrace of this Thread is available. See stacktrace.

And here is the stacktrace...

HTTP Worker [@662359980]

  at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:25)

  at java.lang.reflect.Array.newInstance(Ljava/lang/Class;I)Ljava/lang/Object; (Array.java:52)

  at oracle.jdbc.driver.BufferCache.get(Ljava/lang/Class;I)Ljava/lang/Object; (BufferCache.java:226)

  at oracle.jdbc.driver.PhysicalConnection.getByteBuffer(I)[B (PhysicalConnection.java:7665)

  at oracle.jdbc.driver.T4C8TTIClob.read([BJJZ[CI)J (T4C8TTIClob.java:226)

  at oracle.jdbc.driver.T4CConnection.getChars(Loracle/sql/CLOB;JI[C)I (T4CConnection.java:3211)

  at oracle.sql.CLOB.getChars(JI[C)I (CLOB.java:459)

  at oracle.jdbc.driver.OracleClobReader.needChars(I)Z (OracleClobReader.java:187)

  at oracle.jdbc.driver.OracleClobReader.read([CII)I (OracleClobReader.java:142)

  at java.io.Reader.read([C)I (Reader.java:123)

  at com.sap.sql.jdbc.common.CommonWrappedReader.read([C)I (CommonWrappedReader.java:108)

  at com.technidata.core.persistency.sql.QueryResult.getString(Ljava/lang/String;)Ljava/lang/String; (QueryResult.java:190)

  at com.technidata.em.facility.controller.PC_EmissionRes.getEmissionResBo(Lde/technidata/ecs/persist/sql/persistency/iface/IQueryResult;)Lcom/technidata/em/facility/detail/EmissionResBo; (PC_EmissionRes.java:2235)

  at com.technidata.em.facility.controller.PC_EmissionRes.readData(JJJJLjava/sql/Date;Ljava/sql/Date;Ljava/lang/String;ZJI)Ljava/util/List; (PC_EmissionRes.java:682)

  at com.technidata.em.facility.controller.PC_EmissionRes.readData(JJJLjava/sql/Date;Ljava/sql/Date;ZI)Ljava/util/List; (PC_EmissionRes.java:599)

  at com.technidata.em.facility.controller.PC_EmissionRes.readData(Lcom/technidata/em/facility/TypeCapArgument;)Ljava/util/List; (PC_EmissionRes.java:573)

  at com.technidata.em.facility.ESourceBo.getTypeCapData(Ljava/lang/String;Lcom/technidata/em/facility/TypeCapArgument;Ljava/lang/String;Ljava/lang/String;)Ljava/util/List; (ESourceBo.java:707)

  at com.technidata.em.facility.mgmt.EmissionResTable.initialize(Z)V (EmissionResTable.java:1793)

  at com.technidata.em.facility.mgmt.EmissionResTable.init(Lcom/sap/tc/webdynpro/progmodel/api/IWDCustomEvent;Ljava/lang/String;)V (EmissionResTable.java:1454)

  at com.technidata.em.facility.mgmt.wdp.InternalEmissionResTable.wdInvokeEventHandler(Ljava/lang/String;Lcom/sap/tc/webdynpro/progmodel/api/IWDCustomEvent;)Ljava/lang/Object; (InternalEmissionResTable.java:723)

  at com.sap.tc.webdynpro.progmodel.generation.DelegatingView.invokeEventHandler(Ljava/lang/String;Lcom/sap/tc/webdynpro/progmodel/api/IWDCustomEvent;)Ljava/lang/Object; (DelegatingView.java:142)

  at com.sap.tc.webdynpro.progmodel.components.Component.fireEvent(Ljava/lang/String;Ljava/lang/String;Ljava/util/Map;)V (Component.java:492)

  at com.technidata.em.facility.mgmt.wdp.InternalTabsInterface.wdFireEventInitTab(Ljava/lang/String;)V (InternalTabsInterface.java:398)

  at com.technidata.em.facility.mgmt.EmissionResWinInterfaceView.onPlugIn(Lcom/sap/tc/webdynpro/progmodel/api/IWDCustomEvent;)V (EmissionResWinInterfaceView.java:97)

  at com.technidata.em.facility.mgmt.wdp.InternalEmissionResWinInterfaceView.wdInvokeEventHandler(Ljava/lang/String;Lcom/sap/tc/webdynpro/progmodel/api/IWDCustomEvent;)Ljava/lang/Object; (InternalEmissionResWinInterfaceView.java:95)

  at com.sap.tc.webdynpro.progmodel.generation.DelegatingWindow.invokeEventHandler(Ljava/lang/String;Lcom/sap/tc/webdynpro/progmodel/api/IWDCustomEvent;)Ljava/lang/Object; (DelegatingWindow.java:121)

please help me fix the performance issue. And let me know if I can provide some details to help you analyse further.

Thanks,

Kumar

divyanshu_srivastava3
Active Contributor
0 Kudos

Hi Kumar,

Please attach the thread dumps for all server nodes that you took and if possible load your heap dumps n share the link.

As far as from the above info, I can see that there is leaking loader and not much can be acquired from that.

Regards,

Divyanshu