Frequent heap dumps and system load average

[balajik]

Dear Experts,

Performance of our development has gone worst. There has been frequent Heap Dumps and Paging, also System Load Average '/Kernel/SAPJVM/SystemProblemReporting/System Load Average' is over 133%.

We are using NW731 with EP core.

JVM parameters arememory are according to SAP recommendations

-Xms4096m

-Xmx4096m

-XX:NewSize=1365m

-XX:MaxNewSize=1365m

-XX:PermSize=1024m

-XX:MaxPermSize=1024m

-XX:NewRatio=6

SCS, Application and Database are on single host.

Number of nodes=2.

Physical memory size is 15GB

Appreciate your assistance in analyzing and fixing the performance issue.

Thanks,

Kumar

[balajik]

Hi Divyanshu,

Either zip or direct thread dump format b00 is not supported for attaching. OOM and java_PID**** files are more than 4 GB in size.....

Is there any other way to send the files....

[divyanshu_srivastava3]

Hi Kumar,

You can share your files using external spaces like Dropbox etc.

Also, as server0 log files(dev_server0) contains thread dumps, you can zip and attach this file.

Please upload these files to SAP as well, they will provide you a container for uploading heap dumps.

Regards,
Divyanshu

[balajik]

Hi Divyanshu,

I've sent you a mail with thread attached.....please check and assist.

Thanks,

Kumar.

[divyanshu_srivastava3]

Hi Kumar,

I scanned your dumped and there is not much to say until I have the heap dump and some other details.

Regards,

Divyanshu

[balajik]

Currently SAP has provided latest patch to fine tune the application (add-on on NW) with few indexes. Performance has been increased to a good extent. I'll post once if any further performance issues encounter.

Thank you all

Kumar

[Former Member]

Hi Balaji,

Please try to generate heap hprof dump, by adding below parameters in configtool

-XX:+HeapDumpOnOutOfMemoryError

-XX:+HeapDumpOnCtrlBreak

And after generating hprof file you can analyze it using memory analyzer.

Regards,

Jitendra

[Former Member]

Hi Balaji,

Please use below doc to know more about using memory analyzer.

http://wiki.scn.sap.com/wiki/display/TechTSG/(JSTTSG)(P4)+Produce+Heap+Dump+in+Java+Serverhttp://

Regards,

Jitendra

[divyanshu_srivastava3]

Hi Kumar,

I would suggest you to take 7-8 thread dumps in interval 10 15-20 sec when the utilization is high. Then find and map the corresponding Kernel ID with Thread ID of the processes causing high utilization. Also, use MAT for HEAP dumps to find problem in heaps, check for leaking loaders and memory leaks.

Regards,

Divyanshu

[balajik]

Hi Divyanshu,

Thanks for your quick reply. I just downloaded memory analyzer tool, will analyze recent OOM.hprof file. Btw....how to takte thread dumps, please assist. I am using NWA don't have access to configtool.

Thanks,

Kumar

[divyanshu_srivastava3]

Hi,

Simply use sap mmc or sap mc(host:5XX13) and right click on server node and click dump stack. Then u can check corresponding dev_serverX.out file in work directory in which thread dumps are generated.

Other simple way is use jcmon to take thread dumps.

Regards,

Divyanshu

[davefitzgibbon]

Hello Kumar,

Have a look at the following link which shows you how to obtain thread dumps.

http://wiki.sdn.sap.com/wiki/x/VYZ3Cw

Regards,

David

[divyanshu_srivastava3]

Since you don't have access to config tool, use JVVMON to generate HEAP dumps.

Regards,

Divyanshu

[divyanshu_srivastava3]

Hi Balaji,

How is it working ? Any updates ..

Regards,

DV

[balajik]

Hi Divyanshu,

I was away on to another project for configuring change control tool.

As mentioned earlier I don't have OS access.

But could able to download Thread Dumps via NWA as mentioned by David...., but couldn't able to open them.

Opened OOM.hprof file through MAT to check leak suspects etc. I could hardly understand as it is my first Java AS project ....here is bits and pieces from the dump analyzer tool.

Leap Suspects:

Problem Suspect 1:

The thread com.sap.engine.core.thread.execution.CentralExecutor$SingleThread @ 0x58dcd248 HTTP Worker [@662359980] keeps local variables with total size 3,66,84,54,240 (89.70%) bytes.

The memory is accumulated in one instance of "java.lang.Object[]" loaded by "<system class loader>".

The stacktrace of this Thread is available. See stacktrace.

And here is the stacktrace...

HTTP Worker [@662359980]

  at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:25)

  at java.lang.reflect.Array.newInstance(Ljava/lang/Class;I)Ljava/lang/Object; (Array.java:52)

  at oracle.jdbc.driver.BufferCache.get(Ljava/lang/Class;I)Ljava/lang/Object; (BufferCache.java:226)

  at oracle.jdbc.driver.PhysicalConnection.getByteBuffer(I)[B (PhysicalConnection.java:7665)

  at oracle.jdbc.driver.T4C8TTIClob.read([BJJZ[CI)J (T4C8TTIClob.java:226)

  at oracle.jdbc.driver.T4CConnection.getChars(Loracle/sql/CLOB;JI[C)I (T4CConnection.java:3211)

  at oracle.sql.CLOB.getChars(JI[C)I (CLOB.java:459)

  at oracle.jdbc.driver.OracleClobReader.needChars(I)Z (OracleClobReader.java:187)

  at oracle.jdbc.driver.OracleClobReader.read([CII)I (OracleClobReader.java:142)

  at java.io.Reader.read([C)I (Reader.java:123)

  at com.sap.sql.jdbc.common.CommonWrappedReader.read([C)I (CommonWrappedReader.java:108)

  at com.technidata.core.persistency.sql.QueryResult.getString(Ljava/lang/String;)Ljava/lang/String; (QueryResult.java:190)

  at com.technidata.em.facility.controller.PC_EmissionRes.getEmissionResBo(Lde/technidata/ecs/persist/sql/persistency/iface/IQueryResult;)Lcom/technidata/em/facility/detail/EmissionResBo; (PC_EmissionRes.java:2235)

  at com.technidata.em.facility.controller.PC_EmissionRes.readData(JJJJLjava/sql/Date;Ljava/sql/Date;Ljava/lang/String;ZJI)Ljava/util/List; (PC_EmissionRes.java:682)

  at com.technidata.em.facility.controller.PC_EmissionRes.readData(JJJLjava/sql/Date;Ljava/sql/Date;ZI)Ljava/util/List; (PC_EmissionRes.java:599)

  at com.technidata.em.facility.controller.PC_EmissionRes.readData(Lcom/technidata/em/facility/TypeCapArgument;)Ljava/util/List; (PC_EmissionRes.java:573)

  at com.technidata.em.facility.ESourceBo.getTypeCapData(Ljava/lang/String;Lcom/technidata/em/facility/TypeCapArgument;Ljava/lang/String;Ljava/lang/String;)Ljava/util/List; (ESourceBo.java:707)

  at com.technidata.em.facility.mgmt.EmissionResTable.initialize(Z)V (EmissionResTable.java:1793)

  at com.technidata.em.facility.mgmt.EmissionResTable.init(Lcom/sap/tc/webdynpro/progmodel/api/IWDCustomEvent;Ljava/lang/String;)V (EmissionResTable.java:1454)

  at com.technidata.em.facility.mgmt.wdp.InternalEmissionResTable.wdInvokeEventHandler(Ljava/lang/String;Lcom/sap/tc/webdynpro/progmodel/api/IWDCustomEvent;)Ljava/lang/Object; (InternalEmissionResTable.java:723)

  at com.sap.tc.webdynpro.progmodel.generation.DelegatingView.invokeEventHandler(Ljava/lang/String;Lcom/sap/tc/webdynpro/progmodel/api/IWDCustomEvent;)Ljava/lang/Object; (DelegatingView.java:142)

  at com.sap.tc.webdynpro.progmodel.components.Component.fireEvent(Ljava/lang/String;Ljava/lang/String;Ljava/util/Map;)V (Component.java:492)

  at com.technidata.em.facility.mgmt.wdp.InternalTabsInterface.wdFireEventInitTab(Ljava/lang/String;)V (InternalTabsInterface.java:398)

  at com.technidata.em.facility.mgmt.EmissionResWinInterfaceView.onPlugIn(Lcom/sap/tc/webdynpro/progmodel/api/IWDCustomEvent;)V (EmissionResWinInterfaceView.java:97)

  at com.technidata.em.facility.mgmt.wdp.InternalEmissionResWinInterfaceView.wdInvokeEventHandler(Ljava/lang/String;Lcom/sap/tc/webdynpro/progmodel/api/IWDCustomEvent;)Ljava/lang/Object; (InternalEmissionResWinInterfaceView.java:95)

  at com.sap.tc.webdynpro.progmodel.generation.DelegatingWindow.invokeEventHandler(Ljava/lang/String;Lcom/sap/tc/webdynpro/progmodel/api/IWDCustomEvent;)Ljava/lang/Object; (DelegatingWindow.java:121)

please help me fix the performance issue. And let me know if I can provide some details to help you analyse further.

Thanks,

Kumar

[divyanshu_srivastava3]

Hi Kumar,

Please attach the thread dumps for all server nodes that you took and if possible load your heap dumps n share the link.

As far as from the above info, I can see that there is leaking loader and not much can be acquired from that.

Regards,

Divyanshu