Hello,
In our XI environment, we have two server nodes, quite offen i see lots of javacore?????????.txt is getting generated which causing /usr/sap/XIX is filled, due to this java goes down.
As a temp fix, i am removing those files and restarting java and server backs to normal.
I cheked std_serverx logs and default trace file. not able to get any concrete information about the cause of the crash. I could see this in std_serverx 'Caught OutOfMemoryError! '
we are using IBM JVM.
Below are the logs
tpd0101:xsxadm 195> grep Error std_server0.out
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at java.net.PlainSocketImpl.socketAccept(Native Method)
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:356)JVMDUMP006I Processing Dump Event "uncaught", detail "java/lang/OutOfMemoryError" - Please Wait.
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at java.net.PlainSocketImpl.socketAccept(Native Method)
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at java.security.AccessController.doPrivileged(AccessController.java:207)
java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at com.sap.engine.core.thread.impl3.SingleThread.run(SingleThread.java:170)
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at java.net.PlainSocketImpl.socketAccept(Native Method)
java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at java.net.PlainSocketImpl.socketAccept(Native Method)
java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666 at java.net.PlainSocketImpl.socketAccept(Native Method)
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
FATAL: Caught OutOfMemoryError! Node will exit with exit code 666java.lang.OutOfMemoryError
tpd0101:xixadm 196> ls
tpd0101:xixadm 109> grep dump javacore.20080102.085845.4885.txt
0SECTION TITLE subcomponent dump routine
0SECTION GPINFO subcomponent dump routine
1XHERROR2 Register dump section only produced for SIGSEGV, SIGILL or SIGFPE.
0SECTION ENVINFO subcomponent dump routine
2CIUSERARG -Xdump:heap:none
2CIUSERARG -Xdump
0SECTION MEMINFO subcomponent dump routine
0SECTION LOCKS subcomponent dump routine
0SECTION THREADS subcomponent dump routine
2XMFULLTHDDUMP Full thread dump J9SE VM (J2RE 1.4.2 IBM J9 2.2 Linux amd64-64 build 20061121_1035_LHdSMr, native threads):
0SECTION CLASSES subcomponent dump routine
3CLTEXTCLASS com/seeburger/xi/module/dump/PayloadDump(0x0000002B32F0A0D8)
0SECTION Javadump End section
tpd0101:xixadm 110> vi
0SECTION MEMINFO subcomponent dump routine
NULL =================================
1STHEAPFREE Bytes of Heap Space Free: 0SECTION MEMINFO subcomponent dump routine
NULL =================================
1STHEAPFREE Bytes of Heap Space Free: 1f57158
1STHEAPALLOC Bytes of Heap Space Allocated: 80000000
NULL
1STHEAPALLOC Bytes of Heap Space Allocated: 80000000
NULL
Can you please advice me on this.
Thanks,
Venkatesh