Java process server0 restarts for single activity in EP

[Former Member]

HI Experts,

We installed the Netwever Application server with EP+PI component. The installation is successful. But when we are trying to do any activity in portal, the server0 of J2ee process table in MMC gets restarts. We are getting message saying that the message server is nt available, follow the note 943498 and 764417 to resolve the problem.

After following the note 943498 and 764417 also we are unable to resolve this problem. The output of dev_jcontrol is as given below -

The java version is

C:\Documents and Settings\tndadm>java -version

java version "1.4.2_17"

Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_17-b06)

Java HotSpot(TM) 64-Bit Server VM (build 1.4.2_17-b06, mixed mode)

Please reply soon.

[Former Member · Accepted solution]

Hi,

Error is FATAL: Caught OutOfMemoryError! Node will exit with exit code 666

Memory issue.

Is your j2ee heap memory size is 1G. and perm size is 256M(log says). Check it.

Make max heap memory 2G and Perm 512M

For 2 GB heap -XX:NewSize and -XX:MaxNewSize will be >=341m.

You can change from configtool. Perm size should be >=512M for 64 bit.

Suman

[Former Member · Accepted solution]

Hi Suman,

hanks for your reply. I got the same answer from SAP support also. Thanks for the answers.

But can you please tell me any note which contents the steps to do this changes or just steps to do it in my windows system.

As I am working newly to the java stack. I scared to do any changes.

Please guide me.

Thanks in Advance.

Regards,

Jayendra.

[Former Member · Accepted solution]

Hi,

GO to

C:\usr\sap\SID\JC00\j2ee\configtool\configtool.bat

and change the settings

config tool->clusterdata->dispatcher XY->server XY-> general is the right path to set the Xmx and Xss paramters.

Regards,

Ravi

[JPReyes · Accepted solution]

Did you read SAP Note 723909 - Java VM settings for J2EE 6.40/7.0 as suggested ?

You'll find all the info you need there.

Never be scared to do changes... This is the only way to learn.

Regards

Juan

[Former Member]

Hi All,

The problem solved. The paramter setting I found in Java are similar like ABAP

The >= value is not allowed for maxpersize and maxnewsize parameters. I made it = 512MB. Then its started working.

Thanks Jaun to encourage me to resolve this problem.

Ones again thanks to all.

Regards,

Jayendra.

[Former Member]

Thanks Jaun / Ravi,

I succseed in changing but the server0 is not starting. I think I need to keep the value of maxpersize diffrent. This is the log of developer trace of server0. I think I am near to my solution. Just suggest me about this parameter.

JStartupIReadSection: read node properties [ID3788750]

-> node name : server0

-> node type : server

-> node execute : yes

-> jlaunch parameters :

-> java path : C:\j2sdk1.4.2_17-x64

-> java parameters : -Djco.jarm=1 -XX:MaxPermSize>=512M -XX:PermSize=512M -XX:NewSize=512M -XX:MaxNewSize>=512M -XX:DisableExplicitGC -verbose:gc -XX:PrintGCDetails -XX:+PrintGCTimeStamps -Djava.awt.headless=true -Dsun.io.useCanonCaches=false -XX:SoftRefLRUPolicyMSPerMB=1 -XX:SurvivorRatio=2 -XX:TargetSurvivorRatio=90 -Djava.security.policy=./java.policy -Djava.security.egd=file:/dev/urandom -Dorg.omg.CORBA.ORBClass=com.sap.engine.system.ORBProxy -Dorg.omg.CORBA.ORBSingletonClass=com.sap.engine.system.ORBSingletonProxy -Djavax.rmi.CORBA.PortableRemoteObjectClass=com.sap.engine.system.PortableRemoteObjectProxy

-> java vm version : 1.4.2_17-b06

-> java vm vendor : Java HotSpot(TM) 64-Bit Server VM (Sun Microsystems Inc.)

-> java vm type : server

-> java vm cpu : amd64

-> heap size : 2048M

-> init heap size : 1024M

-> root path : C:\usr\sap\TND\DVEBMGS00\j2ee\cluster\server0

-> class path : .\bin\boot\boot.jar;.\bin\boot\jaas.jar;.\bin\system\bytecode.jar;.

-> OS libs path : C:\usr\sap\TND\DVEBMGS00\j2ee\os_libs

-> main class : com.sap.engine.boot.Start

-> framework class : com.sap.bc.proj.jstartup.JStartupFramework

-> registr. class : com.sap.bc.proj.jstartup.JStartupNatives

-> framework path : C:\usr\sap\TND\DVEBMGS00\exe\jstartup.jar;C:\usr\sap\TND\DVEBMGS00\exe\jvmx.jar

-> shutdown class : com.sap.engine.boot.Start

-> parameters :

-> debuggable : no

-> debug mode : no

-> debug port : 50021

-> shutdown timeout : 120000

**********************************************************************

[Thr 4176] JLaunchISetDebugMode: set debug mode [no]

[Thr 4644] JLaunchIStartFunc: Thread 4644 started as Java VM thread.

[Thr 4644] [JHVM_PrepareVMOptions] use java parameters set by profile parameter

Java Parameters: -Xss2m

**********************************************************************

JHVM_LoadJavaVM: VM Arguments of node [server0]

-> stack : 1048576 Bytes

-> arg[ 0]: exit

-> arg[ 1]: abort

-> arg[ 2]: vfprintf

-> arg[ 3]: -Djco.jarm=1

-> arg[ 4]: -XX:MaxPermSize>=512M

-> arg[ 5]: -XX:PermSize=512M

-> arg[ 6]: -XX:NewSize=512M

-> arg[ 7]: -XX:MaxNewSize>=512M

-> arg[ 8]: -XX:+DisableExplicitGC

-> arg[ 9]: -verbose:gc

-> arg[ 10]: -XX:+PrintGCDetails

-> arg[ 11]: -XX:+PrintGCTimeStamps

-> arg[ 12]: -Djava.awt.headless=true

-> arg[ 13]: -Dsun.io.useCanonCaches=false

-> arg[ 14]: -XX:SoftRefLRUPolicyMSPerMB=1

-> arg[ 15]: -XX:SurvivorRatio=2

-> arg[ 16]: -XX:TargetSurvivorRatio=90

-> arg[ 17]: -Djava.security.policy=./java.policy

-> arg[ 18]: -Djava.security.egd=file:/dev/urandom

-> arg[ 19]: -Dorg.omg.CORBA.ORBClass=com.sap.engine.system.ORBProxy

-> arg[ 20]: -Dorg.omg.CORBA.ORBSingletonClass=com.sap.engine.system.ORBSingletonProxy

-> arg[ 21]: -Djavax.rmi.CORBA.PortableRemoteObjectClass=com.sap.engine.system.PortableRemoteObjectProxy

-> arg[ 22]: -Dsys.global.dir=C:\usr\sap\TND\SYS\global

-> arg[ 23]: -Dapplication.home=C:\usr\sap\TND\DVEBMGS00\exe

-> arg[ 24]: -Djava.class.path=C:\usr\sap\TND\DVEBMGS00\exe\jstartup.jar;C:\usr\sap\TND\DVEBMGS00\exe\jvmx.jar;.\bin\boot\boot.jar;.\bin\boot\jaas.jar;.\bin\system\bytecode.jar;.

-> arg[ 25]: -Djava.library.path=C:\j2sdk1.4.2_17-x64\jre\bin\server;C:\j2sdk1.4.2_17-x64\jre\bin;C:\j2sdk1.4.2_17-x64\bin;C:\usr\sap\TND\DVEBMGS00\j2ee\os_libs;C:\Program Files\HP\NCU;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\system32\WindowsPowerShell\v1.0;C:\j2sdk1.4.2_17-x64\bin;C:\Program Files\Microsoft SQL Server\90\Tools\binn\;C:\Program Files (x86)\Microsoft SQL Server\90\Tools\binn\;C:\Program Files (x86)\Microsoft SQL Server\90\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\90\Tools\Binn\VSShell\Common7\IDE\;C:\Program Files\Microsoft SQL Server\90\DTS\Binn\;C:\usr\sap\TND\SYS\exe\uc\NTAMD64

-> arg[ 26]: -Dmemory.manager=2048M

-> arg[ 27]: -Xmx2048M

-> arg[ 28]: -Xms1024M

-> arg[ 29]: -DLoadBalanceRestricted=no

-> arg[ 30]: -Djstartup.mode=JCONTROL

-> arg[ 31]: -Djstartup.ownProcessId=4056

-> arg[ 32]: -Djstartup.ownHardwareId=Q0970199207

-> arg[ 33]: -Djstartup.whoami=server

-> arg[ 34]: -Djstartup.debuggable=no

-> arg[ 35]: -Xss2m

-> arg[ 36]: -DSAPINFO=TND_00_server

-> arg[ 37]: -DSAPSTART=1

-> arg[ 38]: -DCONNECT_PORT=2891

-> arg[ 39]: -DSAPSYSTEM=00

-> arg[ 40]: -DSAPSYSTEMNAME=TND

-> arg[ 41]: -DSAPMYNAME=TNWDEV_TND_00

-> arg[ 42]: -DSAPPROFILE=C:\usr\sap\TND\SYS\profile\TND_DVEBMGS00_TNWDEV

-> arg[ 43]: -DFRFC_FALLBACK=ON

-> arg[ 44]: -DFRFC_FALLBACK_HOST=localhost

-> arg[ 45]: -DSAPSTARTUP=1

-> arg[ 46]: -DSAPSYSTEM=00

-> arg[ 47]: -DSAPSYSTEMNAME=TND

-> arg[ 48]: -DSAPMYNAME=TNWDEV_TND_00

-> arg[ 49]: -DSAPDBHOST=TNWDEV

-> arg[ 50]: -Dj2ee.dbhost=TNWDEV

**********************************************************************

Unrecognized VM option 'MaxPermSize>=512M'

[Thr 4644] *** ERROR => JHVM_LoadJavaVM: Cannot create Java VM (rc=-6) [jhvmxx.c 549]

[Thr 4644] *** ERROR => Cannot load Java VM (server) (rc=-1) [jlnchxxi.c 752]

[Thr 4644] **********************************************************************

• ERROR => Java VM initialization failed.

• Please see SAP Note 943602 , section 'Java VM initialization issues'

• for additional information and trouble shooting.

**********************************************************************

[Thr 4644] JLaunchCloseProgram: good bye (exitcode = -1)

[Former Member]

Can you post server0 trace.

Suman

[JPReyes]

The output of dev_jcontrol is as given below 

Can't see it...

Also check std_server0.out and std_dispatcher.out

Regards

Juan

[Former Member]

>

> 

The output of dev_jcontrol is as given below

> Can't see it...
> 
> Also check std_server0.out and std_dispatcher.out
> 
> Regards
> Juan

>

> 

The output of dev_jcontrol is as given below

> Can't see it...
> 
> Also check std_server0.out and std_dispatcher.out
> 
> Regards
> Juan

Hi All,

The some output of dev_server0 is as follow

java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)

- locked <0x000000001ab10220> (a java.lang.ref.Reference$Lock)

"main" prio=5 tid=0x000000000168a9f0 nid=0x16b8 in Object.wait() [0x000000000235f000..0x000000000235fc68]

at java.lang.Object.wait(Native Method)

- waiting on <0x000000001ab18270> (a com.sap.engine.boot.StartFrame)

at java.lang.Object.wait(Object.java:429)

at com.sap.engine.boot.StartFrame.work(StartFrame.java:121)

- locked <0x000000001ab18270> (a com.sap.engine.boot.StartFrame)

at com.sap.engine.boot.Start.main(Start.java:34)

"VM Thread" prio=5 tid=0x0000000001755d10 nid=0x1710 runnable

"VM Periodic Task Thread" prio=10 tid=0x0000000001755f10 nid=0x1698 waiting on condition

"Suspend Checker Thread" prio=10 tid=0x0000000001755e10 nid=0x12d4 runnable

[Thr 5904] Wed Jul 15 16:21:23 2009

[Thr 5904] JLaunchIExitJava: exit hook is called (rc = 666)

[Thr 5904] **********************************************************************

• ERROR => The Java VM terminated with a non-zero exit code.

• Please see SAP Note 943602 , section 'J2EE Engine exit codes'

• for additional information and trouble shooting.

**********************************************************************

[Thr 5904] JLaunchCloseProgram: good bye (exitcode = 666)

-

---

trc file: "C:\usr\sap\TND\DVEBMGS00\work\dev_server0", trc level: 1, release: "700"

-

---

node name : ID3788750

pid : 7916

system name : TND

system nr. : 00

started at : Wed Jul 15 16:21:25 2009

arguments :

arg[00] : C:\usr\sap\TND\DVEBMGS00\exe\jlaunch.exe

arg[01] : pf=C:\usr\sap\TND\SYS\profile\TND_DVEBMGS00_TNWDEV

arg[02] : -DSAPINFO=TND_00_server

arg[03] : pf=C:\usr\sap\TND\SYS\profile\TND_DVEBMGS00_TNWDEV

arg[04] : -DSAPSTART=1

arg[05] : -DCONNECT_PORT=1485

arg[06] : -DSAPSYSTEM=00

arg[07] : -DSAPSYSTEMNAME=TND

arg[08] : -DSAPMYNAME=TNWDEV_TND_00

arg[09] : -DSAPPROFILE=C:\usr\sap\TND\SYS\profile\TND_DVEBMGS00_TNWDEV

arg[10] : -DFRFC_FALLBACK=ON

arg[11] : -DFRFC_FALLBACK_HOST=localhost

[Thr 7708] Wed Jul 15 16:21:25 2009

[Thr 7708] *** WARNING => INFO: Unknown property [instance.box.number=TNDDVEBMGS00tnwdev] [jstartxx.c 841]

[Thr 7708] *** WARNING => INFO: Unknown property [instance.en.host=TNWDEV] [jstartxx.c 841]

[Thr 7708] *** WARNING => INFO: Unknown property [instance.en.port=3201] [jstartxx.c 841]

[Thr 7708] *** WARNING => INFO: Unknown property [instance.system.id=0] [jstartxx.c 841]

**********************************************************************

JStartupReadInstanceProperties: read instance properties [C:\usr\sap\TND\DVEBMGS00\j2ee\cluster\instance.properties]

-> ms host : TNWDEV

-> ms port : 3901

-> OS libs : C:\usr\sap\TND\DVEBMGS00\j2ee\os_libs

-> Admin URL :

-> run mode : NORMAL

-> run action : NONE

-> enabled : yes

**********************************************************************

**********************************************************************

Used property files

-> files [00] : C:\usr\sap\TND\DVEBMGS00\j2ee\cluster\instance.properties

**********************************************************************

**********************************************************************

Instance properties

-> ms host : TNWDEV

-> ms port : 3901

-> os libs : C:\usr\sap\TND\DVEBMGS00\j2ee\os_libs

-> admin URL :

-> run mode : NORMAL

-> run action : NONE

-> enabled : yes

The Output of std_server0.out is

sun.reflect.GeneratedSerializationConstructorAccessor651]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor650]

: 382150K->382158K(873472K), 1.3991216 secs] 383686K->382158K(1004800K), [Perm : 262143K->262143K(262144K)], 1.3992528 secs]

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666

================================================================================

getThreadDump : Thu Jul 16 12:08:22 2009

FATAL: Caught OutOfMemoryError! Node will exit with exit code 666

================================================================================

Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.4.2_17-b06 mixed mode):

"Thread-143" daemon prio=5 tid=0x0000000006c90520 nid=0x1390 in Object.wait() [0x000000006a60f000..0x000000006a60fb80]

at java.lang.Object.wait(Native Method)

- waiting on <0x00000000306f0b80> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)

- locked <0x00000000306f0b80> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)

at com.sap.engine.cache.util.ReferenceQueueRunnable.run(ReferenceQueueRunnable.java:43)

at java.lang.Thread.run(Thread.java:534)

Some output of std_dispactcher.out is

stdout/stderr redirect

-

---

node name : dispatcher

pid : 4916

system name : TND

system nr. : 00

started at : Wed Jul 15 15:45:25 2009

[Thr 6088] MtxInit: -2 0 0

CompilerOracle: exclude com/sapportals/portal/pb/layout/taglib/ContainerTag addIviewResources

CompilerOracle: exclude com/sap/engine/services/keystore/impl/security/CodeBasedSecurityConnector getApplicationDomain

CompilerOracle: exclude com/sap/engine/services/rmi_p4/P4StubSkeletonGenerator generateStub

CompilerOracle: exclude com/sapportals/portal/prt/util/StringUtils escapeToJS

CompilerOracle: exclude com/sapportals/portal/prt/core/broker/PortalServiceItem startServices

CompilerOracle: exclude com/sap/engine/services/webservices/server/deploy/WSConfigurationHandler downloadFile

CompilerOracle: exclude com/sapportals/portal/prt/jndisupport/util/AbstractHierarchicalContext lookup

CompilerOracle: exclude com/sapportals/portal/navigation/cache/CacheNavigationNode getAttributeValue

CompilerOracle: exclude com/sapportals/portal/navigation/TopLevelNavigationiView PrintNode

CompilerOracle: exclude com/sapportals/wcm/service/ice/wcm/ICEPropertiesCoder encode

CompilerOracle: exclude com/sap/lcr/pers/delta/importing/ObjectLoader loadObjects

CompilerOracle: exclude com/sap/engine/services/webservices/jaxrpc/encoding/InstanceBuilder readElement

CompilerOracle: exclude com/sap/engine/services/webservices/jaxrpc/encoding/InstanceBuilder readSequence

CompilerOracle: exclude com/sap/engine/services/webservices/jaxrpc/encoding/TypeMappingImpl initializeRelations

CompilerOracle: exclude com/sap/engine/services/webservices/jaxrpc/encoding/GeneratedComplexType _loadInto

SAP J2EE Engine Version 7.00 PatchLevel 109886.44 is starting...

Loading: LogManager ... 687 ms.

Loading: PoolManager ... 0 ms.

Loading: ThreadManager ... 62 ms.

Loading: IpVerificationManager ... 0 ms.

Loading: ConnectionsManipulator ... 16 ms.

Loading: ClassLoaderManager ... 16 ms.

Loading: ClusterManager ... 140 ms.

Loading: PortsManager ... 0 ms.

Loading: LockingManager ... 125 ms.

Loading: ConfigurationManager ... [GC 46720K->3355K(168256K), 0.0173971 secs]

1656 ms.

Loading: LicensingManager ... 16 ms.

Loading: CacheManager ... 156 ms.

Loading: ServiceManager ...

Loading services.:

Service security started. (0 ms).

Service iiop started. (109 ms).

Service sld started. (31 ms).

Service timeout started. (15 ms).

Service memory started. (0 ms).

Service r3startup started. (31 ms).

Service licensing started. (16 ms).

Service httptunneling started. (47 ms).

Service webservices started. (156 ms).

Service shell started. (109 ms).

Service keystore started. (16 ms).

Service p4 started. (390 ms).

Service classload started. (16 ms).

Service jmx_notification started. (15 ms).

Service telnet started. (157 ms).

[GC 50075K->5669K(168256K), 0.0233330 secs]

Service ssl started. (63 ms).

Service jmx started. (188 ms).

Service log_configurator started. (1468 ms).

Service configuration started. (0 ms).

[GC 52389K->7439K(168256K), 0.0188114 secs]

Service basicadmin started. (188 ms).

[Full GC 13410K->7946K(168256K), 0.0831450 secs]

Service adminadapter started. (0 ms).

Service tc.monitoring.logviewer started. (515 ms).

Service monitor started. (1000 ms).

Service http started. (2953 ms).

Service jms_provider started. (2859 ms).

ServiceManager started for 6171 ms.

Framework started for 9296 ms.

SAP J2EE Engine Version 7.00 PatchLevel 109886.44 is running!

PatchLevel 109886.44 November 17, 2007 10:27 GMT

>

Login :[GC 54661K->9834K(168256K), 0.0125017 secs]

[Full GC 25727K->11080K(168256K), 0.1504584 secs]

[GC 57768K->13834K(168256K), 0.0096830 secs]

[GC 60554K->13770K(168256K), 0.0074174 secs]

[GC 60490K->12333K(168256K), 0.0034877 secs]

[GC 59053K->12353K(168256K), 0.0032919 secs]

[GC 59073K->12362K(168256K), 0.0032960 secs]

[GC 59082K->13914K(168256K), 0.0147358 secs]

Service msp started. (0 ms).

[Former Member]

Hi All,

The most of the time it is showing as "caught out of memory error"

The system has 8GB RAM and page memory set to 32GB. I did watch the page memory usage in system but it is constant around 16GB at all the time.

I think I want to do some parameter settings in JVM. But as I am new to this NW Admin I am unable to find out the solution.

Please provide me an optimal solution.

Thanks in advance.

Jayendra

[JPReyes]

You need to increase the memory for your engine.

Read,

Note 723909 - Java VM settings for J2EE 6.40/7.0

Regards

Juan

[Former Member]

Refer to following troubleshooting guides and notes in there. Check Xmx and Xms settings as per the notes in following link.

[https://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/e09e4a5e-8280-2a10-ca98-a59dd47d396d?overridelayout=true]

[Former Member]

Hi,

Please also make heap memory and the XMS parameter to 2048 in Configtool under Server0.

When you just click on server you shall see these parameter in the right hand side.

Thanks

Rishi Abrol