Skip to Content
0
Jan 08, 2009 at 12:55 PM

J2EE engine hangs on status Starting Apps

464 Views

Hi All

We are currently experiencing issues with our J2EE engine. When we start it, it goes up to the status Starting Apps and never leaves that status. There are no errors on any of the logs/traces under work directory or the server0/log directory.

We are running EP 700 with Kernel Patch 146 on windows 2003 server. If anyone has seen this type of issue, please advice.

I have attached a java dump from std-server0.out file. I hope that helps

Reserved 1610612736 (0x60000000) bytes before loading DLLs.

[Thr 4320] MtxInit: -2 0 0

CompilerOracle: exclude com/sapportals/portal/pb/layout/taglib/ContainerTag addIviewResources

CompilerOracle: exclude com/sap/engine/services/keystore/impl/security/CodeBasedSecurityConnector getApplicationDomain

CompilerOracle: exclude com/sap/engine/services/rmi_p4/P4StubSkeletonGenerator generateStub

CompilerOracle: exclude com/sapportals/portal/prt/util/StringUtils escapeToJS

CompilerOracle: exclude com/sapportals/portal/prt/core/broker/PortalServiceItem startServices

CompilerOracle: exclude com/sap/engine/services/webservices/server/deploy/WSConfigurationHandler downloadFile

CompilerOracle: exclude com/sapportals/portal/prt/jndisupport/util/AbstractHierarchicalContext lookup

CompilerOracle: exclude com/sapportals/portal/navigation/cache/CacheNavigationNode getAttributeValue

CompilerOracle: exclude com/sapportals/portal/navigation/TopLevelNavigationiView PrintNode

CompilerOracle: exclude com/sapportals/wcm/service/ice/wcm/ICEPropertiesCoder encode

CompilerOracle: exclude com/sap/lcr/pers/delta/importing/ObjectLoader loadObjects

CompilerOracle: exclude com/sap/engine/services/webservices/jaxrpc/encoding/InstanceBuilder readElement

CompilerOracle: exclude com/sap/engine/services/webservices/jaxrpc/encoding/InstanceBuilder readSequence

CompilerOracle: exclude com/sap/engine/services/webservices/jaxrpc/encoding/TypeMappingImpl initializeRelations

CompilerOracle: exclude com/sap/engine/services/webservices/jaxrpc/encoding/GeneratedComplexType _loadInto

SAP J2EE Engine Version 7.00 PatchLevel 110760.44 is starting...

Loading: LogManager ... 953 ms.

Loading: PoolManager ... 0 ms.

Loading: ApplicationThreadManager ... 47 ms.

Loading: ThreadManager ... 0 ms.

Loading: IpVerificationManager ... 16 ms.

Loading: ClassLoaderManager ... 0 ms.

Loading: ClusterManager ... 453 ms.

Loading: LockingManager ... 31 ms.

Loading: ConfigurationManager ... 1203 ms.

Loading: LicensingManager ... 16 ms.

Loading: CacheManager ... 109 ms.

Loading: ServiceManager ...

Loading services.:

Service memory started. (15 ms).

Service DQE started. (0 ms).

Service timeout started. (0 ms).

Service runtimeinfo started. (0 ms).

Service cross started. (704 ms).

Service file started. (938 ms).

Service cafeuodi~mnuacc started. (0 ms).

Service trex.service started. (16 ms).

Service p4 started. (954 ms).

Service classpath_resolver started. (125 ms).

Service cafeucc~api started. (4844 ms).

Service userstore started. (15 ms).

Service jmx_notification started. (15 ms).

Service log_configurator started. (10594 ms).

Service locking started. (0 ms).

Service naming started. (172 ms).

Service ts started. (32 ms).

Service failover started. (0 ms).

Service licensing started. (16 ms).

Service appclient started. (16 ms).

Service javamail started. (578 ms).

Service iiop started. (578 ms).

Service jmsconnector started. (46 ms).

Service http started. (156 ms).

Service connector started. (78 ms).

Service deploy started. (7719 ms).

Service configuration started. (31 ms).

Service com.sap.portal.runtime.config.kmreadonly started. (0 ms).

Service MigrationService started. (63 ms).

Service bimmrdeployer started. (16 ms).

Service webservices started. (1953 ms).

Service dbpool started. (1563 ms).

Service UT started. (0 ms).

Service cafeugpmailcf started. (15 ms).

Service com.sap.security.core.ume.service started. (1531 ms).

Service security started. (922 ms).

Service jmx started. (125 ms).

Service servlet_jsp started. (297 ms).

Service basicadmin started. (187 ms).

Service adminadapter started. (47 ms).

Service classload started. (0 ms).

Service applocking started. (15 ms).

Service cafruntimeconnectivity~impl started. (9344 ms).

Service tclmctcconfsservice_sda started. (31 ms).

Service CUL started. (16 ms).

Service com.sap.portal.pcd.gl started. (0 ms).

Service com.sap.nw.kmc.readonly started. (0 ms).

Service jms_provider started. (1422 ms).

Service cafummetadata~imp started. (110 ms).

Service com.sap.nw.kmc.readonlymbean started. (0 ms).

Service com.adobe~DataManagerService started. (110 ms).

Service com.adobe~LicenseService started. (812 ms).

Service shell started. (5969 ms).

Service keystore started. (6141 ms).

Service tcsecsecurestorage~service started. (31 ms).

Service dsr started. (5859 ms).

Service cafumrelgroups~imp started. (93 ms).

Service tceCATTPingservice started. (0 ms).

Service tcseccertrevoc~service started. (47 ms).

Service ssl started. (0 ms).

Service apptracing started. (6907 ms).

Service monitor started. (4266 ms).

Service ejb started. (8766 ms).

Service prtbridge started. (2062 ms).

Service telnet started. (3031 ms).

Service tc.CBS.Service started. (953 ms).

Service tcdisdic~srv started. (14516 ms).

Service com.adobe~DocumentServicesBinaries2 started. (11001 ms).

Service webdynpro started. (7313 ms).

Service com.adobe~DocumentServicesLicenseSupportService started. (11000 ms).

Service com.adobe~XMLFormService started. (11016 ms).

Service com.sap.portal.prt.sapj2ee started. (4782 ms).

Service developmentserver started. (4610 ms).

Service cafeugp~model started. (16 ms).

Service tceujwfuiwizsvc started. (0 ms).

Service com.adobe~DocumentServicesConfiguration started. (5891 ms).

Service sld started. (6094 ms).

Service tcsecwssec~service started. (5703 ms).

Service tcsecdestinations~service started. (7110 ms).

Service rfcengine started. (63 ms).

Service tcsecvsi~service started. (62 ms).

Service pmi started. (15 ms).

Service com.adobe~PDFManipulation started. (13032 ms).

Service com.adobe~TrustManagerService started. (47 ms).

Service com.adobe~DocumentServicesDestProtoService started. (31 ms).

Service cafeuer~service started. (2313 ms).

Service com.adobe~FontManagerService started. (14719 ms).

Service tc.monitoring.logviewer started. (17251 ms).

ServiceManager started for 39674 ms.

Framework started for 42939 ms.

SAP J2EE Engine Version 7.00 PatchLevel 110760.44 is running!

PatchLevel 110760.44 February 15, 2008 20:58 GMT

>

Login :

================================================================================

dump stack trace : Thu Jan 08 14:29:30 2009

================================================================================

Full thread dump Java HotSpot(TM) Server VM (1.4.2_13-b06 mixed mode):

"Thread-71" daemon prio=5 tid=0x0502e580 nid=0x138c runnable [0x08f0f000..0x08f0fd94]

at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.read(SocketInputStream.java:129)

at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)

at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)

at java.io.BufferedInputStream.read(BufferedInputStream.java:277)

- locked <0x15590888> (a java.io.BufferedInputStream)

at com.sun.jndi.ldap.Connection.run(Connection.java:784)

at java.lang.Thread.run(Thread.java:534)

"RTCBaseService" prio=5 tid=0x0849cdc8 nid=0x1df4 runnable [0x0fe5f000..0x0fe5fd94]

at java.lang.Object.wait(Native Method)

- waiting on <0x15593348> (a com.sap.jms.client.session.WaitBlockableQueue)

at com.sap.jms.client.session.WaitBlockableQueue.dequeue(WaitBlockableQueue.java:215)

- locked <0x15593348> (a com.sap.jms.client.session.WaitBlockableQueue)

at com.sap.jms.client.session.Session.provideMessage(Session.java:1656)

at com.sap.jms.client.session.MessageConsumer.receive(MessageConsumer.java:177)

at com.sap.ip.collaboration.core.api.rtmf.core.RTMFMessaging.peekServiceMessage(RTMFMessaging.java:448)

- locked <0x15593420> (a com.sap.ip.collaboration.core.api.rtmf.core.RTMFMessaging)

at com.sap.ip.collaboration.core.api.semantic.services.RTCBaseService$RTCServiceProcessor.run(RTCBaseService.java:368)

at java.lang.Thread.run(Thread.java:534)

"ClientAwarnessChecker" prio=5 tid=0x08b9e5e0 nid=0x1ca0 waiting on condition [0x0fe1f000..0x0fe1fd94]

at java.lang.Thread.sleep(Native Method)

at com.sap.ip.collaboration.core.api.semantic.services.RTCClientAwareness$ClientAwarnessChecker.run(RTCClientAwareness.java:896)

at java.lang.Thread.run(Thread.java:534)

"Thread-65" prio=5 tid=0x08475008 nid=0x18c4 waiting on condition [0x0fddf000..0x0fddfd94]

at java.lang.Thread.sleep(Native Method)

at com.sap.ip.collaboration.core.api.rtmf.serverbridge.hitsPerSecond.HPS.run(HPS.java:64)

at java.lang.Thread.run(Thread.java:534)

"Thread-64" prio=5 tid=0x090ecdb0 nid=0x15ec waiting on condition [0x0fd9f000..0x0fd9fd94]

at java.lang.Thread.sleep(Native Method)

at com.sap.ip.collaboration.core.api.rtmf.core.RTMFMessaging$JMSPolling.run(RTMFMessaging.java:1211)

at java.lang.Thread.run(Thread.java:534)

"UWL Pooled Thread:4" prio=5 tid=0x04c97400 nid=0x18c in Object.wait() [0x0fd5f000..0x0fd5fd94]

at java.lang.Object.wait(Native Method)

- waiting on <0x15593620> (a java.util.LinkedList)

at java.lang.Object.wait(Object.java:429)

at com.sap.netweaver.bc.uwl.core.ThreadManager$PoolWorker.run(ThreadManager.java:148)

- locked <0x15593620> (a java.util.LinkedList)

"UWL Pooled Thread:3" prio=5 tid=0x04c97298 nid=0x110c in Object.wait() [0x0fd1f000..0x0fd1fd94]

at java.lang.Object.wait(Native Method)

- waiting on <0x15593620> (a java.util.LinkedList)

at java.lang.Object.wait(Object.java:429)

at com.sap.netweaver.bc.uwl.core.ThreadManager$PoolWorker.run(ThreadManager.java:148)

- locked <0x15593620> (a java.util.LinkedList)

"UWL Pooled Thread:2" prio=5 tid=0x097a3d88 nid=0x1158 in Object.wait() [0x0fcdf000..0x0fcdfd94]

at java.lang.Object.wait(Native Method)

- waiting on <0x15593620> (a java.util.LinkedList)

at java.lang.Object.wait(Object.java:429)

at com.sap.netweaver.bc.uwl.core.ThreadManager$PoolWorker.run(ThreadManager.java:148)

- locked <0x15593620> (a java.util.LinkedList)

"UWL Pooled Thread:1" prio=5 tid=0x08b62dd0 nid=0x1cd0 in Object.wait() [0x0fc9f000..0x0fc9fd94]

at java.lang.Object.wait(Native Method)

- waiting on <0x15593620> (a java.util.LinkedList)

at java.lang.Object.wait(Object.java:429)

at com.sap.netweaver.bc.uwl.core.ThreadManager$PoolWorker.run(ThreadManager.java:148)

- locked <0x15593620> (a java.util.LinkedList)

"UWL Pooled Thread:0" prio=5 tid=0x05050640 nid=0x1600 in Object.wait() [0x0fc5f000..0x0fc5fd94]

at java.lang.Object.wait(Native Method)

- waiting on <0x15593620> (a java.util.LinkedList)

at java.lang.Object.wait(Object.java:429)

at com.sap.netweaver.bc.uwl.core.ThreadManager$PoolWorker.run(ThreadManager.java:148)

- locked <0x15593620> (a java.util.LinkedList)

"Thread-63" daemon prio=5 tid=0x04993d50 nid=0xb84 in Object.wait() [0x0fc1f000..0x0fc1fd94]

at java.lang.Object.wait(Native Method)

- waiting on <0x203eef48> (a java.util.TaskQueue)

at java.util.TimerThread.mainLoop(Timer.java:429)

- locked <0x203eef48> (a java.util.TaskQueue)

at java.util.TimerThread.run(Timer.java:382)

"ConnectionSetCleaner Thread (UNMANAGED)." prio=5 tid=0x050413c8 nid=0x1974 in Object.wait() [0x0fbdf000..0x0fbdfd94]

at java.lang.Object.wait(Native Method)

- waiting on <0x1fcb8988> (a com.sap.engine.services.connector.jca.ConnectionHashSet$ConnectionSetCleaner)

at com.sap.engine.services.connector.jca.ConnectionHashSet$ConnectionSetCleaner.run(ConnectionHashSet.java:708)

- locked <0x1fcb8988> (a com.sap.engine.services.connector.jca.ConnectionHashSet$ConnectionSetCleaner)

"Thread-61" daemon prio=5 tid=0x091a0bd0 nid=0x168 in Object.wait() [0x0fb9f000..0x0fb9fd94]

at java.lang.Object.wait(Native Method)

- waiting on <0x1e1c96f0> (a java.util.TaskQueue)

at java.util.TimerThread.mainLoop(Timer.java:429)

- locked <0x1e1c96f0> (a java.util.TaskQueue)

at java.util.TimerThread.run(Timer.java:382)

"ConfigurationEventDispatcher" daemon prio=5 tid=0x07f55e40 nid=0x16fc waiting on condition [0x0fb5f000..0x0fb5fd94]

at java.lang.Thread.sleep(Native Method)

at com.sapportals.config.event.ConfigEventDispatcher.run(ConfigEventDispatcher.java:114)

"Thread-60" daemon prio=5 tid=0x08001408 nid=0x14a8 in Object.wait() [0x0fb1f000..0x0fb1fd94]

at java.lang.Object.wait(Native Method)

- waiting on <0x1e1c97b0> (a java.util.TaskQueue)

at java.util.TimerThread.mainLoop(Timer.java:429)

- locked <0x1e1c97b0> (a java.util.TaskQueue)

at java.util.TimerThread.run(Timer.java:382)

"config_context_cleaner" daemon prio=5 tid=0x05265e90 nid=0x18a4 in Object.wait() [0x0fadf000..0x0fadfd94]

at java.lang.Object.wait(Native Method)

- waiting on <0x1e1c97f0> (a com.sapportals.config.fwk.util.ConfigurationCache)

at com.sapportals.config.fwk.util.ConfigurationCache.run(ConfigurationCache.java:198)

- locked <0x1e1c97f0> (a com.sapportals.config.fwk.util.ConfigurationCache)

"com.sap.pcd.gl.xfs.LocalNotificationThread" daemon prio=5 tid=0x04a31698 nid=0x1a0c in Object.wait() [0x0fa9f000..0x0fa9fd94]

at java.lang.Object.wait(Native Method)

- waiting on <0x1e1c98a0> (a com.sapportals.portal.pcd.gl.xfs.notif.XfsNotificationEngine)

at java.lang.Object.wait(Object.java:429)

at com.sapportals.portal.pcd.gl.xfs.notif.XfsNotificationEngine.run(XfsNotificationEngine.java:239)

- locked <0x1e1c98a0> (a com.sapportals.portal.pcd.gl.xfs.notif.XfsNotificationEngine)

at java.lang.Thread.run(Thread.java:534)

"Soft cache optimizer - appl. PCD" daemon prio=5 tid=0x04e5b5f0 nid=0x1084 waiting on condition [0x0fa5f000..0x0fa5fd94]

at java.lang.Thread.sleep(Native Method)

at com.sapportals.portal.softcache.CacheOptimizer.threadSleep(CacheOptimizer.java:139)

at com.sapportals.portal.softcache.CacheOptimizer.run(CacheOptimizer.java:119)

at java.lang.Thread.run(Thread.java:534)

"com.sapportals.portal.softcache.CacheCleaner" daemon prio=5 tid=0x051c1008 nid=0x1ffc waiting on condition [0x0fa1f000..0x0fa1fd94]

at java.lang.Thread.sleep(Native Method)

at com.sapportals.portal.softcache.CacheCleaner.run(CacheCleaner.java:89)

at java.lang.Thread.run(Thread.java:534)

"PRT-AppBrkCleanRef-60000" prio=5 tid=0x093bc848 nid=0x1a30 waiting on condition [0x0f9df000..0x0f9dfd94]

at java.lang.Thread.sleep(Native Method)

at com.sapportals.portal.prt.core.broker.PortalAppBroker$ReferenceCleaner.run(PortalAppBroker.java:2090)

"SeedGenerator Thread" daemon prio=2 tid=0x08a24408 nid=0x1a94 in Object.wait() [0x0f95f000..0x0f95fd94]

at java.lang.Object.wait(Native Method)

- waiting on <0x1db29c28> (a sun.security.provider.SeedGenerator$ThreadedSeedGenerator)

at java.lang.Object.wait(Object.java:429)

at sun.security.provider.SeedGenerator$ThreadedSeedGenerator.run(SeedGenerator.java:282)

- locked <0x1db29c28> (a sun.security.provider.SeedGenerator$ThreadedSeedGenerator)

at java.lang.Thread.run(Thread.java:534)

"DSR_Write_Rate_Monitor: zadoluipsap07_EPP_7810350" prio=5 tid=0x094c1ad0 nid=0x1e18 waiting on condition [0x0f99f000..0x0f99fd94]

at java.lang.Thread.sleep(Native Method)

at com.sap.engine.services.dsr.DSRComponentImpl0$SubmitRateMonitor.run(DSRComponentImpl0.java:1498)

"Thread-50" prio=5 tid=0x04b42090 nid=0x1a64 waiting on condition [0x0f91f000..0x0f91fd94]

at java.lang.Thread.sleep(Native Method)

at com.sap.caf.eu.gp.model.forms.co.IFormsTimeoutProcessor.run(IFormsTimeoutProcessor.java:151)

at java.lang.Thread.run(Thread.java:534)

"Thread-49" prio=5 tid=0x095af038 nid=0x1a6c waiting on condition [0x0f8df000..0x0f8dfd94]

at java.lang.Thread.sleep(Native Method)

at com.sap.caf.eu.gp.model.forms.co.IFormsTimeoutProcessor.run(IFormsTimeoutProcessor.java:151)

at java.lang.Thread.run(Thread.java:534)

"Thread-48" prio=5 tid=0x08dbe208 nid=0x11b8 waiting on condition [0x0f89f000..0x0f89fd94]

at java.lang.Thread.sleep(Native Method)

at com.sap.caf.eu.gp.model.forms.co.IFormsTimeoutProcessor.run(IFormsTimeoutProcessor.java:151)

at java.lang.Thread.run(Thread.java:534)

"Thread-47" prio=5 tid=0x08dbe008 nid=0x1b68 waiting on condition [0x0f85f000..0x0f85fd94]

at java.lang.Thread.sleep(Native Method)

at com.sap.caf.eu.gp.model.forms.co.IFormsTimeoutProcessor.run(IFormsTimeoutProcessor.java:151)

at java.lang.Thread.run(Thread.java:534)

"Thread-46" prio=5 tid=0x08a7a918 nid=0x1a90 waiting on condition [0x0f81f000..0x0f81fd94]

at java.lang.Thread.sleep(Native Method)

at com.sap.caf.eu.gp.model.forms.co.IFormsTimeoutProcessor.run(IFormsTimeoutProcessor.java:151)

at java.lang.Thread.run(Thread.java:534)

"Thread-45" prio=9 tid=0x08c816e0 nid=0x17f4 in Object.wait() [0x0f7df000..0x0f7dfd94]

at java.lang.Object.wait(Native Method)

- waiting on <0x1d5365a8> (a com.sap.tc.webdynpro.services.session.WatchDog)

at java.lang.Object.wait(Object.java:429)

at com.sap.tc.webdynpro.services.session.WatchDog$WatchDogRunnable.run(WatchDog.java:387)

- locked <0x1d5365a8> (a com.sap.tc.webdynpro.services.session.WatchDog)

at java.lang.Thread.run(Thread.java:534)

"Thread-44" prio=5 tid=0x07f14d70 nid=0x1fdc waiting on condition [0x0f79f000..0x0f79fd94]

at java.lang.Thread.sleep(Native Method)

at com.sap.caf.eu.gp.model.context.impl.UserConfigCache$ValidationThread.run(UserConfigCache.java:148)

"Thread-43" prio=5 tid=0x08844210 nid=0x16b8 in Object.wait() [0x0f75f000..0x0f75fd94]

at java.lang.Object.wait(Native Method)

- waiting on <0x1d536668> (a java.util.TaskQueue)

at java.util.TimerThread.mainLoop(Timer.java:429)

- locked <0x1d536668> (a java.util.TaskQueue)

at java.util.TimerThread.run(Timer.java:382)

"Thread-42" prio=5 tid=0x04a3fba0 nid=0x1a88 waiting on condition [0x0f71f000..0x0f71fd94]

at java.lang.Thread.sleep(Native Method)

at com.sap.caf.eu.gp.model.fnd.cache.impl.ServerCacheManagerImpl$ValidationThread.run(ServerCacheManagerImpl.java:159)

"Thread-40" prio=5 tid=0x08e63e40 nid=0x1a7c waiting on condition [0x0f6df000..0x0f6dfd94]

at java.lang.Thread.sleep(Native Method)

at com.sap.caf.eu.gp.base.caching.MemoryCache$ValidationThread.run(MemoryCache.java:152)

"Thread-39" prio=5 tid=0x04d05708 nid=0x1f4c waiting on condition [0x0f69f000..0x0f69fd94]

at java.lang.Thread.sleep(Native Method)

at com.sap.caf.eu.gp.model.forms.co.IFormsTimeoutProcessor.run(IFormsTimeoutProcessor.java:151)

at java.lang.Thread.run(Thread.java:534)

"JCO.ServerThread-21" prio=5 tid=0x08c5d880 nid=0x1864 in Object.wait() [0x0f65f000..0x0f65fd94]

at java.lang.Object.wait(Native Method)

- waiting on <0x1d5367f0> (a com.sap.mw.jco.RequestListenerGroup$RequestQueue)

at java.lang.Object.wait(Object.java:429)

at com.sap.mw.jco.RequestListenerGroup$RequestQueue.getRequest(RequestListenerGroup.java:719)

- locked <0x1d5367f0> (a com.sap.mw.jco.RequestListenerGroup$RequestQueue)

at com.sap.mw.jco.JCO$Server.loop(JCO.java:8239)

- locked <0x1d5367f0> (a com.sap.mw.jco.RequestListenerGroup$RequestQueue)

at com.sap.mw.jco.JCO$Server.run(JCO.java:8165)

at java.lang.Thread.run(Thread.java:534)

"JCO.ServerThread-20" prio=5 tid=0x056c3808 nid=0xa58 in Object.wait() [0x0f61f000..0x0f61fd94]

at java.lang.Object.wait(Native Method)

- waiting on <0x1d5367f0> (a com.sap.mw.jco.RequestListenerGroup$RequestQueue)

at java.lang.Object.wait(Object.java:429)

at com.sap.mw.jco.RequestListenerGroup$RequestQueue.getRequest(RequestListenerGroup.java:719)

- locked <0x1d5367f0> (a com.sap.mw.jco.RequestListenerGroup$RequestQueue)

at com.sap.mw.jco.JCO$Server.loop(JCO.java:8239)

- locked <0x1d5367f0> (a com.sap.mw.jco.RequestListenerGroup$RequestQueue)

at com.sap.mw.jco.JCO$Server.run(JCO.java:8165)

at java.lang.Thread.run(Thread.java:534)