Skip to Content
avatar image
Former Member

Java Server unavailable (in yellow )- status "starting apps for a long time

Hi Gurus,

J2ee Server unavailable in the NWDS instance and the dispatcher is running but the status of the Java Server is "Starting Apps " for a long time( in yellow).can anyone help me out in this ,solutions rewarded

We are having the 2 server nodes

Trace of node o:

-


stdout/stderr redirect

-


node name : server0

pid : 4168

system name : KDS

system nr. : 00

started at : Thu Dec 20 14:27:26 2007

Reserved 1610612736 (0x60000000) bytes before loading DLLs.

Thr 2904 MtxInit: -2 0 0

CompilerOracle: exclude com/sapportals/portal/pb/layout/taglib/ContainerTag addIviewResources

CompilerOracle: exclude com/sap/engine/services/keystore/impl/security/CodeBasedSecurityConnector getApplicationDomain

CompilerOracle: exclude com/sap/engine/services/rmi_p4/P4StubSkeletonGenerator generateStub

CompilerOracle: exclude com/sapportals/portal/prt/util/StringUtils escapeToJS

CompilerOracle: exclude com/sapportals/portal/prt/core/broker/PortalServiceItem startServices

CompilerOracle: exclude com/sap/engine/services/webservices/server/deploy/WSConfigurationHandler downloadFile

CompilerOracle: exclude com/sapportals/portal/prt/jndisupport/util/AbstractHierarchicalContext lookup

CompilerOracle: exclude com/sapportals/portal/navigation/cache/CacheNavigationNode getAttributeValue

CompilerOracle: exclude com/sapportals/portal/navigation/TopLevelNavigationiView PrintNode

CompilerOracle: exclude com/sapportals/wcm/service/ice/wcm/ICEPropertiesCoder encode

CompilerOracle: exclude com/sap/lcr/pers/delta/importing/ObjectLoader loadObjects

CompilerOracle: exclude com/sap/engine/services/webservices/jaxrpc/encoding/InstanceBuilder readElement

CompilerOracle: exclude com/sap/engine/services/webservices/jaxrpc/encoding/InstanceBuilder readSequence

CompilerOracle: exclude com/sap/engine/services/webservices/jaxrpc/encoding/TypeMappingImpl initializeRelations

CompilerOracle: exclude com/sap/engine/services/webservices/jaxrpc/encoding/GeneratedComplexType _loadInto

SAP J2EE Engine Version 7.00 PatchLevel 108458.44 is starting...

Loading: LogManager ... 3190 ms.

Loading: PoolManager ... 16 ms.

Loading: ApplicationThreadManager ... 265 ms.

Loading: ThreadManager ... 47 ms.

Loading: IpVerificationManager ... 32 ms.

Loading: ClassLoaderManager ... 31 ms.

Loading: ClusterManager ... 719 ms.

Loading: LockingManager ... 125 ms.

Loading: ConfigurationManager ... 12946 ms.

Loading: LicensingManager ... 47 ms.

Loading: CacheManager ... 469 ms.

Loading: ServiceManager ...

Loading services.:

Service cafeuodi~mnuacc started. (0 ms).

Service cross started. (16 ms).

Service memory started. (16 ms).

Service file started. (16 ms).

Service DQE started. (0 ms).

Service cafeucc~api started. (16 ms).

Service userstore started. (15 ms).

Service runtimeinfo started. (266 ms).

Service jmx_notification started. (188 ms).

Service timeout started. (360 ms).

Service trex.service started. (313 ms).

Service p4 started. (954 ms).

Service classpath_resolver started. (15 ms).

Service log_configurator started. (22420 ms).

Service locking started. (16 ms).

Service http started. (1141 ms).

Service naming started. (1329 ms).

Service failover started. (109 ms).

Service appclient started. (266 ms).

Service jmsconnector started. (329 ms).

Service javamail started. (798 ms).

Service ts started. (375 ms).

Service licensing started. (15 ms).

Service connector started. (548 ms).

Service webservices started. (1219 ms).

Service iiop started. (766 ms).

Service deploy started. (71073 ms).

Service MigrationService started. (1110 ms).

Service configuration started. (141 ms).

Service bimmrdeployer started. (203 ms).

service MobileSetupGeneration ================= ERROR =================

Service MobileArchiveContainer started. (31 ms).

Service dbpool started. (3924 ms).

Service cafeugpmailcf started. (78 ms).

Service com.sap.security.core.ume.service started. (8364 ms).

Service tcdisdic~srv started. (1297 ms).

Service security started. (6926 ms).

Service classload started. (266 ms).

Service applocking started. (281 ms).

Service ejb started. (656 ms).

Service shell started. (563 ms).

Service tceCATTPingservice started. (125 ms).

Service telnet started. (1266 ms).

Service cafummetadata~imp started. (2204 ms).

Service webdynpro started. (1782 ms).

Service developmentserver started. (2579 ms).

Service servlet_jsp started. (3846 ms).

Service dsr started. (3283 ms).

Service keystore started. (3330 ms).

Service ssl started. (0 ms).

Service tcseccertrevoc~service started. (391 ms).

Service tcsecsecurestorage~service started. (516 ms).

Service cafeugp~model started. (32 ms).

Service cafeuer~service started. (0 ms).

Service tceujwfuiwizsvc started. (0 ms).

Service cafruntimeconnectivity~impl started. (98747 ms).

Service cafumrelgroups~imp started. (3815 ms).

Service jmx started. (4080 ms).

Service tclmctcconfsservice_sda started. (1048 ms).

Service CUL started. (1126 ms).

Service rfcengine started. (3143 ms).

Service basicadmin started. (5065 ms).

Service com.adobe~LicenseService started. (954 ms).

Service tcsecwssec~service started. (10319 ms).

Service com.adobe~DocumentServicesBinaries2 started. (5518 ms).

Service com.adobe~PDFManipulation started. (3236 ms).

Service com.adobe~DocumentServicesLicenseSupportService started. (3251 ms).

Service com.adobe~DataManagerService started. (5550 ms).

Service com.adobe~FontManagerService started. (5612 ms).

Service adminadapter started. (1626 ms).

Service com.sap.portal.pcd.gl started. (16 ms).

Service apptracing started. (2846 ms).

Service monitor started. (1298 ms).

Service tcsecdestinations~service started. (13054 ms).

Service sld started. (13492 ms).

Service com.sap.portal.prt.sapj2ee started. (1923 ms).

Service pmi started. (1610 ms).

Service com.adobe~XMLFormService started. (8302 ms).

Service prtbridge started. (11335 ms).

Service com.adobe~DocumentServicesDestProtoService started. (1767 ms).

Service tcsecvsi~service started. (6300 ms).

Service jms_provider started. (15931 ms).

Service com.adobe~DocumentServicesConfiguration started. (12366 ms).

Service com.adobe~TrustManagerService started. (188 ms).

Service tc.monitoring.logviewer started. (23701 ms).

ServiceManager started for 142475 ms.

Framework started for 161863 ms.

SAP J2EE Engine Version 7.00 PatchLevel 108458.44 is running!

PatchLevel 108458.44 May 05, 2007 14:23 GMT

Trace of Node 1:

-


stdout/stderr redirect

-


node name : server1

pid : 4168

system name : KDS

system nr. : 00

started at : Thu Dec 20 14:27:26 2007

Reserved 1610612736 (0x60000000) bytes before loading DLLs.

Thr 6244 MtxInit: -2 0 0

CompilerOracle: exclude com/sapportals/portal/pb/layout/taglib/ContainerTag addIviewResources

CompilerOracle: exclude com/sap/engine/services/keystore/impl/security/CodeBasedSecurityConnector getApplicationDomain

CompilerOracle: exclude com/sap/engine/services/rmi_p4/P4StubSkeletonGenerator generateStub

CompilerOracle: exclude com/sapportals/portal/prt/util/StringUtils escapeToJS

CompilerOracle: exclude com/sapportals/portal/prt/core/broker/PortalServiceItem startServices

CompilerOracle: exclude com/sap/engine/services/webservices/server/deploy/WSConfigurationHandler downloadFile

CompilerOracle: exclude com/sapportals/portal/prt/jndisupport/util/AbstractHierarchicalContext lookup

CompilerOracle: exclude com/sapportals/portal/navigation/cache/CacheNavigationNode getAttributeValue

CompilerOracle: exclude com/sapportals/portal/navigation/TopLevelNavigationiView PrintNode

CompilerOracle: exclude com/sapportals/wcm/service/ice/wcm/ICEPropertiesCoder encode

CompilerOracle: exclude com/sap/lcr/pers/delta/importing/ObjectLoader loadObjects

CompilerOracle: exclude com/sap/engine/services/webservices/jaxrpc/encoding/InstanceBuilder readElement

CompilerOracle: exclude com/sap/engine/services/webservices/jaxrpc/encoding/InstanceBuilder readSequence

CompilerOracle: exclude com/sap/engine/services/webservices/jaxrpc/encoding/TypeMappingImpl initializeRelations

CompilerOracle: exclude com/sap/engine/services/webservices/jaxrpc/encoding/GeneratedComplexType _loadInto

SAP J2EE Engine Version 7.00 PatchLevel 108458.44 is starting...

Loading: LogManager ... 1720 ms.

Loading: PoolManager ... 16 ms.

Loading: ApplicationThreadManager ... 250 ms.

Loading: ThreadManager ... 78 ms.

Loading: IpVerificationManager ... 31 ms.

Loading: ClassLoaderManager ... 16 ms.

Loading: ClusterManager ... 500 ms.

Loading: LockingManager ... 704 ms.

Loading: ConfigurationManager ... 10491 ms.

Loading: LicensingManager ... 31 ms.

Loading: CacheManager ... 235 ms.

Loading: ServiceManager ...

Loading services.:

0.000: [GC 0.000: DefNew: 87040K->8111K(130560K), 0.0858880 secs] 87040K->8111K(1005056K), 0.0860550 secs

Service cafeuodi~mnuacc started. (0 ms).

Service DQE started. (297 ms).

Service cafeucc~api started. (109 ms).

Service cross started. (390 ms).

Service file started. (390 ms).

Service runtimeinfo started. (422 ms).

Service userstore started. (376 ms).

Service memory started. (673 ms).

Service jmx_notification started. (453 ms).

Service timeout started. (875 ms).

Service trex.service started. (704 ms).

Service p4 started. (656 ms).

Service classpath_resolver started. (46 ms).

14.066: [GC 14.066: DefNew: 95151K->10393K(130560K), 0.0932969 secs] 95151K->10393K(1005056K), 0.0934684 secs

19.895: [GC 19.896: DefNew: 97433K->11125K(130560K), 0.0509232 secs] 97433K->11125K(1005056K), 0.0511944 secs

Service log_configurator started. (24420 ms).

Service locking started. (16 ms).

25.557: [GC 25.557: DefNew: 98165K->13919K(130560K), 0.0770372 secs] 98165K->13919K(1005056K), 0.0771960 secs

Service http started. (1282 ms).

Service naming started. (1392 ms).

Service failover started. (1235 ms).

Service appclient started. (1485 ms).

Service javamail started. (1829 ms).

Service ts started. (1767 ms).

Service jmsconnector started. (1876 ms).

Service licensing started. (110 ms).

Service connector started. (907 ms).

Service iiop started. (532 ms).

Service webservices started. (2438 ms).

48.200: [GC 48.200: DefNew: 100959K->25542K(130560K), 0.1360955 secs] 100959K->25542K(1005056K), 0.1363436 secs

54.474: [GC 54.474: DefNew: 112582K->20164K(130560K), 0.1008394 secs] 112582K->20164K(1005056K), 0.1009996 secs

69.199: [GC 69.200: DefNew: 107204K->22875K(130560K), 0.1197005 secs] 107204K->22875K(1005056K), 0.1200568 secs

Service deploy started. (72543 ms).

Service configuration started. (140 ms).

Service bimmrdeployer started. (31 ms).

Service MigrationService started. (562 ms).

Service MobileArchiveContainer started. (1516 ms).

Service MobileSetupGeneration started. (1595 ms).

Service dbpool started. (5206 ms).

Service cafeugpmailcf started. (63 ms).

Service com.sap.security.core.ume.service started. (8161 ms).

87.782: [GC 87.782: DefNew: 109911K->29413K(130560K), 0.1891978 secs] 109911K->29413K(1005056K), 0.1893808 secs

Service tcdisdic~srv started. (1282 ms).

Service security started. (3033 ms).

Service classload started. (156 ms).

Service applocking started. (157 ms).

Service shell started. (313 ms).

Service tceCATTPingservice started. (94 ms).

Service telnet started. (312 ms).

Service ejb started. (922 ms).

Service developmentserver started. (453 ms).

Service webdynpro started. (547 ms).

Service servlet_jsp started. (1173 ms).

Service dsr started. (1063 ms).

Service keystore started. (1344 ms).

Service ssl started. (16 ms).

Service tcseccertrevoc~service started. (578 ms).

Service jmx started. (1110 ms).

Service tcsecsecurestorage~service started. (1047 ms).

94.120: [GC 94.120: DefNew: 116453K->43520K(130560K), 0.2516995 secs] 116453K->55613K(1005056K), 0.2519682 secs

Service tclmctcconfsservice_sda started. (2580 ms).

Service CUL started. (3549 ms).

Service cafummetadata~imp started. (6254 ms).

Service cafeugp~model started. (1329 ms).

Service rfcengine started. (3737 ms).

Service cafeuer~service started. (0 ms).

Service tceujwfuiwizsvc started. (16 ms).

Service com.adobe~DocumentServicesLicenseSupportService started. (4300 ms).

Service com.adobe~FontManagerService started. (5425 ms).

Service com.adobe~DataManagerService started. (5582 ms).

Service com.adobe~DocumentServicesBinaries2 started. (5910 ms).

Service com.adobe~PDFManipulation started. (4315 ms).

Service basicadmin started. (7254 ms).

Service cafruntimeconnectivity~impl started. (100294 ms).

Service com.adobe~LicenseService started. (297 ms).

Service prtbridge started. (8098 ms).

Service apptracing started. (3643 ms).

Service tcsecwssec~service started. (8208 ms).

Service adminadapter started. (1532 ms).

Service cafumrelgroups~imp started. (5613 ms).

Service com.sap.portal.pcd.gl started. (438 ms).

104.035: [GC 104.036: DefNew: 130560K->43520K(130560K), 0.4183386 secs] 142653K->62215K(1005056K), 0.4185225 secs

Service com.adobe~XMLFormService started. (9162 ms).

Service monitor started. (3268 ms).

Service sld started. (12382 ms).

Service com.sap.portal.prt.sapj2ee started. (125 ms).

106.513: [GC 106.513: DefNew: 130545K->40021K(130560K), 0.2210693 secs] 149241K->63511K(1005056K), 0.2212475 secs

Service com.adobe~DocumentServicesConfiguration started. (7645 ms).

Service com.adobe~TrustManagerService started. (94 ms).

Service tcsecdestinations~service started. (14790 ms).

110.328: [GC 110.329: DefNew: 126991K->37367K(130560K), 0.1896564 secs] 150481K->64176K(1005056K), 0.1898111 secs

Service pmi started. (1939 ms).

Service com.adobe~DocumentServicesDestProtoService started. (78 ms).

Service tcsecvsi~service started. (2079 ms).

114.407: [GC 114.407: DefNew: 124395K->38325K(130560K), 0.1181515 secs] 151204K->65134K(1005056K), 0.1183338 secs

119.591: [GC 119.591: DefNew: 125360K->39243K(130560K), 0.1979451 secs] 152169K->66052K(1005056K), 0.1981060 secs

122.205: [GC 122.205: DefNew: 126283K->39785K(130560K), 0.1415175 secs] 153092K->66851K(1005056K), 0.1416860 secs

Service tc.monitoring.logviewer started. (30502 ms).

Service jms_provider started. (31486 ms).

ServiceManager started for 142552 ms.

Framework started for 158391 ms.

SAP J2EE Engine Version 7.00 PatchLevel 108458.44 is running!

PatchLevel 108458.44 May 05, 2007 14:23 GMT

145.476: [GC 145.476: DefNew: 126825K->40239K(130560K), 0.1088881 secs] 153891K->69357K(1005056K), 0.1090310 secs

154.856: [GC 154.856: DefNew: 127279K->41929K(130560K), 0.1665652 secs] 156397K->74781K(1005056K), 0.1667231 secs

402.301: [GC 402.301: DefNew: 128878K->18046K(130560K), 0.1411321 secs] 161730K->73978K(1005056K), 0.1413031 secs

404.085: [GC 404.085: DefNew: 105086K->18796K(130560K), 0.0750212 secs] 161018K->74729K(1005056K), 0.0751696 secs

409.852: [GC 409.852: DefNew: 105836K->26140K(130560K), 0.1006458 secs] 161769K->82072K(1005056K), 0.1008288 secs

regards,

S.Rajeshkumar

Edited by: Rajesh kumar on Dec 20, 2007 3:18 PM

Add comment
10|10000 characters needed characters exceeded

  • Get RSS Feed

9 Answers

  • Best Answer
    avatar image
    Former Member
    Jan 15, 2008 at 10:05 PM

    Hi Rajesh,

    Have you tried taking threaddumps as per my previous memo on Jan 10, 2008 3:46 PM ? That is the option to find out what is causing the delay.

    Also, you can check server.n.log inside server\log\system to see which applications are already loaded/how long they took/any exceptions.

    You could also logon using telnet administrator, jump to server node, add deploy and check using list_app about the applications that are currently in 'Starting' state.

    Hope this helps.

    Regards,

    Snehal

    Add comment
    10|10000 characters needed characters exceeded

    • Former Member

      Hi snehal,

      Thanks a lot for ur effort ,i have tried a lot with ur clues but nothing will be done at end ,iam trying still ......to get the solution

      thanks in advance !

      regards,

      S.RAjeshkumar

  • avatar image
    Former Member
    Dec 22, 2007 at 02:16 PM

    Hi,

    Go to config tool

    under the global server configuration select managers

    under thread manager -->>select option --->intial thread count increase that to 100

    Restart the server.

    Reward the points if use ful.

    Regards,

    Maheshwer p

    Add comment
    10|10000 characters needed characters exceeded

    • Former Member Former Member

      Hi,

      I have not any errors on this log . check can u able to login into visual admin . if it is possible try to start the application which are not started.

  • Jan 14, 2008 at 10:24 PM

    Hi,

    Here come just a few hints of what these log messages mean

    The message "SAP J2EE Engine Version 7.00 PatchLevel 108458.44 is running!" does not mean, that everything has been started. It just means that the J2EE engine with the configured services has been started.

    Afterwards all the applications are started, which can take a couple of minutes depending of which applications you have and how many you have.

    To check how long the engine needs to startup you can search for the String "State changed" in you dev_server* file.

    By monitoring your CPU and GC log during the startup you will usually find that the J2EE is doing a lot of work to start all the applications.

    Regards,

    Markus

    Add comment
    10|10000 characters needed characters exceeded

    • Former Member

      Hi Markus,

      I have checked for a day the status remains the same "Starting Apps" ....u have any altenate

      regards,

      S.Rajeshkumar

  • avatar image
    Former Member
    Jan 25, 2008 at 06:55 PM

    Hi Rajesh,

    Could you forward me the threaddumps (taken during starting applications phase) and server.n.log, defaultrace file from the time of restart ?

    Email : snehal.bhaidasna@sap.com

    I'll take a look at it and paste the soln here.

    Regards,

    Snehal

    Add comment
    10|10000 characters needed characters exceeded

  • avatar image
    Former Member
    Feb 03, 2008 at 08:01 AM

    Hi Rajesh,

    Check the parameters in Configtool:

    -> java parameters : -Djava.io.tmpdir=D:\ep_temp -Dcm.tmpdir=D:\ep_temp -Djava.security.policy=./java.policy -Djava.security.egd=file:/dev/urandom -Dorg.omg.CORBA.ORBClass=com.sap.engine.system.ORBProxy -Dorg.omg.CORBA.ORBSingletonClass=com.sap.engine.system.ORBSingletonProxy -Djavax.rmi.CORBA.PortableRemoteObjectClass=com.sap.engine.system.PortableRemoteObjectProxy -XX:MaxPermSize=192M -XX:PermSize=192M -XX:NewSize=170M -XX:MaxNewSize=170M -XX:DisableExplicitGC -verbose:gc -Xloggc:GC.log -XX:PrintGCDetails -XX:PrintGCTimeStamps -Djava.awt.headless=true -Dsun.io.useCanonCaches=false -XX:SoftRefLRUPolicyMSPerMB=1 -XX:SurvivorRatio=2 -XX:TargetSurvivorRatio=90 -Dorg.omg.PortableInterceptor.ORBInitializerClass.com.sap.engine.services.ts.jts.ots.PortableInterceptor.JTSInitializer -XX:UseTLAB -XX:+UseParNewGC

    If you have any other system then try to match the parameter and there value or select the derault parameters and try to start again.

    Best Wishes

    Kumar

    Add comment
    10|10000 characters needed characters exceeded

  • avatar image
    Former Member
    Jan 10, 2008 at 06:46 PM

    Rajesh

    Did you manage to resolve this ?

    I have a similar result myself

    Add comment
    10|10000 characters needed characters exceeded

  • avatar image
    Former Member
    Jan 10, 2008 at 11:46 PM

    Hello Rajesh/Wai,

    You could do the following :

    - Retstart the j2ee server

    - Monitor dev_server0/1 (or SAPMMC if windows to see 'Starting Applications' status) file to reach :

    JLaunchISetState: change state from [Starting (2)] to [Starting

    applications (10)]

    -Now wait for some time and take 5 threaddumps as per note 710154 at an interval of 20 seconds each.

    -Use Angel Penkov's Threaddump viewer tool attached with note 1020246 to check the threaddumps. If you need help analyzing this, please send me the std_server0 file with the threaddumps at snehal dot bhaidasna at sap.com.

    -You could also refer to server.<n>.log under server0\log\system for the lines like :

    startApp on application sap.com/caf~km.proxies finished on current cluster node for 6578 ms

    that shows the start time taken by that particular application. You could scan through this file after the engine is in running state to find out which application took the longest.

    Threaddumps help to identify the exact cause though.

    Hope this helps.

    Best Regards,

    Snehal Bhaidasna

    Development Support

    Palo Alto, CA(USA)

    Add comment
    10|10000 characters needed characters exceeded

  • avatar image
    Former Member
    Feb 03, 2008 at 09:53 PM

    A few things.

    Your JDK looks a little bit low.... 1.4.2_09????

    Your GC's are going to GC.log,,, can you check the frequency and speed of the GC's... I have seen poor paging cripple a J2EE starting Apps.

    Certtainly a Maxpermsize of 192 I would have increased to 256... you can always set the printheapatgc paramater to get a better idea, a very handy param.

    As the engine is up, you can use VA and go to the performance Tracing service on the server and under JARM - Threads you can see an 'SM50' style view that is like a thrad dump (but not as useful). This should give you an idea of what app may be hanging as can the locking adapter service in the VA.

    I would use the stand alone log viewer to look in default.trc and under applications and Server log nodes for additional clues.

    As it is NWDI it is quite possible something is a little messed up in the CBS, have you tried restarting this service in the VA once the system is up.

    Add comment
    10|10000 characters needed characters exceeded

  • avatar image
    Former Member
    Feb 04, 2008 at 08:29 AM

    Hello,

    Would yo plese increse the Application Thread manager count (should be atleast 150-250) and restart the J2EE Engine.

    Regards

    Kumaraswamy

    Add comment
    10|10000 characters needed characters exceeded