Skip to Content
author's profile photo Former Member
Former Member

Server processes struck in Starting Applications Phase

Hello All,

After EHP4 pgrade of abap+java, J2ee was not coming up and we had to change a lot of profile

parameters to get it up.

But now JCmon shows the following state:

|Idx|Name               |PID     |State                |Error|Restart|
|---|-------------------|--------|---------------------|-----|-------|
|  0|dispatcher         |   21292|Running              |    0|yes    |
|  1|server0            |   23019|Starting applications|    0|yes    |
|  2|server1            |   23143|Starting applications|    0|yes    |
|  3|SDM                |   21295|Running              |    0|yes    |
------------------------------------------------------------

It stays in the Starting applications state and the webpage http://hostname/50400 results the

message "Application Stopped" "You have requested an application that is currently stopped".

I am not able to find any descrepancy in the logs right now.

Following is the dev_jcontrol log:
[Thr  1] Sat Oct  9 02:51:29 2010
[Thr  1] JStartupICreateProcess: fork process (pid 22248)
[Thr  1] JControlICheckProcessList: process server0 started (PID:22248)

[Thr  1] Sat Oct  9 02:53:59 2010
[Thr  1] JControlICheckProcessList: process server1 (pid:21294) died (RUN-FLAG)
[Thr  1] JControlIResetProcess: reset process server1
[Thr  1] JControlICheckProcessList: running flight recorder:
        /usr/j2se/bin/java -classpath 

../j2ee/cluster/bootstrap/sap.com~tc~bl~offline_launcher~impl.jar 

com.sap.engine.offline.OfflineToolStart com.sap.engine.flightrecorder.core.Collector 

../j2ee/cluster/bootstrap -node ID44702651 1286610937 -bz /usr/sap/DV2/SYS/global

**********************************************************************
JControlStartJLaunch: program = /usr/sap/DV2/SYS/exe/run/jlaunch
-> arg[00] = /usr/sap/DV2/SYS/exe/run/jlaunch
-> arg[01] = pf=/usr/sap/DV2/SYS/profile/DV2_DVEBMGS04_aeerpdv1
-> arg[02] = -DSAPINFO=DV2_04_server
-> arg[03] = -nodeId=2
-> arg[04] = -file=/usr/sap/DV2/DVEBMGS04/j2ee/cluster/instance.properties
-> arg[05] = -syncSem=352321557
-> arg[06] = -nodeName=ID44702651
-> arg[07] = -jvmOutFile=/usr/sap/DV2/DVEBMGS04/work/jvm_server1.out
-> arg[08] = -jvmOutMode=append
-> arg[09] = -stdOutFile=/usr/sap/DV2/DVEBMGS04/work/std_server1.out
-> arg[10] = -stdOutMode=append
-> arg[11] = -traceMode=append
-> arg[12] = -locOutFile=/usr/sap/DV2/DVEBMGS04/work/dev_server1
-> arg[13] = -mode=JCONTROL
-> arg[14] = pf=/usr/sap/DV2/SYS/profile/DV2_DVEBMGS04_aeerpdv1
-> arg[15] = -DSAPSTART=1
-> arg[16] = -DCONNECT_PORT=58682
-> arg[17] = -DSAPSYSTEM=04
-> arg[18] = -DSAPSYSTEMNAME=DV2
-> arg[19] = -DSAPMYNAME=aeerpdv1_DV2_04
-> arg[20] = -DSAPPROFILE=/usr/sap/DV2/SYS/profile/DV2_DVEBMGS04_aeerpdv1
-> arg[21] = -DFRFC_FALLBACK=ON
-> arg[22] = -DFRFC_FALLBACK_HOST=localhost
-> lib path = 

LD_LIBRARY_PATH=/usr/j2se/jre/lib/sparcv9/server:/usr/j2se/jre/lib/sparcv9:/usr/j2se/jre/../lib/

sparcv9:/usr/sap/DV2/DVEBMGS04/j2ee/os_libs:/usr/sap/DV2/SYS/exe/run:/usr/sap/DV2/SYS/exe/run:/u

sr/j2se/jre/lib/sparc/client:/usr/j2se/jre/lib/sparc:/usr/j2se/jre/../lib/sparc:/usr/sap/DV2/SYS

/exe/run:/usr/sap/DV2/SYS/exe/runU:/usr/lib:/usr/sap/DV2/DVEBMGS04/j2ee/os_libs:/usr/sap/DV2/DVE

BMGS04/j2ee/os_libs:/usr/sap/DV2/SYS/exe/run:/usr/sap/DV2/SYS/exe/run:/oracle/client/10x_64/inst

antclient
-> exe path = 

PATH=/usr/j2se/bin:/usr/sap/DV2/DVEBMGS04/j2ee/os_libs:/usr/j2se/bin:/oracle/DV2/102_64/bin:/exp

ort/home/dv2adm:/usr/sap/DV2/SYS/exe/run:/usr/bin:.:/usr/ccs/bin:/usr/ucb
**********************************************************************

[Thr  1] JStartupICreateProcess: fork process (pid 22304)
[Thr  1] JControlICheckProcessList: process server1 started (PID:22304)

[Thr  1] Sat Oct  9 03:45:55 2010
[Thr  1] JControlICheckProcessList: process server0 (pid:22248) died (RUN-FLAG)
[Thr  1] JControlIResetProcess: reset process server0
[Thr  1] JControlICheckProcessList: running flight recorder:
        /usr/j2se/bin/java -classpath 

../j2ee/cluster/bootstrap/sap.com~tc~bl~offline_launcher~impl.jar 

com.sap.engine.offline.OfflineToolStart com.sap.engine.flightrecorder.core.Collector 

../j2ee/cluster/bootstrap -node ID44702650 1286614289 -bz /usr/sap/DV2/SYS/global

**********************************************************************
JControlStartJLaunch: program = /usr/sap/DV2/SYS/exe/run/jlaunch
-> arg[00] = /usr/sap/DV2/SYS/exe/run/jlaunch
-> arg[01] = pf=/usr/sap/DV2/SYS/profile/DV2_DVEBMGS04_aeerpdv1
-> arg[02] = -DSAPINFO=DV2_04_server
-> arg[03] = -nodeId=1
-> arg[04] = -file=/usr/sap/DV2/DVEBMGS04/j2ee/cluster/instance.properties
-> arg[05] = -syncSem=352321557
-> arg[06] = -nodeName=ID44702650
-> arg[07] = -jvmOutFile=/usr/sap/DV2/DVEBMGS04/work/jvm_server0.out
-> arg[08] = -jvmOutMode=append
-> arg[09] = -stdOutFile=/usr/sap/DV2/DVEBMGS04/work/std_server0.out
-> arg[10] = -stdOutMode=append
-> arg[11] = -traceMode=append
-> arg[12] = -locOutFile=/usr/sap/DV2/DVEBMGS04/work/dev_server0
-> arg[13] = -mode=JCONTROL
-> arg[14] = pf=/usr/sap/DV2/SYS/profile/DV2_DVEBMGS04_aeerpdv1
-> arg[15] = -DSAPSTART=1
-> arg[16] = -DCONNECT_PORT=58682
-> arg[17] = -DSAPSYSTEM=04
-> arg[18] = -DSAPSYSTEMNAME=DV2
-> arg[19] = -DSAPMYNAME=aeerpdv1_DV2_04
-> arg[20] = -DSAPPROFILE=/usr/sap/DV2/SYS/profile/DV2_DVEBMGS04_aeerpdv1
-> arg[21] = -DFRFC_FALLBACK=ON
-> arg[22] = -DFRFC_FALLBACK_HOST=localhost
-> lib path = 

LD_LIBRARY_PATH=/usr/j2se/jre/lib/sparcv9/server:/usr/j2se/jre/lib/sparcv9:/usr/j2se/jre/../lib/

sparcv9:/usr/sap/DV2/DVEBMGS04/j2ee/os_libs:/usr/sap/DV2/SYS/exe/run:/usr/sap/DV2/SYS/exe/run:/u

sr/j2se/jre/lib/sparc/client:/usr/j2se/jre/lib/sparc:/usr/j2se/jre/../lib/sparc:/usr/sap/DV2/SYS

/exe/run:/usr/sap/DV2/SYS/exe/runU:/usr/lib:/usr/sap/DV2/DVEBMGS04/j2ee/os_libs:/usr/sap/DV2/DVE

BMGS04/j2ee/os_libs:/usr/sap/DV2/SYS/exe/run:/usr/sap/DV2/SYS/exe/run:/oracle/client/10x_64/inst

antclient
-> exe path = 

PATH=/usr/j2se/bin:/usr/sap/DV2/DVEBMGS04/j2ee/os_libs:/usr/j2se/bin:/oracle/DV2/102_64/bin:/exp

ort/home/dv2adm:/usr/sap/DV2/SYS/exe/run:/usr/bin:.:/usr/ccs/bin:/usr/ucb
**********************************************************************


[Thr  1] Sat Oct  9 03:45:56 2010
[Thr  1] JStartupICreateProcess: fork process (pid 23019)
[Thr  1] JControlICheckProcessList: process server0 started (PID:23019)

[Thr  1] Sat Oct  9 03:52:42 2010
[Thr  1] JControlICheckProcessList: process server1 (pid:22304) died (RUN-FLAG)
[Thr  1] JControlIResetProcess: reset process server1
[Thr  1] JControlICheckProcessList: running flight recorder:
        /usr/j2se/bin/java -classpath 

../j2ee/cluster/bootstrap/sap.com~tc~bl~offline_launcher~impl.jar 

com.sap.engine.offline.OfflineToolStart com.sap.engine.flightrecorder.core.Collector 

../j2ee/cluster/bootstrap -node ID44702651 1286614439 -bz /usr/sap/DV2/SYS/global

**********************************************************************
JControlStartJLaunch: program = /usr/sap/DV2/SYS/exe/run/jlaunch
-> arg[00] = /usr/sap/DV2/SYS/exe/run/jlaunch
-> arg[01] = pf=/usr/sap/DV2/SYS/profile/DV2_DVEBMGS04_aeerpdv1
-> arg[02] = -DSAPINFO=DV2_04_server
-> arg[03] = -nodeId=2
-> arg[04] = -file=/usr/sap/DV2/DVEBMGS04/j2ee/cluster/instance.properties
-> arg[05] = -syncSem=352321557
-> arg[06] = -nodeName=ID44702651
-> arg[07] = -jvmOutFile=/usr/sap/DV2/DVEBMGS04/work/jvm_server1.out
-> arg[08] = -jvmOutMode=append
-> arg[09] = -stdOutFile=/usr/sap/DV2/DVEBMGS04/work/std_server1.out
-> arg[10] = -stdOutMode=append
-> arg[11] = -traceMode=append
-> arg[12] = -locOutFile=/usr/sap/DV2/DVEBMGS04/work/dev_server1
-> arg[13] = -mode=JCONTROL
-> arg[14] = pf=/usr/sap/DV2/SYS/profile/DV2_DVEBMGS04_aeerpdv1
-> arg[15] = -DSAPSTART=1
-> arg[16] = -DCONNECT_PORT=58682
-> arg[17] = -DSAPSYSTEM=04
-> arg[18] = -DSAPSYSTEMNAME=DV2
-> arg[19] = -DSAPMYNAME=aeerpdv1_DV2_04
-> arg[20] = -DSAPPROFILE=/usr/sap/DV2/SYS/profile/DV2_DVEBMGS04_aeerpdv1
-> arg[21] = -DFRFC_FALLBACK=ON
-> arg[22] = -DFRFC_FALLBACK_HOST=localhost
-> lib path = 

LD_LIBRARY_PATH=/usr/j2se/jre/lib/sparcv9/server:/usr/j2se/jre/lib/sparcv9:/usr/j2se/jre/../lib/

sparcv9:/usr/sap/DV2/DVEBMGS04/j2ee/os_libs:/usr/sap/DV2/SYS/exe/run:/usr/sap/DV2/SYS/exe/run:/u

sr/j2se/jre/lib/sparc/client:/usr/j2se/jre/lib/sparc:/usr/j2se/jre/../lib/sparc:/usr/sap/DV2/SYS

/exe/run:/usr/sap/DV2/SYS/exe/runU:/usr/lib:/usr/sap/DV2/DVEBMGS04/j2ee/os_libs:/usr/sap/DV2/DVE

BMGS04/j2ee/os_libs:/usr/sap/DV2/SYS/exe/run:/usr/sap/DV2/SYS/exe/run:/oracle/client/10x_64/inst

antclient
-> exe path = 

PATH=/usr/j2se/bin:/usr/sap/DV2/DVEBMGS04/j2ee/os_libs:/usr/j2se/bin:/oracle/DV2/102_64/bin:/exp

ort/home/dv2adm:/usr/sap/DV2/SYS/exe/run:/usr/bin:.:/usr/ccs/bin:/usr/ucb
**********************************************************************

[Thr  1] JStartupICreateProcess: fork process (pid 23143)
[Thr  1] JControlICheckProcessList: process server1 started (PID:23143)

[Thr  1] Sat Oct  9 04:42:58 2010
[Thr  1] JControlICheckProcessList: process server0 (pid:23019) died (RUN-FLAG)
[Thr  1] JControlIResetProcess: reset process server0
[Thr  1] JControlICheckProcessList: running flight recorder:
        /usr/j2se/bin/java -classpath 

../j2ee/cluster/bootstrap/sap.com~tc~bl~offline_launcher~impl.jar 

com.sap.engine.offline.OfflineToolStart com.sap.engine.flightrecorder.core.Collector 

../j2ee/cluster/bootstrap -node ID44702650 1286617556 -bz /usr/sap/DV2/SYS/global

**********************************************************************
JControlStartJLaunch: program = /usr/sap/DV2/SYS/exe/run/jlaunch
-> arg[00] = /usr/sap/DV2/SYS/exe/run/jlaunch
-> arg[01] = pf=/usr/sap/DV2/SYS/profile/DV2_DVEBMGS04_aeerpdv1
-> arg[02] = -DSAPINFO=DV2_04_server
-> arg[03] = -nodeId=1
-> arg[04] = -file=/usr/sap/DV2/DVEBMGS04/j2ee/cluster/instance.properties
-> arg[05] = -syncSem=352321557
-> arg[06] = -nodeName=ID44702650
-> arg[07] = -jvmOutFile=/usr/sap/DV2/DVEBMGS04/work/jvm_server0.out
-> arg[08] = -jvmOutMode=append
-> arg[09] = -stdOutFile=/usr/sap/DV2/DVEBMGS04/work/std_server0.out
-> arg[10] = -stdOutMode=append
-> arg[11] = -traceMode=append
-> arg[12] = -locOutFile=/usr/sap/DV2/DVEBMGS04/work/dev_server0
-> arg[13] = -mode=JCONTROL
-> arg[14] = pf=/usr/sap/DV2/SYS/profile/DV2_DVEBMGS04_aeerpdv1
-> arg[15] = -DSAPSTART=1
-> arg[16] = -DCONNECT_PORT=58682
-> arg[17] = -DSAPSYSTEM=04
-> arg[18] = -DSAPSYSTEMNAME=DV2
-> arg[19] = -DSAPMYNAME=aeerpdv1_DV2_04
-> arg[20] = -DSAPPROFILE=/usr/sap/DV2/SYS/profile/DV2_DVEBMGS04_aeerpdv1
-> arg[21] = -DFRFC_FALLBACK=ON
-> arg[22] = -DFRFC_FALLBACK_HOST=localhost
-> lib path = 

LD_LIBRARY_PATH=/usr/j2se/jre/lib/sparcv9/server:/usr/j2se/jre/lib/sparcv9:/usr/j2se/jre/../lib/

sparcv9:/usr/sap/DV2/DVEBMGS04/j2ee/os_libs:/usr/sap/DV2/SYS/exe/run:/usr/sap/DV2/SYS/exe/run:/u

sr/j2se/jre/lib/sparc/client:/usr/j2se/jre/lib/sparc:/usr/j2se/jre/../lib/sparc:/usr/sap/DV2/SYS

/exe/run:/usr/sap/DV2/SYS/exe/runU:/usr/lib:/usr/sap/DV2/DVEBMGS04/j2ee/os_libs:/usr/sap/DV2/DVE

BMGS04/j2ee/os_libs:/usr/sap/DV2/SYS/exe/run:/usr/sap/DV2/SYS/exe/run:/oracle/client/10x_64/inst

antclient
-> exe path = 

PATH=/usr/j2se/bin:/usr/sap/DV2/DVEBMGS04/j2ee/os_libs:/usr/j2se/bin:/oracle/DV2/102_64/bin:/exp

ort/home/dv2adm:/usr/sap/DV2/SYS/exe/run:/usr/bin:.:/usr/ccs/bin:/usr/ucb
**********************************************************************

[Thr  1] JStartupICreateProcess: fork process (pid 24667)
[Thr  1] JControlICheckProcessList: process server0 started (PID:24667)

Following are the Heap Size parameters set up:
Dispatcher 		MaxHeapSize=544
Server0 		MaxHeapSize=2048
Server1			MaxHeapSize=2048
Bootstrap instance	MaxHeapSize=128
bootstrap_Dispatcher	MaxHeapSize=544
bootstrap_Server0	MaxHeapSize=512
bootstrap_Server1	MaxHeapSize=512

Thanks,

Antarpreet

Add a comment
10|10000 characters needed characters exceeded

Related questions

3 Answers

  • author's profile photo Former Member
    Former Member
    Posted on Oct 11, 2010 at 07:02 AM

    Hello Antarpreet,

    Have you checked that you have enough applicationm threads to start all the applications? Please, increase them and try again:

    Configtool->Cluster-data-> Global Server configuration->

    ThreadManager/Application Thread Manager -> MaxThreadCount

    If the problem persists, please add the std_server.out error shows when the server fails.

    Kind regards,

    Blanca

    Add a comment
    10|10000 characters needed characters exceeded

    • Former Member

      Hello Blanca,

      Thanks for your Reply.

      The Thread count have been set to the default value of 40. Should I change this value and by what number should I increase this value?

      We have 2 server processes.

      Thanks,

      Antarpreet

  • author's profile photo Former Member
    Former Member
    Posted on Oct 13, 2010 at 06:52 AM

    Hello Antarpreet,

    40 is an small value, please, set it to 150 and try again. Set it in both nodes.

    I hope this helps you.

    Regards,

    Blanca

    Add a comment
    10|10000 characters needed characters exceeded

  • author's profile photo Former Member
    Former Member
    Posted on Oct 13, 2010 at 04:50 PM

    Hello Antarpreet,

    As Blanca suggested increasing the threadcount for ApplicationThreadManager is a first step.

    If you continue to see the issue even after that, you can take threaddumps as per note 710154 on the server process. Usually, we suggest taking 5 threaddumps during the time of issue of hang/slow startup/no response of servers taken at an interval of 1 min each.

    You can get these analyzed by SAP support or you can yourself analyze them using the tool called threaddump viewer as per this article:

    http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/2015bb75-a9da-2910-e0b1-c5ba9fe90012?quicklink=index&overridelayout=true

    Best Regards,

    Snehal/ SAP/Palo Alto(USA)

    Add a comment
    10|10000 characters needed characters exceeded

Before answering

You should only submit an answer when you are proposing a solution to the poster's problem. If you want the poster to clarify the question or provide more information, please leave a comment instead, requesting additional details. When answering, please include specifics, such as step-by-step instructions, context for the solution, and links to useful resources. Also, please make sure that you answer complies with our Rules of Engagement.
You must be Logged in to submit an answer.

Up to 10 attachments (including images) can be used with a maximum of 1.0 MB each and 10.5 MB total.