Startup of Instance Service failed

[former_member280828]

Hi Experts,

Our Linux team told us that our "/ (Slash)" mount point of SOLMAN got 100% utilization so we told them to clear something and make the space available and we stopped the SAP and DB on SOLMAN machine. So they freed some space and after that when I am trying to start SAP instance of SOLMAN server but it is showing "Startup of Instance Service failed"  and the logs shows as below.

SOLMAN:smdadm 48> startsap

Checking SMD Database

Database is running

-------------------------------------------

Starting Startup Agent sapstartsrv

FAIL: Timeout

Startup of Instance Service failed

See /home/smdadm/startsap_SCS01.log for details

-------------------------------------------

SOLMAN:smdadm 50> cat /home/smdadm/startsap_SCS01.log

Trace of system startup/check of SAP System SMD on Wed Jan 20 16:59:04 IST 2016

Called command: /usr/sap/SMD/SYS/exe/run/startsap start

-------------------------------------------

16:59:05

Starting Startup Agent sapstartsrv

-------------------------------------------

17:01:05

Startup of Instance Service failed

-------------------------------------------

17:01:05

See /home/smdadm/startsap_SCS01.log for details

Please suggest me what to do on this situation

Thanks in advanced.

Regards,

Musa.

[feng_shi · Accepted solution]

Hi,

The problem should be the lock file permission.

please check the permission of the following hidden file:

/tmp/.sapstartsrv01_sapstartsrv.log

the default/correct permission should be "-rw-rw-rw- 1 <sid>adm  sapsys" .

other permissions will cause "Unable to open trace file sapstartsrv.log. (Error 13 Permission denied) " error when you run

"sapstartsrv pf=/sapmnt/SMD/profile/START_SCS01_SOLMAN" command.

you can just delete /tmp/.sapstartsrv01_sapstartsrv.log file and it will be generated again when sapstartsrv service start again.

Thanks and best regards,

Shi Feng

[former_member280828 · Accepted solution]

Hey Feng,

I just saw your suggestion and found that the permission of sapstartsrv00_sapstartsrv.log and sapstartsrv01_sapstartsrv.log is root root

So now i am trying to delete it and will start the instance once again and will let you know the status very soon.

Thanks for your suggestion.

Regards,

Musaddik.

[former_member280828]

SOLUTION

Hey Feng,

Thanks a lot for your suggestion it solved my issue.

I deleted old log files below

sapstartsrv00_sapstartsrv.log

sapstartsrv01_sapstartsrv.log

which had root root permission and then gave command startsap.

The result is below.

SOLMAN:smdadm 45> startsap

Checking SMD Database

Database is not available via R3trans

-------------------------------------------

Trying to start SMD database ...

Log file: /home/smdadm/startdb.log

SMD database started

Starting Startup Agent sapstartsrv

OK

Instance Service on host SOLMAN started

-------------------------------------------

starting SAP Instance SCS01

Startup-Log is written to /home/smdadm/startsap_SCS01.log

-------------------------------------------

/usr/sap/SMD/SCS01/exe/sapcontrol -prot NI_HTTP -nr 01 -function Start

Instance on host SOLMAN started

Starting Startup Agent sapstartsrv

OK

Instance Service on host SOLMAN started

-------------------------------------------

starting SAP Instance DVEBMGS00

Startup-Log is written to /home/smdadm/startsap_DVEBMGS00.log

-------------------------------------------

/usr/sap/SMD/DVEBMGS00/exe/sapcontrol -prot NI_HTTP -nr 00 -function Start

Instance on host SOLMAN started

Thanks Feng and everybody who gave their valuable thoughts on my problem.

May God Bless everybody.

Regards,

Musaddik.

[feng_shi]

Hi,

I'm glad to hear that the issue has been solved by my solution.

Please mark this thread as "Answered".

Thanks and best regards,

Shi Feng

[former_member280828]

Hey Feng,

Yes I marked this thread as Answered,

Thanks and regards,

Musaddik.

[feng_shi]

Hi,

By the way, /tmp/.sapstartsrv01_sapstartsrv.log is the lock file for sapstartsrv,  you must not delete it if there is sapstartsrv process already running, otherwise, there will be more than one  sapstartsrv can be started.

Best regards,

Shi Feng

[former_member280828]

SOLUTION

Hey Feng,

Thanks a lot for your suggestion it solved my issue.

I deleted old log files below

sapstartsrv00_sapstartsrv.log

sapstartsrv01_sapstartsrv.log

which had root root permission and then gave command startsap.

The result is below.

SOLMAN:smdadm 45> startsap

Checking SMD Database

Database is not available via R3trans

-------------------------------------------

Trying to start SMD database ...

Log file: /home/smdadm/startdb.log

SMD database started

Starting Startup Agent sapstartsrv

OK

Instance Service on host SOLMAN started

-------------------------------------------

starting SAP Instance SCS01

Startup-Log is written to /home/smdadm/startsap_SCS01.log

-------------------------------------------

/usr/sap/SMD/SCS01/exe/sapcontrol -prot NI_HTTP -nr 01 -function Start

Instance on host SOLMAN started

Starting Startup Agent sapstartsrv

OK

Instance Service on host SOLMAN started

-------------------------------------------

starting SAP Instance DVEBMGS00

Startup-Log is written to /home/smdadm/startsap_DVEBMGS00.log

-------------------------------------------

/usr/sap/SMD/DVEBMGS00/exe/sapcontrol -prot NI_HTTP -nr 00 -function Start

Instance on host SOLMAN started

Thanks Feng and everybody who gave their valuable thoughts on my problem.

May God Bless everybody.

Regards,

Musaddik..

[former_member185239]

Dear RBK B,

I think there is a permission issue some where.

Just run the below command

sapstartsrv pf=<give the instance profile of SCS>

Paste the error here.

With Regards

Ashutosh Chaturvedi

[former_member280828]

Hey Ashutosh,

see the output

SOLMAN:smdadm 10> sapstartsrv pf=/sapmnt/SMD/profile/START_SCS01_SOLMAN

Unable to open trace file sapstartsrv.log. (Error 13 Permission denied) [ntservsserver.cpp 3483]

[former_member185239]

Hi Rbk,

The error is simple , it is not able to open the sapstartsrv.log file.

You need to assign the permission.

Follow the below steps.

- goto the location of sapstartsrv.log file

- Assign the correct permission on the folder where sapstartsrv reside.

With Regards

Ashutosh Chaturvedi

[former_member280828]

Hey Ashutosh,

Please see the rights and permission of the sapstartsrv.log file.

SOLMAN:smdadm 32> ls -l /usr/sap/SMD/DVEBMGS00/work/sapstartsrv.log

-rw-r--r-- 1 smdadm sapsys 681 Jan  8 11:37 /usr/sap/SMD/DVEBMGS00/work/sapstartsrv.log

SOLMAN:smdadm 33>  ls -l /usr/sap/SMD/SCS01/work/sapstartsrv.log

-rw-r--r-- 1 smdadm sapsys 673 Jan  8 11:37 /usr/sap/SMD/SCS01/work/sapstartsrv.log

Regards,

Musa

[former_member185239]

Dear Musa,

File was last updated on 8th of January.

Can you check the log file in home directory of smdadm.

Try to give the file 777 permission.

With Regards

Ashutosh Chaturvedi

[former_member280828]

Hey Ashutosh,

On which files should I give 777 permission ???

Regards,

Musa.

[former_member185239]

Hi Musa,

Kindly follow the below steps

- Goto the home/smdadm directory

- Check the permission of smdadm folder

- check is there any sapstartsrv.log file is there or not.

if it is there , then check the permission of the folder and file.

With Regards

Ashutosh Chaturvedi

[former_member280828]

Dear Ashutosh,

smdadm.sapsys is the permission for smdadm folder.

There is no any file name sapstartsrv.log in smdadm directory

regards,

Musa.

[former_member185239]

Hi Rbk,

Just run the command

touch text.txt in  /home/smdadm folder.

Just try to give full permission 777 to sapstartsrv.log file in SCS01/work directory.

With Regards

Ashutosh Chaturvedi

[isaias_freitas]

Hello,

SOLMAN:smdadm 10> sapstartsrv pf=/sapmnt/SMD/profile/START_SCS01_SOLMAN

Unable to open trace file sapstartsrv.log. (Error 13 Permission denied) [ntservsserver.cpp 3483]

What are the permissions of the /tmp folder?

The command "ls -lad /tmp" should return this.

If this has been changed, reset it to the default (usually "drwxrwxrwt").

Regards,

Isaías

[former_member280828]

Hey Ahsutosh,

Giving permission of 777 to sapstartsrv.log file in SCS01/work directory is causing no effect still the system is not starting.

Regards,

Musa.

[former_member185239]

Hi Musa,

has given the correct suggestion.

Kindly check the sapnote

1762827 - "Startup of Instance Service failed" in UNIX or Linux environment

With Regards

Ashutosh Chaturvedi

[former_member280828]

Hey Isaias,

please look the command output

SOLMAN:smdadm 1> ls -lad /tmp

drwxrwxrwt 13 root root 4096 Jan 21 18:31 /tmp

[isaias_freitas]

Hello,

That looks correct.

Please attach the following profiles to this thread.

• DEFAULT.PFL
• SMD_SCS01_SOLMAN
• START_SCS01_SOLMAN

There is an option for you to attach the files themselves, instead of having to copy/paste their content to the thread.

Regards,

Isaías

[former_member280828]

Hey Isaias,

I have uploaded 3 pictures of profiles please have a look.

Regards,

Musa.

[isaias_freitas]

Hello Musa,

The profiles look ok.

Can you please check the permissions of the "work" folder and of the existing sapstartsrv log files?

ls -lad /usr/sap/SMD/SCS01/work

ls -la /usr/sap/SMD/SCS01/work/sapstartsrv.*

Regards,

Isaías

[former_member230159]

please attach sapstartsrv.log , stderr* logs , startsap.trc here.

best regards,

Manjunath Hanmantgad

[former_member280828]

Dear Manjunath,

Please take a look at "sapstartsrv.log"

SOLMAN:smdadm 158> cat sapstartsrv.log

---------------------------------------------------

trc file: "sapstartsrv.log", trc level: 0, release: "720"

---------------------------------------------------

pid        7224

[Thr 47483764347440] Fri Jan  8 11:37:20 2016

No halib defined => HA support disabled

CCMS agent initialization for instance type DOUBLESTACK: return code 0.

CCMS agent start: return code 255.

Initializing SAPControl Webservice

Starting AutoRestart thread

SapSSLInit failed => https support disabled

Starting WebService thread

[Thr 1095481664] Fri Jan  8 11:37:22 2016

AutoRestart thread started

Webservice thread started, listening on port 50013

soap_bind failed, Webservice thread enters retry loop

[feng_shi]

Hi,

---------

Webservice thread started, listening on port 50013

soap_bind failed, Webservice thread enters retry loop

---------

==> Which indicate that sapstartsrv does not work correctly due to the fact that port 50013 is occupied by other process.

Please see 2208264 sap KBA for the root cause and solution.

2208264 - Sapstartsrv - Soap_bind failed

Best regards,

Shi Feng

[former_member280828]

Hey Feng Shi,

I have checked the status of 50013 port via netstat but the port is free and open.

Thanks for your valuable reply.

Regards,

Musa.

[feng_shi]

Hi,

-------------------

Webservice thread started, listening on port 50013

soap_bind failed, Webservice thread enters retry loop

-------------------

this trace log indeed indicates that port 50013 was occupied by other process at that moment.

if you are sure that 50013 port is free and open now, I think you can try to start the system again.

Meanwhile, I just review all the history.

As far as I can see, the problem is that sapstartsrv of SCS01 instance could not be started.

however, as you mentioned, the sapstartsrv.log is as below:

=======================

... ...

Webservice thread started, listening on port 50013

soap_bind failed, Webservice thread enters retry loop

=======================

Normally, the HTTP port of sapstartsrv on SCS01 instance should be 50113, but not 50013, since SCS instance number is 01.

where did you get the sapstartsrv.log file ?

if you got it from /usr/sap/<SID>/SCS01/work/, maybe there are some problem of sapstartsrv service definition or SCS01 instance profile.

if system still can not be started with the same error, please also upload :

1)the result of command: ps -ef | grep sapstartsrv

2)all the profiles which under /sapmnt/<SID>/profile directory

Best regards,

Shi Feng

[former_member280828]

Hi Feng,

Again it failed. see

startsap

Checking SMD Database

Database is running

-------------------------------------------

Starting Startup Agent sapstartsrv

FAIL: Timeout

Startup of Instance Service failed

See /home/smdadm/startsap_SCS01.log for details

output of command ps -ef |grepsapstartsrv is

SOLMAN:smdadm 2> ps -ef |grep sapstartsrv

smdadm   26911 26634  0 13:46 pts/0    00:00:00 grep sapstartsrv

[former_member280828]

Feng,

Now which profile of the below your want

-rw-rw---- 1 smdadm sapsys 2297 Oct 18  2013 DEFAULT.BAK

-rwxrwxrwx 1 smdadm sapsys 2412 Oct 18  2013 DEFAULT.PFL

-rwxrwxrwx 1 smdadm sapsys 3544 Jul  9  2014 SMD_DVEBMGS00_SOLMAN

-rwxrwxrwx 1 smdadm sapsys 1395 Sep 27  2013 SMD_DVEBMGS00_SOLMAN.1

-rwxrwxrwx 1 smdadm sapsys 1453 Sep 27  2013 SMD_DVEBMGS00_SOLMAN.2

-rw-rw---- 1 smdadm sapsys 3194 Jul  9  2014 SMD_DVEBMGS00_SOLMAN.BAK

-rwxrwxrwx 1 smdadm sapsys  950 Oct  9  2013 SMD_SCS01_SOLMAN

-rwxrwxrwx 1 smdadm sapsys 3530 Oct 18  2013 START_DVEBMGS00_SOLMAN

-rwxrwxrwx 1 smdadm sapsys 1631 Sep 27  2013 START_SCS01_SOLMAN

[Former Member]

Hello Rbk,

Please execute command sapcontrol -function StopService -nr 00 with SIDADM.

Confirm whether its stopped or not. If not kill the process and restart the system again.

Regards

Anand

[former_member280828]

Hi Anand,

i executed he Command which you have suggested and the output is as below.

SOLMAN:smdadm 56> sapcontrol -function StopService -nr 00

20.01.2016 17:50:46

StopService

FAIL: NIECONN_REFUSED (Connection refused), NiRawConnect failed in plugin_fopen()

SOLMAN:smdadm 57>

[Former Member]

Rbk,

execute stopsap...so that you can bring down DB also.

after check ps -ef|grep sap or ps -ef|grep SMD.

if any process in running such as refering SMD (SID of the system) kill the process and try again.

Regards

Anand

[JPReyes]

Hi Rbk,

Sounds like the crash has left processes/ports been hugged by orphaned PID's.

I think you should do a clean restart of the host and try to start it again.

Also, you should be able to find more information of the developer traces.

Regards, Juan

[former_member280828]

Hi Juan,

Should i restart the server and then try starting the SAP???

Thanks for your valuable reply Juan,

Regards,

Musa.

[JPReyes]

Hi Musa,

That is certainly one of the options, the other one as suggested above is to look for the hanging processes and kill them manually.

Regards, Juan

[former_member280828]

Hi Juan,

I tried executing all the above suggestions and all of them help me understanding the issue but didnt found any exact solution. Still i am trying to rectify the issue.

I will try to restart the server as you said.

Regards,

Musa.

[isaias_freitas]

Hello Musa,

Maybe this WIKI page can help you.

SAP Startup Troubleshooting Guide for Netweaver Application Server - Application Server Infrastructu...

Regards,

Isaías

[former_member185239]

Hi Rbk B,

Just run the command

sapstartsrv pf=<location of start profile of SCS >

You will receive the exact message which is causing the TIMEOUT issue.

With Regards

Ashutosh Chaturvedi

[former_member280828]

Hi Ashutosh,

I tried using your command and found the location of .log and .trc files but when i tried to open then it is showing permission error.

Thanks for your answer.

[former_member185239]

Hi Rbk B,

Can you paste the screenshot of the error which you are getting on running the command as suggested.

So there will be a permission issue. So correct the permission and then try to start the system

With Regards

Ashutosh Chaturvedi

[Sriram2009]

Hi Musa.

1.  Could you share the log /startsap_SCS01.log

2. Refer the SAP KBA's 1762827  1766358 for startup issue -

Regards

[former_member280828]

Thanks  Sriram for your valuable reply

Q . Could you share the log /startsap_SCS01.log ??

Ans yes please see below.

SOLMAN:smdadm 50> cat /home/smdadm/startsap_SCS01.log

Trace of system startup/check of SAP System SMD on Wed Jan 20 16:59:04 IST 2016

Called command: /usr/sap/SMD/SYS/exe/run/startsap start

-------------------------------------------

16:59:05

Starting Startup Agent sapstartsrv

-------------------------------------------

17:01:05

Startup of Instance Service failed

-------------------------------------------

17:01:05

See /home/smdadm/startsap_SCS01.log for details