multiple instances' sapstartsrv processes

[benoit-schmid]

Hello,

After restarting SAP I regularly have multiple sapstartsrv for a single instance running:

# ps -ef | grep -i /usr/sap/SID/DVEBMGS00/exe/sapstartsrv | grep -v grep
sidadm   12959     1  0 Jun10 ?        00:02:10 /usr/sap/SID/DVEBMGS00/exe/sapstartsrv pf=/usr/sap/SID/SYS/profile/START_DVEBMGS00_host -D
sidadm   39648     1  0 Jun21 ?        00:00:05 /usr/sap/SID/DVEBMGS00/exe/sapstartsrv pf=/usr/sap/SID/SYS/profile/START_DVEBMGS00_host -D

1. Is it a bug or a classical behaviour?

2. What are the consequences of having these duplicated processes?

Basically should I care about it?

Thanks in advance for your answer.

[benoit-schmid · Accepted solution]

Hello,

BTW, I have learned that tmpwatch is activated by default on Redhat.

It cleans /tmp after 10d from wht I see:

# cat /etc/cron.daily/tmpwatch
#! /bin/sh
flags=-umc
/usr/sbin/tmpwatch "$flags" -x /tmp/.X11-unix -x /tmp/.XIM-unix \
    -x /tmp/.font-unix -x /tmp/.ICE-unix -x /tmp/.Test-unix \
    -X '/tmp/hsperfdata_*' 10d /tmp
/usr/sbin/tmpwatch "$flags" 30d /var/tmp
for d in /var/{cache/man,catman}/{cat?,X11R6/cat?,local/cat?}; do
    if [ -d "$d" ]; then
    /usr/sbin/tmpwatch "$flags" -f 30d "$d"
    fi
done

I guess this is the root cause of my duplicated process.

For the one that use Redhat, do you deactivate tmpwatch by commenting the lines in etc/cron.daily/tmpwatch?

Regards,

[divyanshu_srivastava3 · Accepted solution]

Hi Schmid,

It looks like this was the cause of multiple process. However, rather than disabling, why not create a copy of this script, custom version to clean tmp without removing any sap files.

Regards,

Divyanshu

[Former Member · Accepted solution]

For the one that use Redhat, do you deactivate tmpwatch by commenting the lines in etc/cron.daily/tmpwatch?

This can be done, but before that you have to check any impact if you disable the tmpwatch job. Because it deactivates entire cleanups.

Regards,

Nick Loy

[benoit-schmid · Accepted solution]

Hello,

Divyanshu Srivastava wrote:

It looks like this was the cause of multiple process. However, rather than disabling, why not create a copy of this script, custom version to clean tmp without removing any sap files.

I have have find out .sap* and jstartup_*.

What are the other "sap files"?

Regards,

[benoit-schmid · Accepted solution]

Hello,

Nick Loy wrote:

This can be done, but before that you have to check any impact if you disable the tmpwatch job. Because it deactivates entire cleanups.

On Solaris, I had not see big fs usage of /tmp.

Is it different with Redhat?

Regards,

[divyanshu_srivastava3 · Accepted solution]

That's all I believe, just take care of .sap* files, as tmp is a non-sap space and sap won't make any files on such location without using 'sap' as a key word in file name.

Also, by saying 'any sap file' I meant sapstartsrv lock files .

Regards,

Divyanshu

[benoit-schmid · Accepted solution]

Hello,

Divyanshu Srivastava wrote:

That's all I believe, just take care of .sap* files, as tmp is a non-sap space and sap won't make any files on such location without using 'sap' as a key word in file name.

Also, by saying 'any sap file' I meant sapstartsrv lock files .

What are the side effects of deleteing the .jstartup_*.fifo?

Regards,

[divyanshu_srivastava3 · Accepted solution]

Hi Schmid,

Check 1551881. It tells about this file.

This file is created by startup framework for named pipes

Regards,

Divyanshu

[divyanshu_srivastava3 · Accepted solution]

So, these files should be removed. However, not at the time of java restart.

Cheers,

Divyanshu

[divyanshu_srivastava3 · Accepted solution]

Hi Schmid,

If you got your replies then you can mark and close the thread .

Regards.

Divyanshu

[benoit-schmid · Accepted solution]

Hello,

Divyanshu Srivastava wrote:

Hi Schmid,

If you got your replies then you can mark and close the thread .

Regards.
Divyanshu

Are you sure that SAP does not need others files sap* and jstartup_* in /tmp?

Basically, I can blindly delete them and there is no side effect for SAP.

Only if it is the case, I can close the thread.

regards,

[divyanshu_srivastava3 · Accepted solution]

Please give read/write access to /tmp file system to <sid>adm user, it will take care own its own all temp files.

When your will stop SAP and execute cleanipc command successfully with <sidadm> all files which are not required should be removed. For java fifo files, as said before, they can be removed any time but not at system start up, same applies for other files generated at the time of startup.

Regards,

Divyanshu

[benoit-schmid · Accepted solution]

Hello,

Divyanshu Srivastava wrote:

Please give read/write access to /tmp file system to <sid>adm user, it will take care own its own all temp files.

When your will stop SAP and execute cleanipc command successfully with <sidadm> all files which are not required should be removed. For java fifo files, as said before, they can be removed any time but not at system start up, same applies for other files generated at the time of startup.

Regards,
Divyanshu

The problem is not that /tmp is not available.

The problem is due to the fact that Redhat delete sap files.

On this thread, I have been advised to add filters so that the files created and used by sap

are not deleted.

Therefore wht I need to know is the following:

Are you sure that SAP does not need others files sap* and jstartup_* in /tmp?

Basically, I can blindly delete them and there is no side effect for SAP.

Only if it is the case, I can close the thread.

Regards,

[divyanshu_srivastava3 · Accepted solution]

No, you cannot blindly delete these files. Only when the system is down and cleanipc is not able to clean, you can delete them, it's that time when SAP don't need them.

Regards,

Divyanshu

[benoit-schmid · Accepted solution]

Divyanshu Srivastava wrote:

No, you cannot blindly delete these files. Only when the system is down and cleanipc is not able to clean, you can delete them, it's that time when SAP don't need them.

Regards,
Divyanshu

I know that I can not blindly delete them.

But what I would like to know is

"Are you sure that SAP does not need others files sap* and jstartup_* in /tmp?"

In other words, what are the other files that should not be blindly deleted 🙂

Is it clear?

Regards,

[divyanshu_srivastava3 · Accepted solution]

Hi Schmid,

I am not sure what all other files can be deleted or not. All I can tell you is the application which generated the files in /tmp directory should be using this directory is a temporary storage of short lived files. Till that application is UP and running, these files should bot be removed. And yes, once the system is not running, these files should not exists.

Regards,

Divyanshu

[Former Member]

Hi,

Can you post ps -ef|grep sapstartsrv output?

Regards,

Nick Loy

[benoit-schmid]

Hello,

Nick Loy wrote:

Hi,

Can you post ps -ef|grep sapstartsrv output?



Regards,
Nick Loy

I have killed the second instance for instance 00:

sidadm   11433     1  0 Jun10 ?        00:01:24 /usr/sap/SID/SCS01/exe/sapstartsrv pf=/usr/sap/SID/SYS/profile/START_SCS01_hostnameX -D
sapadm   14324     1  0 Jun10 ?        00:00:46 /usr/sap/hostctrl/exe/sapstartsrv pf=/usr/sap/hostctrl/exe/host_profile -D
sidadm   38117     1  0 Jun21 ?        00:00:04 /usr/sap/SID/SCS01/exe/sapstartsrv pf=/usr/sap/SID/SYS/profile/START_SCS01_hostnameX -D
sidadm   39648     1  0 Jun21 ?        00:00:05 /usr/sap/SID/DVEBMGS00/exe/sapstartsrv pf=/usr/sap/SID/SYS/profile/START_DVEBMGS00_hostnameX -D
root     42955 41556  0 10:35 pts/1    00:00:00 grep sapstartsrv

Regards,

[Former Member]

I have killed the second instance for instance 00:

But there is no difference between initial output and the latest one.

Still it shows 4 entries (there should be 2)!

Have your tried to restart the entire system to clean up the hung processes?

Regards,

Nick Loy

[benoit-schmid]

Hello,

Nick Loy wrote:
 
But there is no difference between initial output and the latest one.
Still it shows 4 entries (there should be 2)!

Have your tried to restart the entire system to clean up the hung processes?

No there is a difference because it is not duplicated for DVEBMGS00.

By restaring, which command exactly would you expect me to perform?

Regards.

[Former Member]

Yes,

You are right, seems SCS01 has 2 processes.

By restaring, which command exactly would you expect me to perform?

I am talking about reboot of the server (If possible).

Regards,

Nick Loy

[benoit-schmid]

Hello,

Nick Loy wrote:

I am talking about reboot of the server (If possible).

I can tell you the result without rebooting it.

Then I would have a single sapstartsrv.

But I do not see why it would be helpfull to reboot.

Regards.

[divyanshu_srivastava3]

Hi Schmid,

sapstartsrv is a process responsible for starting an instance. If you have multiple process running of sapstartsrv with same profile/user, if all is technically fine, the problem might be the way you are re-starting your system. Perhaps, the shared memory and semaphores were still latched when you have restated the system.

And old orphaned process generally has no references to harm your current process. However, keep in mind, the process can cause problems at times.

sapstartsrv uses locking OS locking mechanism on UNIX. If you go to /tmp file system, you can see a lockfile of this process .sapstartsrv<XX>_sapstartsrv.log. So make sure that no one is deleting these lock files.

Regards,

Divyanshu

[benoit-schmid]

Hello,

Divyanshu Srivastava wrote:

So make sure that no one is deleting these lock files.

What is the problem with not deleting the lock file?

Regards,

[divyanshu_srivastava3]

Hi Schmid,

These files will not allow any other process of sapstarsrv for same instance to start on the operating system level automatically. Thus, putting a constraint on multiple sapstartsrv process for a same instance.

Regards,
Divyanshu

[benoit-schmid]

Hello,

Divyanshu Srivastava wrote:

These files will not allow any other process of sapstarsrv for same instance to start on the operating system level automatically. Thus, putting a constraint on multiple sapstartsrv process for a same instance.

Is it correct to say that the most probable source of the problem is a batch job of redhat that cleans

/tmp?

Regards,

[divyanshu_srivastava3]

Not sure if any house keeping OS job kills these files in your case but yes, it could be of the prime reasons why this happens on your OS. Proper access control and groups assignment can prevent such things.

Regards,

Divyanshu

[Former Member]

Dear ,

This is not correctly performed stopsap script. I think you should use the following command after the stopsap:

kill -9 | <sid>adm

cleanipc 00 remove

if you have central services instance

cleanipc <number CS> remove

ipcs | grep <sid>adm

if any exist, remove it use ipcrm command.

With best regards,

Alexander

[ashish_vikas]

I see in your ps -ef, one process is active since 10 June and another one from 21 june.

Basically, when you stop sap with stopsap command, sapstartsrv do not die. So I kill them manually.

Also check,

1516245 - sapstartsrv: several processes might be started on UNIX

Hope it helps.

best regards

ashish