Kernel Upgrade on Solaris Cluster Failed

[Former Member]

Dear Experts,

We have our ECC6 - PRD server as:

CI - node1

DB - node2

[node1 & node2 are in cluster]

application server - node3

I upgraded the kernel from 83 to 206. Ran saproot.sh script successfully.

On command prompt, under <sid>adm user, I ran disp+work & got the kernel level as 206 successfully.

Now when we tried to start the cluster, all oracle Resource Groups were started successfully.

ciCEP-HA-RS -> started successfully

ciCEP-enq-RS -> started successfully

ciCEP-msg-RS -> start up failed

ciCEP-msg-j2ee-RS -> start up failed

Finally our SAP was not able to start. We reverted back to the old kernel 83 & the cluster got started successfully.

Can anyone please throw some light on this issue. is there any special measures for kernel upgrade in cluster environment?

Regards,

Ashish.

[markus_doehr2 · Accepted solution]

> Finally our SAP was not able to start. We reverted back to the old kernel 83 & the cluster got started successfully.

If a system is not coming up the first thing to check is

dev_disp

dev_w0

You can still check them as dev_disp.old or dev_w0.old

Markus

[Former Member · Accepted solution]

Dear Markus,

Thanks for your input, but I have already checked these logs.

dev_disp log contains entries of SAP startup only with kernel level 83. It has no entry with kernel level 206.

Also dev_disp.old does not have anything related to kernel 206.

dev_w0 - also contains no entries for kernel 206.

Is it so that these logs capture kernel entries only on successfull SAP startup & they do not capture half way startup of SAP with the new kernel?

Please advice on this.

Regards,

Ashish.

[markus_doehr2 · Accepted solution]

> dev_disp log contains entries of SAP startup only with kernel level 83. It has no entry with kernel level 206.

> Also dev_disp.old does not have anything related to kernel 206.

And exactly that may be your problem.

Where did you unpack the new kernel?

Markus

[Former Member · Accepted solution]

Dear Markus,

First I unpacked the new kernel to /usr/sap/CEP/DVEBMGS01/exe, run saproot.sh & then run disp+work which returned the old kernel level 83.

Then again I unpacked the new kernel to /sapmnt/CEP/exe, run saproot.sh & then run disp+work which returned the new kernel level 206.

After this we tried to start the cluster.

Regards,

Ashish.

[markus_doehr2 · Accepted solution]

Please check

Note 1104735 - Upgrade to new instance-specific directory on UNIX

You may have "confused" the system by copying the kernel to the wrong directory.

Markus

[Former Member · Accepted solution]

Dear Markus,

We have all our parameters set exactly as mentioned in the Note 1104735, specified by you.

The steps which I performed are:

1. Stop Application server.

2. Bring down the cluster.

3. Copied the new kernel 206 in /usr/sap/CEP/DVEBMGS01/exe directory.

4. Run saproot.sh

5. Run disp+work, got kernel level as 83.

6. Copied the new kernel 206 in /sapmnt/CEP/exe directory.

7. Run saproot.sh

8. Run disp+work, got kernel level as 206.

9. Start the cluster, cluster failed to start.

10. Restored the old kernel 83 to /sapmnt/CEP/exe , but I did not restored the kernel 83 to /usr/sap/CEP/DVEBMGS01/exe directory.

11. Run disp+work, got kernel level as 83.

12. Start the cluster & SAP started successfully.

Now, as per the Note 1104735, it says that the sapcpe program copies the Kernel from central directory to other instance directories. That is why, now the file modification dates that I see in the /usr/sap/CEP/DVEBMGS01/exe directory are of 2007 & not of the 26 July 2009, when I copied the new Kernel to /usr/sap/CEP/DVEBMGS01/exe directory.

This means that sapcpe program did copied the old kernel 83 from /sapmnt/CEP/exe directory to /usr/sap/CEP/DVEBMGS01/exe directory. Now the system is in its original state.

Dear Markus - Please suggest now, how do we proceed again to upgrade the Kernel successfully this time. The client is very much concerned about the downtime & provides very limited downtime. In my first attempt, I failed to upgrade the Kernel, I do not want to fail the 2nd time. Please suggest all Do's & Dont's for the Kernel upgrade process in Cluster environments.

Our Platform is Sun Solaris 5.10 / Oracle 10.2.0.2 / Sun Cluster 3.1 / ECC6.0

Looking forward for your kind help.

Regards,

Ashish.

[markus_doehr2 · Accepted solution]

> 3. Copied the new kernel 206 in /usr/sap/CEP/DVEBMGS01/exe directory.

That is the runtime directory, don't copy there!

> 4. Run saproot.sh

> 5. Run disp+work, got kernel level as 83.

> 6. Copied the new kernel 206 in /sapmnt/CEP/exe directory.

> 7. Run saproot.sh

> 8. Run disp+work, got kernel level as 206.

You have to execute the shell command "rehash" (if you shell is /bin/csh) to make sure, the system is not using a cached version/path of the files.

> 9. Start the cluster, cluster failed to start.

> Now, as per the Note 1104735, it says that the sapcpe program copies the Kernel from central directory to other instance directories. That is why, now the file modification dates that I see in the /usr/sap/CEP/DVEBMGS01/exe directory are of 2007 & not of the 26 July 2009, when I copied the new Kernel to /usr/sap/CEP/DVEBMGS01/exe directory.

The problem seems to be, that the system tried to execute cleanipc of the old kernel by loading the shared libraries from the new kernel in the runtime.

> Dear Markus - Please suggest now, how do we proceed again to upgrade the Kernel successfully this time. The client is very much concerned about the downtime & provides very limited downtime. In my first attempt, I failed to upgrade the Kernel, I do not want to fail the 2nd time. Please suggest all Do's & Dont's for the Kernel upgrade process in Cluster environments.

There's not much difference between updating a kernel in a cluster or in a non-cluster environment.

I would do:

- stopsap r3 (which keeps the database running)

- saposcol -k

- ps -ef | grep sap (check for running programs e. g. sapstartsrv, sapxpg, sapevt etc. - kill them)

- as user root: cd /sapmnt/SID && chown -Rf <sid>adm:sapsys exe

- as user <sid>adm: unpack all in /sapmnt/<SID>/exe (SAPCAR -xvf SAPEXE, SAPEXEDB, DBATL, IGSEXE*)

- download all patches that are newer (by date) as the SAPEXE* you downloaded (e. g saposcol*.SAR)

- SAPCAR -xvf <all-other-patches>

- su root (without dash)

- saproot.sh <SID>

- startsap

it's important to check if there are programs from the kernel directory running (e. g. sapstartsrv, which is started by /etc/init.d/sapinit) and saposcol, otherwise they may not be correctly overwritten and you still have old kernel parts running while the binaries in the directory are new.

To minimize downtime you can copy the actual kernel in /sapmnt/<SID>/exe to /sapmnt/<SID>/exenew, unpack everything as above and then just rename the directory, run saproot.sh and restart the system.

Markus

[Former Member · Accepted solution]

Dear Markus,

Thank you very much for your kind help. I will follow the way you have mentioned. I will try this on next downtime.

Regards,

Ashish.

[Former Member · Accepted solution]

Dear Markus,

We are planning to do kernel upgrade today night.

One last doubt that I have in my mind is that do I need to upgrade the Kernel of CI & Application separately:

For CI -> /sapmnt/CEP/exe

For Application server -> /usr/sap/CEP/D00/exe

Our CI has /usr/sap as a local mount point & /sapmnt/CEP as a global file system which is shared through CI.

Markus, please suggest, shall I upgrade kernel in /sapmnt/CEP/exe alone or in /usr/sap/CEP/D00/exe also?

Regards,

Ashish

[markus_doehr2 · Accepted solution]

> Our CI has /usr/sap as a local mount point & /sapmnt/CEP as a global file system which is shared through CI.

Good!

> Markus, please suggest, shall I upgrade kernel in /sapmnt/CEP/exe alone or in /usr/sap/CEP/D00/exe also?

I'd just update /sapmnt/CEP/exe.

Markus

[Former Member · Accepted solution]

Check your database client, whether its supported new Kernel or not.

Ravi Saluja

[Former Member · Accepted solution]

Dear Markus,

Yes, the kernel upgrade was successful this time. I only upgraded the /sapmnt/CEP/exe directory.

But the kernel upgrade still didnt worked until we restored the startsap script from the old kernel. I got to know from one of my friend that maybe the guy who had implemented ECC6 here might have changed the startsap script according to the cluster setup here.

After restoring the old startsap script our SAP cluster was up & running successfully.

Thank you very much Markus for your kind & expert support.

Regards,

Ashish.

[Former Member · Accepted solution]

Nothing to do with startsap script.

If you followed above.

Cluster script will call all new executables to start sap.

Dharmendra