SAP only starts "half", no access possible

[Former Member]

Hi all

I did a migration of three SAP instances to another server where two other SAP instances were already running. (SAP 4.6B, HP Unix 11.23)

The migration itself went fine up until the moment where I tried to access the SAP instance that was last migrated via SAP Client:

"WSAECONNREFUSED (10061) Connection refused".

The corresponding SAP instance started fine, no problems in the start log. DB therefore was also running fine.

What I discovered is, that some of the TCP ports of that SAP instance where not "open" (meaning, nothing behind them handling requests...):

PORT STATE SERVICE

3201/tcp closed unknown

3301/tcp closed unknown

3601/tcp open unknown

4801/tcp closed unknown

So, checking with ps -ef to see where the corresponding processes have gone, I only saw a few of the SAP processes that normally are running: ms.sapSID, se.sapSID, co.sapSID, /usr/sap/SID/SYS/exe/run/sapstart and 8 "ora_" processes. The list of processes of the other SAP instances fill more than one screen...

What I also discovered is the fact that when I stop the half started SAP instance, then stop one of the other properly running SAP instances, start the one that was "half" started again and AFTER that the one that run properly previously, then the situation has turned around:

- The previously fine running SAP instance, that now was started last, is not reachable, ports colsed & processes are missing.

- The previously "half" started SAP instance is now running fine.

That leads to the conclusion, that it is not the SAP instance itself that is "wrong". It has to be a shared resource that runs out of capacity during the fifth start of a SAP instances, regardless witch of the five. So when the fifth SAP instance comes up, it can not start completely as not enough of this resource is left over for it.

But what is this resource? And how can I change the consumption of it?

And again: Startlog does not tell me anything and after starting I got the message: "Instance on host ... started" No complains, no errors, no whatsoever....

Any ideas welcome!

Thanks!

[Former Member · Accepted solution]

First thing that comes to my mind:

Are system numbers different for your SAP instances?

As you probably know, you cannot have two instances with the same number on one server.

[Former Member · Accepted solution]

Yes they are different.

Proven by the fact that the port-scans to the different SAP TCP port (3201 for Nr 01, 3202 for Nr. 02) show different results, depending witch SAP instance was started last.

[Former Member · Accepted solution]

Did you check trace files of work processes?

dev_disp, dev_w0, ...

[Former Member · Accepted solution]

So I compared the dev_disp file of a running SAP instance with the one of the "half" started, and it starts to differ at the following line:

Properly running SAP instance:

EsIUnamFileMapInit: ES base = 0xc000000100000000

EsStdInit: 1020 MB successfully allocated

Half started SAP instance:

EsIUnamFileMapInit: shmget(IPC_PRIVATE,1069547520,IPC_CREAT | 0777) failed.

errno = 12(Not enough space)

EsStdInit: unable to allocate 1020 MB

So I have to by more memory? Or is there a parameter I can set?

Goot hint, thanks!

[Former Member · Accepted solution]

Can you check the SWAP size , and also the free memory at OS level?

Regards.

[Former Member · Accepted solution]

So I did some checks with "swapinfo". And I tested it with different SAP Instances running, so the results are not dependant on what Instances are running, it only matters how many of them.

And even with all of them running, still 50% of the 8GB RAM is free.... Problem on OS level? Any restrictions in HP Unix on an HP rx2660 box???

5 Instances running:

Kb Kb Kb PCT START/ Kb

TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME

dev 8388608 0 8388608 0% 0 - 1 /dev/vg00/lvol2

dev 16777216 712 16776504 0% 0 - 0 /dev/vg00/lvol9

reserve - 25163780 -25163780

memory 8368192 4200496 4167696 50%

4 Instances running:

Kb Kb Kb PCT START/ Kb

TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME

dev 8388608 0 8388608 0% 0 - 1 /dev/vg00/lvol2

dev 16777216 712 16776504 0% 0 - 0 /dev/vg00/lvol9

reserve - 25163768 -25163768

memory 8368192 3722020 4646172 44%

3 Instances running

root@gpoch2s03:/usr/sap/ECH/DVEBMGS01/work # swapinfo

Kb Kb Kb PCT START/ Kb

TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME

dev 8388608 0 8388608 0% 0 - 1 /dev/vg00/lvol2

dev 16777216 712 16776504 0% 0 - 0 /dev/vg00/lvol9

reserve - 20256372 -20256372

memory 8368192 2258824 6109368 27%

2 running

Kb Kb Kb PCT START/ Kb

TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME

dev 8388608 0 8388608 0% 0 - 1 /dev/vg00/lvol2

dev 16777216 712 16776504 0% 0 - 0 /dev/vg00/lvol9

reserve - 13885496 -13885496

memory 8368192 2250704 6117488 27%

1 Instance running:

Kb Kb Kb PCT START/ Kb

TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME

dev 8388608 0 8388608 0% 0 - 1 /dev/vg00/lvol2

dev 16777216 712 16776504 0% 0 - 0 /dev/vg00/lvol9

reserve - 7240320 -7240320

memory 8368192 2221636 6146556 27%

[Former Member · Accepted solution]

Look's like you have litlle swap space configured ...

Check the note

Note 146289 - Parameter Recommendations for 64-Bit SAP Kernel

-->

When implementing the 64-bit kernel, we recommend at least
20 GByte of swap space. (Plus approx.10 GB for every additional 64-bit instance on the same computer) See note 153641 for more information

Also in your situation i will be recheck the

Note 172747 - SAP on HP-UX: OS kernel parameter recommendations

may be some parametrs need to be extended because 5 App servers on 1 host are not so good )...

Regards.

[Former Member · Accepted solution]

Meanwhile I fixed it: SWAP!

First I set the kernel parameters shmmax, shmmni and shmseg to the recommended values. (As SAP tried to allocate four chunks of 1024MB memory for a total aof 4092MB Extended Memory. But I think this has nothing to do with the solution. Only "cosmetics" ...

Then I had to set the kernel parameter swchunk from default 2048 to recommended 8192. Only with that it was possible to extend the SWAP.

After reboot I assigned another 16GB logical volume to the SWAP space, now a total of 40GB.

And now the fifth SAP instance starts fine (at leas in the test environment where I tried this out).

Thanks a lot for all your help!

About the 5 SAP instances on one server: The server is an HP Itanium rx2660, 2xCPU, 8GB RAM, productive SAP system has 50 something users, where two dozens are logged in simultaneously, where as only a handful of the logged in users are doing "serious" work in there (some finance, some purchase, some warehouse, ...). The other SAP instances (Validation, Test, Dev and a Sandbox) are used rarely. I guess at a time max two or three users logged in in any of them...

So, server is mainly bored...

Concerning SWAP: I know with the new total of 40GB I am still far bellow the by SAP recomenden 70GB SWAP space (20GB + 5 x 10GB). But it now runs with 40GB, all HDD slots are occupied, there are another 17GB not used I could assign to SWAP if needed. Only if no other things help we are going to exchange the current HDDs for larger ones!

Edited by: Frank Huber on Aug 25, 2010 8:45 AM

Edited by: Frank Huber on Aug 25, 2010 9:00 AM

[Former Member]

First of all in situations like this check the logs dev_disp and dev_w0.

Usually form this log's you find all errors and can find why your sap are Half runnig

Regards.