Dispatcher does not start during Refresh

[Former Member]

Hi all,

We are currently carrying out System Refresh from our Production to Quality system using Oracle backup and recovery method.

We are currently able to restore & recover the Production backup on Quality System. We have tried to restart the database & application to continue with Post-refresh activities. However, we are unable to start the SAP application ( Note: Both SAP application & database resides on different host. Also, we have other SAP applications running on the same server).

Following are the details:

SAP ECC 6.0 (RMCA)

SunOS sun4u sparc SUNW

Oracle 10.2.0.4.

When checked , dev_disp shows the below logs :

EsStdUnamFileMapInit: Info: mmap() failed. errno = 11(Resource temporarily unavailable)

EsStdInit: unable to allocate 4 MB

EsStdInit: try to allocate 0 MB

• ERROR => EsStdInit: unable to allocate 20576 MB

• Error 11 while initializing OS dependent part.

• ERROR => DpEmInit: EmInit (1)

• ERROR => DpMemInit: DpEmInit (-1)

• DP_FATAL_ERROR => DpSapEnvInit: DpMemInit

• DISPATCHER EMERGENCY SHUTDOWN ***

increase tracelevel of WPs

NiWait: sleep (10000ms) ...

NiISelect: timeout 10000ms

NiISelect: maximum fd=1

NiISelect: read-mask is NULL

NiISelect: write-mask is NULL

Fri Dec 16 14:59:01 2011

NiISelect: TIMEOUT occured (10000ms)

Also, the startup logs shows as below:

Starting Programs

-

---

SAP-R/3-Startup Program Rel 700 V1.8 (2003/04/24)

-

---

Starting at 2011/12/16 14:57:46

Startup Profile: "/usr/sap/QUL/SYS/profile/START_DVEBMGS03"

Setup Environment Variables

-

---

(735) SETENV LD_LIBRARY_PATH=/usr/sap/QUL/DVEBMGS03/exe:/usr/sap/QUL/DVEBMGS03/exe:/usr/sap/QUL/SYS/exe/run:/oracle/client/10x_64/instantclient

(735) SETENV SHLIB_PATH=/usr/sap/QUL/DVEBMGS03/exe:

(735) SETENV LIBPATH=/usr/sap/QUL/DVEBMGS03/exe:

Update local Kernel Files

-

---

(737) Local: /usr/sap/QUL/SYS/exe/run/sapcpe name=QUL

(735) system(/usr/sap/QUL/SYS/exe/run/sapcpe name=QUL) returns 1

(735) Return-Code 1 in Local-Kernel-Update. See sapcpe.log.

Execute Pre-Startup Commands

-

---

(743) Local: /usr/sap/QUL/SYS/exe/run/sapcpe pf=/sapmnt/QUL/profile/QUL_DVEBMGS03

(735) system(/usr/sap/QUL/SYS/exe/run/sapcpe pf=/sapmnt/QUL/profile/QUL_DVEBMGS03) returns 1

(749) Local: /usr/sap/QUL/DVEBMGS03/exe/sapmscsa pf=/sapmnt/QUL/profile/QUL_DVEBMGS03_ -n

/usr/sap/QUL/DVEBMGS03/exe/sapmscsa: make new mode. SCSA currently non existent.

sapcscsa: SCSA defined. sapscsaId == 500318684 == 1e00089c

sapcscsa: SCSA attached at address ffffffff7e000000

sapcscsa: SCSA initialized.

rslgwr1(21): Searching for overlap point in pre-existing SysLog file...

/usr/sap/QUL/DVEBMGS03/exe/sapmscsa: finished.

(751) Local: rm -f ms.sapQUL_DVEBMGS03

(753) Local: ln -s -f /usr/sap/QUL/DVEBMGS03/exe/msg_server ms.sapQUL_DVEBMGS03

(755) Local: rm -f dw.sapQUL_DVEBMGS03

(757) Local: ln -s -f /usr/sap/QUL/DVEBMGS03/exe/disp+work dw.sapQUL_DVEBMGS03

(759) Local: rm -f co.sapQUL_DVEBMGS03

(761) Local: ln -s -f /usr/sap/QUL/DVEBMGS03/exe/rslgcoll co.sapQUL_DVEBMGS03

(763) Local: rm -f se.sapQUL_DVEBMGS03

(765) Local: ln -s -f /usr/sap/QUL/DVEBMGS03/exe/rslgsend se.sapQUL_DVEBMGS03

(767) Local: rm -f ig.sapQUL_DVEBMGS03

(769) Local: ln -s -f /usr/sap/QUL/DVEBMGS03/exe/igswd_mt ig.sapQUL_DVEBMGS03

Starting Programs

-

---

16.12.2011 14:57:47

ShmDetach

OK

(794) Starting: local ms.sapQUL_DVEBMGS03 pf=/sapmnt/QUL/profile/QUL_DVEBMGS03_

(795) Starting: local dw.sapQUL_DVEBMGS03 pf=/sapmnt/QUL/profile/QUL_DVEBMGS03_

(796) Starting: local co.sapQUL_DVEBMGS03 pf=/sapmnt/QUL/profile/QUL_DVEBMGS03_ -F

(798) Starting: local se.sapQUL_DVEBMGS03 pf=/sapmnt/QUL/profile/QUL_DVEBMGS03_ -F

(799) Starting: local ig.sapQUL_DVEBMGS03 -mode=profile pf=/sapmnt/QUL/profile/QUL_DVEBMGS03_

(735) Waiting for Child Processes to terminate.

(735) **** 2011/12/16 14:59:07 Child 795 terminated with Status 0 . ****

(795) **** 2011/12/16 14:59:07 No RestartProgram command for program 1 ****

:QULadm 38%

Can you please help us on how to resolve this error as soon as possible?

Your help in this regard will be highly appreciated.

With regards,

Raj

Edited by: Arul Sekar on Dec 19, 2011 6:25 AM

[Former Member]

Hi all,

Thanks for your help.

We have reduced the value for parameter "abap/buffer_size" on the server and it worked.

with regards,

Raja

[vincentlim826]

• ERROR => EsStdInit: unable to allocate 20576 MB

• Error 11 while initializing OS dependent part.

This shows that probably not enough swap space to fulfil request

Maybe reduce em/initial_size_MB or increase your swap space..

You can check the maximum values by 'memlimits -l 20576' command run by <sid>adm.

For 64 bits kernel, it is preferable to have at least 20GB swap space. (SAP note 153641)

Do check following SAP Notes:

300464 Shared Memory Parameters in Solaris

395438 Parameter settings for Solaris 9 and below

724713 Parameter settings for Solaris 10

Hope this helps.

Regards,

Vincent

[Former Member]

Hi Vincent,

Thanks for your reply!!

I have the below output for memlimits -l

Maximum heap size per process........: 9628 MB

Maximum mapped file size (mmap)......: 7420 MB

this value is probably limited by swap space

Maximum protectable size (mprotect)..: 7420 MB

em/initial_size_MB > 7420 MB will not work

Maximum address space per process....: 15768 MB

Total available swap space...........: 10572 MB

main memory size x 3 recommended , minimum 1 GB

What does this suggest? Does this require change of memory settings? Kindly suggest.

with regards,

Raj

[sunny_pahuja2]

Hi,

You need to decrease memory in these parameters as your system does not have memory to allocate as per logs.

Thanks

Sunny

[Former Member]

Hi all,

Thanks for your reply!!

I have the below output for memlimits -l

Maximum heap size per process........: 9628 MB

Maximum mapped file size (mmap)......: 7420 MB

this value is probably limited by swap space

Maximum protectable size (mprotect)..: 7420 MB

em/initial_size_MB > 7420 MB will not work

Maximum address space per process....: 15768 MB

Total available swap space...........: 10572 MB

main memory size x 3 recommended , minimum 1 GB

Based on the input, we have tried changing the Extended memory settings for the server ( em/initial_size_MB = 7000 , ztta/roll_extension = 2147483648) This time the SAP application started, but did not last for long. The Dispatcher died soon with the below error:

Mon Dec 19 12:11:17 2011

• ERROR => DpWpCreate: fork (12: Not enough space)

***LOG Q0I=> DpWpCreate, fork ( fork 12)

• ERROR => DpWpDynCreate: DpWpCreate

• ERROR => DpProcCreate: DpWpDynCreate

• DP_FATAL_ERROR => DpSapEnvInit: DpProcCreate

• DISPATCHER EMERGENCY SHUTDOWN ***

increase trace level of WPs

kill(22278,0) successful -> process alive

killing W0-22278 (SIGUSR2)

Kindly let us know how can this be resolved?

with regards,

Raja

[vincentlim826]

>> ERROR => DpWpCreate: fork (12: Not enough space)

This is resource problem. Most likely the swap. So probably you need to delete some files in order to release swap memory.

How much physical memory do you have? How much swap do you currently have configured? Sap recommends that swap= 3x RAM. Please also make sure the kernel parameters recommended by SAP are used(see installation guide).

The error "fork 12" indicates problems with memory allocation. Usually this indicates oversizing of memory allocation parameters for R/3 and database (together their memory allocation requirements exceeded the virtual memory, i.e. physical + swap memory, available on the server).

Check the SAP Note 12715 and use "ipcs -ma" to list ALL shared memory segments allocated by <sid>adm user and do "ipcrm" for these shared memories, semaphores etc. This will definitely remove any remain shared memories and let the system use the full memory again.

Do the "ipcs" for ora<sid> user too and remove any shared memories for Oracle when the database is down. Simply put ipcs shouldn't show any shared memories for any application user when that application (R/3, Oracle etc) are shutdown.

Also check the SAP Note 146289 if you have a 64bit kernel.

Hope this helps.

Regards,

Vincent

[Former Member]

Hello Raj,

As suggested by Sunny and other folks, please reduce the memory param values and run SAPPFPAR PF=</sapmnt/SID/profile/instance profile> CHECK for errors in shared pool.

Thanks,

Siva Kumar

[Former Member]

*** ERROR => EsStdInit: unable to allocate 20576 MB
*** Error 11 while initializing OS dependent part.
*** ERROR => DpEmInit: EmInit (1) 
*** ERROR => DpMemInit: DpEmInit (-1) 
*** DP_FATAL_ERROR => DpSapEnvInit: DpMemInit
*** DISPATCHER EMERGENCY SHUTDOWN ***

Issue seems with insufficient memory, try to reboot the server and then retry.

Also paste the ERROR lines from dev_w0 log file.

Regards,

Nick Loy