Dispatcher stopped working

1490 Views

hi,

my Dispatcher in sap stopped working. can anybody guide me to solve this issue, please? my developer trace content is below

---------------------------------------------------

trc file: "dev_disp", trc level: 1, release: "700"

---------------------------------------------------

sysno 00

sid ECC

systemid 560 (PC with Windows NT)

relno 7000

patchlevel 0

patchno 75

intno 20050900

make: multithreaded, Unicode, optimized

pid 3396

Mon Aug 19 19:55:45 2013
kernel runs with dp version 217000(ext=109000) (@(#) DPLIB-INT-VERSION-217000-UC)
length of sys_adm_ext is 572 bytes
*** SWITCH TRC-HIDE on ***
***LOG Q00=> DpSapEnvInit, DPStart (00 3396) [dpxxdisp.c 1237]
shared lib "dw_xml.dll" version 75 successfully loaded
shared lib "dw_xtc.dll" version 75 successfully loaded
shared lib "dw_stl.dll" version 75 successfully loaded
shared lib "dw_gui.dll" version 75 successfully loaded
shared lib "dw_mdm.dll" version 75 successfully loaded
rdisp/softcancel_sequence : -> 0,5,-1
use internal message server connection to port 3900

Mon Aug 19 19:55:49 2013
*** WARNING => DpNetCheck: NiAddrToHost(1.0.0.0) took 4 seconds
***LOG GZZ=> 1 possible network problems detected - check tracefile and adjust the DNS settings [dpxxtool2.c 5273]
MtxInit: 30000 0 0
DpSysAdmExtInit: ABAP is active
DpSysAdmExtInit: VMC (JAVA VM in WP) is not active
DpIPCInit2: start server >gcecc62_ECC_00 <
DpShMCreate: sizeof(wp_adm) 12672 (1408)
DpShMCreate: sizeof(tm_adm) 3994272 (19872)
DpShMCreate: sizeof(wp_ca_adm) 24000 (80)
DpShMCreate: sizeof(appc_ca_adm) 8000 (80)
DpCommTableSize: max/headSize/ftSize/tableSize=500/8/528056/528064
DpShMCreate: sizeof(comm_adm) 528064 (1048)
DpFileTableSize: max/headSize/ftSize/tableSize=0/0/0/0
DpShMCreate: sizeof(file_adm) 0 (72)
DpShMCreate: sizeof(vmc_adm) 0 (1440)
DpShMCreate: sizeof(wall_adm) (38456/34360/64/184)
DpShMCreate: sizeof(gw_adm) 48
DpShMCreate: SHM_DP_ADM_KEY (addr: 05CF0040, size: 4647736)
DpShMCreate: allocated sys_adm at 05CF0040
DpShMCreate: allocated wp_adm at 05CF1E40
DpShMCreate: allocated tm_adm_list at 05CF4FC0
DpShMCreate: allocated tm_adm at 05CF4FF0
DpShMCreate: allocated wp_ca_adm at 060C4290
DpShMCreate: allocated appc_ca_adm at 060CA050
DpShMCreate: allocated comm_adm at 060CBF90
DpShMCreate: system runs without file table
DpShMCreate: allocated vmc_adm_list at 0614CE50
DpShMCreate: allocated gw_adm at 0614CE90
DpShMCreate: system runs without vmc_adm
DpShMCreate: allocated ca_info at 0614CEC0
DpShMCreate: allocated wall_adm at 0614CEC8

Mon Aug 19 19:55:50 2013
MBUF state OFF
DpCommInitTable: init table for 500 entries
EmInit: MmSetImplementation( 2 ).
MM global diagnostic options set: 0

Mon Aug 19 19:55:51 2013
<ES> client 0 initializing ....
<ES> InitFreeList
<ES> block size is 1024 kByte.
Using implementation flat
<EsNT> Memory Reset disabled as NT default
<ES> 511 blocks reserved for free list.
ES initialized.

Mon Aug 19 19:55:52 2013
rdisp/http_min_wait_dia_wp : 1 -> 1

Mon Aug 19 19:55:53 2013
***LOG CPS=> DpLoopInit, ICU ( 3.0 3.0 4.0.1) [dpxxdisp.c 1623]
***LOG Q0K=> DpMsAttach, mscon ( gcecc62) [dpxxdisp.c 11586]
DpStartStopMsg: send start message (myname is >gcecc62_ECC_00 <)
DpStartStopMsg: start msg sent

Mon Aug 19 19:55:55 2013
CCMS: AlInitGlobals : alert/use_sema_lock = TRUE.
CCMS: Initalizing shared memory of size 60000000 for monitoring segment.

Mon Aug 19 19:56:10 2013
CCMS: start to initalize 3.X shared alert area (first segment).
DpMsgAdmin: Set release to 7000, patchlevel 0
MBUF state PREPARED
MBUF component UP
DpMBufHwIdSet: set Hardware-ID
***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c 1050]
DpMsgAdmin: Set patchno for this platform to 75
Release check o.K.

Mon Aug 19 19:56:33 2013
*** ERROR => W0 (pid 3748) died [dpxxdisp.c 14241]
*** ERROR => W1 (pid 3764) died [dpxxdisp.c 14241]
*** ERROR => W2 (pid 3708) died [dpxxdisp.c 14241]
my types changed after wp death/restart 0xbf --> 0xbe
*** ERROR => W3 (pid 3772) died [dpxxdisp.c 14241]
my types changed after wp death/restart 0xbe --> 0xbc
*** ERROR => W4 (pid 3776) died [dpxxdisp.c 14241]
my types changed after wp death/restart 0xbc --> 0xb8
*** ERROR => W5 (pid 2752) died [dpxxdisp.c 14241]
*** ERROR => W6 (pid 3696) died [dpxxdisp.c 14241]
my types changed after wp death/restart 0xb8 --> 0xb0
*** ERROR => W7 (pid 2772) died [dpxxdisp.c 14241]
my types changed after wp death/restart 0xb0 --> 0xa0
*** ERROR => W8 (pid 2760) died [dpxxdisp.c 14241]
my types changed after wp death/restart 0xa0 --> 0x80
*** DP_FATAL_ERROR => DpWPCheck: no more work processes
*** DISPATCHER EMERGENCY SHUTDOWN ***
increase tracelevel of WPs
NiWait: sleep (10000ms) ...
NiISelect: timeout 10000ms
NiISelect: maximum fd=1653
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Aug 19 19:56:43 2013
NiISelect: TIMEOUT occured (10000ms)
dump system status
Workprocess Table (long) Mon Aug 19 14:26:43 2013
========================

No Ty. Pid Status Cause Start Err Sem CPU Time Program Cl User Action Table
-----------------------------------------------------------------------------------------------------------------------
0 DIA 3748 Ended no 1 0 0
1 DIA 3764 Ended no 1 0 0
2 DIA 3708 Ended no 1 0 0
3 UPD 3772 Ended no 1 0 0
4 ENQ 3776 Ended no 1 0 0
5 BTC 2752 Ended no 1 0 0
6 BTC 3696 Ended no 1 0 0
7 SPO 2772 Ended no 1 0 0
8 UP2 2760 Ended no 1 0 0
Dispatcher Queue Statistics Mon Aug 19 14:26:43 2013
===========================

+------+--------+--------+--------+--------+--------+
| Typ | now | high | max | writes | reads |
+------+--------+--------+--------+--------+--------+
| NOWP | 0 | 2 | 2000 | 7 | 7 |
+------+--------+--------+--------+--------+--------+
| DIA | 5 | 5 | 2000 | 5 | 0 |
+------+--------+--------+--------+--------+--------+
| UPD | 0 | 0 | 2000 | 0 | 0 |
+------+--------+--------+--------+--------+--------+
| ENQ | 0 | 0 | 2000 | 0 | 0 |
+------+--------+--------+--------+--------+--------+
| BTC | 0 | 0 | 2000 | 0 | 0 |
+------+--------+--------+--------+--------+--------+
| SPO | 0 | 0 | 2000 | 0 | 0 |
+------+--------+--------+--------+--------+--------+
| UP2 | 0 | 0 | 2000 | 0 | 0 |
+------+--------+--------+--------+--------+--------+

max_rq_id 12
wake_evt_udp_now 0

wake events total 8, udp 7 ( 87%), shm 1 ( 12%)
since last update total 8, udp 7 ( 87%), shm 1 ( 12%)

Dump of tm_adm structure: Mon Aug 19 14:26:43 2013
=========================

Term uid man user term lastop mod wp ta a/i (modes)

Workprocess Comm. Area Blocks Mon Aug 19 14:26:43 2013
=============================

Slots: 300, Used: 1, Max: 0
+------+--------------+----------+-------------+
| id | owner | pid | eyecatcher |
+------+--------------+----------+-------------+
| 0 | DISPATCHER | -1 | *WPCAAD000* |

NiWait: sleep (5000ms) ...
NiISelect: timeout 5000ms
NiISelect: maximum fd=1653
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Aug 19 19:56:48 2013
NiISelect: TIMEOUT occured (5000ms)
DpHalt: shutdown server >gcecc62_ECC_00 < (normal)
DpJ2eeDisableRestart
DpModState: buffer in state MBUF_PREPARED

NiBufSend starting
NiIWrite: hdl 2 sent data (wrt=110,pac=1,MESG_IO)
MsINiWrite: sent 110 bytes
MsIModState: change state to SHUTDOWN
DpModState: change server state from STARTING to SHUTDOWN
Switch off Shared memory profiling
ShmProtect( 57, 3 )
ShmProtect(SHM_PROFILE, SHM_PROT_RW
ShmProtect( 57, 1 )
ShmProtect(SHM_PROFILE, SHM_PROT_RD
DpWakeUpWps: wake up all wp's
Stop work processes
Stop gateway
killing process (3720) (SOFT_KILL)
Stop icman
killing process (3740) (SOFT_KILL)
Terminate gui connections
wait for end of work processes
wait for end of gateway
[DpProcDied] Process lives (PID:3720 HANDLE:1624)
waiting for termination of gateway ...
NiWait: sleep (1000ms) ...
NiISelect: timeout 1000ms
NiISelect: maximum fd=1653
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Aug 19 19:56:49 2013
NiISelect: TIMEOUT occured (1000ms)
[DpProcDied] Process died (PID:3720 HANDLE:1624)
wait for end of icman
[DpProcDied] Process lives (PID:3740 HANDLE:1628)
waiting for termination of icman ...
NiWait: sleep (1000ms) ...
NiISelect: timeout 1000ms
NiISelect: maximum fd=1653
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Aug 19 19:56:50 2013
NiISelect: TIMEOUT occured (1000ms)
[DpProcDied] Process lives (PID:3740 HANDLE:1628)
waiting for termination of icman ...
NiWait: sleep (1000ms) ...
NiISelect: timeout 1000ms
NiISelect: maximum fd=1653
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Aug 19 19:56:51 2013
NiISelect: TIMEOUT occured (1000ms)
[DpProcDied] Process lives (PID:3740 HANDLE:1628)
waiting for termination of icman ...
NiWait: sleep (1000ms) ...
NiISelect: timeout 1000ms
NiISelect: maximum fd=1653
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Aug 19 19:56:52 2013
NiISelect: TIMEOUT occured (1000ms)
[DpProcDied] Process lives (PID:3740 HANDLE:1628)
waiting for termination of icman ...
NiWait: sleep (1000ms) ...
NiISelect: timeout 1000ms
NiISelect: maximum fd=1653
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Aug 19 19:56:53 2013
NiISelect: TIMEOUT occured (1000ms)
[DpProcDied] Process lives (PID:3740 HANDLE:1628)
waiting for termination of icman ...
NiWait: sleep (1000ms) ...
NiISelect: timeout 1000ms
NiISelect: maximum fd=1653
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Aug 19 19:56:54 2013
NiISelect: TIMEOUT occured (1000ms)
[DpProcDied] Process lives (PID:3740 HANDLE:1628)
waiting for termination of icman ...
NiWait: sleep (1000ms) ...
NiISelect: timeout 1000ms
NiISelect: maximum fd=1653
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Aug 19 19:56:55 2013
NiISelect: TIMEOUT occured (1000ms)
[DpProcDied] Process lives (PID:3740 HANDLE:1628)
waiting for termination of icman ...
NiWait: sleep (1000ms) ...
NiISelect: timeout 1000ms
NiISelect: maximum fd=1653
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Aug 19 19:56:56 2013
NiISelect: TIMEOUT occured (1000ms)
[DpProcDied] Process lives (PID:3740 HANDLE:1628)
waiting for termination of icman ...
NiWait: sleep (1000ms) ...
NiISelect: timeout 1000ms
NiISelect: maximum fd=1653
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Aug 19 19:56:57 2013
NiISelect: TIMEOUT occured (1000ms)
[DpProcDied] Process lives (PID:3740 HANDLE:1628)
waiting for termination of icman ...
NiWait: sleep (1000ms) ...
NiISelect: timeout 1000ms
NiISelect: maximum fd=1653
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Aug 19 19:56:58 2013
NiISelect: TIMEOUT occured (1000ms)
[DpProcDied] Process lives (PID:3740 HANDLE:1628)
waiting for termination of icman ...
NiWait: sleep (1000ms) ...
NiISelect: timeout 1000ms
NiISelect: maximum fd=1653
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Aug 19 19:56:59 2013
NiISelect: TIMEOUT occured (1000ms)
[DpProcDied] Process lives (PID:3740 HANDLE:1628)
waiting for termination of icman ...
NiWait: sleep (1000ms) ...
NiISelect: timeout 1000ms
NiISelect: maximum fd=1653
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Aug 19 19:57:00 2013
NiISelect: TIMEOUT occured (1000ms)
[DpProcDied] Process died (PID:3740 HANDLE:1628)
DpStartStopMsg: send stop message (myname is >gcecc62_ECC_00 <)
NiIMyHostName: hostname = 'gcecc62'
AdGetSelfIdentRecord: > <
AdCvtRecToExt: opcode 60 (AD_SELFIDENT), ser 0, ex 0, errno 0
AdCvtRecToExt: opcode 4 (AD_STARTSTOP), ser 0, ex 0, errno 0
DpConvertRequest: net size = 189 bytes

NiBufSend starting
NiIWrite: hdl 2 sent data (wrt=562,pac=1,MESG_IO)
MsINiWrite: sent 562 bytes
send msg (len 110+452) to name -, type 4, key -
DpStartStopMsg: stop msg sent
NiIRead: hdl 2 received data (rcd=274,pac=1,MESG_IO)
NiBufIIn: NIBUF len=274
NiBufIIn: packet complete for hdl 2

NiBufReceive starting
MsINiRead: received 274 bytes
MSG received, len 110+164, flag 1, from MSG_SERVER , typ 0, key -
DpHalt: received 164 bytes from message server
NiIRead: hdl 2 received data (rcd=274,pac=1,MESG_IO)
NiBufIIn: NIBUF len=274
NiBufIIn: packet complete for hdl 2

NiBufReceive starting
MsINiRead: received 274 bytes
MSG received, len 110+164, flag 1, from MSG_SERVER , typ 0, key -
DpHalt: received 164 bytes from message server
NiIRead: hdl 2 recv would block (errno=EAGAIN)
NiIRead: read for hdl 2 timed out (0ms)
DpHalt: no more messages from the message server
DpHalt: send keepalive to synchronize with the message server

NiBufSend starting
NiIWrite: hdl 2 sent data (wrt=114,pac=1,MESG_IO)
MsINiWrite: sent 114 bytes
send msg (len 110+4) to name MSG_SERVER, type 0, key -
MsSndName: MS_NOOP ok
Send 4 bytes to MSG_SERVER
NiIRead: hdl 2 received data (rcd=114,pac=1,MESG_IO)
NiBufIIn: NIBUF len=114
NiBufIIn: packet complete for hdl 2

NiBufReceive starting
MsINiRead: received 114 bytes
MSG received, len 110+4, flag 3, from MSG_SERVER , typ 0, key -
Received 4 bytes from MSG_SERVER
Received opcode MS_NOOP from msg_server, reply MSOP_OK
MsOpReceive: ok
MsSendKeepalive : keepalive sent to message server
NiIRead: hdl 2 recv would block (errno=EAGAIN)
Mon Aug 19 19:57:01 2013
NiIPeek: peek for hdl 2 timed out (r; 1000ms)
NiIRead: read for hdl 2 timed out (1000ms)
DpHalt: no more messages from the message server
DpHalt: sync with message server o.k.
detach from message server
***LOG Q0M=> DpMsDetach, ms_detach () [dpxxdisp.c 11899]

NiBufSend starting
NiIWrite: hdl 2 sent data (wrt=110,pac=1,MESG_IO)
MsINiWrite: sent 110 bytes
MsIDetach: send logout to msg_server
MsIDetach: call exit function
DpMsShutdownHook called
NiBufISelUpdate: new MODE -- (r-) for hdl 2 in set0
SiSelNSet: set events of sock 1580 to: ---
NiBufISelRemove: remove hdl 2 from set0
SiSelNRemove: removed sock 1580 (pos=2)
SiSelNRemove: removed sock 1580
NiSelIRemove: removed hdl 2
MBUF state OFF
AdGetSelfIdentRecord: > <
AdCvtRecToExt: opcode 60 (AD_SELFIDENT), ser 0, ex 0, errno 0
AdCvtRecToExt: opcode 40 (AD_MSBUF), ser 0, ex 0, errno 0
AdCvtRecToExt: opcode 40 (AD_MSBUF), ser 0, ex 0, errno 0
blks_in_queue/wp_ca_blk_no/wp_max_no = 1/300/9
LOCK WP ca_blk 1
make DISP owner of wp_ca_blk 1
DpRqPutIntoQueue: put request into queue (reqtype 1, prio LOW, rq_id 15)
MBUF component DOWN
NiICloseHandle: shutdown and close hdl 2 / sock 1580
NiBufIClose: clear extension for hdl 2
MsIDetach: detach MS-system
cleanup EM
EsCleanup ....
EmCleanup() -> 0
Es2Cleanup: Cleanup ES2
***LOG Q05=> DpHalt, DPStop ( 3396) [dpxxdisp.c 10259]
Good Bye .....