Hi All,
We had a power maintenance in our data center so I shutdown the solman server yesterday. Now when I started it, disp+work stops.
listed below is the dev_disp log. We are using 640 with oracle 9i. I tried restarting everything but still i have the problem.
-
trc file: "dev_disp", trc level: 1, release: "620"
-
Sun Sep 21 19:55:03 2008
kernel runs with dp version 13(ext=2) (@(#) DPLIB-INT-VERSION-13)
length of sys_adm_ext is 304 bytes
systemid 561 (PC with Windows NT)
relno 6200
patchlevel 0
patchno 1773
intno 20020600
make: multithreaded, ASCII, 64 bit
pid 1212
***LOG Q00=> DpSapEnvInit, DPStart (00 1212) [dpxxdisp.c 1022]
shared lib "dw_xml.dll" version 1773 successfully loaded
shared lib "dw_xtc.dll" version 1773 successfully loaded
shared lib "dw_stl.dll" version 1773 successfully loaded
Sun Sep 21 19:55:08 2008
WARNING => DpNetCheck: NiAddrToHost(1.0.0.0) took 5 seconds
***LOG GZZ=> 1 possible network problems detected - check tracefile and adjust the DNS settings [dpxxtool2.c 3452]
MtxInit: -2 0 0
DpShMCreate: sizeof(wp_adm) 12920 (760)
DpShMCreate: sizeof(tm_adm) 23419704 (11704)
DpShMCreate: sizeof(wp_ca_adm) 60000 (60)
DpShMCreate: sizeof(appc_ca_adm) 120000 (60)
DpShMCreate: sizeof(comm_adm) 848000 (424)
DpShMCreate: sizeof(wall_adm) (224040/346312/56/104)
DpShMCreate: SHM_DP_ADM_KEY (addr: 0000000004A00050, size: 25032712)
DpShMCreate: allocated sys_adm at 0000000004A00050
DpShMCreate: allocated wp_adm at 0000000004A00698
DpShMCreate: allocated tm_adm_list at 0000000004A03910
DpShMCreate: allocated tm_adm at 0000000004A03938
DpShMCreate: allocated wp_ca_adm at 0000000006059470
DpShMCreate: allocated appc_ca_adm at 0000000006067ED0
DpShMCreate: allocated comm_adm_list at 0000000006085390
DpShMCreate: allocated comm_adm at 00000000060853A8
DpShMCreate: allocated ca_info at 0000000006154428
DpShMCreate: allocated wall_adm at 0000000006154430
MBUF state OFF
EmInit: MmSetImplementation( 2 ).
<ES> client 0 initializing ....
<ES> InitFreeList
<ES> block size is 4096 kByte.
<ES> Info: em/initial_size_MB( 2453MB) not multiple of em/blocksize_KB( 4096KB)
<ES> Info: em/initial_size_MB rounded up to 2456MB
Using implementation std
<EsNT> Memory Reset enabled as NT default
<EsNT> EsIUnamFileMapInit: Initialize the memory 2456 MB
<ES> 613 blocks reserved for free list.
ES initialized.
rdisp/http_min_wait_dia_wp : 1 -> 1
***LOG Q0K=> DpMsAttach, mscon ( s11d001z) [dpxxdisp.c 9465]
CCMS: Initalizing shared memory of size 20000000 for monitoring segment.
CCMS: start to initalize 3.X shared alert area (first segment).
Sun Sep 21 19:55:13 2008
DpMsgAdmin: Set release to 6200, patchlevel 0
MBUF state PREPARED
MBUF component UP
DpMBufHwIdSet: set Hardware-ID
***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c 941]
DpMsgAdmin: Set patchno for this platform to 1773
Release check o.K.
Sun Sep 21 19:55:48 2008
ERROR => W0 (pid 528) died [dpxxdisp.c 11866]
ERROR => W1 (pid 2996) died [dpxxdisp.c 11866]
ERROR => W2 (pid 2316) died [dpxxdisp.c 11866]
ERROR => W3 (pid 3168) died [dpxxdisp.c 11866]
ERROR => W4 (pid 2584) died [dpxxdisp.c 11866]
ERROR => W5 (pid 3344) died [dpxxdisp.c 11866]
my types changed after wp death/restart 0xbf --> 0xbe
ERROR => W6 (pid 3032) died [dpxxdisp.c 11866]
ERROR => W7 (pid 2344) died [dpxxdisp.c 11866]
ERROR => W8 (pid 3296) died [dpxxdisp.c 11866]
ERROR => W9 (pid 848) died [dpxxdisp.c 11866]
ERROR => W10 (pid 668) died [dpxxdisp.c 11866]
my types changed after wp death/restart 0xbe --> 0xbc
ERROR => W11 (pid 3904) died [dpxxdisp.c 11866]
my types changed after wp death/restart 0xbc --> 0xb8
ERROR => W12 (pid 3300) died [dpxxdisp.c 11866]
ERROR => W13 (pid 1120) died [dpxxdisp.c 11866]
my types changed after wp death/restart 0xb8 --> 0xb0
ERROR => W14 (pid 1164) died [dpxxdisp.c 11866]
my types changed after wp death/restart 0xb0 --> 0xa0
ERROR => W15 (pid 3404) died [dpxxdisp.c 11866]
ERROR => W16 (pid 2952) died [dpxxdisp.c 11866]
my types changed after wp death/restart 0xa0 --> 0x80
DP_FATAL_ERROR => DpEnvCheck: no more work processes
DISPATCHER EMERGENCY SHUTDOWN ***
increase tracelevel of WPs
killing W0-528 (SIGUSR2)
ERROR => DpWpKill(528, SIGUSR2) failed [dpxxtool.c 2645]
killing W1-2996 (SIGUSR2)
ERROR => DpWpKill(2996, SIGUSR2) failed [dpxxtool.c 2645]
killing W2-2316 (SIGUSR2)
ERROR => DpWpKill(2316, SIGUSR2) failed [dpxxtool.c 2645]
killing W3-3168 (SIGUSR2)
ERROR => DpWpKill(3168, SIGUSR2) failed [dpxxtool.c 2645]
killing W4-2584 (SIGUSR2)
ERROR => DpWpKill(2584, SIGUSR2) failed [dpxxtool.c 2645]
killing W5-3344 (SIGUSR2)
ERROR => DpWpKill(3344, SIGUSR2) failed [dpxxtool.c 2645]
killing W6-3032 (SIGUSR2)
ERROR => DpWpKill(3032, SIGUSR2) failed [dpxxtool.c 2645]
killing W7-2344 (SIGUSR2)
ERROR => DpWpKill(2344, SIGUSR2) failed [dpxxtool.c 2645]
killing W8-3296 (SIGUSR2)
ERROR => DpWpKill(3296, SIGUSR2) failed [dpxxtool.c 2645]
killing W9-848 (SIGUSR2)
ERROR => DpWpKill(848, SIGUSR2) failed [dpxxtool.c 2645]
killing W10-668 (SIGUSR2)
ERROR => DpWpKill(668, SIGUSR2) failed [dpxxtool.c 2645]
killing W11-3904 (SIGUSR2)
ERROR => DpWpKill(3904, SIGUSR2) failed [dpxxtool.c 2645]
killing W12-3300 (SIGUSR2)
ERROR => DpWpKill(3300, SIGUSR2) failed [dpxxtool.c 2645]
killing W13-1120 (SIGUSR2)
ERROR => DpWpKill(1120, SIGUSR2) failed [dpxxtool.c 2645]
killing W14-1164 (SIGUSR2)
ERROR => DpWpKill(1164, SIGUSR2) failed [dpxxtool.c 2645]
killing W15-3404 (SIGUSR2)
ERROR => DpWpKill(3404, SIGUSR2) failed [dpxxtool.c 2645]
killing W16-2952 (SIGUSR2)
ERROR => DpWpKill(2952, SIGUSR2) failed [dpxxtool.c 2645]
NiWait: sleep (10000 msecs) ...
NiISelect: timeout 10000 ms
Sun Sep 21 19:55:58 2008
NiISelect: TIMEOUT occured(10000 msecs)
dump system status
Workprocess Table (long) Sun Sep 21 09:55:58 2008
========================
No Ty. Pid Status Cause Start Err Sem CPU Time Program Cl User Action Table
-
0 DIA 528 Ended no 1 0 0
1 DIA 2996 Ended no 1 0 0
2 DIA 2316 Ended no 1 0 0
3 DIA 3168 Ended no 1 0 0
4 DIA 2584 Ended no 1 0 0
5 DIA 3344 Ended no 1 0 0
6 UPD 3032 Ended no 1 0 0
7 UPD 2344 Ended no 1 0 0
8 UPD 3296 Ended no 1 0 0
9 UPD 848 Ended no 1 0 0
10 UPD 668 Ended no 1 0 0
11 ENQ 3904 Ended no 1 0 0
12 BTC 3300 Ended no 1 0 0
13 BTC 1120 Ended no 1 0 0
14 SPO 1164 Ended no 1 0 0
15 UP2 3404 Ended no 1 0 0
16 UP2 2952 Ended no 1 0 0
Dispatcher Queue Statistics Sun Sep 21 09:55:58 2008
===========================
--------
+
Typ
now
high
max
writes
reads
--------
+
NOWP
0
4
2000
4
4
--------
+
DIA
3
3
2000
3
0
--------
+
UPD
0
0
2000
0
0
--------
+
ENQ
0
0
2000
0
0
--------
+
BTC
0
0
2000
0
0
--------
+
SPO
0
0
2000
0
0
--------
+
UP2
0
0
2000
0
0
--------
+
max_rq_id 10
wake_evt_udp_now 0
wake events total 6, udp 2 ( 33%), shm 4 ( 66%)
since last update total 6, udp 2 ( 33%), shm 4 ( 66%)
Dump of tm_adm structure: Sun Sep 21 09:55:58 2008
=========================
Term uid man user term lastop mod wp ta a/i (modes)
Workprocess Comm. Area Blocks Sun Sep 21 09:55:58 2008
=============================
Slots: 1000, Used: 1, Max: 0
--------
+
id
owner
pid
eyecatcher
--------
+
0
DISPATCHER
-1
WPCAAD000
NiWait: sleep (5000 msecs) ...
NiISelect: timeout 5000 ms
Sun Sep 21 19:56:03 2008
NiISelect: TIMEOUT occured(5000 msecs)
DpModState: buffer in state MBUF_PREPARED
NiBufSend starting
NiIWrite: write 110, 1 packs, MESG_IO, handle 3, data complete
MsINiWrite: sent 110 bytes
MsIModState: change state to SHUTDOWN
DpModState: change server state from STARTING to SHUTDOWN
Switch off Shared memory profiling
ShmProtect( 57, 3 )
ShmProtect(SHM_PROFILE, SHM_PROT_RW
ShmProtect( 57, 1 )
ShmProtect(SHM_PROFILE, SHM_PROT_RD
DpWakeUpWps: wake up all wp's
killing process (3472) (SOFT_KILL)
killing process (1620) (SOFT_KILL)
[DpProcDied] Process lives (PID:3472 HANDLE:1584)
wait for end of gateway
NiWait: sleep (1000 msecs) ...
NiISelect: timeout 1000 ms
Sun Sep 21 19:56:04 2008
NiISelect: TIMEOUT occured(1000 msecs)
[DpProcDied] Process died (PID:3472 HANDLE:1584)
[DpProcDied] Process lives (PID:1620 HANDLE:1576)
wait for end of icman
NiWait: sleep (1000 msecs) ...
NiISelect: timeout 1000 ms
Sun Sep 21 19:56:05 2008
NiISelect: TIMEOUT occured(1000 msecs)
[DpProcDied] Process lives (PID:1620 HANDLE:1576)
wait for end of icman
NiWait: sleep (1000 msecs) ...
NiISelect: timeout 1000 ms
Sun Sep 21 19:56:06 2008
NiISelect: TIMEOUT occured(1000 msecs)
[DpProcDied] Process lives (PID:1620 HANDLE:1576)
wait for end of icman
NiWait: sleep (1000 msecs) ...
NiISelect: timeout 1000 ms
Sun Sep 21 19:56:07 2008
NiISelect: TIMEOUT occured(1000 msecs)
[DpProcDied] Process lives (PID:1620 HANDLE:1576)
wait for end of icman
NiWait: sleep (1000 msecs) ...
NiISelect: timeout 1000 ms
Sun Sep 21 19:56:08 2008
NiISelect: TIMEOUT occured(1000 msecs)
[DpProcDied] Process lives (PID:1620 HANDLE:1576)
wait for end of icman
NiWait: sleep (1000 msecs) ...
NiISelect: timeout 1000 ms
Sun Sep 21 19:56:09 2008
NiISelect: TIMEOUT occured(1000 msecs)
[DpProcDied] Process lives (PID:1620 HANDLE:1576)
wait for end of icman
NiWait: sleep (1000 msecs) ...
NiISelect: timeout 1000 ms
Sun Sep 21 19:56:10 2008
NiISelect: TIMEOUT occured(1000 msecs)
[DpProcDied] Process lives (PID:1620 HANDLE:1576)
wait for end of icman
NiWait: sleep (1000 msecs) ...
NiISelect: timeout 1000 ms
Sun Sep 21 19:56:11 2008
NiISelect: TIMEOUT occured(1000 msecs)
[DpProcDied] Process lives (PID:1620 HANDLE:1576)
wait for end of icman
NiWait: sleep (1000 msecs) ...
NiISelect: timeout 1000 ms
Sun Sep 21 19:56:12 2008
NiISelect: TIMEOUT occured(1000 msecs)
[DpProcDied] Process lives (PID:1620 HANDLE:1576)
wait for end of icman
NiWait: sleep (1000 msecs) ...
NiISelect: timeout 1000 ms
Sun Sep 21 19:56:13 2008
NiISelect: TIMEOUT occured(1000 msecs)
[DpProcDied] Process died (PID:1620 HANDLE:1576)
DpHalt: cancel all lcom connections
MPI CancelAll 2 -> 0
MPI DeleteAll 2 -> 0
AdGetSelfIdentRecord: > <
AdCvtRecToExt: opcode 60, ser 0, ex 0, errno 0
AdCvtRecToExt: opcode 4, ser 0, ex 0, errno 0
DpConvertRequest: net size = 163 bytes
NiBufSend starting
NiIWrite: write 562, 1 packs, MESG_IO, handle 3, data complete
MsINiWrite: sent 562 bytes
send msg (len 110+452) to name -, type 4, key -
***LOG Q0M=> DpMsDetach, ms_detach () [dpxxdisp.c 9691]
NiBufSend starting
NiIWrite: write 110, 1 packs, MESG_IO, handle 3, data complete
MsINiWrite: sent 110 bytes
MsIDetach: send logout to msg_server
MsIDetach: call exit function
DpMsShutdownHook called
NiSelClear: removed hdl 3 from selectset
MBUF state OFF
AdGetSelfIdentRecord: > <
AdCvtRecToExt: opcode 60, ser 0, ex 0, errno 0
AdCvtRecToExt: opcode 40, ser 0, ex 0, errno 0
AdCvtRecToExt: opcode 40, ser 0, ex 0, errno 0
blks_in_queue/wp_ca_blk_no/wp_max_no = 1/1000/17
LOCK WP ca_blk 1
make DISP owner of wp_ca_blk 1
DpRqPutIntoQueue: put request into queue (reqtype 1, prio LOW, rq_id 12)
MBUF component DOWN
NiBufClose: clear extensions for handle 3
NiBufSetStat: bufstat of hdl 3 changed from OK to OFF
NiICloseHandle: shutdown and close nihandle-socket 3-1592
MsIDetach: detach MS-system
EsCleanup ....
***LOG Q05=> DpHalt, DPStop ( 1212) [dpxxdisp.c 8225]
Good Bye .....