System went down while trying to configure TREX
Hello colleagues,
I am managing an ERP 6.0 EHP6 system running on HANA and was trying to configure TREX installed on the same host. While running the report ESH_ADM_INDEX_ALL_SC in SE38 and is not coming up after that. I tried a kernel upgrade and system came up once but again went down while running the report.
While starting the system disp+work goes to state "System is STARTING" and stops. In the current status of system in SAP MMC, I can see an error in the DatabaseClient category. ABAP_SQL_ERROR:[Error DBIF_RSQL_SQL_ERROR_OCCURED........
Below is the developer trace of disp+work.
************************************************************************************************************************************
---------------------------------------------------
trc file: "dev_disp", trc level: 1, release: "740"
---------------------------------------------------
sysno 00
sid BO4
systemid 562 (PC with Windows NT)
relno 7400
patchlevel 0
patchno 37
intno 20020600
make multithreaded, Unicode, 64 bit, optimized
profile \\p246198\sapmnt\BO4\SYS\profile\BO4_DVEBMGS00_p246198
pid 5212
Wed Nov 06 15:41:51 2013
kernel runs with dp version 184000(ext=116000) (@(#) DPLIB-INT-VERSION-184000-UC)
length of sys_adm_ext is 500 bytes
*** SWITCH TRC-HIDE on ***
***LOG Q00=> DpSapEnvInit, DPStart (00 5212) [dpInit.c 548]
shared lib "dw_xml.dll" version 37 successfully loaded
shared lib "dw_xtc.dll" version 37 successfully loaded
shared lib "dw_stl.dll" version 37 successfully loaded
shared lib "dw_gui.dll" version 37 successfully loaded
shared lib "dw_mdm.dll" version 37 successfully loaded
shared lib "dw_rndrt.dll" version 37 successfully loaded
shared lib "dw_abp.dll" version 37 successfully loaded
DpCheckMulticast: check of multicast address o.k.
rdisp/softcancel_sequence : -> 0,5,-5
use internal message server connection to port 3901
rdisp/shutdown/disable_login : 0
rdisp/snapshot(CREATE): DP_CS_RESOURCES_EXHAUSTED
rdisp/snapshot(PERIOD):300 sec
rdisp/snapshot(LINES):10000 lines
DpCommonParamInit: rdisp/core_file_size = default --> no change
Wed Nov 06 15:41:56 2013
*** WARNING => DpNetCheck: NiAddrToHost(1.0.0.0) took 5 seconds
***LOG GZZ=> 1 possible network problems detected - check tracefile and adjust the DNS settings [dpxxtool2.c 4927]
MtxInit: 30000 0 0
DpIPCInit2: write dp-profile-values into sys_adm_ext
DpIPCInit2: start server >p246198_BO4_00 <
DpShMCreate: alloate/attach shared memory (mode=CREATE)
DpShMCreate: sizeof(wp_adm) 22848 (1088)
DpShMCreate: sizeof(tm_adm) 7055904 (35104)
DpShMCreate: sizeof(ca_adm) 432000 (72)
DpCommTableSize: max/headSize/ftSize/tableSize=500/16/1336064/1455152
DpShMCreate: sizeof(comm_adm) 1455152 (2648)
DpSlockTableSize: max/headSize/ftSize/fiSize/tableSize=0/0/0/0/0
DpShMCreate: sizeof(slock_adm) 0 (296)
DpFileTableSize: max/headSize/ftSize/tableSize=0/0/0/0
DpShMCreate: sizeof(file_adm) 0 (80)
DpSockTableSize: max/headSize/ftSize/tableSize=500/16/520064/520080
DpShMCreate: sizeof(sock_adm) 520080 (1016)
DpShMCreate: sizeof(vmc_adm) 0 (3208)
DpShMCreate: sizeof(wall_adm) (ft=41664/fi=67472/hd=64/rec=192)
DpShMCreate: sizeof(amc_rec_adm) (ft=86464/fi=67472/hd=64/rec=408)
DpShMCreate: sizeof(websocket_adm) (ft=38464/hd=64/rec=176)
DpShMCreate: sizeof(gw_adm) 48
DpShMCreate: sizeof(j2ee_adm) 3936
DpShMCreate: SHM_DP_ADM_KEY (addr: 00000000129C0050, size: 9896768)
DpShMCreate: allocated sys_adm at 00000000129C0250
DpShMCreate: allocated wp_adm_list at 00000000129D1C80
DpShMCreate: allocated wp_adm at 00000000129D1FA0
DpShMCreate: allocated tm_adm_list at 00000000129D7AE0
DpShMCreate: allocated tm_adm at 00000000129D7D20
DpShMCreate: allocated ca_adm at 0000000013092940
DpShMCreate: allocated comm_adm at 00000000130FC2C0
DpShMCreate: system runs without slock table
DpShMCreate: allocated sock_adm at 000000001325F8F0
DpShMCreate: allocated vmc_adm_list at 00000000132DEA80
DpShMCreate: system runs without VMC
DpShMCreate: allocated gw_adm at 00000000132DED20
DpShMCreate: allocated j2ee_adm at 00000000132DEF50
DpShMCreate: allocated ca_info at 00000000132E00B0
DpShMCreate: allocated wall_adm (ft) at 00000000132E02F0
DpShMCreate: allocated wall_adm (fi) at 00000000132EA7B0
DpShMCreate: allocated wall_adm (head) at 00000000132FB140
DpShMCreate: allocated amc_rec_adm (ft) at 00000000132FB380
DpShMCreate: allocated amc_rec_adm (fi) at 0000000013310740
DpShMCreate: allocated amc_rec_adm (head) at 00000000133210D0
DpShMCreate: allocated websocket_adm (ft) at 0000000013321310
DpShMCreate: allocated websocket_adm (head) at 000000001332AB50
DpShMCreate: initialized 21 eyes
DpSysAdmIntInit: initialize sys_adm
Scheduler info
--------------
WP info
#dia = 10
#btc = 0
#standby = 0
#max = 21
General Scheduler info
preemptionInfo.isActive = true
preemptionInfo.timeslice = 500
preemptionInfo.checkLoad = true
Prio Class High
maxRuntime[RQ_Q_PRIO_HIGH] = 600 sec
maxRuntimeHalf[RQ_Q_PRIO_HIGH] = 300 sec
Prio Class Normal
maxRuntime[RQ_Q_PRIO_NORMAL] = 3600 sec
maxRuntimeHalf[RQ_Q_PRIO_NORMAL] = 1800 sec
maxTicketsForPrio[RQ_Q_PRIO_NORMAL] = 8
withPrioTickets[RQ_Q_PRIO_NORMAL] = true
Prio Class Low
maxRuntime[RQ_Q_PRIO_LOW] = infinite
maxRuntimeHalf[RQ_Q_PRIO_LOW] = infinite
maxTicketsForPrio[RQ_Q_PRIO_LOW] = 8
withPrioTickets[RQ_Q_PRIO_LOW] = true
DpCommAttachTable: attached comm table (header=00000000130FC2C0/ft=00000000130FC2D0/fi=00000000132425D0)
DpSockAttachTable: attached sock table (header=000000001325F8F0/ft=000000001325F900)
MBUF state OFF
DpCommInitTable: init table for 500 entries
DpSockInitTable: init table for 500 entries
EmInit: MmSetImplementation( 2 ).
MM global diagnostic options set: 0
<ES> client 21 initializing ....
<ES> EsILock: use spinlock for locking
<ES> InitFreeList
<ES> block size is 4096 kByte.
<ES> Info: em/initial_size_MB( 34399MB) not multiple of em/blocksize_KB( 4096KB)
<ES> Info: em/initial_size_MB rounded up to 34400MB
<ES> Info: disclaim threshold = 0 MB
<ES> Info: disclaim coasting/alloc = 300 s
<ES> Info: disclaim coasting/free = 0 s
<ES> Info: blockdisclaimsize_KB = 0 KB
Using implementation view
<EsNT> Using memory model view.
<EsNT> Memory Reset disabled as NT default
<ES> 8599 blocks reserved for free list.
ES initialized.
mm.dump: set maximum dump mem to 96 MB
rdisp/calculateLoadAverage : 1
=================================================
= SSL Initialization platform tag=(ntamd64-msc16)
= (740_REL,Sep 27 2013,mt,ascii-uc,SAP_UC/size_t/void* = 16/64/64)
DIR_INSTANCE="F:\usr\sap\BO4\DVEBMGS00"
DIR_LIBRARY="F:\usr\sap\BO4\DVEBMGS00\exe"
= Initialization with _no_ default credentials
ssl/ssl_lib="F:\usr\sap\BO4\DVEBMGS00\exe\sapcrypto.dll"
profile param "ssl/ssl_lib" = "F:\usr\sap\BO4\DVEBMGS00\exe\sapcrypto.dll"
resulting Filename = "F:\usr\sap\BO4\DVEBMGS00\exe\sapcrypto.dll"
= found CommonCryptoLib (SAPCRYPTOLIB) Version 8.4.7 pl35 (2.0 SP1 Patch 2) (Aug 13 2013) MT-safe
= current UserID: P246198\SAPServiceBO4
= found SECUDIR environment variable
= using SECUDIR=F:\usr\sap\BO4\DVEBMGS00\sec
ssl/ciphersuites="193:HIGH:MEDIUM:+e3DES"
ssl/client_ciphersuites="192:HIGH:MEDIUM:+e3DES"
= Success -- SapCryptoLib SSL ready!
=================================================
***LOG Q0K=> DpMsAttach, mscon ( p246198) [dpMessageSer 1513]
MBUF state LOADING
DpStartStopMsg: send start message (myname is >p246198_BO4_00 <)
DpStartStopMsg: start msg sent to message server o.k.
MPI: dynamic quotas disabled.
MPI init: pipes=4000 buffers=6395 reserved=1918 quota=10%
CCMS: Initialized monitoring segment of size 60000000.
CCMS: Initialized CCMS Headers in the shared monitoring segment.
CCMS: Checking Downtime Configuration of Monitoring Segment.
CCMS: AlMsUpload called by wp 1024.
Wed Nov 06 15:41:59 2013
CCMS: AlMsUpload successful for F:\usr\sap\BO4\DVEBMGS00\log\ALMTTREE.DAT (5685 MTEs).
Wed Nov 06 15:42:00 2013
CCMS: start to initalize 3.X shared alert area (first segment).
DpCheckPreemptionTicker: created thread for DpPremptionTicker
DpMBufHwIdSet: set Hardware-ID
***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c 1330]
MBUF state ACTIVE
DpMBufTypeMod: MBUF component UP (event=MBUF_DPEVT_UP)
DpMsgProcess: 1 server in MBUF
DpMsCheckServices()
DpMsCheckServices: send DpDelService(J2EE) to msgserver
Wed Nov 06 15:42:36 2013
*** ERROR => DpHdlDeadWp: W0 (pid 180) died (severity=0, status=0) [dpxxwp.c 1277]
********** SERVER SNAPSHOT 1 (Reason: Workprocess 0 died) - begin **********
Scheduler info
--------------
WP info
DpNumberOfDiaWps: dia_wps 9, standby_wps 0
#dia = 9
#btc = 3
#standby = 0
#max = 21
General Scheduler info
preemptionInfo.isActive = true
preemptionInfo.timeslice = 500
preemptionInfo.checkLoad = true
Prio Class High
maxRuntime[RQ_Q_PRIO_HIGH] = 600 sec
maxRuntimeHalf[RQ_Q_PRIO_HIGH] = 300 sec
Prio Class Normal
maxRuntime[RQ_Q_PRIO_NORMAL] = 3600 sec
maxRuntimeHalf[RQ_Q_PRIO_NORMAL] = 1800 sec
maxTicketsForPrio[RQ_Q_PRIO_NORMAL] = 8
withPrioTickets[RQ_Q_PRIO_NORMAL] = true
Prio Class Low
maxRuntime[RQ_Q_PRIO_LOW] = infinite
maxRuntimeHalf[RQ_Q_PRIO_LOW] = infinite
maxTicketsForPrio[RQ_Q_PRIO_LOW] = 8
withPrioTickets[RQ_Q_PRIO_LOW] = true
Actual tickets in use
actTicketsInUseForPrio[RQ_Q_PRIO_NORMAL] = 0
actTicketsInUseForPrio[RQ_Q_PRIO_LOW] = 0
Queue Statistics Wed Nov 06 14:42:36 2013
------------------------------------------------------------
Number of lost wakeup datagrams: 0
Max. number of queue elements : 14000
NOWP: 0 (peak 5, writeCount 8, readCount 8)
DIA: 4 (peak 4, writeCount 4, readCount 0)
UPD: 0 (peak 0, writeCount 0, readCount 0)
ENQ: 0 (peak 0, writeCount 0, readCount 0)
BTC: 0 (peak 0, writeCount 0, readCount 0)
SPO: 0 (peak 0, writeCount 0, readCount 0)
UP2: 0 (peak 0, writeCount 0, readCount 0)
Session queue dump (high priority, 1 elements, peak 1):
-1 <- 26 < EmbryoQueue_DIA> -> -1
Session queue dump (normal priority, 0 elements, peak 0):
Session queue dump (low priority, 0 elements, peak 0):
Dump of queue <EmbryoQueue_DIA> in slot 26 (4 requests, port=0):
-1 <- 4 (rq_id 7, DIA, REQ_HANDLER_AUTOTH) -> 5
4 <- 5 (rq_id 9, DIA, REQ_HANDLER_MSADM) -> 2
5 <- 2 (rq_id 13, DIA, REQ_HANDLER_AUTOABAP) -> 3
2 <- 3 (rq_id 15, DIA, REQ_HANDLER_BGRFC_WATCHDOG) -> -1
Infos about some special queues:
Queue <DispatcherQueue> in slot 0 (port=55510) has no requests
Queue <GatewayQueue> in slot 1 (port=0) has no requests
Queue <IcmanQueue> in slot 2 (port=55596) has no requests
Queue <StartServiceQueue> in slot 3 (port=0) has no requests
Workprocess Table (long) Wed Nov 06 14:42:36 2013
------------------------------------------------------------
Current snapshot id: 1
Number of preemptions : 0
|No |Pid |Type|State |Cause|Err|Prio|Sess-Key |Sess-Type|Locked|Sem|Time |Program |Cli|User |Action |Action-Info |
|---+------+----+-------+-----+---+----+----------------+---------+------+---+-----+----------------------------------------+---+------------+--------------------+--------------------|
| 0| |DIA |WP_KILL| |1 | | | | | | | | | | | |
| 1|4296 |DIA |WP_NEW | | | | | | | | | | | | | |
| 2|5164 |DIA |WP_NEW | | | | | | | | | | | | | |
| 3|5156 |DIA |WP_NEW | | | | | | | | | | | | | |
| 4|6064 |DIA |WP_NEW | | | | | | | | | | | | | |
| 5|6096 |DIA |WP_NEW | | | | | | | | | | | | | |
| 6|6092 |DIA |WP_NEW | | | | | | | | | | | | | |
| 7|6084 |DIA |WP_NEW | | | | | | | | | | | | | |
| 8|1172 |DIA |WP_NEW | | | | | | | | | | | | | |
| 9|6136 |DIA |WP_NEW | | | | | | | | | | | | | |
| 10|5940 |UPD |WP_NEW | | | | | | | | | | | | | |
| 11|3940 |BTC |WP_NEW | | | | | | | | | | | | | |
| 12|5160 |BTC |WP_NEW | | | | | | | | | | | | | |
| 13|2712 |BTC |WP_NEW | | | | | | | | | | | | | |
| 14|6132 |SPO |WP_NEW | | | | | | | | | | | | | |
| 15|4616 |UP2 |WP_NEW | | | | | | | | | | | | | |
Session Table Wed Nov 06 14:42:36 2013
------------------------------------------------------------
No logons/sessions found
CA Blocks
0 INVALID -1
1 ca_blk slots of 6000 in use, 1 currently unowned (in request queues)
********** SERVER SNAPSHOT 1 (Reason: Workprocess 0 died) - end **********
***LOG Q41=> DpDumpInternalTables, () [dpxxdisp.c 3069]
my types changed after wp death/restart: 0x9b --> 0xbb
*** ERROR => DpHdlDeadWp: W1 (pid 4296) died (severity=0, status=0) [dpxxwp.c 1277]
DpSkipSnapshot: delayed snapshot exist, skip new snapshot
*** ERROR => DpHdlDeadWp: W2 (pid 5164) died (severity=0, status=0) [dpxxwp.c 1277]
DpSkipSnapshot: delayed snapshot exist, skip new snapshot
*** ERROR => DpHdlDeadWp: W3 (pid 5156) died (severity=0, status=0) [dpxxwp.c 1277]
DpSkipSnapshot: delayed snapshot exist, skip new snapshot
*** ERROR => DpHdlDeadWp: W4 (pid 6064) died (severity=0, status=0) [dpxxwp.c 1277]
DpSkipSnapshot: delayed snapshot exist, skip new snapshot
*** ERROR => DpHdlDeadWp: W5 (pid 6096) died (severity=0, status=0) [dpxxwp.c 1277]
DpSkipSnapshot: delayed snapshot exist, skip new snapshot
*** ERROR => DpHdlDeadWp: W6 (pid 6092) died (severity=0, status=0) [dpxxwp.c 1277]
DpSkipSnapshot: delayed snapshot exist, skip new snapshot
*** ERROR => DpHdlDeadWp: W7 (pid 6084) died (severity=0, status=0) [dpxxwp.c 1277]
DpSkipSnapshot: delayed snapshot exist, skip new snapshot
*** ERROR => DpHdlDeadWp: W8 (pid 1172) died (severity=0, status=0) [dpxxwp.c 1277]
DpSkipSnapshot: delayed snapshot exist, skip new snapshot
*** ERROR => DpHdlDeadWp: W9 (pid 6136) died (severity=0, status=0) [dpxxwp.c 1277]
DpSkipSnapshot: delayed snapshot exist, skip new snapshot
*** ERROR => DpHdlDeadWp: W10 (pid 5940) died (severity=0, status=0) [dpxxwp.c 1277]
DpSkipSnapshot: delayed snapshot exist, skip new snapshot
*** ERROR => DpHdlDeadWp: W11 (pid 3940) died (severity=0, status=0) [dpxxwp.c 1277]
DpSkipSnapshot: delayed snapshot exist, skip new snapshot
*** ERROR => DpHdlDeadWp: W12 (pid 5160) died (severity=0, status=0) [dpxxwp.c 1277]
DpSkipSnapshot: delayed snapshot exist, skip new snapshot
*** ERROR => DpHdlDeadWp: W13 (pid 2712) died (severity=0, status=0) [dpxxwp.c 1277]
DpSkipSnapshot: delayed snapshot exist, skip new snapshot
*** ERROR => DpHdlDeadWp: W14 (pid 6132) died (severity=0, status=0) [dpxxwp.c 1277]
DpSkipSnapshot: delayed snapshot exist, skip new snapshot
*** ERROR => DpHdlDeadWp: W15 (pid 4616) died (severity=0, status=0) [dpxxwp.c 1277]
DpSkipSnapshot: delayed snapshot exist, skip new snapshot
*** DP_FATAL_ERROR => DpWpCheck: no more work processes
*** DISPATCHER EMERGENCY SHUTDOWN ***
increase tracelevel of WPs
NiIWait: sleep (10000ms) ...
NiISelect: timeout 10000ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Wed Nov 06 15:42:37 2013
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Wed Nov 06 15:42:38 2013
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Wed Nov 06 15:42:39 2013
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Wed Nov 06 15:42:40 2013
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Wed Nov 06 15:42:41 2013
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Wed Nov 06 15:42:42 2013
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Wed Nov 06 15:42:43 2013
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Wed Nov 06 15:42:44 2013
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Wed Nov 06 15:42:45 2013
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Wed Nov 06 15:42:46 2013
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
NiISelect: TIMEOUT occured (10000ms)
dump system status
Workprocess Table (long) Wed Nov 06 14:42:46 2013
------------------------------------------------------------
DpMBufServGetOverloaded: load info of server >p246198_BO4_00 < is DP_NORMAL_LOAD
Current snapshot id: 1
Number of preemptions : 0
|No |Pid |Type|State |Cause|Err|Prio|Sess-Key |Sess-Type|Locked|Sem|Time |Program |Cli|User |Action |Action-Info |
|---+------+----+-------+-----+---+----+----------------+---------+------+---+-----+----------------------------------------+---+------------+--------------------+--------------------|
| 0| |DIA |WP_KILL| |1 | | | | | | | | | | | |
| 1| |DIA |WP_KILL| |1 | | | | | | | | | | | |
| 2| |DIA |WP_KILL| |1 | | | | | | | | | | | |
| 3| |DIA |WP_KILL| |1 | | | | | | | | | | | |
| 4| |DIA |WP_KILL| |1 | | | | | | | | | | | |
| 5| |DIA |WP_KILL| |1 | | | | | | | | | | | |
| 6| |DIA |WP_KILL| |1 | | | | | | | | | | | |
| 7| |DIA |WP_KILL| |1 | | | | | | | | | | | |
| 8| |DIA |WP_KILL| |1 | | | | | | | | | | | |
| 9| |DIA |WP_KILL| |1 | | | | | | | | | | | |
| 10| |UPD |WP_KILL| |1 | | | | | | | | | | | |
| 11| |BTC |WP_KILL| |1 | | | | | | | | | | | |
| 12| |BTC |WP_KILL| |1 | | | | | | | | | | | |
| 13| |BTC |WP_KILL| |1 | | | | | | | | | | | |
| 14| |SPO |WP_KILL| |1 | | | | | | | | | | | |
| 15| |UP2 |WP_KILL| |1 | | | | | | | | | | | |
Queue Statistics Wed Nov 06 14:42:46 2013
------------------------------------------------------------
Number of lost wakeup datagrams: 0
Max. number of queue elements : 14000
NOWP: 0 (peak 5, writeCount 8, readCount 8)
DIA: 4 (peak 4, writeCount 4, readCount 0)
UPD: 0 (peak 0, writeCount 0, readCount 0)
ENQ: 0 (peak 0, writeCount 0, readCount 0)
BTC: 0 (peak 0, writeCount 0, readCount 0)
SPO: 0 (peak 0, writeCount 0, readCount 0)
UP2: 0 (peak 0, writeCount 0, readCount 0)
Session queue dump (high priority, 1 elements, peak 1):
-1 <- 26 < EmbryoQueue_DIA> -> -1
Session queue dump (normal priority, 0 elements, peak 0):
Session queue dump (low priority, 0 elements, peak 0):
Dump of queue <EmbryoQueue_DIA> in slot 26 (4 requests, port=0):
-1 <- 4 (rq_id 7, DIA, REQ_HANDLER_AUTOTH) -> 5
4 <- 5 (rq_id 9, DIA, REQ_HANDLER_MSADM) -> 2
5 <- 2 (rq_id 13, DIA, REQ_HANDLER_AUTOABAP) -> 3
2 <- 3 (rq_id 15, DIA, REQ_HANDLER_BGRFC_WATCHDOG) -> -1
Infos about some special queues:
Queue <DispatcherQueue> in slot 0 (port=55510) has no requests
Queue <GatewayQueue> in slot 1 (port=0) has no requests
Queue <IcmanQueue> in slot 2 (port=55596) has no requests
Queue <StartServiceQueue> in slot 3 (port=0) has no requests
Logon Table Wed Nov 06 14:42:46 2013
------------------------------------------------------------
No logons/sessions found
CA Blocks
0 INVALID -1
1 ca_blk slots of 6000 in use, 1 currently unowned (in request queues)
NiIWait: sleep (5000ms) ...
NiISelect: timeout 5000ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Wed Nov 06 15:42:47 2013
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Wed Nov 06 15:42:48 2013
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Wed Nov 06 15:42:49 2013
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Wed Nov 06 15:42:50 2013
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Wed Nov 06 15:42:51 2013
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
NiISelect: TIMEOUT occured (500ms)
NiISelect: timeout 500ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
NiISelect: TIMEOUT occured (5000ms)
DpSkipSnapshot: delayed snapshot exist, skip new snapshot
DpHalt: shutdown server >p246198_BO4_00 < (normal)
DpUpdateStatusFile: update status file F:\usr\sap\BO4\DVEBMGS00\work\disp+work.status, unique number = 2
DpJ2eeDisableRestart
DpMBufRead: read server list from MBUF (1 entries)
DpMBufMod: name [p246198_BO4_00 ], types 155 state 3 (0x0 0x0 0x0 0x0)
DpMBufMod: call hooks for event MBUF_DPEVT_DSTATE (5->3)
AdGetSelfIdentRecord: > <
AdCvtRecToExt: opcode 60 (AD_SELFIDENT), ser 0, ex 0, errno 0
AdCallRegisteredCvtToExt: opcode 60
AdCallRegisteredCvtToExt: opcode 60 call 0000000141553610
AdCallRegisteredCvtToExt: opcode 60 exit rc=SAP_O_K
AdCvtRecToExt: opcode 40 (AD_MSBUF), ser 0, ex 0, errno 0
AdCallRegisteredCvtToExt: opcode 40
AdCallRegisteredCvtToExt: opcode 40 call 00000001415543E0
AdCallRegisteredCvtToExt: opcode 40 exit rc=SAP_O_K
AdCvtRecToExt: opcode 40 (AD_MSBUF), ser 0, ex 0, errno 0
AdCallRegisteredCvtToExt: opcode 40
AdCallRegisteredCvtToExt: opcode 40 call 00000001415543E0
AdCallRegisteredCvtToExt: opcode 40 exit rc=SAP_O_K
AdCvtRecToExt: opcode 40 (AD_MSBUF), ser 0, ex 0, errno 0
AdCallRegisteredCvtToExt: opcode 40
AdCallRegisteredCvtToExt: opcode 40 call 00000001415543E0
AdCallRegisteredCvtToExt: opcode 40 exit rc=SAP_O_K
AdCvtRecToExt: opcode 40 (AD_MSBUF), ser 0, ex 0, errno 0
AdCallRegisteredCvtToExt: opcode 40
AdCallRegisteredCvtToExt: opcode 40 call 00000001415543E0
AdCallRegisteredCvtToExt: opcode 40 exit rc=SAP_O_K
AdCvtRecToExt: opcode 40 (AD_MSBUF), ser 0, ex 0, errno 0
AdCallRegisteredCvtToExt: opcode 40
AdCallRegisteredCvtToExt: opcode 40 call 00000001415543E0
AdCallRegisteredCvtToExt: opcode 40 exit rc=SAP_O_K
AdCvtRecToExt: opcode 40 (AD_MSBUF), ser 0, ex 0, errno 0
AdCallRegisteredCvtToExt: opcode 40
AdCallRegisteredCvtToExt: opcode 40 call 00000001415543E0
AdCallRegisteredCvtToExt: opcode 40 exit rc=SAP_O_K
AdCvtRecToExt: opcode 40 (AD_MSBUF), ser 0, ex 0, errno 0
AdCallRegisteredCvtToExt: opcode 40
AdCallRegisteredCvtToExt: opcode 40 call 00000001415543E0
AdCallRegisteredCvtToExt: opcode 40 exit rc=SAP_O_K
AdCvtRecToExt: opcode 40 (AD_MSBUF), ser 0, ex 0, errno 0
AdCallRegisteredCvtToExt: opcode 40
AdCallRegisteredCvtToExt: opcode 40 call 00000001415543E0
AdCallRegisteredCvtToExt: opcode 40 exit rc=SAP_O_K
AdCvtRecToExt: opcode 40 (AD_MSBUF), ser 0, ex 0, errno 0
AdCallRegisteredCvtToExt: opcode 40
AdCallRegisteredCvtToExt: opcode 40 call 00000001415543E0
AdCallRegisteredCvtToExt: opcode 40 exit rc=SAP_O_K
AdCvtRecToExt: opcode 40 (AD_MSBUF), ser 0, ex 0, errno 0
AdCallRegisteredCvtToExt: opcode 40
AdCallRegisteredCvtToExt: opcode 40 call 00000001415543E0
AdCallRegisteredCvtToExt: opcode 40 exit rc=SAP_O_K
DpRqCreate: creating REQ_HANDLER_MSADM request (rq_id 17, slot 13)
DpCaLockBlk: locked ca_blk 1
DpRqSendRequest: REQ_HANDLER_MSADM (rq_id 17) from DISP to DISP
DpCaGiveUpOwnerOfBlk: give up ownership of ca_blk 1
rq_id 17
-OUT- ca_blk 1 len 1180
-OUT- reqHdlId REQ_HANDLER_MSADM
-OUT- sender address:
-OUT- type LOCAL
-OUT- agentType DISP
-OUT- receiver address:
-OUT- type LOCAL
-OUT- agentType DISP
RqQAddRequest: Add request rq_id 17 (NOWP) to queue <DispatcherQueue> in slot 0 at the end
Dump of queue <DispatcherQueue> in slot 0 (1 requests, in use, port=55510):
-1 <- 0 (rq_id 17, NOWP, REQ_HANDLER_MSADM) -> -1
RqQQueueNotifySelectableHandle: Reuse NI connection for queue <DispatcherQueue> in slot 0 on port 55510 (10026)
RqQQueueNotifySelectableHandle: Signal selectable handle for queue <DispatcherQueue> in slot 0 on port 55510 (id=6)
NiDgHdlSend: hdl 361 sent 1 of 1 bytes to 127.0.0.1:55510
RqQQueueGetNumberOfRequests: Queue <DispatcherQueue> in slot 0 contains 1 requests of type NOWP
DpRqPutIntoQueue: put request into DISP queue (workerType 0, prio LOW, rq_id 17), queue has 1 requests
DpRqFree: freeing REQ_HANDLER_MSADM request (rq_id 17, slot 13)
DpMBufTypeMod: MBUF server state SHUTDOWN (event=MBUF_DPEVT_DSTATE)
DpModState: change state STARTING -> SHUTDOWN
NiBufIAlloc: malloc MSLIB-BUF, to 32110 bytes
NiBufSend starting
NiIWrite: hdl 353 sent data (wrt=110,pac=1,MESG_IO)
<<< NiBufISelUpdate: hdl 353(0, SAP_O_K)
MsINiWrite: sent 110 bytes
MsIModState: change state to SHUTDOWN
DpModState: change server state from STARTING to SHUTDOWN
DpHalt: switch off Shared memory profiling
ShmProtect( 57, 3 )
ShmProtect(SHM_PROFILE, SHM_PROT_RW
ShmProtect( 57, 1 )
ShmProtect(SHM_PROFILE, SHM_PROT_RD
DpWakeUpWps: wake up all wp's
DpHalt: stop work processes
DpHalt: stop gateway
killing process (6108) (SOFT_KILL)
DpHalt: stop icman
killing process (4248) (SOFT_KILL)
DpHalt: terminate gui connections
DpHalt: wait for end of work processes
DpHalt: wait for end of gateway
[DpProcDied] Process lives (PID:6108 HANDLE:1120)
DpHalt: waiting for termination of gateway ...
NiIWait: sleep (1000ms) ...
NiISelect: timeout 1000ms
NiISelect: maximum fd=1221
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Wed Nov 06 15:42:52 2013
NiISelect: TIMEOUT occured (500ms)
NiISelect: TIMEOUT occured (1000ms)
[DpProcDied] Process died (PID:6108 HANDLE:1120 rc:0x0)
DpHalt: wait for end of icman
[DpProcDied] Process died (PID:4248 HANDLE:1128 rc:0x0)
DpStartStopMsg: send stop message (myname is >p246198_BO4_00 <)
DpStartStopMsg: Write AD_STARTSTOP message with type= 0, name=p246198_BO4_00 , sapsysnr= 0, hostname=p246198
AdGetSelfIdentRecord: > <
AdCvtRecToExt: opcode 60 (AD_SELFIDENT), ser 0, ex 0, errno 0
AdCallRegisteredCvtToExt: opcode 60
AdCallRegisteredCvtToExt: opcode 60 call 0000000141553610
AdCallRegisteredCvtToExt: opcode 60 exit rc=SAP_O_K
AdCvtRecToExt: opcode 4 (AD_STARTSTOP), ser 0, ex 0, errno 0
AdCallRegisteredCvtToExt: opcode 4
AdCallRegisteredCvtToExt: opcode 4 call 0000000141550780
AdCallRegisteredCvtToExt: opcode 4 exit rc=SAP_O_K
DpRqCreate: creating REQ_HANDLER_ADM request (rq_id 19, slot 14)
DpRqConvertToNetForm: converted from intern to net (164 bytes), rq_id 19
DpRqFree: freeing REQ_HANDLER_ADM request (rq_id 19, slot 14)
NiBufIAlloc: malloc MSLIB-BUF, to 32110 bytes
NiBufSend starting
NiIWrite: hdl 353 sent data (wrt=866,pac=1,MESG_IO)
<<< NiBufISelUpdate: hdl 353(0, SAP_O_K)
MsINiWrite: sent 866 bytes
MsISndTypeOnce: send msg (ms hdr/msg 110/756 bytes) to MSG_SERVER, type 1
DpStartStopMsg: stop msg sent to message server o.k.
NiBufIAlloc: malloc MSLIB-BUF, to 32110 bytes
NiBufIAlloc: malloc NIBUF-IN, to 32110 bytes
NiIRead: hdl 353 received data (rcd=274,pac=1,MESG_IO)
NiBufIIn: NIBUF len=274
NiBufIIn: packet complete for hdl 353
NiBufReceive starting
<<< NiBufISelUpdate: hdl 353(0, SAP_O_K)
MsINiRead: received 274 bytes
MsIReceive: received msg (ms hdr/msg 110/164 bytes), flag 1, from MSG_SERVER , typ 0, key -
DpHalt: received 164 bytes from message server
NiBufIAlloc: malloc NIBUF-IN, to 32110 bytes
NiIRead: hdl 353 received data (rcd=274,pac=1,MESG_IO)
NiBufIIn: NIBUF len=274
NiBufIIn: packet complete for hdl 353
NiBufReceive starting
<<< NiBufISelUpdate: hdl 353(0, SAP_O_K)
MsINiRead: received 274 bytes
MsIReceive: received msg (ms hdr/msg 110/164 bytes), flag 1, from MSG_SERVER , typ 0, key -
DpHalt: received 164 bytes from message server
NiIRead: hdl 353 recv would block (errno=EAGAIN)
NiIRead: read for hdl 353 timed out (0ms)
DpHalt: more messages from the message server
DpHalt: send keepalive to synchronize with the message server
NiBufSend starting
NiIWrite: hdl 353 sent data (wrt=114,pac=1,MESG_IO)
<<< NiBufISelUpdate: hdl 353(0, SAP_O_K)
MsINiWrite: sent 114 bytes
MsISnd2: send msg (ms hdr/msg 110/4 bytes) to name MSG_SERVER, type 0, key -
MsSndName: MS_NOOP ok
Send 4 bytes to MSG_SERVER
NiBufIAlloc: malloc MSLIB-BUF, to 32110 bytes
NiIRead: hdl 353 recv would block (errno=EAGAIN)
NiIPeek: peek successful for hdl 353 (r)
NiBufIAlloc: malloc NIBUF-IN, to 32110 bytes
NiIRead: hdl 353 received data (rcd=114,pac=1,MESG_IO)
NiBufIIn: NIBUF len=114
NiBufIIn: packet complete for hdl 353
NiBufReceive starting
<<< NiBufISelUpdate: hdl 353(0, SAP_O_K)
MsINiRead: received 114 bytes
MsIReceive: received msg (ms hdr/msg 110/4 bytes), flag 3, from MSG_SERVER , typ 0, key -
Received 4 bytes from MSG_SERVER
Received opcode MS_NOOP from msg_server, reply MSOP_OK
MsOpReceive: ok
MsISendKeepalive : keepalive sent to message server
NiIRead: hdl 353 recv would block (errno=EAGAIN)
Wed Nov 06 15:42:53 2013
NiIPeek: peek for hdl 353 timed out (r; 1000ms)
NiIRead: read for hdl 353 timed out (1000ms)
DpHalt: no more messages from the message server
DpHalt: sync with message server o.k.
DpHalt: detach from message server
***LOG Q0M=> DpMsDetach, ms_detach () [dpMessageSer 1856]
NiBufSend starting
NiIWrite: hdl 353 sent data (wrt=110,pac=1,MESG_IO)
<<< NiBufISelUpdate: hdl 353(0, SAP_O_K)
MsINiWrite: sent 110 bytes
MsIDetach: send logout to msg_server (p246198 / 3901)
MsIDetach: call exit function
DpMsShutdownHook called
NiBufISelUpdate: new MODE -- (r-) for hdl 353 in set0
SiSelNSet: set events of sock 1080 to: ---
<<< NiBufISelUpdate: hdl 353(0, SAP_O_K)
NiBufISelRemove: remove hdl 353 from set0
<<< NiBufISelUpdate: hdl 353(0, SAP_O_K)
SiSelNRemove: removed sock 1080
NiSelIRemove: removed hdl 353
DpDelSocketInfo: del info for nihdl 353 (pos/type=44/4)
DpMBufClearGlobalHighLoadCounter: reset global load flags
MBUF state OFF
AdGetSelfIdentRecord: > <
AdCvtRecToExt: opcode 60 (AD_SELFIDENT), ser 0, ex 0, errno 0
AdCallRegisteredCvtToExt: opcode 60
AdCallRegisteredCvtToExt: opcode 60 call 0000000141553610
AdCallRegisteredCvtToExt: opcode 60 exit rc=SAP_O_K
AdCvtRecToExt: opcode 40 (AD_MSBUF), ser 0, ex 0, errno 0
AdCallRegisteredCvtToExt: opcode 40
AdCallRegisteredCvtToExt: opcode 40 call 00000001415543E0
AdCallRegisteredCvtToExt: opcode 40 exit rc=SAP_O_K
AdCvtRecToExt: opcode 40 (AD_MSBUF), ser 0, ex 0, errno 0
AdCallRegisteredCvtToExt: opcode 40
AdCallRegisteredCvtToExt: opcode 40 call 00000001415543E0
AdCallRegisteredCvtToExt: opcode 40 exit rc=SAP_O_K
DpRqCreate: creating REQ_HANDLER_MSADM request (rq_id 20, slot 15)
DpCaLockBlk: locked ca_blk 2
DpRqSendRequest: REQ_HANDLER_MSADM (rq_id 20) from DISP to DISP
DpCaGiveUpOwnerOfBlk: give up ownership of ca_blk 2
rq_id 20
-OUT- ca_blk 2 len 348
-OUT- reqHdlId REQ_HANDLER_MSADM
-OUT- sender address:
-OUT- type LOCAL
-OUT- agentType DISP
-OUT- receiver address:
-OUT- type LOCAL
-OUT- agentType DISP
RqQAddRequest: Add request rq_id 20 (NOWP) to queue <DispatcherQueue> in slot 0 at the end
Dump of queue <DispatcherQueue> in slot 0 (3 requests, in use, port=55510):
-1 <- 0 (rq_id 17, NOWP, REQ_HANDLER_MSADM) -> 1
0 <- 1 (rq_id 18, NOWP, REQ_HANDLER_ICMAN_OPCODE) -> 6
1 <- 6 (rq_id 20, NOWP, REQ_HANDLER_MSADM) -> -1
RqQQueueNotifySelectableHandle: Reuse NI connection for queue <DispatcherQueue> in slot 0 on port 55510 (10026)
RqQQueueNotifySelectableHandle: Signal selectable handle for queue <DispatcherQueue> in slot 0 on port 55510 (id=8)
NiDgHdlSend: hdl 361 sent 1 of 1 bytes to 127.0.0.1:55510
RqQQueueGetNumberOfRequests: Queue <DispatcherQueue> in slot 0 contains 3 requests of type NOWP
DpRqPutIntoQueue: put request into DISP queue (workerType 0, prio LOW, rq_id 20), queue has 3 requests
DpRqFree: freeing REQ_HANDLER_MSADM request (rq_id 20, slot 15)
DpMBufTypeMod: MBUF component DOWN (event=MBUF_DPEVT_DOWN)
NiICloseHandle: shutdown and close hdl 353/sock 1080
NiBufIClose: clear extension for hdl 353
MsIDetach: detach MS-system (p246198 / 3901)
DpHalt: cleanup EM
EsCleanup( )
EsCleanup ....
EmCleanup() -> 0
Es2Cleanup: Cleanup ES2
***LOG Q05=> DpHalt, DPStop ( 5212) [dpInit.c 3353]
DpHalt: Good Bye .....
************************************************************************************************************************************
Could you please help me here ? Please let me know if you require any additional information.
Thanks and regards,
Sudarsan.