Hi,
Our production server stopped today with the dispatcher giving the follwoing message
"Dispatcher runing,dialogue queue standstill"
We found all dialog work processes being filled up y user SAPSYS.
We restarted SAP and this resolved the issue.
I would like to investigate further to know the rootcause of the issue.
Here I have attached the dev_w0 file.
Thanks,
Rajiv
Wed Apr 28 15:50:57 2010
C User requested cancel of current SQL on connection 0
M program canceled
M reason = soft cancel
M user = SAPSYS
M client = 000
M terminal = 10.61.10.207
M report = SAPLSDCL2
C OCI-call failed with -1=OCI_ERROR
C SQL error 1013: 'ORA-01013: user requested cancel of current operation'
C SQL cancelled; calling registered fct. at 00000000004BC6D0
M program canceled
M reason = soft cancel
M user = SAPSYS
M client = 000
M terminal = 10.61.10.207
M report = SAPLSDCL2
M in_ThErrHandle: 1
M ThSoftCancel: no login (step 4, th_errno 50, action 1, level 1)
M ThResetEmMagic: reset em magic for T30/M0/I0
M PfStatDisconnect: disconnect statistics
M Entering ThSetStatError
M ThIRollBack
M ThIRollBack: call db_rollback
M ThCheckComOrRb (event=0, full_commit=1)
M ThCallHooks: call hook >ab_SosRfcBeforeRollback< for event BEFORE_ROLLBACK
M ThCallHooks: call hook >ASTAT-collect commit handling< for event BEFORE_ROLLBACK
M PfAStatRevertCollected: nothing to do
M ThCallHooks: call hook >dyKeyTableReset< for event BEFORE_ROLLBACK
M ThCallHooks: call hook >SpoolHandleHook< for event BEFORE_ROLLBACK
M ThCheckComOrRb (event=2, full_commit=1)
M ThCallHooks: call hook >ThVBICmRbHook< for event AFTER_ROLLBACK
M ThVBICmRbHook: called for rollback
M ThVBIRollBackRedo: got 0 entries to delete again
M ThShortCommit: call db_commit98 (keep resources)
M ThCheckComOrRb (event=1, full_commit=0)
M ThCheckComOrRb (event=3, full_commit=0)
M ThShortCommit o.k.
M ThCallHooks: call hook >ThNoClearPrevErr< for event AFTER_ROLLBACK
M ThNoClearPrevErr: clear prev no err
M ThCallHooks: call hook >rsts_after_rollback< for event AFTER_ROLLBACK
M ThCallHooks: call hook >ThNoResetMutex< for event AFTER_ROLLBACK
M ThNoCheckMutex: check for remaining locked intervals
M ThCallHooks: call hook >SpoolHandleHook< for event AFTER_ROLLBACK
M ThICommit3: full commit, set time, free resources, redispatch
M ThICommit3: commit and free resources
M ThCheckComOrRb (event=1, full_commit=1)
M ThCallHooks: call hook >ab_SosShmBeforeCommit< for event BEFORE_COMMIT
M ThCallHooks: call hook >ab_SosRfcBeforeCommit< for event BEFORE_COMMIT
M ThCallHooks: call hook >ASTAT-collect commit handling< for event BEFORE_COMMIT
M ThCallHooks: call hook >rsts_before_commit< for event BEFORE_COMMIT
M ThCheckComOrRb (event=3, full_commit=1)
M ThCallHooks: call hook >ThVBICmRbHook< for event AFTER_COMMIT
M ThVBICmRbHook: called for commit
M ThCallHooks: call hook >dyKeyTableReset< for event AFTER_COMMIT
M ThCallHooks: call hook >rsts_after_commit< for event AFTER_COMMIT
M ThCallHooks: call hook >ThNoClearPrevErr< for event AFTER_COMMIT
M ThNoClearPrevErr: clear prev no err
M ThCallHooks: call hook >SpoolHandleHook< for event AFTER_COMMIT
M ThrClearMemory o.k.
M ThCallHooks: call hook >rsts_storage_free< for event AFTER_STORAGE_FREE
M ThCallHooks: call hook >db_x_short_mem_sync_point< for event AFTER_STORAGE_FREE
M ThAlarm: set alarm to 7200 sec
M ThICommit3 o.k.
M ThIErrHandle: do not call ThrCoreInfo (no_core_info=0, in_dynp_env=1)
M ThIErrHandle: Entering ThErrHdlUser
M ThIAMDel: delete tid/mode 30/0 (th_errno 50, release 1)
M ThTSCancel: set T30/M0 in status MODE_CANCEL
M ThIDeleteMode (30, 0, 3, ><, 0, 255, TRUE)
M ThIDeleteMode: no modes found ..
M no sub modes
M ThCheckMemoryState (0, 0, 1)
M ThIAMDel2: last active mode (0) delete tid (30)
M ThIUsrDel (50, 1)
M ThCheckMemoryState (0, 0, 1)
M ThIUsrDel: set status of T30 to TM_NEND
M ThResFree: free resources of U4951 M0 I0 (normal mode, complete free) at level 3, errno=50, db_action=TH_DB_ROLLBACK, pooling=1
M ThResFree: reset spa state for user T30/U4951/M0
M ThCheckMemoryState (0, 0, 1)
M ThCallHooks: call hook >ITSP: EndOfSession< for event BEFORE_SESSION_CANCEL
M ThCallHooks: call hook >abRtmClearSession< for event BEFORE_SESSION_CANCEL
M ThCallHooks: call hook >HttpClearSession< for event BEFORE_SESSION_CANCEL
M ThCallHooks: call hook >SpoolHandleHook< for event BEFORE_SESSION_CANCEL
M ThIRollBack
M ThIRollBack: call db_rollback
M ThCheckComOrRb (event=0, full_commit=1)
M ThCallHooks: call hook >ab_SosRfcBeforeRollback< for event BEFORE_ROLLBACK
M ThCallHooks: call hook >ASTAT-collect commit handling< for event BEFORE_ROLLBACK
M PfAStatRevertCollected: nothing to do
M ThCallHooks: call hook >dyKeyTableReset< for event BEFORE_ROLLBACK
M ThCallHooks: call hook >SpoolHandleHook< for event BEFORE_ROLLBACK
M ThCheckComOrRb (event=2, full_commit=1)
M ThCallHooks: call hook >ThVBICmRbHook< for event AFTER_ROLLBACK
M ThVBICmRbHook: called for rollback
M ThVBIRollBackRedo: got 0 entries to delete again
M ThShortCommit: call db_commit98 (keep resources)
M ThCheckComOrRb (event=1, full_commit=0)
M ThCheckComOrRb (event=3, full_commit=0)
M ThShortCommit o.k.
M ThCallHooks: call hook >ThNoClearPrevErr< for event AFTER_ROLLBACK
M ThNoClearPrevErr: clear prev no err
M ThCallHooks: call hook >rsts_after_rollback< for event AFTER_ROLLBACK
M ThCallHooks: call hook >ThNoResetMutex< for event AFTER_ROLLBACK
M ThNoCheckMutex: check for remaining locked intervals
M ThCallHooks: call hook >SpoolHandleHook< for event AFTER_ROLLBACK
M ThICommit3: full commit, set time, free resources, redispatch
M ThICommit3: commit and free resources
M ThCheckComOrRb (event=1, full_commit=1)
M ThCallHooks: call hook >ab_SosShmBeforeCommit< for event BEFORE_COMMIT
M ThCallHooks: call hook >ab_SosRfcBeforeCommit< for event BEFORE_COMMIT
M ThCallHooks: call hook >ASTAT-collect commit handling< for event BEFORE_COMMIT
M ThCallHooks: call hook >rsts_before_commit< for event BEFORE_COMMIT
M ThCheckComOrRb (event=3, full_commit=1)
M ThCallHooks: call hook >ThVBICmRbHook< for event AFTER_COMMIT
M ThVBICmRbHook: called for commit
M ThCallHooks: call hook >dyKeyTableReset< for event AFTER_COMMIT
M ThCallHooks: call hook >rsts_after_commit< for event AFTER_COMMIT
M ThCallHooks: call hook >ThNoClearPrevErr< for event AFTER_COMMIT
M ThNoClearPrevErr: clear prev no err
M ThCallHooks: call hook >SpoolHandleHook< for event AFTER_COMMIT
M ThrClearMemory o.k.
M ThCallHooks: call hook >rsts_storage_free< for event AFTER_STORAGE_FREE
M ThCallHooks: call hook >db_x_short_mem_sync_point< for event AFTER_STORAGE_FREE
M ThAlarm: set alarm to 7200 sec
M ThICommit3 o.k.
M PfStatDisconnect: disconnect statistics
M ThDealComm: T30/U4951/M0 has no cpic conns
M clean up all internal modes of T30/U4951/M0
M call ab_endmode (END_MODE_CLEAR_ALL)
M RstrNotifyUserChange: user/client = (SAPSYS /000)
M ***LOG R47=> ThResFree, delete (001050) [thxxmode.c 1384]
M ThVmcCleanMode (deleteOfMainMode=1, AoS=0)
M DpVmcGetVmByTmAdm: no VM found for T30/M0/I0
M ThSesGetSessionType: no associated entry
M ThSesIsJavaSessionId: no associated entry
M ThVmcCleanMode o.k.
M ThGetEnqInfo: no enq info found
M ThRmLocQueue: no async request found
M ThPlgCleanAll: clean all plugins of T30/M0/I0 (th_errno=50, level=3)
M ThPlgClean: clean plugin 0 of T30/M0 (th_errno=50)
M ThPlgClean: call cleanup handler
M ThPlgClean: clean server plugin
M DpSetContextCounter: num_used_contexts[1] = 6
M ThPlgClean: errno=50, state=DP_PLUGIN_START_WRITE, cancel plugins
M ThPlgClean: cancel read mpi <7d>
M ThPlgClean: cancel write mpi <7e>
M ThPageFree: free (4951,0,0, mode = 3)
M ThRollFree: free (4951,0,0, mode = 3)
M ThRmFree: cleanup resources for T30 M0
M RmCleanUpResources3: hdr/tbl/ext_tbl/free/keep = 000000000FC4F948/000007FF3AE9A040/0/1/0
M RmCleanUpResources3: call resource handler 00000000009530D0 for key/entry 10/0
M RmInitResourceTblEntry: free resource description at 000007FF3AE9F0E0 (ext_tbl=0)
M RmIFreeTable: free resource table at 000007FF3AE9A040 (ext_tbl=0)
M ThRmFree: reset eg state of T30 M0
M ThRmFree cleanup resources for T30
M RmCleanUpResources3: hdr/tbl/ext_tbl/free/keep = 000000000FC55578/0000000000000000/1/1/0
M RmCleanUpResources3: no resources registered
M ThRmFree: reset eg state of T30
M ThIEmFree: delete hyper context (hyperHdl=000007FF3AE88E50)
M ThIEmFree: call EmContextDelete (em_hdl=1)
M ThResFree: save spa pointer 000007FFD1443CD0 and install dummy spa
M ThClearMode: clear tid/mode 30/0 (next/prev 0/0)
M ThIUsrDel: set state of term T30 to TM_NEND
M ThISend: (tm/user/type/info = 30/SAPSYS /0x41/0x0, mode_deleted=1)
M TASK_SWITCH
M DISP
M ThCheckAbapDebugging: abap check returned FALSE
M ThCheckAbapDebugging: message for dispatcher, debugging ok
M DpDebuggingActive: check T30/M0
M ThCheckAbapDebugging: return 0
M ThJCheckJavaDebugging: return 0
M abap strategy ROLL / O.K.
M ThISend: set TH_LOGOFF for T30 (user=SAPSYS ) in state TM_NEND
M ThRqOutCheck: o.k.
M ThISend: allowed rq_type of T30/M0 = TH_ALL_RQ
M ThNewWpStat (type=0x10000041, task_switch=1, inline_hold=0, hand_shake=0, debug=0, ..)
M ThNewWpStat: new MODE_REC = 0x40
M ThNewWpStat: set mode wait
M ThNewWpStat: new state of T30/M0 = 0x41
M ThISend: new wp stat: 0x2
M ThDtWpBlk: detach WP ca_blk 66
M ThISend: complete stat record
M ThISend: stat_state = TH_STAT_WRITE
M ThISend: th_wp_waits_for_rq = TH_WP_WAITS_FOR_RQ
M make DISP owner of wp_ca_blk 66
M DpRqPutIntoQueue: put request into queue (reqtype 0, prio LOW, rq_id 53483)
M -OUT- sender_id WORK_PROCESS tid 30 wp_ca_blk 66 wp_id 0
M -OUT- action SEND_TO_DP uid 4951 appc_ca_blk -1 type NOWP
M -OUT- new_stat WP_WAIT mode 0 len 8 rq_id 53483
M -OUT- req_info LOGOFF
M wake_evt_udp_now 1, no dgram needed
M invalidate act fields from T30/M0
M REL APPC ca_blk 132
M ThDtAppcBlk: detach APPC ca_blk 132
M PfClearAllRec: clear statistic area
M PfStatDisconnect: disconnect statistics
M ThCallHooks: call hook >SAP-Trace buffer write< for event BEFORE_TASK_SWITCH
M TrThHookFunc: called for task switch
M ThCallHooks: call hook >ImcResetPtr< for event BEFORE_TASK_SWITCH
M ThCallHooks: call hook >DbgBeforeTaskSwitch< for event BEFORE_TASK_SWITCH
M ThCallHooks: call hook >HttpResetPtr< for event BEFORE_TASK_SWITCH
M Pipe read request: 2 anz: 64
M NiWait: sleep (1000ms) ...
M Signal handling in the signal thread (signal 2)
M SigIRegisterRoutine: handler for signal 2 installed (DpSigInt)
M ThWpSoftCancelHandler: during nosync step, reject softcancel
M Wed Apr 28 15:50:58 2010
M TskhLoop: waited 1 seconds for arrival of softcancel signalTskhLoop: switch back to normal tracel level 1
M
M Wed Apr 28 15:50:59 2010
M CrtChkOsSem: create check mutex and RC=[already exists] (Key:33/Name:SAP_SYSTEM_MUTEX_20_33/Hdl:1224)