SAP system is not starting

[Former Member]

Dear Experts,

Currently, we're not able to start the SAP system from the operating system level. From the dev_disp log we found the below error message:

DpUpdateStatusFileWith: state=YELLOW, reason=Request handling without progress *** WARNING => DpRequestProcessingCheck: potential request processing problem detected (123. check) [dpxxwp.c 4705]

SAP NW system is running with ADA database and operating system Solaris 10.

Any ideas are appreciated.

Thanks & best regards,

Sreenu

[isaias_freitas]

Hi,

Can you please close this question if further assistance is not required?

Thanks!

[prithvirajr]

Hi Sreeni,

Background work process number 4 is the culprit here, as shown in dispatcher log.

Try killing the process if that helps. If not, look at the trace file dev_w10 to see what is causing it to hold Sem 10 for so long.

|No |Pid     |Type|State  |Cause|Err|Prio|Sess-Key        |Sess-Type|Locked|Sem|
|---|--------|----|-------|-----|---|----|----------------|---------|------|---|
|  0|6563    |DIA |WP_NEW |     |   |    |                |         |      |10 |
|  1|6564    |DIA |WP_NEW |     |   |    |                |         |      |10 |
|  2|6565    |DIA |WP_NEW |     |   |    |                |         |      |10 |
|  3|6566    |UPD |WP_NEW |     |   |    |                |         |      |10 |
|  4|6567    |BTC |WP_NEW |     |   |    |                |         |10    |   |
|  5|6568    |BTC |WP_NEW |     |   |    |                |         |      |10 |
|  6|6569    |SPO |WP_NEW |     |   |    |                |         |      |10 |
|  7|6570    |UP2 |WP_NEW |     |   |    |                |         |      |10 |

Regards,

Prithviraj

[Former Member]

Hello Prithviraj,

Thank you for the proposal.

I don't have the trace file dev_w10 but i see the error "ThSemAlreadyLocked: waiting for semaphore 10 held by process 6719 since 5218s" in trace file dev_w7 (see attached file).

Based on your Information, i killed the process 6719 and restart the System but still the issue exists.

Any other ideas are appreciated.

Thank you,

Sreenu

[JPReyes]

The WP died waiting for a semaphore to be released,

"ThSemAlreadyLocked: waiting for semaphore 10 held by process 6719 since XX"

Check what is running on PID 6719, you might have to kill the process to release the locked semaphore.

Regards, JP

[isaias_freitas]

Hi!

Minding the SAP was starting up, I believe that killing the semaphore holder would not help.

The next work process that wants to use the semaphore will take it, but will be stuck waiting for the database again 😉

Regards,

Isaías

[JPReyes]

Hence I said "check whats running on that PID" 🙂

[raquel_gomez]

Hi,

Checking dispatcher trace file, and snapshot is created:
********** SERVER SNAPSHOT 141 (Reason: Request handling without progress) - begin **********

|No |Pid |Type|State |Cause|Err|Prio|Sess-Key |Sess-Type|Locked|Sem|Time |Program |Cli|User |Action |Action-Info |
|---|--------|----|-------|-----|---|----|----------------|---------|------|---|-----|----------------------------------------|---|------------|--------------------|--------------------|
| 0|6563 |DIA |WP_NEW | | | | | | |10 | | | | | | |
| 1|6564 |DIA |WP_NEW | | | | | | |10 | | | | | | |
| 2|6565 |DIA |WP_NEW | | | | | | |10 | | | | | | |
| 3|6566 |UPD |WP_NEW | | | | | | |10 | | | | | | |
| 4|6567 |BTC |WP_NEW | | | | | |10 | | | | | | | |
| 5|6568 |BTC |WP_NEW | | | | | | |10 | | | | | | |
| 6|6569 |SPO |WP_NEW | | | | | | |10 | | | | | | |
| 7|6570 |UP2 |WP_NEW | | | | | | |10 | | | | | |

Semaphore 10 is being held by WP4, which is in status WP_NEW. The rest of the work processes are waiting for it.
Is it DB up and correctly running. Normally those kind of situations are related to wrong connection from WP to DB.
Please provide dev_w4 trace file, to check c-stack generated on its trace.

Thanks,
Raquel

[Former Member]

Hello Gomez,

Thank you for the Information.

Please find the file dev_w4 in attachment. Use the Notepad++ to open the file.

incase of ideas, please let us know.

thanks sreenu

[raquel_gomez]

Hi,
C-stack generated on the trace shows:

M ------------------ C-STACK ----------------------
[0] SunDoStack2, at 0x1eb6c59
[1] CTrcStack2, at 0x1eb67f3
[2] __1cOThStackHandler6F_v_, at 0x1c83c76
[3] __1cKDpTrcOnOff6Fi_v_, at 0x1b7ea6f
[4] __sighndlr, at 0xfffffd7ff5fedd16
[5] call_user_handler, at 0xfffffd7ff5fe25e2
[6] sigacthandler, at 0xfffffd7ff5fe280e
[7] ????????, at 0xffffffffffffffff
[8] SemTimedOp, at 0x1c28e9a
[9] RqOsSem, at 0x1c25bc3
[10] SemRq, at 0x1c26772
[11] __1cJThIPCInit6FpC_i_, at 0x1c859ce
[12] __1cGThInit6F_i_, at 0x1c85067
[13] __1cHThStart6F_v_, at 0x1c843be
[14] DpMain, at 0x1b1d2d9
M -------------------------------------------------

Is DB up and correctly runnign? Please provide also dev_w0 trace.

Regards,
Raquel

[isaias_freitas]

Hi!

Providing the complete "dev_disp" and "dev_w0" might help too ;-).

Please "zip" and attach them to this thread.

Cheers!

Isaías

[Former Member]

Hello Freitas,

Please find the dev_disp and dev_w0 trace files in attachments.

Thank you,

Sreenu

[isaias_freitas]

Hello Sreenivas!

Did you see the reply from Prithviraj?

The analysis is correct, we need the dev_w10 to continue analyzing the case.

Cheers!

Isaías

[isaias_freitas]

---> now I see that I have mentioned the incorrect work process trace... we need dev_w4, not dev_w10... (mixed up the semaphore number :D)

[Former Member]

Hello Freitas,

Please find the dev_w4 in attachment, use the Notepad++ to open the file.

I restarted the Server but System start issue still exists.

Please let me know if you have any ideas.

Thank you,

Sreenu

[isaias_freitas]

Hello Sreenu,

Since the system has been restarted, we need to start the analysis from the beginning.

This is because the issue could be occurring at a different work process now.

Or you could provide the "dev_w4.old" trace. If the system was restarted only once, the ".old" trace will be from the time of the first occurrence of the issue.

If the ".old" file (with entries from the previous startup) is not available anymore, check the current "dev_disp" trace and locate the work process that is holding the semaphore (see Raquel's reply for an example). Then, share the corresponding dev_wXX.

Feel free to share the dev_disp too, if you want us to identify the affected process.

Cheers!

Isaías

[Former Member]

Hello Freitas,

Please find dev_w4.old.txt in attachment, use the Notepad++ to open the file.

According to Prithvi and Raquel, i killed the BTC process PID 6567. After that, Dispatcher was not running. We're getting Dispatcher error "DpMonAttach failed - possibly no dispatcher running".

Anyway, please find the dev_disp file in attachment.

Thank you for your Support.

Best regards,

Sreenu

[isaias_freitas]

Hello Sreenu,

The dev_w4old.txt that you provided matches the previous traces we were analyzing.

It shows that the work process is waiting for the database (MaxDB; check lines "[8]" to "[20]", they related to a DB request):

M Mon Mar 12 09:54:30:566 2018
M  
M  ********** SERVER SNAPSHOT 1 (Reason: Request handling without progress) - begin **********
M  
M  ------------------ C-STACK ----------------------
[0] SunDoStack2, at 0x1eb6c59
[1] CTrcStack2, at 0x1eb67f3
[2] __1cOThStackHandler6F_v_, at 0x1c83c76
[3] __1cKDpTrcOnOff6Fi_v_, at 0x1b7ea6f
[4] __sighndlr, at 0xfffffd7ff5fedd16
[5] call_user_handler, at 0xfffffd7ff5fe25e2
[6] sigacthandler, at 0xfffffd7ff5fe280e
[7] ????????, at 0xffffffffffffffff
[8] __1cSRTEInternalClient_bRRTECKC_ClientLegacyCommunicationSegmentUNIXHReceive6MpFpv_i2pnKrte_header_IIrnNtsp00_CString4ibO___nStsp01_CommErr_Enum__, at 0xfffffd7f9be85fa6
[9] __1cNsql33_receive6FpnPconnection_info_rnNtsp00_CString4ibO___nStsp01_CommErr_Enum__, at 0xfffffd7f9be83f82
[10] __1cNsql03_receive6FIppvrirnNtsp00_CString4ibO___nStsp01_CommErr_Enum__, at 0xfffffd7f9be7fc0d
[11] sqlareceive, at 0xfffffd7f9be7c66b
[12] __1cUSQdDLDBC_ClientRuntimeHreceive6MlppvrirnPSQdDLDBC_IRuntimeFError__b_, at 0xfffffd7f9bcd8fef
[13] __1cbCSQdDLDBC_SingleThreadedRuntimeHreceive6MlppvrirnPSQdDLDBC_IRuntimeFError__b_, at 0xfffffd7f9bcda414
[14] __1cOIFR_ConnectionLsqlaexecute6MrnXIFRPacket_RequestPacket_rnVIFRPacket_ReplyPacket_n0AKAppendMode_rnNIFR_ErrorHndl_pnWIFRUtil_AsyncOperation__nLIFR_Retcode__, at 0xfffffd7f9bd3dea5
[15] __1cOIFR_ConnectionLsqlaexecute6MrnXIFRPacket_RequestPacket_rnVIFRPacket_ReplyPacket_n0AKAppendMode_rnNIFR_ErrorHndl_pnWIFRUtil_AsyncOperation__nLIFR_Retcode__, at 0xfffffd7f9bd3dc30
[16] __1cQIFR_PreparedStmtHexecute6M_nLIFR_Retcode__, at 0xfffffd7f9bda200d
[17] __1cQIFR_PreparedStmtMexecuteBatch6M_nLIFR_Retcode__, at 0xfffffd7f9bdc0d7a
[18] __1cGSQdDLDBCYSQdDLDBC_PreparedStatementHexecute6M_nOSQdDLDBC_Retcode__, at 0xfffffd7f9bcca491
[19] __1cLCSdbDbslSQdDLHexecute6MIIC_i_, at 0xfffffd7f9c57a849
[20] __1cMstmt_execute6FpvpnHDBSL_SS_nEIO_T_pnHDBSL_DA__nLDBSL_RETURN__, at 0xfffffd7f9c5b574b
[21] __1cLexec_modify6FpvpnHDBSL_SS_CnEIO_T_pnHDBSL_DA__nLDBSL_RETURN__, at 0xfffffd7f9c5b32c0
[22] __1cNDbSlSdbModify6FhpnHDBSL_SS_nSDBSL_OPERATIONTYPE_pnHDBSL_DA__nLDBSL_RETURN__, at 0xfffffd7f9c5a0602
[23] dbsl_modify, at 0x37e8f20
[24] __1cJdbsl_call6FpnPrtTrtabCursor_s_nMrtDbSlFunc_t_nSDBSL_OPERATIONTYPE__nLDBSL_RETURN__, at 0x36d66c6
[25] __1cJdbsl_exec6FpnPrtTrtabCursor_s_nPRT_FUNCTIONCODE_nMrtDbSlFunc_t_nSDBSL_OPERATIONTYPE__i_, at 0x36d656d
[26] __1cNmodify_single6FpnPrtTrtabCursor_s_nPRT_FUNCTIONCODE__i_, at 0x36d3436
[27] __1cJtran_rtab6FpnNrtCrtabCrsr_s_nPRT_FUNCTIONCODE__i_, at 0x35d25c6
[28] __1cJrtab_exec6FpnNrtCrtabCrsr_s_nPRT_FUNCTIONCODE_pnQRT_OUTBUFFERSPEC__i_, at 0x35cf025
[29] db_rtab0, at 0x35ccef8
[30] NotifyPatchHistory, at 0x1b1be9c
[31] __1cJThShMInit6F_i_, at 0x1c86321
[32] __1cJThIPCInit6FpC_i_, at 0x1c85a7c
[33] __1cGThInit6F_i_, at 0x1c85067
[34] __1cHThStart6F_v_, at 0x1c843be
[35] DpMain, at 0x1b1d2d9

At this point, I would expect that "R3trans -d" fails too.

Please check what is happening at the DB.

If you need further assistance, please update the tags from this question, changing them to MaxDB related tags.

Best regards,

Isaías

[Former Member]

Hello Isaias,

I see R3trans -d is working.

Anyway thanks for the Suggestion.

Best regards,

Sreenu

[isaias_freitas]

hmmm that's odd... R3trans fails in such cases, usually...

In any case, the work processes are waiting for the DB, so we must move the analysis there.

I have no knowledge on DBs, thus the suggestion to change the tags from this question to MaxDB related ones.

Best regards,

Isaías

[isaias_freitas]

Hello Sreenu,

Has the issue been solved? Can we provide further assistance? 🙂

If the issue was solved, please close this question and select the best answer :).

Best regards,

Isaías

[Matt_Fraser]

Sreenu,

If you want help with this, you have to provide quite a bit more information. I recommend having a look at this blog:

https://blogs.sap.com/2013/06/20/logical-basic-abap-system-start-troubleshooting-sequence/

If you go through the basic steps outlined there, chances are you'll resolve your own problem (and if so, please do come back and tell us how you solved it!). And if not, well, you'll probably at least narrow it down to one or more components of the system, which should provide a clue on what logs to provide here to get more help.

Version information is always helpful, too.

Cheers,
Matt

[Former Member]

Hello Matt,

Thank you for the blog. In my case, I see all the traces regarding the message Server, Dispatcher and work process traces in the OS Level.

For more Information, please find the attached trace files dev_disp and dev_w0.

Thank you,

Sreenu