on 03-12-2018 8:15 PM
Dear Experts,
Currently, we're not able to start the SAP system from the operating system level. From the dev_disp log we found the below error message:
DpUpdateStatusFileWith: state=YELLOW, reason=Request handling without progress *** WARNING => DpRequestProcessingCheck: potential request processing problem detected (123. check) [dpxxwp.c 4705]
SAP NW system is running with ADA database and operating system Solaris 10.
Any ideas are appreciated.
Thanks & best regards,
Sreenu
Hi Sreeni,
Background work process number 4 is the culprit here, as shown in dispatcher log.
Try killing the process if that helps. If not, look at the trace file dev_w10 to see what is causing it to hold Sem 10 for so long.
|No |Pid |Type|State |Cause|Err|Prio|Sess-Key |Sess-Type|Locked|Sem|
|---|--------|----|-------|-----|---|----|----------------|---------|------|---|
| 0|6563 |DIA |WP_NEW | | | | | | |10 |
| 1|6564 |DIA |WP_NEW | | | | | | |10 |
| 2|6565 |DIA |WP_NEW | | | | | | |10 |
| 3|6566 |UPD |WP_NEW | | | | | | |10 |
| 4|6567 |BTC |WP_NEW | | | | | |10 | |
| 5|6568 |BTC |WP_NEW | | | | | | |10 |
| 6|6569 |SPO |WP_NEW | | | | | | |10 |
| 7|6570 |UP2 |WP_NEW | | | | | | |10 |
Regards,
Prithviraj
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hello Prithviraj,
Thank you for the proposal.
I don't have the trace file dev_w10 but i see the error "ThSemAlreadyLocked: waiting for semaphore 10 held by process 6719 since 5218s" in trace file dev_w7 (see attached file).
Based on your Information, i killed the process 6719 and restart the System but still the issue exists.
Any other ideas are appreciated.
Thank you,
Sreenu
The WP died waiting for a semaphore to be released,
"ThSemAlreadyLocked: waiting for semaphore 10 held by process 6719 since XX"
Check what is running on PID 6719, you might have to kill the process to release the locked semaphore.
Regards, JP
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi,
Checking dispatcher trace file, and snapshot is created:
********** SERVER SNAPSHOT 141 (Reason: Request handling without progress) - begin **********
|No |Pid |Type|State |Cause|Err|Prio|Sess-Key |Sess-Type|Locked|Sem|Time |Program |Cli|User |Action |Action-Info |
|---|--------|----|-------|-----|---|----|----------------|---------|------|---|-----|----------------------------------------|---|------------|--------------------|--------------------|
| 0|6563 |DIA |WP_NEW | | | | | | |10 | | | | | | |
| 1|6564 |DIA |WP_NEW | | | | | | |10 | | | | | | |
| 2|6565 |DIA |WP_NEW | | | | | | |10 | | | | | | |
| 3|6566 |UPD |WP_NEW | | | | | | |10 | | | | | | |
| 4|6567 |BTC |WP_NEW | | | | | |10 | | | | | | | |
| 5|6568 |BTC |WP_NEW | | | | | | |10 | | | | | | |
| 6|6569 |SPO |WP_NEW | | | | | | |10 | | | | | | |
| 7|6570 |UP2 |WP_NEW | | | | | | |10 | | | | | |
Semaphore 10 is being held by WP4, which is in status WP_NEW. The rest of the work processes are waiting for it.
Is it DB up and correctly running. Normally those kind of situations are related to wrong connection from WP to DB.
Please provide dev_w4 trace file, to check c-stack generated on its trace.
Thanks,
Raquel
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi,
C-stack generated on the trace shows:
M ------------------ C-STACK ----------------------
[0] SunDoStack2, at 0x1eb6c59
[1] CTrcStack2, at 0x1eb67f3
[2] __1cOThStackHandler6F_v_, at 0x1c83c76
[3] __1cKDpTrcOnOff6Fi_v_, at 0x1b7ea6f
[4] __sighndlr, at 0xfffffd7ff5fedd16
[5] call_user_handler, at 0xfffffd7ff5fe25e2
[6] sigacthandler, at 0xfffffd7ff5fe280e
[7] ????????, at 0xffffffffffffffff
[8] SemTimedOp, at 0x1c28e9a
[9] RqOsSem, at 0x1c25bc3
[10] SemRq, at 0x1c26772
[11] __1cJThIPCInit6FpC_i_, at 0x1c859ce
[12] __1cGThInit6F_i_, at 0x1c85067
[13] __1cHThStart6F_v_, at 0x1c843be
[14] DpMain, at 0x1b1d2d9
M -------------------------------------------------
Is DB up and correctly runnign? Please provide also dev_w0 trace.
Regards,
Raquel
Hi!
Providing the complete "dev_disp" and "dev_w0" might help too ;-).
Please "zip" and attach them to this thread.
Cheers!
Isaías
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hello Sreenivas!
Did you see the reply from Prithviraj?
The analysis is correct, we need the dev_w10 to continue analyzing the case.
Cheers!
Isaías
Hello Sreenu,
Since the system has been restarted, we need to start the analysis from the beginning.
This is because the issue could be occurring at a different work process now.
Or you could provide the "dev_w4.old" trace. If the system was restarted only once, the ".old" trace will be from the time of the first occurrence of the issue.
If the ".old" file (with entries from the previous startup) is not available anymore, check the current "dev_disp" trace and locate the work process that is holding the semaphore (see Raquel's reply for an example). Then, share the corresponding dev_wXX.
Feel free to share the dev_disp too, if you want us to identify the affected process.
Cheers!
Isaías
Hello Freitas,
Please find dev_w4.old.txt in attachment, use the Notepad++ to open the file.
According to Prithvi and Raquel, i killed the BTC process PID 6567. After that, Dispatcher was not running. We're getting Dispatcher error "DpMonAttach failed - possibly no dispatcher running".
Anyway, please find the dev_disp file in attachment.
Thank you for your Support.
Best regards,
Sreenu
Hello Sreenu,
The dev_w4old.txt that you provided matches the previous traces we were analyzing.
It shows that the work process is waiting for the database (MaxDB; check lines "[8]" to "[20]", they related to a DB request):
M Mon Mar 12 09:54:30:566 2018
M
M ********** SERVER SNAPSHOT 1 (Reason: Request handling without progress) - begin **********
M
M ------------------ C-STACK ----------------------
[0] SunDoStack2, at 0x1eb6c59
[1] CTrcStack2, at 0x1eb67f3
[2] __1cOThStackHandler6F_v_, at 0x1c83c76
[3] __1cKDpTrcOnOff6Fi_v_, at 0x1b7ea6f
[4] __sighndlr, at 0xfffffd7ff5fedd16
[5] call_user_handler, at 0xfffffd7ff5fe25e2
[6] sigacthandler, at 0xfffffd7ff5fe280e
[7] ????????, at 0xffffffffffffffff
[8] __1cSRTEInternalClient_bRRTECKC_ClientLegacyCommunicationSegmentUNIXHReceive6MpFpv_i2pnKrte_header_IIrnNtsp00_CString4ibO___nStsp01_CommErr_Enum__, at 0xfffffd7f9be85fa6
[9] __1cNsql33_receive6FpnPconnection_info_rnNtsp00_CString4ibO___nStsp01_CommErr_Enum__, at 0xfffffd7f9be83f82
[10] __1cNsql03_receive6FIppvrirnNtsp00_CString4ibO___nStsp01_CommErr_Enum__, at 0xfffffd7f9be7fc0d
[11] sqlareceive, at 0xfffffd7f9be7c66b
[12] __1cUSQdDLDBC_ClientRuntimeHreceive6MlppvrirnPSQdDLDBC_IRuntimeFError__b_, at 0xfffffd7f9bcd8fef
[13] __1cbCSQdDLDBC_SingleThreadedRuntimeHreceive6MlppvrirnPSQdDLDBC_IRuntimeFError__b_, at 0xfffffd7f9bcda414
[14] __1cOIFR_ConnectionLsqlaexecute6MrnXIFRPacket_RequestPacket_rnVIFRPacket_ReplyPacket_n0AKAppendMode_rnNIFR_ErrorHndl_pnWIFRUtil_AsyncOperation__nLIFR_Retcode__, at 0xfffffd7f9bd3dea5
[15] __1cOIFR_ConnectionLsqlaexecute6MrnXIFRPacket_RequestPacket_rnVIFRPacket_ReplyPacket_n0AKAppendMode_rnNIFR_ErrorHndl_pnWIFRUtil_AsyncOperation__nLIFR_Retcode__, at 0xfffffd7f9bd3dc30
[16] __1cQIFR_PreparedStmtHexecute6M_nLIFR_Retcode__, at 0xfffffd7f9bda200d
[17] __1cQIFR_PreparedStmtMexecuteBatch6M_nLIFR_Retcode__, at 0xfffffd7f9bdc0d7a
[18] __1cGSQdDLDBCYSQdDLDBC_PreparedStatementHexecute6M_nOSQdDLDBC_Retcode__, at 0xfffffd7f9bcca491
[19] __1cLCSdbDbslSQdDLHexecute6MIIC_i_, at 0xfffffd7f9c57a849
[20] __1cMstmt_execute6FpvpnHDBSL_SS_nEIO_T_pnHDBSL_DA__nLDBSL_RETURN__, at 0xfffffd7f9c5b574b
[21] __1cLexec_modify6FpvpnHDBSL_SS_CnEIO_T_pnHDBSL_DA__nLDBSL_RETURN__, at 0xfffffd7f9c5b32c0
[22] __1cNDbSlSdbModify6FhpnHDBSL_SS_nSDBSL_OPERATIONTYPE_pnHDBSL_DA__nLDBSL_RETURN__, at 0xfffffd7f9c5a0602
[23] dbsl_modify, at 0x37e8f20
[24] __1cJdbsl_call6FpnPrtTrtabCursor_s_nMrtDbSlFunc_t_nSDBSL_OPERATIONTYPE__nLDBSL_RETURN__, at 0x36d66c6
[25] __1cJdbsl_exec6FpnPrtTrtabCursor_s_nPRT_FUNCTIONCODE_nMrtDbSlFunc_t_nSDBSL_OPERATIONTYPE__i_, at 0x36d656d
[26] __1cNmodify_single6FpnPrtTrtabCursor_s_nPRT_FUNCTIONCODE__i_, at 0x36d3436
[27] __1cJtran_rtab6FpnNrtCrtabCrsr_s_nPRT_FUNCTIONCODE__i_, at 0x35d25c6
[28] __1cJrtab_exec6FpnNrtCrtabCrsr_s_nPRT_FUNCTIONCODE_pnQRT_OUTBUFFERSPEC__i_, at 0x35cf025
[29] db_rtab0, at 0x35ccef8
[30] NotifyPatchHistory, at 0x1b1be9c
[31] __1cJThShMInit6F_i_, at 0x1c86321
[32] __1cJThIPCInit6FpC_i_, at 0x1c85a7c
[33] __1cGThInit6F_i_, at 0x1c85067
[34] __1cHThStart6F_v_, at 0x1c843be
[35] DpMain, at 0x1b1d2d9
At this point, I would expect that "R3trans -d" fails too.
Please check what is happening at the DB.
If you need further assistance, please update the tags from this question, changing them to MaxDB related tags.
Best regards,
Isaías
Sreenu,
If you want help with this, you have to provide quite a bit more information. I recommend having a look at this blog:
https://blogs.sap.com/2013/06/20/logical-basic-abap-system-start-troubleshooting-sequence/
If you go through the basic steps outlined there, chances are you'll resolve your own problem (and if so, please do come back and tell us how you solved it!). And if not, well, you'll probably at least narrow it down to one or more components of the system, which should provide a clue on what logs to provide here to get more help.
Version information is always helpful, too.
Cheers,
Matt
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
User | Count |
---|---|
78 | |
9 | |
9 | |
7 | |
7 | |
6 | |
6 | |
5 | |
5 | |
5 |
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.