ServerCrash

[kk2024]

Dear expert，

The user reported that they were unable to access the server. I used sapcontrol - nr 01- function VNet cessList on the Netweaver Java application server to check the instance running status, and the textstatus of jstart showed "Some Processes running" I found the file in the attachment in the server directory/sapmnt/SID/golbal/sapcontrol/snapshots.

I found a compressed file in the directory, which contains a description.txt file and a log directory. The content of description.txt is 'Servercrash '. I found the jstart file in the log directory and it showed below:

<item>F [Thr 140016985638784] Thu Jan 9 10:53:49 2025</item>
<item>F [Thr 140016985638784] *** LOG =&gt; Send SIGINT to ICM (pid 16077).</item>
<item>F [Thr 140016985638784] *** LOG =&gt; Process icm stopping (pid 16077).</item>
<item>F [Thr 140016985638784] *** LOG =&gt; Instance state is "Some processes running" (RUNNING @ 2, INACTIVE).</item>
<item>F [Thr 140016985638784] *** LOG =&gt; Signal 17 SIGCHLD.</item>
<item>F [Thr 140016985638784] *** WARNING =&gt; process ICM (pid 16077) killed (signal 11). [sfuxlib.hpp 854]</item>
<item>F [Thr 140016985638784] *** LOG =&gt; Process icm died (pid 16077).</item>
<item>F [Thr 140016985638784] *** WARNING =&gt; Node icm failed: result -1, exit code 139. [sfxxnode.hpp 1036]</item>

I checked the memory, but found that it was only half used. What else can I check?

[Isaias_SAP · Accepted solution]

Hi!

The log you have shared shows that the ICM process has crashed with a signal 11.

Thus, you should search for SAP Notes delivering fixes for crashes in the ICM on an SAP kernel patch level higher than the one in use at the affected system.

For further assistance, we would need to know the release and patch level of the SAP kernel, and also the "C-Stack" from the "dev_icm" trace file.

Regards,

Isaías

[kk2024 · Accepted solution]

Dear Isaías,

Thank you very much for your reply！

I found the following information in dev_icm：

trc file: "dev_icm", trc level: 1, release: "749"

sysno 01
sid LMP
systemid 390 (AMD/Intel x86_64 with Linux)
relno 7490
patchlevel 0
patchno 600
intno 20160201
make multithreaded, Unicode, 64 bit, optimized

I couldn't find the C-STACK information when the system crashed. The following is the last part of the ICM file, but the crash occurred on January 9th at 10:54:

....
<item>[Thr 140187815347968] Sat Dec 28 23:13:07:627 2024</item>
<item>[Thr 140187815347968] IcmCreateWorkerThreads: created worker thread 90</item>
<item></item>
<item>[Thr 140187110332160] Sat Dec 28 23:15:21:360 2024</item>
<item>[Thr 140187110332160] *** ERROR =&gt; MtxLock(7f800aed86a4) failed: 12 [mpixx.c 2191]</item>
<item></item>
<item>[Thr 140187089196800] Sat Dec 28 23:18:08:835 2024</item>
<item>[Thr 140187089196800] *** ERROR =&gt; MtxLock(7f800aef36b4) failed: 12 [mpixx.c 2191]</item>
<item></item>
<item>[Thr 140187072288512] Sat Dec 28 23:18:13:286 2024</item>
<item>[Thr 140187072288512] IcmWorkerThread: end worker thread 90</item>
<item></item>
<item>[Thr 140187072816896] Sat Dec 28 23:19:14:327 2024</item>
<item>[Thr 140187072816896] IcmWorkerThread: end worker thread 89</item>
<item></item>
<item>[Thr 140187074402048] Sat Dec 28 23:19:21:071 2024</item>
<item>[Thr 140187074402048] WARNING: re-check of eyecatcher consistency.</item>
<item>[Thr 140187074402048] *** ERROR =&gt; MtxLock(7f800aef8244) failed: 12 [mpixx.c 2191]</item>
<item></item>
<item>[Thr 140187073345280] Sat Dec 28 23:20:15:577 2024</item>
<item>[Thr 140187073345280] IcmWorkerThread: end worker thread 88</item>
<item></item>
<item>[Thr 140187114559232] Sat Dec 28 23:21:11:105 2024</item>
</fields>
</SAPControl:ReadLogFileResponse>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

[Isaias_SAP · Accepted solution]

Hi! You are welcome! If you install the SAP MMC on your (Windows) computer and use it to open the snapshot (the "zip" file mentioned at the initial question), you can right click on the ICM process there and choose to open the developer trace. Once on that screen, use the search feature (on the bottom) to search for "stack" and verify whether the c-stack is there. Without the c-stack, we cannot verify whether this is a known issue (and an SAP Note delivering a fix was already released) or a new issue. Another option to get the c-stack is to check whether a "core" file was created at Linux level (SAP Note 1163337). If yes, the c-stack can be extracted from the "core" file: https://help.sap.com/docs/SUPPORT_CONTENT/si/3362959726.html

[kk2024 · Accepted solution]

Dear Isaías,

Thank you very much again for your professional guidance, the following is the output of the bt command I use gdb in the Linux system:

(gdb) bt
#0 0x00007fdaa1fbc05f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000423398 in ThrEvtWaitReset ()
#2 0x00000000004236ab in ThrSemDown ()
#3 0x000000000058eca2 in Semaphore::Acquire(char const*) ()
#4 0x000000000056af4c in Dispatcher::QueueThread(Dispatcher::tQueueStruct*) ()
#5 0x000000000056bfd0 in RunQueueThread(void*) ()
#6 0x00007fdaa1fb80a4 in start_thread () from /lib64/libpthread.so.0
#7 0x00007fdaa1cee04d in clone () from /lib64/libc.so.6
(gdb) info threads
Id Target Id Frame
16 Thread 0x7fda7dffb700 (LWP 12077) 0x00007fdaa1fbc05f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
15 Thread 0x7fda7effd700 (LWP 12072) 0x00007fdaa1fbc05f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
14 Thread 0x7fda7ffff700 (LWP 12067) 0x00007fdaa1fbc05f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
13 Thread 0x7fda8d0cf700 (LWP 12062) 0x00007fdaa1fbc05f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
12 Thread 0x7fda7e7fc700 (LWP 12076) 0x00007fdaa1fbc05f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
11 Thread 0x7fda7d7fa700 (LWP 12079) 0x00007fdaa1cbfded in nanosleep () from /lib64/libc.so.6
10 Thread 0x7fda7f7fe700 (LWP 12071) 0x00007fdaa1fbc05f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
9 Thread 0x7fda8c8ce700 (LWP 12066) 0x00007fdaa1fbc05f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
8 Thread 0x7fda8d8d0700 (LWP 12061) 0x00007fdaa1fbc05f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
7 Thread 0x7fda8e0d1700 (LWP 12057) 0x00007fdaa1fbc05f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
6 Thread 0x7fda8e8d2700 (LWP 12056) 0x00007fdaa1fbc05f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
5 Thread 0x7fdaa327e740 (LWP 10720) 0x00007fdaa1cbfded in nanosleep () from /lib64/libc.so.6
4 Thread 0x7fda9f5b2700 (LWP 12047) 0x00007fdaa1cee623 in epoll_wait () from /lib64/libc.so.6
3 Thread 0x7fda8f8d4700 (LWP 12051) 0x00007fdaa1fbc05f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
2 Thread 0x7fda8f0d3700 (LWP 12052) 0x00007fdaa1fbc05f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
* 1 Thread 0x7fda9fdb3700 (LWP 12046) 0x00007fdaa1fbc05f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

[Isaias_SAP · Accepted solution]

Hello, you are welcome!

Was that from a "core" file related to the ICM crash? The c-stack does not look like it is related to the ICM.

General comments that I can still make:

• the kernel in use (749 patch 600) is fom 2018 (!), and the kernel 749 is not supported anymore (since 2023, refer to the SAP Note 1969546).
• I have found the SAP Notes 2795949 and 2800543, which address crashes in the ICM and are applicable to the kernel in use. We just do not know (with the information on this thread) whether the crash you faced is from those SAP Notes. In any case, you should consider upgrading the SAP kernel to 753 or 754.

[kk2024 · Accepted solution]

Dear Isaías,

After the ICM crash, I tried stopsap&startsap but failed, I noticed that the core update time was generated when I stopped the system again after startsap, so it should not be related to ICM.

I'll try to update the kernel, thank you for the guidance.