K master has recently crashed several times on Debian 10. Could this have something to do with openssl libs?
-- Juha
Oct 5 08:32:21 lohi systemd[1]: Stopping LSB: Start/stop OpenSIPg SIP Proxy... Oct 5 08:32:21 lohi sip-proxy[19922]: Stopping sip-proxy Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30820]: NOTICE: <core> [main.c:735]: handle_sigs(): Thank you for flying sip-proxy!!! Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30860]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30859]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30858]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30857]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30853]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30856]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30850]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30854]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30834]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30847]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30852]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30849]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30851]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30841]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30830]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30848]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30846]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30836]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30827]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30844]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30845]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30833]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30842]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30839]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30829]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30822]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30840]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30838]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30826]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30837]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30824]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30835]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30823]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30832]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30831]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30828]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30843]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30825]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30821]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30855]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30820]: INFO: <core> [core/sctp_core.c:53]: sctp_core_destroy(): SCTP API not initialized Oct 5 08:32:21 lohi kernel: [176646.851504] sip-proxy[30820]: segfault at 7f84196c7730 ip 00007f8421adce26 sp 00007ffde899c0f0 error 4 in libcrypto.so.1.1[7f84219d8000+19e000] Oct 5 08:32:21 lohi kernel: [176646.851513] Code: 1f 44 00 00 c7 05 9a d9 15 00 01 00 00 00 c7 05 8c d9 15 00 01 00 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 48 85 ff 74 5b 53 <8b> 0f 48 89 fb 85 c9 75 29 8b 53 04 85 d2 75 2e 8b 43 08 85 c0 75
(gdb) where #0 0x00007f8421adce26 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1 #1 0x00007f8421add12f in OPENSSL_cleanup () from /lib/x86_64-linux-gnu/libcrypto.so.1.1 #2 0x00007f8423a87d8c in __run_exit_handlers (status=0, listp=0x7f8423c09718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108 #3 0x00007f8423a87eba in __GI_exit (status=<optimized out>) at exit.c:139 #4 0x000055d40be7bb61 in handle_sigs () at main.c:738 #5 0x000055d40be88966 in main_loop () at main.c:1817 #6 0x000055d40be91e32 in main (argc=17, argv=0x7ffde899c948) at main.c:2861
The backtrace is for the main process during shut down, which can be a side effect of the real issue. You should enable core dump per pid to see if you get core files from more than one process, then grab backtraces from all those core files to figure out what was the first reason of the crash.
Is it the last master version or some older one (kamailio -v should print the commit id, if you compiled from sources)?
Cheers, Daniel
On 05.10.20 13:11, Juha Heinanen wrote:
K master has recently crashed several times on Debian 10. Could this have something to do with openssl libs?
-- Juha
Oct 5 08:32:21 lohi systemd[1]: Stopping LSB: Start/stop OpenSIPg SIP Proxy... Oct 5 08:32:21 lohi sip-proxy[19922]: Stopping sip-proxy Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30820]: NOTICE: <core> [main.c:735]: handle_sigs(): Thank you for flying sip-proxy!!! Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30860]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30859]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30858]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30857]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30853]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30856]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30850]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30854]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30834]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30847]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30852]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30849]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30851]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30841]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30830]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30848]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30846]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30836]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30827]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30844]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30845]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30833]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30842]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30839]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30829]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30822]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30840]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30838]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30826]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30837]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30824]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30835]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30823]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30832]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30831]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30828]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30843]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30825]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30821]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30855]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30820]: INFO: <core> [core/sctp_core.c:53]: sctp_core_destroy(): SCTP API not initialized Oct 5 08:32:21 lohi kernel: [176646.851504] sip-proxy[30820]: segfault at 7f84196c7730 ip 00007f8421adce26 sp 00007ffde899c0f0 error 4 in libcrypto.so.1.1[7f84219d8000+19e000] Oct 5 08:32:21 lohi kernel: [176646.851513] Code: 1f 44 00 00 c7 05 9a d9 15 00 01 00 00 00 c7 05 8c d9 15 00 01 00 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 48 85 ff 74 5b 53 <8b> 0f 48 89 fb 85 c9 75 29 8b 53 04 85 d2 75 2e 8b 43 08 85 c0 75
(gdb) where #0 0x00007f8421adce26 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1 #1 0x00007f8421add12f in OPENSSL_cleanup () from /lib/x86_64-linux-gnu/libcrypto.so.1.1 #2 0x00007f8423a87d8c in __run_exit_handlers (status=0, listp=0x7f8423c09718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108 #3 0x00007f8423a87eba in __GI_exit (status=<optimized out>) at exit.c:139 #4 0x000055d40be7bb61 in handle_sigs () at main.c:738 #5 0x000055d40be88966 in main_loop () at main.c:1817 #6 0x000055d40be91e32 in main (argc=17, argv=0x7ffde899c948) at main.c:2861
Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Daniel-Constantin Mierla writes:
The backtrace is for the main process during shut down, which can be a side effect of the real issue. You should enable core dump per pid to see if you get core files from more than one process, then grab backtraces from all those core files to figure out what was the first reason of the crash.
Sorry, but don't know how to enable core dump per pid. Didn't find any cli option for that.
Is it the last master version or some older one (kamailio -v should print the commit id, if you compiled from sources)?
Just built K from current master and it crashes every time with similar gdb trace when I stop it using 'service sip-proxy stop'.
-- Juha
Juha Heinanen writes:
Just built K from current master and it crashes every time with similar gdb trace when I stop it using 'service sip-proxy stop'.
As a test, djJust build K from latest 5.4 branch and it does not crash at stop with exactly same config setup and environment. So looks like something is broken in master.
-- Juha
The kamailio wiki has a turorial about troubleshooting with core files where it gives the details about how to enable core file per pid (I am on mobile link right now, not easy to give the url).
To understand that it crashes only on shut down anyhow, not a runtime crash?
Cheers, Daniel
On 05.10.20 13:43, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
The backtrace is for the main process during shut down, which can be a side effect of the real issue. You should enable core dump per pid to see if you get core files from more than one process, then grab backtraces from all those core files to figure out what was the first reason of the crash.
Sorry, but don't know how to enable core dump per pid. Didn't find any cli option for that.
Is it the last master version or some older one (kamailio -v should print the commit id, if you compiled from sources)?
Just built K from current master and it crashes every time with similar gdb trace when I stop it using 'service sip-proxy stop'.
-- Juha
Daniel-Constantin Mierla writes:
The kamailio wiki has a turorial about troubleshooting with core files where it gives the details about how to enable core file per pid (I am on mobile link right now, not easy to give the url).
OK, will try to look.
To understand that it crashes only on shut down anyhow, not a runtime crash?
I think that so far only at shutdown.
-- Juha
If it is on shut down, then likely it is only one core file, I thought was at runtime.
Going through difference between the two branches, I think I found the issue -- a PR that was targeting to remove the obsoleted libssl API was removing OPENSSL_cleanup(), which docs recommend not to execute, because it is done automatically by libssl on exit, but in our case it is needed to have access to shared memory before being destroyed. The docs don't say it is obsolete, so it should be kept in Kamailio.
I pushed a commit to fix it, can you try with latest master?
Cheers, Daniel
On 05.10.20 15:13, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
The kamailio wiki has a turorial about troubleshooting with core files where it gives the details about how to enable core file per pid (I am on mobile link right now, not easy to give the url).
OK, will try to look.
To understand that it crashes only on shut down anyhow, not a runtime crash?
I think that so far only at shutdown.
-- Juha
OK, thanks for testing and reporting back.
Cheers, Daniel
On 05.10.20 15:51, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
I pushed a commit to fix it, can you try with latest master?
Tried with latest master and crash does not happen anymore at shutdown.
Thanks, Juha
-
Daniel-Constantin Mierla -
Juha Heinanen