[kamailio/kamailio] Main process exited, code=dumped, status=11/SEGV with http_async (#2616)
### Description I would like to use `http_async_query` and after enabling the module, I notice Kamailio's main process crashes on exit or restart. I have searched and find a number of tickets regarding http_async and crashes, but none related to exit/restart.
### Troubleshooting ``` # coredumpctl info 410246 PID: 410246 (kamailio) UID: 985 (kamailio) GID: 983 (kamailio) Signal: 11 (SEGV) Timestamp: Mon 2021-02-01 00:28:29 CST (11h ago) Command Line: /usr/sbin/kamailio -DD --loadmodule=systemdops.so -m 128 -M 8 Executable: /usr/sbin/kamailio Control Group: /system.slice/kamailio.service Unit: kamailio.service Slice: system.slice Boot ID: 88a3838cfde647a29b3bd62f1e294ce2 Machine ID: e71248e5a94b42069b4603dde95c99b4 Storage: /var/lib/systemd/coredump/core.kamailio.985.88a3838cfde647a29b3bd62f1e294ce2.410246.1612160909000000.zst Message: Process 410246 (kamailio) of user 985 dumped core.
Stack trace of thread 410246: #0 0x00007fd57d84383a BN_clear_free (libcrypto.so.1.1 + 0xb183a) #1 0x00007fd573f9d275 ssh_dh_finalize.part.0 (libssh.so.4 + 0x1e275) #2 0x00007fd573f8ec40 libssh_destructor (libssh.so.4 + 0xfc40) #3 0x00007fd57ead8cd3 _dl_fini (ld-linux-x86-64.so.2 + 0x10cd3) #4 0x00007fd57e7a4237 __run_exit_handlers (libc.so.6 + 0x40237) #5 0x00007fd57e7a43e0 exit (libc.so.6 + 0x403e0) #6 0x000000000041f61a handle_sigs (kamailio + 0x1f61a) #7 0x000000000042c61c main_loop (kamailio + 0x2c61c) #8 0x0000000000434cbd main (kamailio + 0x34cbd) #9 0x00007fd57e78c1e2 __libc_start_main (libc.so.6 + 0x281e2) #10 0x000000000041bbde _start (kamailio + 0x1bbde) ``` #### Reproduction This happens at every kamailio shutdown.
#### Debugging Data
<!-- If you got a core dump, use gdb to extract troubleshooting data - full backtrace, local variables and the list of the code at the issue location.
gdb /path/to/kamailio /path/to/corefile bt full info locals list
If you are familiar with gdb, feel free to attach more of what you consider to be relevant. --> [gdb.txt](https://github.com/kamailio/kamailio/files/5905920/gdb.txt)
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` version: kamailio 5.4.3-5.git06bd17a838.fc33.2 (x86_64/linux) e19ae3 flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: e19ae3 compiled on 00:00:00 Jan 24 2021 with gcc 10.2.1 ```
* **Operating System**:
``` Linux 5.10.10-200.fc33.x86_64 #1 SMP Sun Jan 24 19:58:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux ```
What is the linux distribution?
And to be crystal clear, it happens only on shut down (stop or restart), not at runtime, right?
This is Fedora 33. So far, the http_async_query works well during runtime and the server starts fine. This error seems to get triggered when Kamailio is shutting down.
I forgot to write the important question I had: what is the libssl version?
I forgot to write the important question I had: what is the libssl version?
openssl-1.1.1i-1.fc33.x86_64
From Kamailio start: INFO: tls [tls_mod.c:388]: mod_init(): use OpenSSL version: 1010109f
I also saw in other tickets, sometimes module loading order is an issue. In my case:
``` # cat /lib/systemd/system/kamailio.service [Unit] Description=Kamailio SIP Server Documentation=man:kamailio(8) https://www.kamailio.org/ After=network.target postgresql.service
[Service] Type=notify User=kamailio Environment=SHM_SIZE=128 PKG_SIZE=8 ExecStartPre=/usr/sbin/kamailio -c ExecStart=/usr/sbin/kamailio -DD --loadmodule=systemdops.so -m $SHM_SIZE -M $PKG_SIZE ProtectHome=true Restart=on-failure RestartSec=30
[Install] WantedBy=multi-user.target ```
``` # grep ^loadmodule /etc/kamailio/kamailio.cfg loadmodule "log_systemd.so" loadmodule "db_postgres.so" loadmodule "xhttp.so" loadmodule "jsonrpcs.so" loadmodule "kex.so" loadmodule "corex.so" loadmodule "tm.so" loadmodule "tmx.so" loadmodule "sl.so" loadmodule "stun.so" loadmodule "outbound.so" loadmodule "rr.so" loadmodule "path.so" loadmodule "pv.so" loadmodule "maxfwd.so" loadmodule "usrloc.so" loadmodule "registrar.so" loadmodule "textops.so" loadmodule "siputils.so" loadmodule "sanity.so" loadmodule "ctl.so" loadmodule "cfg_rpc.so" loadmodule "acc.so" loadmodule "counters.so" loadmodule "auth.so" loadmodule "auth_db.so" loadmodule "permissions.so" loadmodule "alias_db.so" loadmodule "speeddial.so" loadmodule "domain.so" loadmodule "presence.so" loadmodule "presence_xml.so" loadmodule "nathelper.so" loadmodule "rtpengine.so" loadmodule "rtpproxy.so" loadmodule "tls.so" loadmodule "htable.so" loadmodule "pike.so" loadmodule "debugger.so" loadmodule "topos.so" loadmodule "jansson.so" loadmodule "dialog.so" loadmodule "group.so" loadmodule "imc.so" loadmodule "lcr.so" loadmodule "app_python3.so" loadmodule "textopsx.so" loadmodule "msilo.so" loadmodule "presence_dialoginfo.so" loadmodule "presence_mwi.so" loadmodule "pua.so" loadmodule "pua_dialoginfo.so" loadmodule "ipops.so" loadmodule "sdpops.so" loadmodule "sipdump.so" loadmodule "sqlops.so" loadmodule "http_async_client.so" loadmodule "uac.so" loadmodule "diversion.so" loadmodule "tmrec.so" loadmodule "websocket.so" loadmodule "auth_ephemeral.so" ```
I think this case is delicate to https://github.com/kamailio/kamailio/issues/2616
Can you try with the master branch and use kamailio with `--no-atexit` command line parameter (which I just added)?
@sergey-safarov - could be, but you didn't provide the gdb backtrace with debugging symbols, as I asked, to be able to relate to the C source code and try to figure out the issue. I saw @henningw wanted to take a different troubleshooting approach and you provided the details he requested, but I did not noticed further updates.
Anyhow, test and if the commit 9f12d314268d1d2cd990cb3c5556f79343869e47 fixes it, then you can close that case as well.
Can you try with the master branch and use kamailio with `--no-atexit` command line parameter (which I just added)?
@miconda I can confirm the patch does work and Kamailio doesn't crash on exit when using `-no-atexit`.
If that's the case, would it always be the preference to have this enabled (no command line switch). What are the downsides?
First version of the code was to see if it actually fixes the crash. Now we should see how common this behaviour is and whether this is going to be the better made default or not. We can flip the parameter to have the option of doing the previous mode of exit. Or maybe make the cli parameter to get a value, so the name can stay, but then we can change the default value in the code, like making the parameter to be `--atexit` and accept `yes` or `no` values. Default now would be `yes`, but if proves that newer Linux distros expose the issue, then set it to `no`.
So far this seems to be specific to newer versions of some distros, the Debian Stable does not expose this behaviour.
I changed the cli parameter from simple `--no-atexit` to `--atexit=yes|no`, so for latest master use `--atexit=no`. For now the default is the equivalent of `--atexit=yes`.
This form allows extensibility in the future, like enabling/disabling at exit callbacks for specific situations (e.g., exit by signal, daemonize, ...).
@miconda, thank you for the explanation. I was thinking this issue was related directly to the http_async_client module, so the fix would be there.
While I'm using your initial patch with small modification to apply on the 5.4 branch, this solution looks like a new feature, rather than a fix. For the bug fix part, can a simple version be backported to 5.4?
It is considered for back porting to branch 5.4. Closing this one.
Closed #2616.
-
Anthony Messina -
Daniel-Constantin Mierla
-
sergey-safarov