Hello,
I'm continuing investigations on Kamailio and stress test. Got it again in the state where it's not accepting any new TCP/TLS connections (UDP still works though), but all looks good from lsof/netnstat part, like system is not reporting any zombie connections. Restart of Kamailio process helps
This time I got output of kamctl trap
Put it here: https://pastebin.com/iYrNZ8U9
kamailio --version version: kamailio 5.6.2 (x86_64/linux) 54a9c1 flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 54a9c1 compiled on 14:01:01 Oct 18 2022 with gcc 4.8.5
It's statically linked with tlsa pointing on openssl-1.1.1q
Settings related to TLS are:
fork=yes children=4 tcp_children=12 enable_tls=yes
tcp_accept_no_cl=yes tcp_max_connections=63536 tls_max_connections=63536 tcp_accept_aliases=no tcp_async=yes tcp_connect_timeout=10 tcp_conn_wq_max=63536 tcp_crlf_ping=yes tcp_delayed_ack=yes tcp_fd_cache=yes tcp_keepalive=yes tcp_keepcnt=3 tcp_keepidle=30 tcp_keepintvl=10 tcp_linger2=30 tcp_rd_buf_size=80000 tcp_send_timeout=10 tcp_wq_blk_size=2100 tcp_wq_max=10485760 open_files_limit=63536
Can you please help to read gdb output and understand where I missed in config?
Thanks in advance!
Daniel,
Thanks for the tip, recompiled and got a new trace
Not sure what happened, but sometimes running kamctl trap restores Kamailio functionality to accept connections.
Thanks in advance!
Le 29/11/2022 à 08:07, Daniel-Constantin Mierla a écrit :
Still now showing the debug symbols for libssl/libcrypto:
1. 0x00007fdc7d07754d in __lll_lock_wait () from /lib64/libpthread.so.0 2. #0 0x00007fdc7d07754d in __lll_lock_wait () from /lib64/libpthread.so.0 3. No symbol table info available. 4. #1 0x00007fdc7d0743d2 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 5. No symbol table info available. 6. #2 0x00007fdc737f2c59 in CRYPTO_THREAD_write_lock () from /usr/local/lib64/kamailio/modules/tlsa.so 7. No symbol table info available. 8. #3 0x00007fdc7379e32e in CRYPTO_free_ex_data () from /usr/local/lib64/kamailio/modules/tlsa.so 9. No symbol table info available. 10. #4 0x00007fdc73708ae2 in SSL_free () from /usr/local/lib64/kamailio/modules/tlsa.so 11. No symbol table info available. 12.
How are these libs installed?
Cheers, Daniel
On 29.11.22 10:38, Ihor Olkhovskyi wrote:
Maybe you can try with git master branch and setting tls moduparam lock_mode to 1 -- let's see if there is any change in behaviour.
Cheers, Daniel
On 08.12.22 10:46, Ihor Olkhovskyi wrote:
Adding a note that the `lock_mode` parameter was renamed to `init_mode`, as I plan to use it for other purposes.
Cheers, Daniel
On 23.01.23 15:23, Daniel-Constantin Mierla wrote:
Daniel,
It does not change the situation, sorry (
Only thing I can confirm, that it's not listener-specific. Means if Kamailio stops answer on TLS on one interface, other interfaces over TLS are also affected (I have explicit listen=)
Le 24/01/2023 à 09:39, Daniel-Constantin Mierla a écrit :
-
Daniel-Constantin Mierla -
Igor Olhovskiy -
Ihor Olkhovskyi