sr-users March 2025

sr-users@lists.kamailio.org

44 participants
34 discussions

Re: send_fd(): sendmsg failed sending 55 on 52: Too many references: cannot splice (109)
by Mathias Schneuwly 24 Mar '25

24 Mar '25

Thank you, I've already changed the value, but that didn't solve my issue with the blocking Kamailio. I did some tests with an older Yocto image which contains Kamailio 5.5.4 and OpenSSL 1.1.1w. With this combination I was not able to reproduce the deadlock. Am Fr., 21. März 2025 um 16:49 Uhr schrieb Harald Kapper <hk(a)kapper.net>: > Hi > > fyi – in case you need this: > > systemctl edit kamailio.service > > > > add: > > ### Editing /etc/systemd/system/kamailio.service.d/override.conf > > ### Anything between here and the comment below will become the contents > of the drop-in file > > > > [Service] > > LimitNOFILE=40960 > > > > ### Edits below this comment will be discarded > > > > (or whatever number you’d like) > > saving saves the override for systemd into (like) > /etc/systemd/system/kamailio.service.d/override.conf > > > > Afterwards you have to restart the service (systemctl restart > kamailio.service) > > > > And you should be fine. > > > > Regards, hk > > > > *Von:* Henning Westerholt via sr-users <sr-users(a)lists.kamailio.org> > *Gesendet:* Donnerstag, 20. März 2025 14:25 > *An:* Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org> > *Cc:* Mathias Schneuwly <mathias(a)schneuwlys.ch>; Henning Westerholt < > hw(a)gilawa.com> > *Betreff:* [SR-Users] Re: send_fd(): sendmsg failed sending 55 on 52: Too > many references: cannot splice (109) > > > > Hello Mathias, > > > > yes, you can set the open files e.g. in the systemd unit file for Kamailio. > > > > Cheers, > > > > Henning > > > > *From:* Mathias Schneuwly via sr-users <sr-users(a)lists.kamailio.org> > *Sent:* Donnerstag, 20. März 2025 13:43 > *To:* sr-users(a)lists.kamailio.org > *Cc:* Mathias Schneuwly <mathias(a)schneuwlys.ch> > *Subject:* [SR-Users] Re: send_fd(): sendmsg failed sending 55 on 52: Too > many references: cannot splice (109) > > > > > I'll also give a try with 6.0.1 later. > > Meanwhile, I've tested 5.8.6 and 6.0.1 and both show the same issue. > > > > Am Do., 20. März 2025 um 10:21 Uhr schrieb Mathias Schneuwly < > mathias(a)schneuwlys.ch>: > > Hi Henning > > > > > Maybe also check the actual limits by “cat /proc/$kamailio-pid/limits” > Not 100% if its related, but if you are looking to scale this would be for > sure required. > > The limit was > > Max open files 1024 524288 files > > and I changed it now to > > Max open files 65536 65536 files > > > > Are there some recommendations for the limit? Would it make sense to > change that in the provided systemd file if you say that 1024 is not that > much? > > > > But anyway, it fails with 1024 and with 65536. > > > > I've also updated to 5.8.5 and it shows the same issue. I'll also give a > try with 6.0.1 later. > > > > > About the tls_thread_mode – you should use mode 2 if you are running > with openssl 3.x, otherwise you probably will observe random crashes. > > We are running OpenSSL 3.2.3 and tls_thread_mode is set to 2. But it still > blocks. > > > > Best regards > > Mathias > > > > Am Do., 20. März 2025 um 08:42 Uhr schrieb Henning Westerholt < > hw(a)gilawa.com>: > > Hello Mathias, > > > > 1024 open files are not that much, you should increase it for the kamailio > user. Maybe also check the actual limits by “cat > /proc/$kamailio-pid/limits” > > Not 100% if its related, but if you are looking to scale this would be for > sure required. > > > > About the tls_thread_mode – you should use mode 2 if you are running with > openssl 3.x, otherwise you probably will observe random crashes. > > > > If you are seeing the mentioned problem with 15 phones, this is clearly a > problem and should be further investigated. > > > > Cheers, > > > > Henning > > > > > > *From:* Mathias Schneuwly via sr-users <sr-users(a)lists.kamailio.org> > *Sent:* Donnerstag, 20. März 2025 08:07 > *To:* sr-users(a)lists.kamailio.org > *Cc:* Mathias Schneuwly <mathias(a)schneuwlys.ch> > *Subject:* [SR-Users] Re: send_fd(): sendmsg failed sending 55 on 52: Too > many references: cannot splice (109) > > > > Hi > > > > Thanks for the reply. We are using the following limits and I guess they > are the default values > > real-time non-blocking time (microseconds, -R) unlimited > core file size (blocks, -c) unlimited > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 15666 > max locked memory (kbytes, -l) 8192 > max memory size (kbytes, -m) unlimited > open files (-n) 1024 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 15666 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > > > > But to me, it looks more like a deadlock somewhere around SSL_accept. > According to the stack trace generated with "kamctl trap", all 8 TCP > receiver processes are waiting in "#1 0x00007fa14ca04830 in > pthread_rwlock_wrlock () from /usr/lib/libc.so.6". I guess the message "Too > many references: cannot splice (109)" is just the consequence as the main > process is no longer able to dispatch jobs to the child workers. > > > > Meanwhile, I'm able to reproduce the issue. It requires a bit of brute > force, but it deadlocks reliably after a few seconds. > > > > I'm flooding Kamailio with register messages using sipexer. I'm running > the following command 3 times in parallel and always after a few seconds, > Kamailio is completely blocked. > > watch -n 0.1 "./sipexer -register -vl 3 -co -com -ex 3600 -fu "2" -au "2" > -ap "test" -tk <path to key>.key -tc <path to cert>.cer -ti -sd -ru > sip:<domain> -cu sip:2@<domain> tls:<fqdn>:5061;sleep 0.1;./sipexer > -register -vl 3 -co -com -ex 0 -fu "2" -au "2" -ap "test" -tk <path to > key>.key -tc <path to cert>.cer -ti -sd -ru sip:<domain> -cu sip:2@<domain> > tls:<fqdn>:5061" > > > > I played a bit with the parameters "modparam("tls", "init_mode", 1)" and > "tls_threads_mode = 2", but it didn't seem to change anything. > > > > Please have in mind that we usually do not stress Kamailio that much. We > have seen the issue a couple of times with around 5 to 15 SIP phones > registered. But in this usual setup, the problem does not occur reliably. > > > > Best regards > > Mathias > > > > Am Di., 18. März 2025 um 21:53 Uhr schrieb Henning Westerholt < > hw(a)gilawa.com>: > > Hello, > > > > check the ulimit settings for the kamailio user and general resource usage > on that system. In development branch there is also a feature present to > print the actual ulimits on startup. > > > > Cheers, > > > > Henning > > > > *From:* Mathias Schneuwly via sr-users <sr-users(a)lists.kamailio.org> > *Sent:* Montag, 17. März 2025 13:41 > *To:* sr-users(a)lists.kamailio.org > *Cc:* Mathias Schneuwly <mathias(a)schneuwlys.ch> > *Subject:* [SR-Users] send_fd(): sendmsg failed sending 55 on 52: Too > many references: cannot splice (109) > > > > Hi > > > > We are using kamailio 5.8.4. > > > > Recently, we noticed, that after a few days, we see the following messages > from Kamailio `2025 Mar 17 10:22:11 ttel CRITICAL: <core> > [core/pass_fd.c:193]: send_fd(): sendmsg failed sending 55 on 52: Too many > references: cannot splice (109)`. > > > > When this happens, devices are no longer able to register. At the moment > we have no idea what might be wrong. > > > > Unfortunately, the internet does not reveal much information regarding > this issue. In one issue related to Kamailio ( > https://lists.kamailio.org/mailman3/hyperkitty/list/sr-users@lists.kamailio…), > I found the command `kamctl trap` and when I execute that, I see one issue, > which might be related, but I'm just guessing... > > > > ``` > > ---start 1782236 ----------------------------------------------------- > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/usr/lib/libthread_db.so.1". > 0x00007fa14c9fb1c6 in ?? () from /usr/lib/libc.so.6 > $1 = 26 > $2 = {pid = 1782236, unix_sock = 36, idx = 0, status = 1, rank = 17, desc > = "tcp receiver (generic) child=0", '\000' <repeats 97 times>} > #0 0x00007fa14c9fb1c6 in ?? () from /usr/lib/libc.so.6 > No symbol table info available. > #1 0x00007fa14ca04830 in pthread_rwlock_wrlock () from /usr/lib/libc.so.6 > No symbol table info available. > #2 0x00007fa14ac20b49 in CRYPTO_THREAD_write_lock () from > /usr/lib/libcrypto.so.3 > No symbol table info available. > #3 0x00007fa14adbd5da in ?? () from /usr/lib/libcrypto.so.3 > No symbol table info available. > #4 0x00007fa14adbd9ca in ?? () from /usr/lib/libcrypto.so.3 > No symbol table info available. > #5 0x00007fa14adbc907 in ?? () from /usr/lib/libcrypto.so.3 > No symbol table info available. > #6 0x00007fa14adbd26c in ?? () from /usr/lib/libcrypto.so.3 > No symbol table info available. > #7 0x00007fa14adbd68e in ?? () from /usr/lib/libcrypto.so.3 > No symbol table info available. > #8 0x00007fa14abe75c6 in ?? () from /usr/lib/libcrypto.so.3 > No symbol table info available. > #9 0x00007fa14abe840b in EVP_RAND_generate () from /usr/lib/libcrypto.so.3 > No symbol table info available. > #10 0x00007fa14acf9282 in RAND_bytes_ex () from /usr/lib/libcrypto.so.3 > No symbol table info available. > #11 0x00007fa14b06cce8 in ?? () from /usr/lib/libssl.so.3 > No symbol table info available. > #12 0x00007fa14b05bdf7 in ?? () from /usr/lib/libssl.so.3 > No symbol table info available. > #13 0x00007fa14b0ef036 in tls_accept (c=c@entry=0x7fa1475c0dc0, > error=error@entry=0x7ffebd9443b8) at tls_server.c:457 > ret = <optimized out> > ssl = warning: could not convert 'ssl_st' from the host encoding > (ANSI_X3.4-1968) to UTF-32. > This normally should not happen, please file a bug report. > 0x7fa147668bd0 > cert = <optimized out> > tls_c = 0x7fa14768f490 > tls_log = <optimized out> > __func__ = "tls_accept" > #14 0x00007fa14b0f7540 in tls_h_read_f (c=0x7fa1475c0dc0, > flags=0x7ffebd9647d0) at tls_server.c:1153 > r = 0x7fa1475c0ee8 > bytes_free = 16383 > bytes_read = <optimized out> > read_size = <optimized out> > ssl_error = 0 > ssl_read = 0 > ssl = 0x7fa147668bd0 > rd_buf = > "\026\003\003\004\016\v\000\004\n\000\004\a\000\004\0040\202\004\0000\202\002\350\240\003\002\001\002\002\002\026\0270\r\006\t*\206H\206\367\r\001\001\v\005\0000b1\v0\t\006\003U\004\006\023\002CH1\r0\v\006\003U\004\b\f\004Bern1\r0\v\006\003U\004\a\f\004Bern1\r0\v\006\003U\004\n\f\004RUAG1\0200\016\006\003U\004\v\f\aDefence1\0240\022\006\003U\004\003\f\versaimfs.ch0\036\027\r230302163519Z\027\r291230163519Z0`1\v0\t\006\003U\004\006\023\002CH1\r0\v\006\003"... > wr_buf = > "\026\003\003\000A\002\000\000=\003\003\371\204\326\361\262-9\300\344gV\2316\373\235D\314\225u8\v}\215\254DOWNGRD\001\000\3000\000\000\025\377\001\000\001\000\000\v\000\004\003\000\001\002\000#\000\000\000\027\000\000\026\003\003\a\267\v\000\a\263\000\a\260\000\004\0040\202\004\0000\202\002\350\240\003\002\001\002\002\002\r>0\r\006\t*\206H\206\367\r\001\001\v\005\0000b1\v0\t\006\003U\004\006\023\002CH1\r0\v\006\003U\004\b\f\004Bern1\r0\v\006\003U\004\a\f\004Bern1\r0\v\006\003U\004\n\f\004RUAG1\0200\016\006\003U\004\v\f\aDefence1\0240\022\006"... > rd = {buf = 0x7ffebd944480 "\026\003\003\004\016\v", pos = 1405, > used = 1405, size = 65536} > wr = {buf = 0x7ffebd954480 "\026\003\003", pos = 0, used = 0, size > = 65536} > tls_c = 0x7fa14768f490 > enc_rd_buf = <optimized out> > n = 0 > flush_flags = 0 > err_src = <optimized out> > ip_buf = '\000' <repeats 63 times> > x = <optimized out> > tls_dbg = <optimized out> > __func__ = "tls_h_read_f" > redo_read = <optimized out> > continue_ssl_read = <optimized out> > __llevel = <optimized out> > __kld = <optimized out> > __llevel = <optimized out> > __kld = <optimized out> > #15 0x000055d2519871b4 in tcp_read_headers (c=c@entry=0x7fa1475c0dc0, > read_flags=read_flags@entry=0x7ffebd9647d0) at core/tcp_read.c:445 > bytes = <optimized out> > remaining = <optimized out> > p = <optimized out> > r = 0x7fa1475c0ee8 > mc = <optimized out> > body_len = <optimized out> > tvnow = {tv_sec = 1742100857, tv_usec = 944923} > tvdiff = <optimized out> > mfline = <optimized out> > mtransid = <optimized out> > __func__ = "tcp_read_headers" > #16 0x000055d251989cef in tcp_read_req (con=0x7fa1475c0dc0, > bytes_read=bytes_read@entry=0x7ffebd9647c8, read_flags=read_flags@entry=0x7ffebd9647d0) > at core/tcp_read.c:1509 > bytes = <optimized out> > total_bytes = 0 > resp = 1 > size = <optimized out> > req = 0x7fa1475c0ee8 > dst = {send_sock = 0x55d251a91020, to = {s = {sa_family = 52024, > sa_data = "\257Q\322U\000\000\002\000\000\000\376\177\000"}, sin = > {sin_family = 52024, sin_port = 20911, sin_addr = {s_addr = 21970}, > sin_zero = "\002\000\000\000\376\177\000"}, sin6 = {sin6_family = 52024, > sin6_port = 20911, sin6_flowinfo = 21970, sin6_addr = {__in6_u = > {__u6_addr8 = > "\002\000\000\000\376\177\000\000\030\000\000\000\000\000\000", __u6_addr16 > = {2, 0, 32766, 0, 24, 0, 0, 0}, __u6_addr32 = {2, 32766, 24, 0}}}, > sin6_scope_id = 0}, sas = {ss_family = 52024, __ss_padding = > "\257Q\322U\000\000\002\000\000\000\376\177\000\000\030", '\000' <repeats > 23 times>, > "\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\262+\226Q\001 > \000\000\200,5L\241\177\000\000 \020\251Q\322U\000\000 > \020\251Q\322U\000\000\200\303\257Q\322U\000\00009SG\241\177\000\000\330G\226\275\376\177\000\000\b\000\000\000\000\000\000", > __ss_align = 20}}, id = 1, send_flags = {f = 1, blst_imask = 0}, proto = 14 > '\016', proto_pad0 = 0 '\000', proto_pad1 = 0} > c = <optimized out> > ret = <optimized out> > again = <optimized out> > __func__ = "tcp_read_req" > #17 0x000055d25198ebec in handle_io (fm=fm@entry=0x7fa14c352c80, > events=events@entry=1, idx=idx@entry=-1) at core/tcp_read.c:1906 > ret = 8 > n = 0 > read_flags = RD_CONN_SHORT_READ > con = 0x7fa1475c0dc0 > s = 14 > resp = <optimized out> > t = <optimized out> > ee = 0x0 > __func__ = "handle_io" > error = <optimized out> > #18 0x000055d25199469b in io_wait_loop_epoll (repeat=repeat@entry=0, t=2, > h=0x55d251c49d60 <io_w>) at core/io_wait.h:1075 > n = 1 > r = 0 > fm = 0x7fa14c352c80 > revents = 1 > __func__ = "io_wait_loop_epoll" > #19 0x000055d251994e87 in tcp_receive_loop (unix_sock=<optimized out>) at > core/tcp_read.c:2026 > __func__ = "tcp_receive_loop" > #20 0x000055d25197e72c in tcp_init_children (woneinit=woneinit@entry=0x7ffebd964d1c) > at core/tcp_main.c:5391 > r = <optimized out> > i = <optimized out> > reader_fd_1 = 39 > pid = <optimized out> > si_desc = "tcp receiver > (generic)\000\000`M\226\275\376\177\000\000\001\000\000\000\000\000\000\000\210\035\223Q\322U\000\000 > \020\251Q\322U\000\000\220\315\257Q\322U\000\000\220\242/L\241\177\000\000\025\000\000\000\000\000\000\000\004\000\000\000\000\000\000\000%\020\251Q\322U\000\000HU\257Q\322U\000\000e\005\000\000\000\000\000\000 > \020\251Q\322U\000\000\3750\235Q\322U\000" > si = <optimized out> > __func__ = "tcp_init_children" > error = <optimized out> > #21 0x000055d2517c6c7e in main_loop () at main.c:1950 > i = <optimized out> > pid = <optimized out> > si = <optimized out> > si_desc = "udp receiver child=7 sock=10.0.90.1:5060\000\000\000\000\322U\000\000?\342\017K\241\177\000\000\022\000\000\000\377\377\377\377\200\006\265L\241\177\000\000\253m\237L\241\177\000\000\361|\003\000\000\000\000\000\t\000\000\000\000\000\000\000Sat > Mar 15 05:25:32 2025\n\000\264L\241\177\000" > nrprocs = <optimized out> > woneinit = 1 > __func__ = "main_loop" > error = <optimized out> > #22 0x000055d2517bc151 in main (argc=<optimized out>, argv=<optimized > out>) at main.c:3256 > cfg_stream = <optimized out> > c = <optimized out> > r = <optimized out> > tmp = 0x7ffebd966e7f "" > tmp_len = 32673 > port = 5060 > proto = 0 > aproto = 0 > ahost = 0x0 > aport = 0 > options = 0x55d251aa74b8 > ":f:cm:M:dVIhEeb:B:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" > ret = -1 > seed = 3209119183 > rfd = <optimized out> > debug_save = <optimized out> > debug_flag = <optimized out> > dont_fork_cnt = 0 > n_lst = <optimized out> > p = <optimized out> > st = {st_dev = 24, st_ino = 46237, st_nlink = 2, st_mode = 16888, > st_uid = 899, st_gid = 899, __pad0 = 0, st_rdev = 0, st_size = 40, > st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1742012732, tv_nsec = > 217328715}, st_mtim = {tv_sec = 1742012732, tv_nsec = 217328715}, st_ctim = > {tv_sec = 1742012732, tv_nsec = 217328715}, __glibc_reserved = {0, 0, 0}} > l1 = <optimized out> > tbuf = '\000' <repeats 32 times>, > "\030\202\237\275\376\177\000\000@\002\000\000@\003\000\000\001", '\000' > <repeats 23 times>, "@\003\000\000@\003\000\000@\003\000\000@\003\000\000@ > \003\000\000@\003\000\000@\003\000\000@\003\000\000@\003\000\000@ > \003\000\000@\003\000\000@\003\000\000@\003\000\000@\003\000\000@ > \003\000\000@\003\000\000@\003\000\000@\003\000\000@\003\000\000@ > \003\000\000@\003\000\000@\003", '\000' <repeats 11 times>, "\001", > '\000' <repeats 118 times>... > option_index = 12 > long_options = {{name = 0x55d251a9230b "help", has_arg = 0, flag = > 0x0, val = 104}, {name = 0x55d251a937a4 "version", has_arg = 0, flag = 0x0, > val = 118}, {name = 0x55d251a9826a "alias", has_arg = 1, flag = 0x0, val = > 1024}, {name = 0x55d251a92310 "subst", has_arg = 1, flag = 0x0, val = > 1025}, {name = 0x55d251a92316 "substdef", has_arg = 1, flag = 0x0, val = > 1026}, {name = 0x55d251a9231f "substdefs", has_arg = 1, flag = 0x0, val = > 1027}, {name = 0x55d251a92329 "server-id", has_arg = 1, flag = 0x0, val = > 1028}, {name = 0x55d251a92333 "loadmodule", has_arg = 1, flag = 0x0, val = > 1029}, {name = 0x55d251a9233e "modparam", has_arg = 1, flag = 0x0, val = > 1030}, {name = 0x55d251a92347 "log-engine", has_arg = 1, flag = 0x0, val = > 1031}, {name = 0x55d251a938c1 "debug", has_arg = 1, flag = 0x0, val = > 1032}, {name = 0x55d251a92352 "cfg-print", has_arg = 0, flag = 0x0, val = > 1033}, {name = 0x55d251a9235c "atexit", has_arg = 1, flag = 0x0, val = > 1034}, {name = 0x55d251a92363 "all-errors", has_arg = 0, flag = 0x0, val = > 1035}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} > __func__ = "main" > [Inferior 1 (process 1782236) detached] > ---end 1782236 ------------------------------------------------------- > > ``` > > > > Does anybody have an idea what this could be? I never saw that with > previous versions, and now we can see it on several instances, but we are > yet unable to reproduce. > > > > Not sure if this might be related, but we also see many messages like `Mar > 17 13:39:03 ttel /usr/sbin/kamailio[2882127]: CRITICAL: <core> > [core/tcp_main.c:5558]: tcp_timer_check_connections(): message processing > timeout on connection id: 402 (state: 0) - closing` > > > > Best regards > > Mathias > >

1 0

Dispatcher module resets URI parameters
by Marat Gareev 22 Mar '25

22 Mar '25

Hello, I encountered a problem that ds_select_domain function does not save URI parameters. I found workaround: if (!ds_select_domain( "1", 4, 3 )) { send_reply("502", "Bad Gateway"); exit; } if (!strempty($(ou{uri.params}))) { $ru = $ru+";"+$(ou{uri.params}); } But this can lead to duplicates if the URI in the dispatcher list contains the same parameter. Is there any other reliable way to keep original URI parameters while using the dispatcher? Thanks

2 1

How to place a delay between cancelling branch and retrying next one on timer expiry?
by Alex A 21 Mar '25

21 Mar '25

Hi All, I have an interesting use-case where I need to place a delay between cancelling branch and retrying next one. *Background: * Running Kamailio 5.6.x Kamailio routes requests via a freeswitch B2BUA server on select call flows (where B2BUA functionality is required). Call flow examples: PSTN --> Kamailio --> Freeswitch --> appserver1 PSTN --> Kamailio --> Freeswitch --> appserver2 On attempt failures, a branch retry is required from appserver1 to appserver2; however freeswitch B2BUA remains the same Since Freeswitch does not quite follow the RFC 3261 timer K specifications (should be 0 seconds for reliable transports, but FS keeps transactions for T4 (set to 2 seconds)); I have implemented an artificial delay in failure route before retrying (as per recommendation described here: https://freeswitch-users.freeswitch.narkive.com/TsSye66D/482-request-merged… ) *Script config: * onreply_route[MANAGE_REPLY] { xlog("L_INFO", "reply $T_reply_code received"); if (t_check_status( "1[1-9][0-9]")) { t_reset_fr(); } else { t_set_fr(120000); } } route(NATDETECT); #!ifdef WITH_RTPENGINE route(RTPMANAGE); #!endif } route[RELAY] { if (is_method("INVITE|BYE|SUBSCRIBE|UPDATE")) { if(!t_is_set("branch_route")) t_on_branch("MANAGE_BRANCH"); } if (is_method("INVITE|SUBSCRIBE|UPDATE")) { if(!t_is_set("onreply_route")) t_on_reply("MANAGE_REPLY"); } if(is_method("INVITE|BYE|UPDATE|CANCEL|ACK")) { setflag(FLT_DLGINFO); } if (!t_relay()) { sl_reply_error(); } exit; } route[TOCARRIER] { process bunch of things, attach custom headers, generate request URI t_on_failure("MANAGE_FAILURE_CARRIER"); route(RELAY); } failure_route[MANAGE_FAILURE_CARRIER] { xlog("L_INFO", "reply $T_reply_code received"); if ($avp(inbound_attempt)<=1){ if ($dlg_var(b2bua)=="true"){ usleep(1900000); } route("TOCARRIER"); } *Call Flows: * The above fix works well when an explicit error reply is received back. *error reply SIP exchange: * SBC --> INVITE(branch0) ---> Freeswitch SBC <-- 503 <-- Freeswitch failure_route executed (sleep runs here) SBC --> INVITE(branch1) ---> Freeswitch call continues as expected. however does not work when NO reply is received at all and the timer expires. *Current timer expiry SIP exchange: * SBC --> INVITE(branch0) ---> Freeswitch SBC <-- 100/180 <-- Freeswitch SBC (timer expires) failure_route executed (sleep runs here) SBC --> INVITE(branch1) ---> Freeswitch SBC <-- 482 Merged <-- Freeswitch SBC --> CANCEL(branch0) ---> Freeswitch I am looking to CANCEL branch0 before the sleep delay runs Based on source code, t_cancel_branches("this") can only be executed in reply_route; however the chellenge is that reply_route never sees the "faked 408" on timer expiry(although the failure route does see $T_reply_code 408). Any suggestions on how to handle this would be greatly appreciated. TIA

6 10

send_fd(): sendmsg failed sending 55 on 52: Too many references: cannot splice (109)
by Mathias Schneuwly 20 Mar '25

20 Mar '25

Hi We are using kamailio 5.8.4. Recently, we noticed, that after a few days, we see the following messages from Kamailio `2025 Mar 17 10:22:11 ttel CRITICAL: <core> [core/pass_fd.c:193]: send_fd(): sendmsg failed sending 55 on 52: Too many references: cannot splice (109)`. When this happens, devices are no longer able to register. At the moment we have no idea what might be wrong. Unfortunately, the internet does not reveal much information regarding this issue. In one issue related to Kamailio ( https://lists.kamailio.org/mailman3/hyperkitty/list/sr-users@lists.kamailio…), I found the command `kamctl trap` and when I execute that, I see one issue, which might be related, but I'm just guessing... ``` ---start 1782236 ----------------------------------------------------- [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". 0x00007fa14c9fb1c6 in ?? () from /usr/lib/libc.so.6 $1 = 26 $2 = {pid = 1782236, unix_sock = 36, idx = 0, status = 1, rank = 17, desc = "tcp receiver (generic) child=0", '\000' <repeats 97 times>} #0 0x00007fa14c9fb1c6 in ?? () from /usr/lib/libc.so.6 No symbol table info available. #1 0x00007fa14ca04830 in pthread_rwlock_wrlock () from /usr/lib/libc.so.6 No symbol table info available. #2 0x00007fa14ac20b49 in CRYPTO_THREAD_write_lock () from /usr/lib/libcrypto.so.3 No symbol table info available. #3 0x00007fa14adbd5da in ?? () from /usr/lib/libcrypto.so.3 No symbol table info available. #4 0x00007fa14adbd9ca in ?? () from /usr/lib/libcrypto.so.3 No symbol table info available. #5 0x00007fa14adbc907 in ?? () from /usr/lib/libcrypto.so.3 No symbol table info available. #6 0x00007fa14adbd26c in ?? () from /usr/lib/libcrypto.so.3 No symbol table info available. #7 0x00007fa14adbd68e in ?? () from /usr/lib/libcrypto.so.3 No symbol table info available. #8 0x00007fa14abe75c6 in ?? () from /usr/lib/libcrypto.so.3 No symbol table info available. #9 0x00007fa14abe840b in EVP_RAND_generate () from /usr/lib/libcrypto.so.3 No symbol table info available. #10 0x00007fa14acf9282 in RAND_bytes_ex () from /usr/lib/libcrypto.so.3 No symbol table info available. #11 0x00007fa14b06cce8 in ?? () from /usr/lib/libssl.so.3 No symbol table info available. #12 0x00007fa14b05bdf7 in ?? () from /usr/lib/libssl.so.3 No symbol table info available. #13 0x00007fa14b0ef036 in tls_accept (c=c@entry=0x7fa1475c0dc0, error=error@entry=0x7ffebd9443b8) at tls_server.c:457 ret = <optimized out> ssl = warning: could not convert 'ssl_st' from the host encoding (ANSI_X3.4-1968) to UTF-32. This normally should not happen, please file a bug report. 0x7fa147668bd0 cert = <optimized out> tls_c = 0x7fa14768f490 tls_log = <optimized out> __func__ = "tls_accept" #14 0x00007fa14b0f7540 in tls_h_read_f (c=0x7fa1475c0dc0, flags=0x7ffebd9647d0) at tls_server.c:1153 r = 0x7fa1475c0ee8 bytes_free = 16383 bytes_read = <optimized out> read_size = <optimized out> ssl_error = 0 ssl_read = 0 ssl = 0x7fa147668bd0 rd_buf = "\026\003\003\004\016\v\000\004\n\000\004\a\000\004\0040\202\004\0000\202\002\350\240\003\002\001\002\002\002\026\0270\r\006\t*\206H\206\367\r\001\001\v\005\0000b1\v0\t\006\003U\004\006\023\002CH1\r0\v\006\003U\004\b\f\004Bern1\r0\v\006\003U\004\a\f\004Bern1\r0\v\006\003U\004\n\f\004RUAG1\0200\016\006\003U\004\v\f\aDefence1\0240\022\006\003U\004\003\f\versaimfs.ch0\036\027\r230302163519Z\027\r291230163519Z0`1\v0\t\006\003U\004\006\023\002CH1\r0\v\006\003"... wr_buf = "\026\003\003\000A\002\000\000=\003\003\371\204\326\361\262-9\300\344gV\2316\373\235D\314\225u8\v}\215\254DOWNGRD\001\000\3000\000\000\025\377\001\000\001\000\000\v\000\004\003\000\001\002\000#\000\000\000\027\000\000\026\003\003\a\267\v\000\a\263\000\a\260\000\004\0040\202\004\0000\202\002\350\240\003\002\001\002\002\002\r>0\r\006\t*\206H\206\367\r\001\001\v\005\0000b1\v0\t\006\003U\004\006\023\002CH1\r0\v\006\003U\004\b\f\004Bern1\r0\v\006\003U\004\a\f\004Bern1\r0\v\006\003U\004\n\f\004RUAG1\0200\016\006\003U\004\v\f\aDefence1\0240\022\006"... rd = {buf = 0x7ffebd944480 "\026\003\003\004\016\v", pos = 1405, used = 1405, size = 65536} wr = {buf = 0x7ffebd954480 "\026\003\003", pos = 0, used = 0, size = 65536} tls_c = 0x7fa14768f490 enc_rd_buf = <optimized out> n = 0 flush_flags = 0 err_src = <optimized out> ip_buf = '\000' <repeats 63 times> x = <optimized out> tls_dbg = <optimized out> __func__ = "tls_h_read_f" redo_read = <optimized out> continue_ssl_read = <optimized out> __llevel = <optimized out> __kld = <optimized out> __llevel = <optimized out> __kld = <optimized out> #15 0x000055d2519871b4 in tcp_read_headers (c=c@entry=0x7fa1475c0dc0, read_flags=read_flags@entry=0x7ffebd9647d0) at core/tcp_read.c:445 bytes = <optimized out> remaining = <optimized out> p = <optimized out> r = 0x7fa1475c0ee8 mc = <optimized out> body_len = <optimized out> tvnow = {tv_sec = 1742100857, tv_usec = 944923} tvdiff = <optimized out> mfline = <optimized out> mtransid = <optimized out> __func__ = "tcp_read_headers" #16 0x000055d251989cef in tcp_read_req (con=0x7fa1475c0dc0, bytes_read=bytes_read@entry=0x7ffebd9647c8, read_flags=read_flags@entry=0x7ffebd9647d0) at core/tcp_read.c:1509 bytes = <optimized out> total_bytes = 0 resp = 1 size = <optimized out> req = 0x7fa1475c0ee8 dst = {send_sock = 0x55d251a91020, to = {s = {sa_family = 52024, sa_data = "\257Q\322U\000\000\002\000\000\000\376\177\000"}, sin = {sin_family = 52024, sin_port = 20911, sin_addr = {s_addr = 21970}, sin_zero = "\002\000\000\000\376\177\000"}, sin6 = {sin6_family = 52024, sin6_port = 20911, sin6_flowinfo = 21970, sin6_addr = {__in6_u = {__u6_addr8 = "\002\000\000\000\376\177\000\000\030\000\000\000\000\000\000", __u6_addr16 = {2, 0, 32766, 0, 24, 0, 0, 0}, __u6_addr32 = {2, 32766, 24, 0}}}, sin6_scope_id = 0}, sas = {ss_family = 52024, __ss_padding = "\257Q\322U\000\000\002\000\000\000\376\177\000\000\030", '\000' <repeats 23 times>, "\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\262+\226Q\001 \000\000\200,5L\241\177\000\000 \020\251Q\322U\000\000 \020\251Q\322U\000\000\200\303\257Q\322U\000\00009SG\241\177\000\000\330G\226\275\376\177\000\000\b\000\000\000\000\000\000", __ss_align = 20}}, id = 1, send_flags = {f = 1, blst_imask = 0}, proto = 14 '\016', proto_pad0 = 0 '\000', proto_pad1 = 0} c = <optimized out> ret = <optimized out> again = <optimized out> __func__ = "tcp_read_req" #17 0x000055d25198ebec in handle_io (fm=fm@entry=0x7fa14c352c80, events=events@entry=1, idx=idx@entry=-1) at core/tcp_read.c:1906 ret = 8 n = 0 read_flags = RD_CONN_SHORT_READ con = 0x7fa1475c0dc0 s = 14 resp = <optimized out> t = <optimized out> ee = 0x0 __func__ = "handle_io" error = <optimized out> #18 0x000055d25199469b in io_wait_loop_epoll (repeat=repeat@entry=0, t=2, h=0x55d251c49d60 <io_w>) at core/io_wait.h:1075 n = 1 r = 0 fm = 0x7fa14c352c80 revents = 1 __func__ = "io_wait_loop_epoll" #19 0x000055d251994e87 in tcp_receive_loop (unix_sock=<optimized out>) at core/tcp_read.c:2026 __func__ = "tcp_receive_loop" #20 0x000055d25197e72c in tcp_init_children (woneinit=woneinit@entry=0x7ffebd964d1c) at core/tcp_main.c:5391 r = <optimized out> i = <optimized out> reader_fd_1 = 39 pid = <optimized out> si_desc = "tcp receiver (generic)\000\000`M\226\275\376\177\000\000\001\000\000\000\000\000\000\000\210\035\223Q\322U\000\000 \020\251Q\322U\000\000\220\315\257Q\322U\000\000\220\242/L\241\177\000\000\025\000\000\000\000\000\000\000\004\000\000\000\000\000\000\000%\020\251Q\322U\000\000HU\257Q\322U\000\000e\005\000\000\000\000\000\000 \020\251Q\322U\000\000\3750\235Q\322U\000" si = <optimized out> __func__ = "tcp_init_children" error = <optimized out> #21 0x000055d2517c6c7e in main_loop () at main.c:1950 i = <optimized out> pid = <optimized out> si = <optimized out> si_desc = "udp receiver child=7 sock=10.0.90.1:5060\000\000\000\000\322U\000\000?\342\017K\241\177\000\000\022\000\000\000\377\377\377\377\200\006\265L\241\177\000\000\253m\237L\241\177\000\000\361|\003\000\000\000\000\000\t\000\000\000\000\000\000\000Sat Mar 15 05:25:32 2025\n\000\264L\241\177\000" nrprocs = <optimized out> woneinit = 1 __func__ = "main_loop" error = <optimized out> #22 0x000055d2517bc151 in main (argc=<optimized out>, argv=<optimized out>) at main.c:3256 cfg_stream = <optimized out> c = <optimized out> r = <optimized out> tmp = 0x7ffebd966e7f "" tmp_len = 32673 port = 5060 proto = 0 aproto = 0 ahost = 0x0 aport = 0 options = 0x55d251aa74b8 ":f:cm:M:dVIhEeb:B:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 3209119183 rfd = <optimized out> debug_save = <optimized out> debug_flag = <optimized out> dont_fork_cnt = 0 n_lst = <optimized out> p = <optimized out> st = {st_dev = 24, st_ino = 46237, st_nlink = 2, st_mode = 16888, st_uid = 899, st_gid = 899, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1742012732, tv_nsec = 217328715}, st_mtim = {tv_sec = 1742012732, tv_nsec = 217328715}, st_ctim = {tv_sec = 1742012732, tv_nsec = 217328715}, __glibc_reserved = {0, 0, 0}} l1 = <optimized out> tbuf = '\000' <repeats 32 times>, "\030\202\237\275\376\177\000\000@ \002\000\000@\003\000\000\001", '\000' <repeats 23 times>, "@\003\000\000@ \003\000\000@\003\000\000@\003\000\000@\003\000\000@\003\000\000@ \003\000\000@\003\000\000@\003\000\000@\003\000\000@\003\000\000@ \003\000\000@\003\000\000@\003\000\000@\003\000\000@\003\000\000@ \003\000\000@\003\000\000@\003\000\000@\003\000\000@\003\000\000@\003", '\000' <repeats 11 times>, "\001", '\000' <repeats 118 times>... option_index = 12 long_options = {{name = 0x55d251a9230b "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x55d251a937a4 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x55d251a9826a "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x55d251a92310 "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x55d251a92316 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x55d251a9231f "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x55d251a92329 "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x55d251a92333 "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x55d251a9233e "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x55d251a92347 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x55d251a938c1 "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x55d251a92352 "cfg-print", has_arg = 0, flag = 0x0, val = 1033}, {name = 0x55d251a9235c "atexit", has_arg = 1, flag = 0x0, val = 1034}, {name = 0x55d251a92363 "all-errors", has_arg = 0, flag = 0x0, val = 1035}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __func__ = "main" [Inferior 1 (process 1782236) detached] ---end 1782236 ------------------------------------------------------- ``` Does anybody have an idea what this could be? I never saw that with previous versions, and now we can see it on several instances, but we are yet unable to reproduce. Not sure if this might be related, but we also see many messages like `Mar 17 13:39:03 ttel /usr/sbin/kamailio[2882127]: CRITICAL: <core> [core/tcp_main.c:5558]: tcp_timer_check_connections(): message processing timeout on connection id: 402 (state: 0) - closing` Best regards Mathias

2 6

Kamailio v5.8.6 Released
by Daniel-Constantin Mierla 20 Mar '25

20 Mar '25

Hello, Kamailio SIP Server v5.8.6 stable release is out. This is a maintenance release of the latest stable branch, 5.8, that includes fixes since the release of v5.8.5. There is no change to database schema or configuration language structure that you have to do on previous installations of v5.8.x. Deployments running previous v5.8.x versions are strongly recommended to be upgraded to v5.8.6. For more details about version 5.8.6 (including links and guidelines to download the tarball or from GIT repository), visit: * https://www.kamailio.org/w/2025/03/kamailio-v5-8-6-released/ RPM, Debian/Ubuntu packages will be available soon as well. Note that 5.8.x is currently the previous stable release series, the latest is 6.0.x and v6.0.1 was released on March 10, 2025. Many thanks to all contributing and using Kamailio! Cheers, Daniel -- Daniel-Constantin Mierla (@ asipto.com) twitter.com/miconda -- linkedin.com/in/miconda Kamailio Consultancy, Training and Development Services -- asipto.com Kamailio Advanced Training, March 24-28, 2025 -- asipto.com Kamailio World Conference, May 12-13, 2025, Berlin -- kamailioworld.com

1 0

Releasing v5.8.6
by Daniel-Constantin Mierla 20 Mar '25

20 Mar '25

Hello, I am considering to release Kamailio v5.8.6 (out of branch 5.8) on Thursday, March 20, 2025. If anyone is aware of issues not yet on the bug tracker, report them there asap in order to have a better chance to be fixed. The most of the commits are backported, should any be found missing, reply to sr-dev mailing list. Cheers, Daniel -- Daniel-Constantin Mierla (@ asipto.com) twitter.com/miconda -- linkedin.com/in/miconda Kamailio Consultancy, Training and Development Services -- asipto.com Kamailio Advanced Training, March 24-28, 2025 -- asipto.com Kamailio World Conference, May 12-13, 2025, Berlin -- kamailioworld.com

1 1

Issue with async_cdp_callback() of MAR diameter message
by Minello, Loredana 18 Mar '25

18 Mar '25

Hello, I have an issue during the registration phase with the Diameter MAR procedure. The issue is as follows: 1. I perform a registration. 2. The MAR is correctly sent using the function: ims_www_challenge("REG_MAR_REPLY", "$var(auth_realm)"); 3. Upon receiving the MAA, after executing async_cdp_callback(), the execution does not return to the Kamailio reply route indicated (REG_MAR_REPLY). I tried replacing t_continue with t_continue_skip_timer in the async_cdp_callback() function of the MAR (see below), and everything works correctly. //tmb.t_continue(data->tindex, data->tlabel, data->act); tmb.t_continue_skip_timer(data->tindex, data->tlabel, data->act); free_saved_transaction_data(data); return; Note: When I send the SAR, I do not have this issue, but I noticed that in the corresponding async_cdp_callback(), the t_continue_skip_timer function is used. Do you think the modification I made to async_cdp_callback() for the MAR is correct? Why does t_continue not work? Thanks.

1 0

dispatcher not distributing well?
by David Villasmil 17 Mar '25

17 Mar '25

Hello guys, i have a kamilio with a dispatcher list in db like +------------------------------------------------+----------------------------------------------+----------------------------+ | id | setid | destination | flags | priority | attrs | description | register_datetime | +------+-------+---------------------+-------+----------+------------------------------+---------------+---------------------+ | 3239 | 430 | sip:gateway-08:5080 | 0 | 10 | duid=gateway-08;weight=50 | gateway-08 | 2023-03-21 00:19:44 | | 3246 | 430 | sip:gateway-11:5080 | 0 | 10 | duid=gateway-11;weight=50 | gateway-11 | 2023-05-15 23:00:28 | | 3247 | 430 | sip:gateway-12:5080 | 0 | 10 | duid=gateway-12;weight=50 | gateway-12 | 2023-05-15 23:06:57 | | 3249 | 430 | sip:gateway-14:5080 | 0 | 10 | duid=gateway-14;weight=50 | gateway-14 | 2023-05-15 23:24:44 | | 3276 | 430 | sip:gateway-15:5080 | 0 | 10 | duid=gateway-15;weight=50 | gateway-15 | 2024-05-01 19:43:19 | | 3277 | 430 | sip:gateway-16:5080 | 0 | 10 | duid=gateway-16;weight=50 | gateway-16 | 2024-05-01 19:51:16 | | 3278 | 430 | sip:gateway-1:5080 | 0 | 10 | duid=gateway-17;weight=50 | gateway-17 | 2024-05-01 19:58:13 | | 3279 | 430 | sip:gateway-17:5080 | 0 | 10 | duid=gateway-18;weight=50 | gateway-18 | 2024-05-01 20:06:44 | | 3280 | 430 | sip:gateway-3:5080 | 0 | 10 | duid=gateway-19;weight=50 | gateway-19 | 2024-05-01 20:14:50 | | 3281 | 430 | sip:gateway-2:5080 | 0 | 10 | duid=gateway-20;weight=50 | gateway-20 | 2024-05-01 20:23:25 | +------+-------+---------------------+-------+----------+------------------------------+---------------+---------------------+ and i'm dispatching with if(!ds_select_dst("$var(dispatcher_id)", "10")) { xlog("[DISPATCH]: ds_select_dst FAILED!'\n"); send_reply("404", "No destination"); exit; } still kamailio isn't distributing at all but for some reason is favoring high-description gateways: kamctl rpc dlg.list 5 gateway-20:5080;transport=udp" 2 gateway-19:5080;transport=udp" it's forwarding calls mostly to gateway-20, gateway-19, gateway-18, gateway-17... in descending order... Why is kamailio doing that? i'd expected it to distribute to all based on the call load on each gateway. Thanks all! Regards, David Villasmil email: david.villasmil.work(a)gmail.com

3 4

Recommendations for paid work
by Anthony Ford 14 Mar '25

14 Mar '25

Hi Can anyone recommend someone that can help us upgrade our 2 Kamailio servers please. After several failed attempts to do this ourselves, it is time to call in the professionals. 5.6.0-dev to the latest The current set up is 2 servers working in tandem, redis and mysql plus nebula. We do have a test server also attached, that can be used for testing. Thanks Tony

6 5

add_contact_alias modifies only one Contact address
by James Browne 13 Mar '25

13 Mar '25

- https://kamailio.org/docs/modules/5.8.x/modules/nathelper.html#nathelper.f.… This add_contact_alias() function changes only the first Contact address in a REGISTER request, even though it's normal (RFC3261, 10.2) for a REGISTER to have multiple Contact addresses. This seems to me to be a kamailio bug. For example, a user might remove one binding while adding a second, in a REGISTER request. If kamailio relays this while adding an alias to only the expires=0 binding, then the new binding may be stored incorrectly in the registrar. James

2 4

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

sr-users March 2025