sr-users March 2019

sr-users@lists.kamailio.org

78 participants
100 discussions

Kamailio crashing and infinite-looping when trying to free memory

by Enrico Bandiera

Hi guys, recently we had some issues with kamailio in production, a proxy at the edge of the infrastructure was crashing seemingly randomly, it happened a couple of times in a timespan of 4-5 days until a point where we found it in an endless loop, unable to process SIP packets anymore and printing in loop messages like these: 50(60) CRITICAL: <core> [core/mem/q_malloc.c:512]: qm_free(): BUG: freeing already freed pointer (0x7ff40a359450), called from core: core/usr_avp.c: destroy_avp_list_unsafe(626), first free core: core/usr_avp.c: destroy_avp_list_unsafe(626) - ignoring 50(60) CRITICAL: <core> [core/mem/q_malloc.c:512]: qm_free(): BUG: freeing already freed pointer (0x7ff409c4fa58), called from core: core/usr_avp.c: destroy_avp_list_unsafe(626), first free core: core/usr_avp.c: destroy_avp_list_unsafe(626) - ignoring We managed to replicate the crash and the loop in a dev environment with a lot of difficulties. It seems not related to the peak load, but to the growing number of processed calls The kamailio version is: version: kamailio 5.2.2 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 5.3.1 Every time this happened it was the timer pid that was crashing or looping, we managed to get 2 core dumps of the crash and 1 of the loop (got it after killing the timer with SIGQUIT). The timer process crashes after attempting to free memory that it's not correctly managed. In particular we saw that it happens inside destroy_avp_list_unsafe() called by the tm timer, when uri_avps_from points to an invalid memory address. We don't use dialog, uac, http_async_client, but we do use topos. In the crash dumps it seems that there's something wrong inside free_cell_helper(), when trying to clean the dead_cell something goes wrong while freeing uri_avps_from, here's the crash log: 61(76) CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 66 0(15) ALERT: <core> [main.c:740]: handle_sigs(): child process 65 exited by a signal 11 0(15) ALERT: <core> [main.c:743]: handle_sigs(): core was generated 0(15) INFO: <core> [main.c:766]: handle_sigs(): terminating due to SIGCHLD <childs getting the SIGTERM signal> 0(15) CRITICAL: <core> [core/mem/q_malloc.c:512]: qm_free(): BUG: freeing already freed pointer (0x7f4d60a2e168), called from htable: ht_api.c: ht_cell_free(217), first free core: core/usr_avp.c: destroy_avp_list_unsafe(626) - ignoring 0(15) INFO: <core> [core/sctp_core.c:53]: sctp_core_destroy(): SCTP API not initialized Before the crash we have no warnings on errors about shared memory allocation backtrace: (gdb) bt full #0 0x00000000006371d7 in destroy_avp_list_unsafe (list=0x7f34f7a411b0) at core/usr_avp.c:625 avp = 0x2 foo = 0x2 __func__ = "destroy_avp_list_unsafe" #1 0x00007f351c6909ab in free_cell_helper (dead_cell=0x7f34f7a41018, silent=0, fname=0x7f351c796a02 "timer.c", fline=689) at h_table.c:260 b = 0x0 i = 1 rpl = 0x0 tt = 0x0 foo = 0x11c68daf2 cbs = 0x0 cbs_tmp = 0x7f34f80c3790 __func__ = "free_cell_helper" #2 0x00007f351c7291e6 in wait_handler (ti=1984410416, wait_tl=0x7f34f7a410a0, data=0x7f34f7a41018) at timer.c:689 p_cell = 0x7f34f7a41018 ret = 1 unlinked = 0 rcount = 1 __func__ = "wait_handler" #3 0x00000000004d0f56 in timer_list_expire (t=1984410416, h=0x7f34f6be7850, slow_l=0x7f34f6bea368, slow_mark=2702) at core/timer.c:874 tl = 0x7f34f7a410a0 ret = 0 #4 0x00000000004d13bd in timer_handler () at core/timer.c:939 saved_ticks = 1984410416 run_slow_timer = 0 i = 654 __func__ = "timer_handler" #5 0x00000000004d1845 in timer_main () at core/timer.c:978 No locals. #6 0x0000000000425cc8 in main_loop () at main.c:1727 i = 8 pid = 0 si = 0x0 si_desc = "udp receiver child=7 sock=127.0.0.1:5060 \000\000\000\000q\000\000\000@6y\000\000\000\000\000\000\b\206|\210X\322:\032\301\255\000\000\000\000\000\000\000\000 \000\000\000\000\000\000\000\002\000\000\000\000\063\275{\000\000\000\000\000>", '\000' <repeats 15 times>, "\320\220\377\226\374\177\000\000\351\022a\000\000\000\000" nrprocs = 8 woneinit = 1 __func__ = "main_loop" #7 0x000000000042ca76 in main (argc=10, argv=0x7ffc96ff9398) at main.c:2696 cfg_stream = 0x25ca010 c = -1 r = 0 tmp = 0x7ffc96ffa4c9 "" tmp_len = -1761635728 port = 32764 proto = -1761635632 options = 0x76c598 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 1492539644 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 2 n_lst = 0x7f351f6f6718 p = 0xffffffff <error: Cannot access memory at address 0xffffffff> st = {st_dev = 2049, st_ino = 1282695, st_nlink = 2, st_mode = 16895, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 4096, st_blksize = 4096, st_blocks = 8, st_atim = {tv_sec = 1552931478, tv_nsec = 972761643}, st_mtim = {tv_sec = 1552931478, tv_nsec = 972761643}, st_ctim = {tv_sec = 1552931478, tv_nsec = 972761643}, __glibc_reserved = {0, 0, 0}} __func__ = "main" In the loop dump we have a different situation, there's a loop in the uri_avps_from structure as you can see from the next pointers: (gdb) p *dead_cell->uri_avps_from $3 = {id = 66, flags = 275, next = 0x7ff40a359450, d = {p = 0x7ff40a9ad540, l = 140686126667072, data = "@\325\232\n\364\177\000"}} (gdb) p *dead_cell->uri_avps_from.next $4 = {id = 66, flags = 275, next = 0x7ff409c4fa58, d = {p = 0x7ff40a359480, l = 140686120031360, data = "\200\224\065\n\364\177\000"}} (gdb) p *dead_cell->uri_avps_from.next.next $5 = {id = 66, flags = 275, next = 0x7ff40a359450, d = {p = 0x7ff409c4fa88, l = 140686112651912, data = "\210\372\304\t\364\177\000"}} backtrace: (gdb) bt full #0 0x00007ff43152e67e in _IO_default_xsputn () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00007ff43150150b in vfprintf () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #2 0x00007ff431502ef1 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #3 0x00007ff43150032d in vfprintf () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #4 0x00007ff4315087f7 in fprintf () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x0000000000686e43 in qm_free (qmp=0x7ff409336000, p=0x7ff409c4fa58, file=0x7e0a75 "core: core/usr_avp.c", func=0x7e2d70 <__func__.8439> "destroy_avp_list_unsafe", line=626, mname=0x7e0a70 "core") at core/mem/q_malloc.c:512 __llevel = -3 qm = 0x7ff409336000 f = 0x7ff409c4fa20 size = 72 next = 0x0 prev = 0x0 __func__ = "qm_free" #6 0x0000000000637209 in destroy_avp_list_unsafe (list=0x7ff409e1b6c8) at core/usr_avp.c:626 avp = 0x7ff40a359450 foo = 0x7ff409c4fa58 __func__ = "destroy_avp_list_unsafe" #7 0x00007ff42ee5d9ab in free_cell_helper (dead_cell=0x7ff409e1b530, silent=0, fname=0x7ff42ef63a02 "timer.c", fline=689) at h_table.c:260 b = 0x0 i = 1 rpl = 0x0 tt = 0x0 foo = 0x12ee5aaf2 cbs = 0x0 cbs_tmp = 0x7ff409c5d420 __func__ = "free_cell_helper" #8 0x00007ff42eef61e6 in wait_handler (ti=89094309, wait_tl=0x7ff409e1b5b8, data=0x7ff409e1b530) at timer.c:689 p_cell = 0x7ff409e1b530 ret = 1 unlinked = 0 rcount = 1 __func__ = "wait_handler" #9 0x00000000004d0f56 in timer_list_expire (t=89094309, h=0x7ff4093b4850, slow_l=0x7ff4093b8058, slow_mark=5981) at core/timer.c:874 tl = 0x7ff409e1b5b8 ret = 0 #10 0x00000000004d13bd in timer_handler () at core/timer.c:939 saved_ticks = 89094309 run_slow_timer = 0 i = 861 __func__ = "timer_handler" #11 0x00000000004d1845 in timer_main () at core/timer.c:978 No locals. #12 0x0000000000425cc8 in main_loop () at main.c:1727 i = 8 pid = 0 si = 0x0 si_desc = "udp receiver child=7 sock=127.0.0.1:5060 \000\000\000\000q\000\000\000@6y\000\000\000\000\000\000\032 t\220\260\242\024\032\301\255\000\000\000\000\000\000\000\000 \000\000\000\000\000\000\000\002\000\000\000\000\063\275{\000\000\000\000\000>", '\000' <repeats 15 times>, "\220\306\336\071\377\177\000\000\351\022a\000\000\000\000" nrprocs = 8 woneinit = 1 __func__ = "main_loop" #13 0x000000000042ca76 in main (argc=10, argv=0x7fff39dec958) at main.c:2696 cfg_stream = 0x2292010 c = -1 r = 0 tmp = 0x7fff39dee4c9 "" tmp_len = 970901552 port = 32767 proto = 970901648 options = 0x76c598 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2710999419 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 2 n_lst = 0x7ff431ec3718 p = 0xffffffff <error: Cannot access memory at address 0xffffffff> st = {st_dev = 2049, st_ino = 1282695, st_nlink = 2, st_mode = 16895, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 4096, st_blksize = 4096, st_blocks = 8, st_atim = {tv_sec = 1552931523, tv_nsec = 918435869}, st_mtim = {tv_sec = 1552994274, tv_nsec = 814438687}, st_ctim = {tv_sec = 1552994274, tv_nsec = 814438687}, __glibc_reserved = {0, 0, 0}} __func__ = "main" We are not sure on how to keep debugging this issue which is causing some serious troubles in our environment, any help is appreciated Here you can find the dumps (beware, 12M download but 1.5G once uncompressed): https://drive.google.com/file/d/1U0LZedyona8jJZHq6HaoqXpfnU83p6Sp/view?usp=… Due thanks to Giacomo Vacca who helped us to dig into this issue. Thanks

5 years, 8 months

Kamailio 5.1.7 crash

by Jon Bonilla (Manwe)

Hi all A customer of mine has some crashes every week. We installed the dbg and finally generated a core. Bt full attached. Kamailio installed from kamailio deb 5.1 repos kamailio -V version: kamailio 5.1.7 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 6.3.0 Any hints? thanks, Jon -- https://pekepbx.com https://www.issabel.com/multitenant

5 years, 8 months

Kamailio as Peer Trunk in Asterisk

by Isravel Raja

Hi, I wanted to achieve the following scenario with Kamailio and Asterisk/FreeSwitch Kamailio as a trunk for Receiving incoming calls and forward to Asterisk and Make outgoing calls. My current working setup like this VoIP Provider -> Kamailio -> Asterisk Clusters Where Kamailio(NAT for public) and Asterisk in the same local network, So I had to do only one thing just create the SIP trunk in Asterisk for authenticating all the Kamailio Incoming calls and making outgoing calls like this [kamailio] host=172.18.1.1 type=friend disallow=all allow=ulaw context=trunkinbound This is IP based authentication, This is working fine since both servers are in the same local network or having the public IP's What I want is, I want to register the Kamailio Trunk in an asterisk with Username Password authentication, My current setup for making that, VoIP Provider -> Kamailio(Public) -> Asterisk(Randomly Changing Public IP, Broadband Connection) I want to register Kamailio SIP in asterisk trunk so that I can forward the incoming calls to asterisk via SIP extension [kamailio] host=172.18.1.1 user=user secret=password type=friend disallow=all allow=ulaw context=trunkinbound Let me know if anyone doesn't understand my concern This e-mail message and any documents attached to it are confidential and may contain information that is protected from disclosure by various federal and state laws, including the HIPAA privacy rule (45 C.F.R., Part 164). This information is intended to be used solely by the entity or individual to whom this message is addressed. If you are not the intended recipient, be advised that any use, dissemination, forwarding, printing, or copying of this message without the sender's written permission is strictly prohibited and may be unlawful. Accordingly, if you have received this message in error, please notify the sender immediately by return e-mail or call (insert phone number), and then delete this message.

5 years, 8 months

RFC: rating - ranking system for community

by Daniel-Constantin Mierla

Hello, starting to continue the discussions here on mailing lists about some of the approached topics during the last IRC meeting -- there will be couple of them. The one now is about finding and deploying a ranking/rating system that could eventually help community members to make decisions easier in regard to what components to use in their voip systems. The need was exemplified by user verticelo with the dificulty to decide what KEMI scripting language to use. While maintaining all the app_xyz are expected to be easy, at least I know that for those I created, the arguments were also from the perspective of the community. Like what others are using, so one can expect good assistance, hints and share of knowledge via community forums. This can be also extended to related tools, like what people use for shared IP high availability of kamailio, preferred database servers, ... This email is to ask if those that didn't participate to IRC devel meeting find such system useful and, if there is a positive feedback, is anyone aware of some OSS that we can deploy on kamailio.org for such purpose? It should be something that allows posting a topic (title and short description) along with a list of answers/options (each can be again like a title and short description) and provide a way to rate the options (like, thumbs up, starts, ...), eventually allowing also comments for each option. I guess it sounds a bit like stackoverflow ... Being users related, the email is sent only to sr-users, let's keep the discussion on this mailing list. Cheers, Daniel -- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - May 6-8, 2019 -- www.kamailioworld.com Kamailio Advanced Training - Mar 25-27, 2019, in Washington, DC, USA -- www.asipto.com

5 years, 8 months

Re: [SR-Users] RFC: roadmap to next major release - v5.3.0

by Ilie Soltanici

Daniel, Sounds good, would be nice to have some improvements on usrloc module, like requested in this topic: https://lists.kamailio.org/pipermail/sr-users/2019-January/104329.html Regards,

5 years, 8 months

Which RTP addon for RFC4040 Clearmode passthrough support ?

by Olivier

Hello, I'm looking for a solution to integrate legacy devices to a SIP network. More precisely, I need to forward to and receive Clearmode RTP traffic (see [1]). 1. Do you know any Kamailio-compliant RTP engine (rtpproxy, rtp engine, ..) that support this on a Debian host ? 2. Suggestion ? [1] https://tools.ietf.org/html/rfc4040 Best regards

5 years, 8 months

rtpengine force-relay

by Vitalii Aleksandrov

Hi there, Well it's mostly rtpengine question but didn't know where should I send it and probably the answer will be more or less useful for kamailio users. By default if an offer has ICE candidates rtpengine leaves them untouched and just inserts itself as an additional low priority candidate and also rewrites c= line for the callee without ICE support. Using ICE=force-relay rtpengine inserts itself as a "relay" candidate what is great but doesn't touch c= line and callee without ICE have no chance to send RTP to rtpengine. Is it possible somehow to make a third behavior like force-relay + rtpengine IP in c= param? It would allow any callee to send media to rtpengine regardless of ICE support.

5 years, 8 months

cdp avps issues in kamailio 5.2

by Pavel S

Hello Mojtava, > Did you use FHOSS or IMS_DIAMETER_SERVER module? No, I am using our own hss server. >Anyway what mandatory parameters do you mean exactly? According to RFC6733 section 5.3.1 - https://tools.ietf.org/html/rfc6733#section-5.3.1 The Capabilities-Exchange-Request (CER), indicated by the Command Code set to 257 and the Command Flags' 'R' bit set, is sent to exchange local capabilities. Upon detection of a transport failure, this message MUST NOT be sent to an alternate peer. When Diameter is run over SCTP [RFC4960] or DTLS/SCTP [RFC6083], which allow for connections to span multiple interfaces and multiple IP addresses, the Capabilities-Exchange-Request message MUST contain one Host-IP-Address AVP for each potential IP address that MAY be locally used when transmitting Diameter messages. Message Format <CER> ::= < Diameter Header: 257, REQ > { Origin-Host } { Origin-Realm } 1* { Host-IP-Address } { Vendor-Id } { Product-Name } [ Origin-State-Id ] * [ Supported-Vendor-Id ] * [ Auth-Application-Id ] * [ Inband-Security-Id ] * [ Acct-Application-Id ] * [ Vendor-Specific-Application-Id ] [ Firmware-Revision ] * [ AVP ] Kamailio 5.1 with loaded cdp, cdr_avp, ims_icsf modules and default configuration described here https://github.com/kamailio/kamailio/tree/master/misc/examples/ims/icscf sends Host-IP-Address but Kamailio 5.2 with the same configuration doesn't. Any idea how to fix it? Regards, Pavel Siderov >Hello, >The CER/CEA messages are used just for connections between two peers >diameter interface. Did you use FHOSS or IMS_DIAMETER_SERVER module? In >both cases, you should not have any issue. Anyway what mandatory parameters >do you mean exactly? >WIth Best Regards. Mojtava ><https://www.kamailio.org/docs/modules/5.1.x/modules/ims_diameter_server.html> >On Tue, Mar 12, 2019 at 8:17 PM Pafel <pafels at gmail.com <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>> wrote: > >>* Hello, *>>>>* I am testing ims setup with kamailio 5.2 using the default configuration *>>* provided here: *>>* https://github.com/kamailio/kamailio/tree/master/misc/examples/ims/ <https://github.com/kamailio/kamailio/tree/master/misc/examples/ims/> . *>>* Тhere is an issue I am facing with cdp avp in request sent to hss. There *>>* are some missing mandatory parameters in capabilities exchange request like *>*> 257 host-ip-address for example. Any idea how to add it via icscf.xml or *>*> kamailio.cfg ? The same parameter is passed to hss with kamailio 5.1 using *>*> default ims configuration. I read cdp and cdp_avp documentation but *>*> couldn't understand if it is possible to add it. *>>>>* Regards, *>>* Pavel Siderov *>>* _______________________________________________ *>>* Kamailio (SER) - Users Mailing List *>>* sr-users at lists.kamailio.org <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users> *>*> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users> *>>

5 years, 8 months

Siremis on Kamailio 5.2

by Joel Serrano

Hi, Just curious if siremis works would work on Kamailio 5.2? The webside might not be updated: Installation Siremis installation is straightforward, being done mostly through a web wizzard. In just a few steps and less than 10 minutes you have your Siremis web site set up and ready to grant you easy administration of your Kamailio SIP server. You need to install first a compatible version of Kamailio SIP Server <https://www.kamailio.org/>. Siremis v5.1.x (devel) *Last release: v5.1.0 (devel version)* *Compatible with: Kamailio 5.1.x, 5.0.x* Thanks! Joel.

5 years, 8 months

Drop 200OK re-transmissions of suspended transaction

by Marco Capetta

Hi All, I started playing with async kamailio feature, in particular using "async_route" function provided by ASYNC module (https://www.kamailio.org/docs/modules/5.1.x/modules/async.html#async.f.asyn…). My scenario is the following: - A calls B - B answer the call - inside onreply_route[reply] I catch the 200OK message and I suspend the transaction for X seconds. - after X seconds the transaction is resumed and the call continues as expected Everything looks good, but I encountered an issue trying to drop reply re-transmission messages. My desire is to drop all the messages received from B (basically 200OK re-transmissions) meanwhile the transaction is suspended. I tested "t_is_retr_async_reply" function provided by TM module (https://www.kamailio.org/docs/modules/5.1.x/modules/tm.html#tm.f.t_is_retr_…), but I'm not able to get it worked as expected: - t_is_retr_async_reply() function can be used only inside named 'onreply_route' (no default 'onreply_route' or standard 'reply_route') - drop() function, on the contrary, doesn't allow to drop 200OK messages inside named 'onreply_route' (but only in default 'onreply_route' or standard 'reply_route'). Do you have any suggestion on how I can achieve my task? Thank you very much for your support. Marco

5 years, 8 months

← Newer
1
2
3
4
5
6
7
8
9
10
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

sr-users March 2019