Hi,
We run Kamailio 3.2.3 (FLAVOUR=ser) on an embedded ARM platform in near- default configuration.
In a duration test, we observe that at a certain moment Kamailio seems to start ignoring all (re-)register messages, and eventually expires the existing registrations.
We have not been able to reproduce the issue using debug-level logging (-dddd). Info-level logging (-ddd) does reproduce the error, but does not produce any error messages.
Configuration: ser-basic.cfg, with the following changes: port=5060 alias=testnet
Command line: ser -m 4 -f /etc/ser/ser-basic.cfg -n3 -l udp:eth0
Load: Bursts of 8 (Re-)REGISTER messages that are repeated every 85 seconds. The specified expiration time is 120 seconds. Bursts of 9 (Re-)INVITE messages that are repeated every 45 seconds.
Note: With this load it takes roughly 1 hour for the error to occur. Using a more standard 3600 second expiration time Kamailio still stalls, it just takes longer.
Does anyone have any idea how to tackle this issue?
Kind regards,
Michiel Veldkamp
Hello,
can you add an xlog() at the start of the main route block and log a message for any request received? Keep the debug level low. It should show if the sip request gets to application level, it may happen to be dropped from the network interface or kernel.
You can put another xlog before the save() function to see if registration requests are getting there. I am not familiar with ser flavour of registrar and usrloc modules. You can switch to kamailio flavour modules and see if reproduces. I haven't seen such issues with k flavour.
Cheers, Daniel
On 1/15/13 11:51 AM, Sotas Development wrote:
Hi,
We run Kamailio 3.2.3 (FLAVOUR=ser) on an embedded ARM platform in near- default configuration.
In a duration test, we observe that at a certain moment Kamailio seems to start ignoring all (re-)register messages, and eventually expires the existing registrations.
We have not been able to reproduce the issue using debug-level logging (-dddd). Info-level logging (-ddd) does reproduce the error, but does not produce any error messages.
Configuration: ser-basic.cfg, with the following changes: port=5060 alias=testnet
Command line: ser -m 4 -f /etc/ser/ser-basic.cfg -n3 -l udp:eth0
Load: Bursts of 8 (Re-)REGISTER messages that are repeated every 85 seconds. The specified expiration time is 120 seconds. Bursts of 9 (Re-)INVITE messages that are repeated every 45 seconds.
Note: With this load it takes roughly 1 hour for the error to occur. Using a more standard 3600 second expiration time Kamailio still stalls, it just takes longer.
Does anyone have any idea how to tackle this issue?
Kind regards,
Michiel Veldkamp
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Daniel,
can you add an xlog() at the start of the main route block and log a
message for any
request received? [...] You can put another xlog before the save()
function to see if
registration requests are getting there. [...] You can switch to kamailio
flavour modules
and see if reproduces.
Thanks for the help! We switched to kamailio flavour, added some xlog messages and managed to reproduce. See below for some logging.
To our surprise, the last message handled was an INVITE. We'll add more logging to see whether it is handled successfully or gets stuck somewhere.
Those numbers at the start of the log message, are these child IDs? At first they alternate, but apparently child 1 and 2 stop running early on (each with an INVITE as last message).
Regards,
Michiel Veldkamp
0(304) WARNING: <core> [socket_info.c:1392]: WARNING: fix_hostname: could not rev. resolve 192.168.10.1 0(304) INFO: <core> [tcp_main.c:4832]: init_tcp: using epoll_lt as the io watch method (auto detected) 0(306) INFO: usrloc [hslot.c:53]: locks array size 512 0(306) INFO: <core> [udp_server.c:179]: INFO: udp_init: SO_RCVBUF is initially 108544 0(306) INFO: <core> [udp_server.c:230]: INFO: udp_init: SO_RCVBUF is finally 217088 7(315) INFO: ctl [io_listener.c:225]: io_listen_loop: using epoll_lt io watch method (config) 1(309) ERROR: <script>: request_route start -- method=REGISTER 2(310) ERROR: <script>: request_route start -- method=REGISTER 3(311) ERROR: <script>: request_route start -- method=REGISTER 3(311) ERROR: <script>: route[REGISTRAR] start 2(310) ERROR: <script>: route[REGISTRAR] start 3(311) ERROR: <script>: route[REGISTRAR] saving location... 2(310) ERROR: <script>: route[REGISTRAR] saving location... 1(309) ERROR: <script>: route[REGISTRAR] start 1(309) ERROR: <script>: route[REGISTRAR] saving location... 1(309) ERROR: <script>: route[REGISTRAR] saving location... done 3(311) ERROR: <script>: route[REGISTRAR] saving location... done 2(310) ERROR: <script>: route[REGISTRAR] saving location... done 3(311) ERROR: <script>: request_route start -- method=REGISTER 3(311) ERROR: <script>: route[REGISTRAR] start 3(311) ERROR: <script>: route[REGISTRAR] saving location... 3(311) ERROR: <script>: route[REGISTRAR] saving location... done 2(310) ERROR: <script>: request_route start -- method=REGISTER 1(309) ERROR: <script>: request_route start -- method=REGISTER 1(309) ERROR: <script>: route[REGISTRAR] start 1(309) ERROR: <script>: route[REGISTRAR] saving location... 2(310) ERROR: <script>: route[REGISTRAR] start 2(310) ERROR: <script>: route[REGISTRAR] saving location... 1(309) ERROR: <script>: route[REGISTRAR] saving location... done 2(310) ERROR: <script>: route[REGISTRAR] saving location... done 3(311) ERROR: <script>: request_route start -- method=INVITE 3(311) ERROR: <script>: route[REGISTRAR] start 1(309) ERROR: <script>: request_route start -- method=INVITE 1(309) ERROR: <script>: route[REGISTRAR] start 2(310) ERROR: <script>: request_route start -- method=INVITE 2(310) ERROR: <script>: route[REGISTRAR] start 1(309) ERROR: <script>: request_route start -- method=INVITE 1(309) ERROR: <script>: route[REGISTRAR] start 2(310) ERROR: <script>: request_route start -- method=INVITE 2(310) ERROR: <script>: route[REGISTRAR] start 2(310) ERROR: <script>: request_route start -- method=ACK 1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered: timestamp=1350034404;method=INVITE;from_tag=966002447;to_tag=3a8473f2;call_id= 331189810@192.168.10.2 ;code=200;reason=OK;src_user=Radio1_device;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=Radio1;dst_user=3;dst_domain=192.168.10.2 3(311) NOTICE: acc [acc.c:275]: ACC: transaction answered: timestamp=1350034404;method=INVITE;from_tag=1718943109;to_tag=22a97aa7;call_id= 956233863@192.168.10.2 ;code=200;reason=OK;src_user=User3;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=IC1;dst_user=1;dst_domain=192.168.10.2 3(311) ERROR: <script>: request_route start -- method=ACK 1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered: timestamp=1350034404;method=INVITE;from_tag=1709043140;to_tag=4aa972a8;call_id= 780056350@192.168.10.2 ;code=200;reason=OK;src_user=User3;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=IC2;dst_user=2;dst_domain=192.168.10.2 1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered: timestamp=1350034404;method=INVITE;from_tag=1506064927;to_tag=5a73a3fc;call_id= 1129604628@192.168.10.2 ;code=200;reason=OK;src_user=User3;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=emergency;dst_user=4;dst_domain=192.168.10.2 1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered: timestamp=1350034404;method=INVITE;from_tag=1200029211;to_tag=14c6d7d4;call_id= 1214006622@192.168.10.2 ;code=200;reason=OK;src_user=User3;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=Radio1;dst_user=3;dst_domain=192.168.10.2 1(309) ERROR: <script>: request_route start -- method=ACK 2(310) ERROR: <script>: request_route start -- method=ACK 3(311) ERROR: <script>: request_route start -- method=ACK 2(310) ERROR: <script>: request_route start -- method=REGISTER 2(310) ERROR: <script>: route[REGISTRAR] start 2(310) ERROR: <script>: route[REGISTRAR] saving location... 1(309) ERROR: <script>: request_route start -- method=REGISTER 1(309) ERROR: <script>: route[REGISTRAR] start 1(309) ERROR: <script>: route[REGISTRAR] saving location... 2(310) ERROR: <script>: route[REGISTRAR] saving location... done 1(309) ERROR: <script>: route[REGISTRAR] saving location... done 3(311) ERROR: <script>: request_route start -- method=INVITE 3(311) ERROR: <script>: route[REGISTRAR] start 3(311) ERROR: <script>: request_route start -- method=ACK 1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered: timestamp=1350034419;method=INVITE;from_tag=22866096;to_tag=461b3848;call_id= 119348927@192.168.10.1 ;code=200;reason=OK;src_user=User1;src_domain=testnet;src_ip=192.168.10.1;dst_ouser=IC1;dst_user=1;dst_domain=192.168.10.2 2(310) ERROR: <script>: request_route start -- method=INVITE 2(310) ERROR: <script>: route[REGISTRAR] start 2(310) ERROR: <script>: request_route start -- method=ACK 3(311) NOTICE: acc [acc.c:275]: ACC: transaction answered: timestamp=1350034419;method=INVITE;from_tag=1086463804;to_tag=1b28a959;call_id= 1666188208@192.168.10.1 ;code=200;reason=OK;src_user=User1;src_domain=testnet;src_ip=192.168.10.1;dst_ouser=IC2;dst_user=2;dst_domain=192.168.10.2 1(309) ERROR: <script>: request_route start -- method=INVITE 1(309) ERROR: <script>: route[REGISTRAR] start 1(309) ERROR: <script>: request_route start -- method=ACK 2(310) NOTICE: acc [acc.c:275]: ACC: transaction answered: timestamp=1350034419;method=INVITE;from_tag=425688445;to_tag=7c6f5fd8;call_id= 138053674@192.168.10.1 ;code=200;reason=OK;src_user=User1;src_domain=testnet;src_ip=192.168.10.1;dst_ouser=emergency;dst_user=4;dst_domain=192.168.10.2 3(311) ERROR: <script>: request_route start -- method=INVITE 3(311) ERROR: <script>: route[REGISTRAR] start 3(311) ERROR: <script>: request_route start -- method=ACK 2(310) NOTICE: acc [acc.c:275]: ACC: transaction answered: timestamp=1350034419;method=INVITE;from_tag=2111220874;to_tag=6a7be79d;call_id= 1381530996@192.168.10.1 ;code=200;reason=OK;src_user=User1;src_domain=testnet;src_ip=192.168.10.1;dst_ouser=Radio1;dst_user=3;dst_domain=192.168.10.2 1(309) ERROR: <script>: request_route start -- method=INVITE 1(309) ERROR: <script>: route[REGISTRAR] start 1(309) ERROR: <script>: request_route start -- method=ACK 2(310) NOTICE: acc [acc.c:275]: ACC: transaction answered: timestamp=1350034419;method=INVITE;from_tag=269879616;to_tag=210d7864;call_id= 907816659@192.168.10.1 ;code=200;reason=OK;src_user=User2;src_domain=testnet;src_ip=192.168.10.1;dst_ouser=IC1;dst_user=1;dst_domain=192.168.10.2 3(311) ERROR: <script>: request_route start -- method=INVITE 3(311) ERROR: <script>: route[REGISTRAR] start 1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered: timestamp=1350034419;method=INVITE;from_tag=1891265955;to_tag=29747a68;call_id= 1205241696@192.168.10.1 ;code=200;reason=OK;src_user=User2;src_domain=testnet;src_ip=192.168.10.1;dst_ouser=emergency;dst_user=4;dst_domain=192.168.10.2 3(311) ERROR: <script>: request_route start -- method=ACK 2(310) ERROR: <script>: request_route start -- method=INVITE 1(309) ERROR: <script>: request_route start -- method=INVITE 2(310) ERROR: <script>: request_route start -- method=ACK 2(310) ERROR: <script>: request_route start -- method=INVITE 3(311) ERROR: <script>: request_route start -- method=ACK 3(311) ERROR: <script>: request_route start -- method=INVITE 2(310) ERROR: <script>: request_route start -- method=INVITE <-- Last message from 2(310) 3(311) ERROR: <script>: request_route start -- method=ACK 3(311) ERROR: <script>: request_route start -- method=ACK 3(311) ERROR: <script>: request_route start -- method=ACK 1(309) ERROR: <script>: request_route start -- method=INVITE <-- Last message from 1(309) 3(311) ERROR: <script>: request_route start -- method=INVITE 3(311) ERROR: <script>: request_route start -- method=ACK 3(311) ERROR: <script>: request_route start -- method=INVITE 3(311) ERROR: <script>: request_route start -- method=ACK 3(311) ERROR: <script>: request_route start -- method=INVITE 3(311) ERROR: <script>: request_route start -- method=ACK 3(311) ERROR: <script>: request_route start -- method=INVITE 3(311) ERROR: <script>: request_route start -- method=ACK ... 3(311) ERROR: <script>: request_route start -- method=REGISTER 3(311) ERROR: <script>: route[REGISTRAR] start 3(311) ERROR: <script>: route[REGISTRAR] saving location... 3(311) ERROR: <script>: route[REGISTRAR] saving location... done 3(311) ERROR: <script>: request_route start -- method=INVITE <-- Last message
Hello Michiel ,
You should check with top to see the status of all kamailio processes. If you see processes that are using 100% CPU, you may have a deadlock. Connect with gdb to the process to investigate what's going on.
Also, you are running on arm. How did you compiled kamailio: native or cross. When you run kamailio on an embedded system, you need to check if you have enough memory. If you start running out of memory, the OS may start killing processes randomly (check your OS logs).
You definitely need to look at this issue from a broader prospective as it's not a regular x86 deployment with plenty of CPU and memory. Also tuning the config (by removing everything that you don't need might help with memory utilization).
Regards, Ovidiu Sas
Hi Ovidiu,
Good points. We're not running out of memory, and all processes keep running. Top shows 0% or 1% CPU load for the Kamailio processes.
We cross-compile with the CodeSourcery toolchain. The default build options drag in the file atomic_unknown.h, that produces the following compile warning:
"no native memory barrier implementations, falling back to slow lock based workarround"
Currently we're testing a version compiled with -DNOSMP (no atomic_unknown.h) that will run over the weekend.
Regards, Michiel Veldkamp
On Thu, Jan 17, 2013 at 3:20 PM, Ovidiu Sas osas@voipembedded.com wrote:
Hello Michiel ,
You should check with top to see the status of all kamailio processes. If you see processes that are using 100% CPU, you may have a deadlock. Connect with gdb to the process to investigate what's going on.
Also, you are running on arm. How did you compiled kamailio: native or cross. When you run kamailio on an embedded system, you need to check if you have enough memory. If you start running out of memory, the OS may start killing processes randomly (check your OS logs).
You definitely need to look at this issue from a broader prospective as it's not a regular x86 deployment with plenty of CPU and memory. Also tuning the config (by removing everything that you don't need might help with memory utilization).
Regards, Ovidiu Sas
-- VoIP Embedded, Inc. http://www.voipembedded.com
If you are running the stable version, there's need for heavy Makefile patching in order to properly cross compile (not to include and link to host libs). The trunk has everything fixed and it's cross-compiling properly for most of the modules. Make sure that your binaries are properly cross compiled.
Depending on your ARM CPU, atomic locks may or may not work. I tested openser without atomic locks (using regular locks) and it worked fine.
Regards, Ovidiu Sas
Hi Ovidiu,
Thanks for the warning! We did not yet have much success running the current master branch, though this may well be a resource problem on the target platform.
For the moment, we decided to switch back to openser 1.3.5 and wait for the official 4.0 release.
Regards, Michiel Veldkamp
On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas osas@voipembedded.com wrote:
If you are running the stable version, there's need for heavy Makefile patching in order to properly cross compile (not to include and link to host libs). The trunk has everything fixed and it's cross-compiling properly for most of the modules. Make sure that your binaries are properly cross compiled.
Depending on your ARM CPU, atomic locks may or may not work. I tested openser without atomic locks (using regular locks) and it worked fine.
Regards, Ovidiu Sas
-- VoIP Embedded, Inc. http://www.voipembedded.com
4.0 (current trunk) is in code freeze. I would suggest to test the trunk version (next 4.0). Even openser 1.3 requires patches to be properly cross compiled.
Regards, Ovidiu Sas
Hi,
Here's an update of this topic. We've tried again with the latest stable version 4.0.4. Unfortunately the problem still exists.
In mails above it is mentioned to use kamailio without atomic locks. How do we this (e.g. which makefile options)?
Kind regards,
Bert (on behalf of Michiel Veldkamp)
On Mon, Jan 28, 2013 at 4:44 PM, Ovidiu Sas osas@voipembedded.com wrote:
4.0 (current trunk) is in code freeze. I would suggest to test the trunk version (next 4.0). Even openser 1.3 requires patches to be properly cross compiled.
Regards, Ovidiu Sas
-- VoIP Embedded, Inc. http://www.voipembedded.com
---------- Forwarded message ---------- From: Sotas Development sotasdev@gmail.com Date: Mon, Jan 28, 2013 at 10:08 AM Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations? To: "SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List" sr-users@lists.sip-router.org
Hi Ovidiu,
Thanks for the warning! We did not yet have much success running the current master branch, though this may well be a resource problem on the target platform.
For the moment, we decided to switch back to openser 1.3.5 and wait for the official 4.0 release.
Regards, Michiel Veldkamp
On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas osas@voipembedded.com wrote:
If you are running the stable version, there's need for heavy Makefile patching in order to properly cross compile (not to include and link to host libs). The trunk has everything fixed and it's cross-compiling properly for most of the modules. Make sure that your binaries are properly cross compiled.
Depending on your ARM CPU, atomic locks may or may not work. I tested openser without atomic locks (using regular locks) and it
worked fine.
Regards, Ovidiu Sas
-- VoIP Embedded, Inc. http://www.voipembedded.com
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
In a previous e-mail, you posted a warning that you had while compiling: "no native memory barrier implementations, falling back to slow lock based workarround" which means that you are already running without atomic locks.
Regards, Ovidiu Sas
On Wed, Nov 13, 2013 at 10:40 AM, Sotas Development sotasdev@gmail.com wrote:
Hi,
Here's an update of this topic. We've tried again with the latest stable version 4.0.4. Unfortunately the problem still exists.
In mails above it is mentioned to use kamailio without atomic locks. How do we this (e.g. which makefile options)?
Kind regards,
Bert (on behalf of Michiel Veldkamp)
On Mon, Jan 28, 2013 at 4:44 PM, Ovidiu Sas osas@voipembedded.com wrote:
4.0 (current trunk) is in code freeze. I would suggest to test the trunk version (next 4.0). Even openser 1.3 requires patches to be properly cross compiled.
Regards, Ovidiu Sas
-- VoIP Embedded, Inc. http://www.voipembedded.com
---------- Forwarded message ---------- From: Sotas Development sotasdev@gmail.com Date: Mon, Jan 28, 2013 at 10:08 AM Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations? To: "SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List" sr-users@lists.sip-router.org
Hi Ovidiu,
Thanks for the warning! We did not yet have much success running the current master branch, though this may well be a resource problem on the target platform.
For the moment, we decided to switch back to openser 1.3.5 and wait for the official 4.0 release.
Regards, Michiel Veldkamp
On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas osas@voipembedded.com wrote:
If you are running the stable version, there's need for heavy Makefile patching in order to properly cross compile (not to include and link to host libs). The trunk has everything fixed and it's cross-compiling properly for most of the modules. Make sure that your binaries are properly cross compiled.
Depending on your ARM CPU, atomic locks may or may not work. I tested openser without atomic locks (using regular locks) and it worked fine.
Regards, Ovidiu Sas
-- VoIP Embedded, Inc. http://www.voipembedded.com
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hi,
In the mean time we have gathered more information on this problem:
As given below, kamailio stops grabbing UDP SIP messages (SIP registrations) after running a while on an embedded ARM and PPC platform (which runs linux 2.6.33 kernel). Some times the hangup occures within hours and some times after couple of days running.
NETSTAT OUTPUT: root# netstat -pl | grep kam udp 1047968 0 (null):sip (null):* 8416/kamailio raw 0 0 (null):255 (null):* 255 8416/kamailio unix 2 [ ACC ] STREAM LISTENING 755205 8429/kamailio /tmp/kamailio_ctl
Kamailio is started with the following options = -m 4 -n 3 -f <cfg> -D
Other relevant info: - When Kamailio hangs, I also noticed that the flag "inuse_transactions" has always the value of '1'. Readout with "kamctl monitor". - A simple cat to /proc/<kamailio_pid>/wchan gives us the function: futex_wait_queue_me. - All possible polling methods are used with -W parameter (sigio_rt, poll, select etc) during these tests. Non of these options did solve this problem. I hope the additional info will clarify more. Thanks in advance.
Best regards, Orhan Yilmaz
On Wed, Nov 13, 2013 at 6:12 PM, Ovidiu Sas osas@voipembedded.com wrote:
In a previous e-mail, you posted a warning that you had while compiling: "no native memory barrier implementations, falling back to slow lock based workarround" which means that you are already running without atomic locks.
Regards, Ovidiu Sas
On Wed, Nov 13, 2013 at 10:40 AM, Sotas Development sotasdev@gmail.com wrote:
Hi,
Here's an update of this topic. We've tried again with the latest stable version 4.0.4. Unfortunately the problem still exists.
In mails above it is mentioned to use kamailio without atomic locks. How
do
we this (e.g. which makefile options)?
Kind regards,
Bert (on behalf of Michiel Veldkamp)
On Mon, Jan 28, 2013 at 4:44 PM, Ovidiu Sas osas@voipembedded.com
wrote:
4.0 (current trunk) is in code freeze. I would suggest to test the trunk version (next 4.0). Even openser 1.3 requires patches to be properly cross compiled.
Regards, Ovidiu Sas
-- VoIP Embedded, Inc. http://www.voipembedded.com
---------- Forwarded message ---------- From: Sotas Development sotasdev@gmail.com Date: Mon, Jan 28, 2013 at 10:08 AM Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations? To: "SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List" sr-users@lists.sip-router.org
Hi Ovidiu,
Thanks for the warning! We did not yet have much success running the current master branch, though this may well be a resource problem on the target platform.
For the moment, we decided to switch back to openser 1.3.5 and wait for the official 4.0 release.
Regards, Michiel Veldkamp
On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas osas@voipembedded.com
wrote:
If you are running the stable version, there's need for heavy Makefile patching in order to properly cross compile (not to include and link to host libs). The trunk has everything fixed and it's cross-compiling properly for most of the modules. Make sure that your binaries are properly cross compiled.
Depending on your ARM CPU, atomic locks may or may not work. I tested openser without atomic locks (using regular locks) and it worked fine.
Regards, Ovidiu Sas
-- VoIP Embedded, Inc. http://www.voipembedded.com
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- VoIP Embedded, Inc. http://www.voipembedded.com
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Try to attach gdb to the kamailio processes and run a full backtrace.
Regards, Ovidiu Sas
On Wed, Nov 27, 2013 at 5:16 AM, Sotas Development sotasdev@gmail.com wrote:
Hi,
In the mean time we have gathered more information on this problem:
As given below, kamailio stops grabbing UDP SIP messages (SIP registrations) after running a while on an embedded ARM and PPC platform (which runs linux 2.6.33 kernel). Some times the hangup occures within hours and some times after couple of days running.
NETSTAT OUTPUT: root# netstat -pl | grep kam udp 1047968 0 (null):sip (null):* 8416/kamailio raw 0 0 (null):255 (null):* 255 8416/kamailio unix 2 [ ACC ] STREAM LISTENING 755205 8429/kamailio /tmp/kamailio_ctl
Kamailio is started with the following options = -m 4 -n 3 -f <cfg> -D
Other relevant info:
- When Kamailio hangs, I also noticed that the flag "inuse_transactions" has
always the value of '1'. Readout with "kamctl monitor".
- A simple cat to /proc/<kamailio_pid>/wchan gives us the function:
futex_wait_queue_me.
- All possible polling methods are used with -W parameter (sigio_rt, poll,
select etc) during these tests. Non of these options did solve this problem.
I hope the additional info will clarify more. Thanks in advance.
Best regards, Orhan Yilmaz
On Wed, Nov 13, 2013 at 6:12 PM, Ovidiu Sas osas@voipembedded.com wrote:
In a previous e-mail, you posted a warning that you had while compiling: "no native memory barrier implementations, falling back to slow lock based workarround" which means that you are already running without atomic locks.
Regards, Ovidiu Sas
On Wed, Nov 13, 2013 at 10:40 AM, Sotas Development sotasdev@gmail.com wrote:
Hi,
Here's an update of this topic. We've tried again with the latest stable version 4.0.4. Unfortunately the problem still exists.
In mails above it is mentioned to use kamailio without atomic locks. How do we this (e.g. which makefile options)?
Kind regards,
Bert (on behalf of Michiel Veldkamp)
On Mon, Jan 28, 2013 at 4:44 PM, Ovidiu Sas osas@voipembedded.com wrote:
4.0 (current trunk) is in code freeze. I would suggest to test the trunk version (next 4.0). Even openser 1.3 requires patches to be properly cross compiled.
Regards, Ovidiu Sas
-- VoIP Embedded, Inc. http://www.voipembedded.com
---------- Forwarded message ---------- From: Sotas Development sotasdev@gmail.com Date: Mon, Jan 28, 2013 at 10:08 AM Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations? To: "SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List" sr-users@lists.sip-router.org
Hi Ovidiu,
Thanks for the warning! We did not yet have much success running the current master branch, though this may well be a resource problem on the target platform.
For the moment, we decided to switch back to openser 1.3.5 and wait for the official 4.0 release.
Regards, Michiel Veldkamp
On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas osas@voipembedded.com wrote:
If you are running the stable version, there's need for heavy Makefile patching in order to properly cross compile (not to include and link to host libs). The trunk has everything fixed and it's cross-compiling properly for most of the modules. Make sure that your binaries are properly cross compiled.
Depending on your ARM CPU, atomic locks may or may not work. I tested openser without atomic locks (using regular locks) and it worked fine.
Regards, Ovidiu Sas
-- VoIP Embedded, Inc. http://www.voipembedded.com
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- VoIP Embedded, Inc. http://www.voipembedded.com
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Attaching with gdb and getting the backtrace (as Ovidiu said), will be very useful
The function futex_wait_queue_me() is from kernel, so it might be an operation from the kernel about reading the udp packages. So we cannot tell without the full trace if a kamailio function is blocking the futex or is other standard library function.
I wanted to mention also that poll methods are for tcp only, for udp kamailio uses recvfrom().
Can you also provide the output for 'kamailio -I' (that's upper case i)?
Cheers, Daniel
On 11/27/13 4:44 PM, Ovidiu Sas wrote:
Try to attach gdb to the kamailio processes and run a full backtrace.
Regards, Ovidiu Sas
On Wed, Nov 27, 2013 at 5:16 AM, Sotas Development sotasdev@gmail.com wrote:
Hi,
In the mean time we have gathered more information on this problem:
As given below, kamailio stops grabbing UDP SIP messages (SIP registrations) after running a while on an embedded ARM and PPC platform (which runs linux 2.6.33 kernel). Some times the hangup occures within hours and some times after couple of days running.
NETSTAT OUTPUT: root# netstat -pl | grep kam udp 1047968 0 (null):sip (null):* 8416/kamailio raw 0 0 (null):255 (null):* 255 8416/kamailio unix 2 [ ACC ] STREAM LISTENING 755205 8429/kamailio /tmp/kamailio_ctl
Kamailio is started with the following options = -m 4 -n 3 -f <cfg> -D
Other relevant info:
- When Kamailio hangs, I also noticed that the flag "inuse_transactions" has
always the value of '1'. Readout with "kamctl monitor".
- A simple cat to /proc/<kamailio_pid>/wchan gives us the function:
futex_wait_queue_me.
- All possible polling methods are used with -W parameter (sigio_rt, poll,
select etc) during these tests. Non of these options did solve this problem.
I hope the additional info will clarify more. Thanks in advance.
Best regards, Orhan Yilmaz
On Wed, Nov 13, 2013 at 6:12 PM, Ovidiu Sas osas@voipembedded.com wrote:
In a previous e-mail, you posted a warning that you had while compiling: "no native memory barrier implementations, falling back to slow lock based workarround" which means that you are already running without atomic locks.
Regards, Ovidiu Sas
On Wed, Nov 13, 2013 at 10:40 AM, Sotas Development sotasdev@gmail.com wrote:
Hi,
Here's an update of this topic. We've tried again with the latest stable version 4.0.4. Unfortunately the problem still exists.
In mails above it is mentioned to use kamailio without atomic locks. How do we this (e.g. which makefile options)?
Kind regards,
Bert (on behalf of Michiel Veldkamp)
On Mon, Jan 28, 2013 at 4:44 PM, Ovidiu Sas osas@voipembedded.com wrote:
4.0 (current trunk) is in code freeze. I would suggest to test the trunk version (next 4.0). Even openser 1.3 requires patches to be properly cross compiled.
Regards, Ovidiu Sas
-- VoIP Embedded, Inc. http://www.voipembedded.com
---------- Forwarded message ---------- From: Sotas Development sotasdev@gmail.com Date: Mon, Jan 28, 2013 at 10:08 AM Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations? To: "SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List" sr-users@lists.sip-router.org
Hi Ovidiu,
Thanks for the warning! We did not yet have much success running the current master branch, though this may well be a resource problem on the target platform.
For the moment, we decided to switch back to openser 1.3.5 and wait for the official 4.0 release.
Regards, Michiel Veldkamp
On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas osas@voipembedded.com wrote:
If you are running the stable version, there's need for heavy Makefile patching in order to properly cross compile (not to include and link to host libs). The trunk has everything fixed and it's cross-compiling properly for most of the modules. Make sure that your binaries are properly cross compiled.
Depending on your ARM CPU, atomic locks may or may not work. I tested openser without atomic locks (using regular locks) and it worked fine.
Regards, Ovidiu Sas
-- VoIP Embedded, Inc. http://www.voipembedded.com
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- VoIP Embedded, Inc. http://www.voipembedded.com
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hi All,
We have solved this issue now. The problem was the wrong interpretation of the Kamailio Makefile.defs makefile;
The ARCH script variable was set to a wrong string by our higher level makefile. Ea; for powerPC, Kamilio.defs expects 'ppc' as the value while the higher level makefile sets the ARCH variable to 'powerpc'. So Kamailio was compiled with wrong and missing GCC arguments. Also some C-defines were not set correctly which resulted in linking wrong 'locking' routines, like atomic_unknown.h. Instead atomic_ppc.h should be linked/compiled.
Thanks for your support anyway.
Best regards, Orhan Yilmaz
On Fri, Nov 29, 2013 at 3:02 PM, Daniel-Constantin Mierla <miconda@gmail.com
wrote:
Attaching with gdb and getting the backtrace (as Ovidiu said), will be very useful
The function futex_wait_queue_me() is from kernel, so it might be an operation from the kernel about reading the udp packages. So we cannot tell without the full trace if a kamailio function is blocking the futex or is other standard library function.
I wanted to mention also that poll methods are for tcp only, for udp kamailio uses recvfrom().
Can you also provide the output for 'kamailio -I' (that's upper case i)?
Cheers, Daniel
On 11/27/13 4:44 PM, Ovidiu Sas wrote:
Try to attach gdb to the kamailio processes and run a full backtrace.
Regards, Ovidiu Sas
On Wed, Nov 27, 2013 at 5:16 AM, Sotas Development sotasdev@gmail.com wrote:
Hi,
In the mean time we have gathered more information on this problem:
As given below, kamailio stops grabbing UDP SIP messages (SIP registrations) after running a while on an embedded ARM and PPC platform (which runs linux 2.6.33 kernel). Some times the hangup occures within hours and some times after couple of days running.
NETSTAT OUTPUT: root# netstat -pl | grep kam udp 1047968 0 (null):sip (null):* 8416/kamailio raw 0 0 (null):255 (null):* 255 8416/kamailio unix 2 [ ACC ] STREAM LISTENING 755205 8429/kamailio /tmp/kamailio_ctl
Kamailio is started with the following options = -m 4 -n 3 -f <cfg> -D
Other relevant info:
- When Kamailio hangs, I also noticed that the flag "inuse_transactions"
has always the value of '1'. Readout with "kamctl monitor".
- A simple cat to /proc/<kamailio_pid>/wchan gives us the function:
futex_wait_queue_me.
- All possible polling methods are used with -W parameter (sigio_rt,
poll, select etc) during these tests. Non of these options did solve this problem.
I hope the additional info will clarify more. Thanks in advance.
Best regards, Orhan Yilmaz
On Wed, Nov 13, 2013 at 6:12 PM, Ovidiu Sas osas@voipembedded.com wrote:
In a previous e-mail, you posted a warning that you had while compiling: "no native memory barrier implementations, falling back to slow lock based workarround" which means that you are already running without atomic locks.
Regards, Ovidiu Sas
On Wed, Nov 13, 2013 at 10:40 AM, Sotas Development <sotasdev@gmail.com
wrote:
Hi,
Here's an update of this topic. We've tried again with the latest stable version 4.0.4. Unfortunately the problem still exists.
In mails above it is mentioned to use kamailio without atomic locks. How do we this (e.g. which makefile options)?
Kind regards,
Bert (on behalf of Michiel Veldkamp)
On Mon, Jan 28, 2013 at 4:44 PM, Ovidiu Sas osas@voipembedded.com wrote:
4.0 (current trunk) is in code freeze. I would suggest to test the trunk version (next 4.0). Even openser 1.3 requires patches to be properly cross compiled.
Regards, Ovidiu Sas
-- VoIP Embedded, Inc. http://www.voipembedded.com
---------- Forwarded message ---------- From: Sotas Development sotasdev@gmail.com Date: Mon, Jan 28, 2013 at 10:08 AM Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations? To: "SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List" sr-users@lists.sip-router.org
Hi Ovidiu,
Thanks for the warning! We did not yet have much success running the current master branch, though this may well be a resource problem on the target platform.
For the moment, we decided to switch back to openser 1.3.5 and wait for the official 4.0 release.
Regards, Michiel Veldkamp
On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas osas@voipembedded.com wrote:
> If you are running the stable version, there's need for heavy > Makefile > patching in order to properly cross compile (not to include and link > to host libs). > The trunk has everything fixed and it's cross-compiling properly for > most of the modules. > Make sure that your binaries are properly cross compiled. > > Depending on your ARM CPU, atomic locks may or may not work. > I tested openser without atomic locks (using regular locks) and it > worked fine. > > Regards, > Ovidiu Sas > > > -- > VoIP Embedded, Inc. > http://www.voipembedded.com > _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- VoIP Embedded, Inc. http://www.voipembedded.com
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Hello,
thanks for reporting back, it's good to know that is not an issue in our Makefile.
Cheers, Daniel
On 12/12/13 11:13, Sotas Development wrote:
Hi All,
We have solved this issue now. The problem was the wrong interpretation of the Kamailio Makefile.defs makefile;
The ARCH script variable was set to a wrong string by our higher level makefile. Ea; for powerPC, Kamilio.defs expects 'ppc' as the value while the higher level makefile sets the ARCH variable to 'powerpc'. So Kamailio was compiled with wrong and missing GCC arguments. Also some C-defines were not set correctly which resulted in linking wrong 'locking' routines, like atomic_unknown.h. Instead atomic_ppc.h should be linked/compiled.
Thanks for your support anyway.
Best regards, Orhan Yilmaz
On Fri, Nov 29, 2013 at 3:02 PM, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
Attaching with gdb and getting the backtrace (as Ovidiu said), will be very useful The function futex_wait_queue_me() is from kernel, so it might be an operation from the kernel about reading the udp packages. So we cannot tell without the full trace if a kamailio function is blocking the futex or is other standard library function. I wanted to mention also that poll methods are for tcp only, for udp kamailio uses recvfrom(). Can you also provide the output for 'kamailio -I' (that's upper case i)? Cheers, Daniel On 11/27/13 4:44 PM, Ovidiu Sas wrote: Try to attach gdb to the kamailio processes and run a full backtrace. Regards, Ovidiu Sas On Wed, Nov 27, 2013 at 5:16 AM, Sotas Development <sotasdev@gmail.com <mailto:sotasdev@gmail.com>> wrote: Hi, In the mean time we have gathered more information on this problem: As given below, kamailio stops grabbing UDP SIP messages (SIP registrations) after running a while on an embedded ARM and PPC platform (which runs linux 2.6.33 kernel). Some times the hangup occures within hours and some times after couple of days running. NETSTAT OUTPUT: root# netstat -pl | grep kam udp 1047968 0 (null):sip (null):* 8416/kamailio raw 0 0 (null):255 (null):* 255 8416/kamailio unix 2 [ ACC ] STREAM LISTENING 755205 8429/kamailio /tmp/kamailio_ctl Kamailio is started with the following options = -m 4 -n 3 -f <cfg> -D Other relevant info: - When Kamailio hangs, I also noticed that the flag "inuse_transactions" has always the value of '1'. Readout with "kamctl monitor". - A simple cat to /proc/<kamailio_pid>/wchan gives us the function: futex_wait_queue_me. - All possible polling methods are used with -W parameter (sigio_rt, poll, select etc) during these tests. Non of these options did solve this problem. I hope the additional info will clarify more. Thanks in advance. Best regards, Orhan Yilmaz On Wed, Nov 13, 2013 at 6:12 PM, Ovidiu Sas <osas@voipembedded.com <mailto:osas@voipembedded.com>> wrote: In a previous e-mail, you posted a warning that you had while compiling: "no native memory barrier implementations, falling back to slow lock based workarround" which means that you are already running without atomic locks. Regards, Ovidiu Sas On Wed, Nov 13, 2013 at 10:40 AM, Sotas Development <sotasdev@gmail.com <mailto:sotasdev@gmail.com>> wrote: Hi, Here's an update of this topic. We've tried again with the latest stable version 4.0.4. Unfortunately the problem still exists. In mails above it is mentioned to use kamailio without atomic locks. How do we this (e.g. which makefile options)? Kind regards, Bert (on behalf of Michiel Veldkamp) On Mon, Jan 28, 2013 at 4:44 PM, Ovidiu Sas <osas@voipembedded.com <mailto:osas@voipembedded.com>> wrote: 4.0 (current trunk) is in code freeze. I would suggest to test the trunk version (next 4.0). Even openser 1.3 requires patches to be properly cross compiled. Regards, Ovidiu Sas -- VoIP Embedded, Inc. http://www.voipembedded.com ---------- Forwarded message ---------- From: Sotas Development <sotasdev@gmail.com <mailto:sotasdev@gmail.com>> Date: Mon, Jan 28, 2013 at 10:08 AM Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations? To: "SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List" <sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org>> Hi Ovidiu, Thanks for the warning! We did not yet have much success running the current master branch, though this may well be a resource problem on the target platform. For the moment, we decided to switch back to openser 1.3.5 and wait for the official 4.0 release. Regards, Michiel Veldkamp On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas <osas@voipembedded.com <mailto:osas@voipembedded.com>> wrote: If you are running the stable version, there's need for heavy Makefile patching in order to properly cross compile (not to include and link to host libs). The trunk has everything fixed and it's cross-compiling properly for most of the modules. Make sure that your binaries are properly cross compiled. Depending on your ARM CPU, atomic locks may or may not work. I tested openser without atomic locks (using regular locks) and it worked fine. Regards, Ovidiu Sas -- VoIP Embedded, Inc. http://www.voipembedded.com _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users -- VoIP Embedded, Inc. http://www.voipembedded.com _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users -- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
-
Daniel-Constantin Mierla -
Ovidiu Sas -
Sotas Development