I am cc-ing sr-dev, since tcp code is from ser and Andrei may have more insights...
On 1/28/10 2:41 PM, Aymeric Moizard wrote:
On Thu, 28 Jan 2010, Henning Westerholt wrote:
On Thursday 28 January 2010, Aymeric Moizard wrote:
here is the backtrace I have. unfortunatly without debug symbol! I found the same for many of the kamailio process. "sched_yield" is pending for ever. My system is a debian/etch.
#0 0xffffe424 in __kernel_vsyscall () #1 0xb7cef4ac in sched_yield () from /lib/tls/i686/cmov/libc.so.6 #2 0x080a93fd in tcp_send () #3 0xb7975679 in send_pr_buffer () from /usr/lib/kamailio/modules/tm.so #4 0xb79789ac in t_forward_nonack () from /usr/lib/kamailio/modules/tm.so #5 0xb7974784 in t_relay_to () from /usr/lib/kamailio/modules/tm.so #6 0xb7983a11 in load_tm () from /usr/lib/kamailio/modules/tm.so #7 0x081cf810 in mem_pool () #8 0x00000000 in ?? ()
I guess most t_relay operation towards my "mobipouce.com" domain with one IP being down breaks each kamailio process one after the other... I'm not sure every such t_relay operation is always breaking exactly one thread each time.
I went through the lock/unlock of tcp_main.c but it seems every lock has an unlock at least...
Hi Aymeric,
i remember that we observed this "sched_yield" problems on one old 0.9 system after some time (like weeks or month). We did not found the solution in this case, after a restart it was gone again..
You mentioned in an earlier mail that you see this related to UDP traffic, but in the log file and also in your investigations you think its related to TPC?
This is the exact case: 1-> SUBSCRIBE sent to/received by over UDP to kamailio. 2-> kamailio does a SRV record lookup for "mobipouce.com" 3-> kamailio try sip2.mobipouce.com (91.199.234.47) over TCP first 4-> connection failed with logs: Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:core:tcp_blocking_connect: poll error: flags 18 Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:core:tcp_blocking_connect: failed to retrieve SO_ERROR (111) Connection refused Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:core:tcpconn_connect: tcp_blocking_connect failed Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:core:tcp_send: connect failed Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:tm:msg_send: tcp_send failed Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:tm:t_forward_nonack: sending request failed 5-> I guess kamailio is supposed to try other SRV record value: sip2.mobipouce.com (91.199.234.46) but it doesn't
Thus, I'm guessing the issue is related to SRV record with failover OR just tcp failure. Not related to UDP at all.
so TCP connect failed, the tcp worker returned as it prints the message and, to be sure I got it right, the UDP worker (the one that received) got blocked?
It's definitly possible to reproduce the issue now!
I guess anyone can try your version of kamailio and t_relay message to "mobipouce.com" and you'll fall in that case! Sending plenty of those messages will finally lock all kamailio process.
All? tcp and udp?
Cheers, Daniel
Regards, Aymeric MOIZARD / ANTISIP amsip - http://www.antisip.com osip2 - http://www.osip.org eXosip2 - http://savannah.nongnu.org/projects/exosip/
Regards,
Henning
Viele Grüße,
Henning
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
some other answer below:
On Thu, 28 Jan 2010, Daniel-Constantin Mierla wrote:
I am cc-ing sr-dev, since tcp code is from ser and Andrei may have more insights...
On 1/28/10 2:41 PM, Aymeric Moizard wrote:
On Thu, 28 Jan 2010, Henning Westerholt wrote:
On Thursday 28 January 2010, Aymeric Moizard wrote:
here is the backtrace I have. unfortunatly without debug symbol! I found the same for many of the kamailio process. "sched_yield" is pending for ever. My system is a debian/etch.
#0 0xffffe424 in __kernel_vsyscall () #1 0xb7cef4ac in sched_yield () from /lib/tls/i686/cmov/libc.so.6 #2 0x080a93fd in tcp_send () #3 0xb7975679 in send_pr_buffer () from /usr/lib/kamailio/modules/tm.so #4 0xb79789ac in t_forward_nonack () from /usr/lib/kamailio/modules/tm.so #5 0xb7974784 in t_relay_to () from /usr/lib/kamailio/modules/tm.so #6 0xb7983a11 in load_tm () from /usr/lib/kamailio/modules/tm.so #7 0x081cf810 in mem_pool () #8 0x00000000 in ?? ()
I guess most t_relay operation towards my "mobipouce.com" domain with one IP being down breaks each kamailio process one after the other... I'm not sure every such t_relay operation is always breaking exactly one thread each time.
I went through the lock/unlock of tcp_main.c but it seems every lock has an unlock at least...
Hi Aymeric,
i remember that we observed this "sched_yield" problems on one old 0.9 system after some time (like weeks or month). We did not found the solution in this case, after a restart it was gone again..
You mentioned in an earlier mail that you see this related to UDP traffic, but in the log file and also in your investigations you think its related to TPC?
This is the exact case: 1-> SUBSCRIBE sent to/received by over UDP to kamailio. 2-> kamailio does a SRV record lookup for "mobipouce.com" 3-> kamailio try sip2.mobipouce.com (91.199.234.47) over TCP first 4-> connection failed with logs: Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:core:tcp_blocking_connect: poll error: flags 18 Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:core:tcp_blocking_connect: failed to retrieve SO_ERROR (111) Connection refused Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:core:tcpconn_connect: tcp_blocking_connect failed Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:core:tcp_send: connect failed Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:tm:msg_send: tcp_send failed Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:tm:t_forward_nonack: sending request failed 5-> I guess kamailio is supposed to try other SRV record value: sip2.mobipouce.com (91.199.234.46) but it doesn't
Thus, I'm guessing the issue is related to SRV record with failover OR just tcp failure. Not related to UDP at all.
so TCP connect failed, the tcp worker returned as it prints the message and, to be sure I got it right, the UDP worker (the one that received) got blocked?
1-> TCP connect failed 2-> second SRV is used: TCP connect succeed, but lock in tcp_send
That's what I understand.
I have tested a TCP connection to my server: It seems to be still working.
It's definitly possible to reproduce the issue now!
I guess anyone can try your version of kamailio and t_relay message to "mobipouce.com" and you'll fall in that case! Sending plenty of those messages will finally lock all kamailio process.
All? tcp and udp?
Only udp! Aymeric
Cheers, Daniel
Regards, Aymeric MOIZARD / ANTISIP amsip - http://www.antisip.com osip2 - http://www.osip.org eXosip2 - http://savannah.nongnu.org/projects/exosip/
Regards,
Henning
Viele Grüße,
Henning
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
-- Daniel-Constantin Mierla
Hello,
On 1/28/10 3:34 PM, Aymeric Moizard wrote:
some other answer below:
On Thu, 28 Jan 2010, Daniel-Constantin Mierla wrote:
I am cc-ing sr-dev, since tcp code is from ser and Andrei may have more insights...
On 1/28/10 2:41 PM, Aymeric Moizard wrote:
On Thu, 28 Jan 2010, Henning Westerholt wrote:
On Thursday 28 January 2010, Aymeric Moizard wrote:
here is the backtrace I have. unfortunatly without debug symbol!
can you recompile with debug symbols? Do you have it installed from package or sources? It will give more hints about the place in the function...
I will try to reproduce, but now I do not have the proper environment for testing...
Thanks, Daniel
I found the same for many of the kamailio process. "sched_yield" is pending for ever. My system is a debian/etch.
#0 0xffffe424 in __kernel_vsyscall () #1 0xb7cef4ac in sched_yield () from /lib/tls/i686/cmov/libc.so.6 #2 0x080a93fd in tcp_send () #3 0xb7975679 in send_pr_buffer () from /usr/lib/kamailio/modules/tm.so #4 0xb79789ac in t_forward_nonack () from /usr/lib/kamailio/modules/tm.so #5 0xb7974784 in t_relay_to () from /usr/lib/kamailio/modules/tm.so #6 0xb7983a11 in load_tm () from /usr/lib/kamailio/modules/tm.so #7 0x081cf810 in mem_pool () #8 0x00000000 in ?? ()
I guess most t_relay operation towards my "mobipouce.com" domain with one IP being down breaks each kamailio process one after the other... I'm not sure every such t_relay operation is always breaking exactly one thread each time.
I went through the lock/unlock of tcp_main.c but it seems every lock has an unlock at least...
Hi Aymeric,
i remember that we observed this "sched_yield" problems on one old 0.9 system after some time (like weeks or month). We did not found the solution in this case, after a restart it was gone again..
You mentioned in an earlier mail that you see this related to UDP traffic, but in the log file and also in your investigations you think its related to TPC?
This is the exact case: 1-> SUBSCRIBE sent to/received by over UDP to kamailio. 2-> kamailio does a SRV record lookup for "mobipouce.com" 3-> kamailio try sip2.mobipouce.com (91.199.234.47) over TCP first 4-> connection failed with logs: Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:core:tcp_blocking_connect: poll error: flags 18 Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:core:tcp_blocking_connect: failed to retrieve SO_ERROR (111) Connection refused Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:core:tcpconn_connect: tcp_blocking_connect failed Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:core:tcp_send: connect failed Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:tm:msg_send: tcp_send failed Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:tm:t_forward_nonack: sending request failed 5-> I guess kamailio is supposed to try other SRV record value: sip2.mobipouce.com (91.199.234.46) but it doesn't
Thus, I'm guessing the issue is related to SRV record with failover OR just tcp failure. Not related to UDP at all.
so TCP connect failed, the tcp worker returned as it prints the message and, to be sure I got it right, the UDP worker (the one that received) got blocked?
1-> TCP connect failed 2-> second SRV is used: TCP connect succeed, but lock in tcp_send
That's what I understand.
I have tested a TCP connection to my server: It seems to be still working.
It's definitly possible to reproduce the issue now!
I guess anyone can try your version of kamailio and t_relay message to "mobipouce.com" and you'll fall in that case! Sending plenty of those messages will finally lock all kamailio process.
All? tcp and udp?
Only udp! Aymeric
Cheers, Daniel
Regards, Aymeric MOIZARD / ANTISIP amsip - http://www.antisip.com osip2 - http://www.osip.org eXosip2 - http://savannah.nongnu.org/projects/exosip/
Regards,
Henning
Viele Grüße,
Henning
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
-- Daniel-Constantin Mierla
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
On Jan 28, 2010 at 14:56, Daniel-Constantin Mierla miconda@gmail.com wrote:
I am cc-ing sr-dev, since tcp code is from ser and Andrei may have more insights...
Is this kamailio 1.5 or kamailio 3.0 (looks like <3.0 to me)?
On 1/28/10 2:41 PM, Aymeric Moizard wrote:
On Thu, 28 Jan 2010, Henning Westerholt wrote:
On Thursday 28 January 2010, Aymeric Moizard wrote:
here is the backtrace I have. unfortunatly without debug symbol! I found the same for many of the kamailio process. "sched_yield" is pending for ever. My system is a debian/etch.
#0 0xffffe424 in __kernel_vsyscall () #1 0xb7cef4ac in sched_yield () from /lib/tls/i686/cmov/libc.so.6 #2 0x080a93fd in tcp_send () #3 0xb7975679 in send_pr_buffer () from /usr/lib/kamailio/modules/tm.so #4 0xb79789ac in t_forward_nonack () from /usr/lib/kamailio/modules/tm.so #5 0xb7974784 in t_relay_to () from /usr/lib/kamailio/modules/tm.so #6 0xb7983a11 in load_tm () from /usr/lib/kamailio/modules/tm.so #7 0x081cf810 in mem_pool () #8 0x00000000 in ?? ()
The backtrace looks strange (mem_pool() and load_tm() for example). It would help greatly to have it compiled with debug symbols from source (or have around the exact source used for the compiled code).
I guess most t_relay operation towards my "mobipouce.com" domain with one IP being down breaks each kamailio process one after the other... I'm not sure every such t_relay operation is always breaking exactly one thread each time.
I went through the lock/unlock of tcp_main.c but it seems every lock has an unlock at least...
Hi Aymeric,
i remember that we observed this "sched_yield" problems on one old 0.9 system after some time (like weeks or month). We did not found the solution in this case, after a restart it was gone again..
If it's kamailio 1.5 (or any version < 3.0) then the sched_yield() means spinning on a lock. However I'm not sure we can trust the backtrace.
You mentioned in an earlier mail that you see this related to UDP traffic, but in the log file and also in your investigations you think its related to TPC?
This is the exact case: 1-> SUBSCRIBE sent to/received by over UDP to kamailio. 2-> kamailio does a SRV record lookup for "mobipouce.com" 3-> kamailio try sip2.mobipouce.com (91.199.234.47) over TCP first 4-> connection failed with logs: Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:core:tcp_blocking_connect: poll error: flags 18 Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:core:tcp_blocking_connect: failed to retrieve SO_ERROR (111) Connection refused Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:core:tcpconn_connect: tcp_blocking_connect failed Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:core:tcp_send: connect failed Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:tm:msg_send: tcp_send failed Jan 27 12:56:38 ns26829 /usr/sbin/kamailio[9763]: ERROR:tm:t_forward_nonack: sending request failed 5-> I guess kamailio is supposed to try other SRV record value: sip2.mobipouce.com (91.199.234.46) but it doesn't
Thus, I'm guessing the issue is related to SRV record with failover OR just tcp failure. Not related to UDP at all.
so TCP connect failed, the tcp worker returned as it prints the message and, to be sure I got it right, the UDP worker (the one that received) got blocked?
My guess (assuming kamailio 1.5) is that the problem is in tm somewhere and not in the TCP code. There are hardly any TCP locks used in this case (connect failure). Most likely the backtrace is bogus.
It's definitly possible to reproduce the issue now!
I guess anyone can try your version of kamailio and t_relay message to "mobipouce.com" and you'll fall in that case! Sending plenty of those messages will finally lock all kamailio process.
All? tcp and udp?
Andrei
On Thu, 28 Jan 2010, Andrei Pelinescu-Onciul wrote:
On Jan 28, 2010 at 14:56, Daniel-Constantin Mierla miconda@gmail.com wrote:
I am cc-ing sr-dev, since tcp code is from ser and Andrei may have more insights...
Is this kamailio 1.5 or kamailio 3.0 (looks like <3.0 to me)?
This is branches/1.5 With svn version 5949.
Here is the debug backtrace: with kamailio-dbg_1.5.0_i386.deb installed:
(gdb) bt #0 0xffffe424 in __kernel_vsyscall () #1 0xb7d694ac in sched_yield () from /lib/tls/i686/cmov/libc.so.6 #2 0x080a93fd in tcp_send (send_sock=0x8159d60, type=3, buf=0xb3992908 "SUBSCRIBE sip:aymeric2@mobipouce.com SIP/2.0\r\nRecord-Route: sip:91.121.81.212:5061;transport=tls;r2=on;lr=on\r\nRecord-Route: sip:91.121.81.212;r2=on;lr=on\r\nVia: SIP/2.0/TLS 91.121.81.212:5061;branc"..., len=645, to=0xb392f494, id=0) at fastlock.h:182 #3 0xb79ef679 in send_pr_buffer (rb=0xb392f480, buf=0xb3992908, len=645) at ../../forward.h:127 #4 0xb79f29ac in t_forward_nonack (t=0xb392f368, p_msg=0x81d02d8, proxy=0x0) at t_fwd.c:691 #5 0xb79ee784 in t_relay_to (p_msg=0x81d02d8, proxy=0x0, flags=<value optimized out>) at t_funcs.c:264 #6 0xb79fda11 in w_t_relay (p_msg=0x81d02d8, proxy=0x0, flags=0x0) at tm.c:1002 #7 0x080551ef in do_action (a=0x8172100, msg=0x81d02d8) at action.c:874 #8 0x080577df in run_action_list (a=0x8172100, msg=0x81d02d8) at action.c:145 #9 0x0808f11b in eval_expr (e=0x8172168, msg=0x81d02d8, val=0x0) at route.c:1171 #10 0x0808ebb0 in eval_expr (e=0x8172190, msg=0x81d02d8, val=0x0) at route.c:1488 #11 0x0808eb3f in eval_expr (e=0x81721b8, msg=0x81d02d8, val=0x0) at route.c:1493 #12 0x08055005 in do_action (a=0x81722d0, msg=0x81d02d8) at action.c:729 #13 0x080577df in run_action_list (a=0x8171928, msg=0x81d02d8) at action.c:145 #14 0x08055e49 in do_action (a=0x816ba50, msg=0x81d02d8) at action.c:120 #15 0x080577df in run_action_list (a=0x816ba50, msg=0x81d02d8) at action.c:145 #16 0x08056d0f in do_action (a=0x816bab8, msg=0x81d02d8) at action.c:746 #17 0x080577df in run_action_list (a=0x81618c0, msg=0x81d02d8) at action.c:145 #18 0x08057b93 in run_top_route (a=0x81618c0, msg=0x81d02d8) at action.c:120 #19 0x08083a0d in receive_msg ( buf=0x81341c0 "SUBSCRIBE sip:aymeric2@mobipouce.com SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.2.3:6010;rport;branch=z9hG4bK972183375\r\nFrom: "aymeric" sip:antisip@sip.antisip.com;tag=286101806\r\nTo: <sip:aymeric2@mobipouce."..., len=692, rcv_info=0xbfc9ad54) at receive.c:175 #20 0x080b3943 in udp_rcv_loop () at udp_server.c:460 #21 0x0806b294 in main (argc=-1211358212, argv=0xb7f61590) at main.c:774
One thing that didn't came up before is that it seems the message is containing TLS, not TCP. I don't have time to analyse it now deeper, but I may try to change the SRV to see how it differ.
Tks, Aymeric
On 1/28/10 8:40 PM, Aymeric Moizard wrote:
On Thu, 28 Jan 2010, Andrei Pelinescu-Onciul wrote:
On Jan 28, 2010 at 14:56, Daniel-Constantin Mierla miconda@gmail.com wrote:
I am cc-ing sr-dev, since tcp code is from ser and Andrei may have more insights...
Is this kamailio 1.5 or kamailio 3.0 (looks like <3.0 to me)?
This is branches/1.5 With svn version 5949.
I thought it is 3.0.0, as all your other emails were related to this version. On another hand, if you run 1.x is better to use the last one, 1.5.3.
Please include the version when you report a problem, otherwise we can hunt in difference places.
Cheers, Daniel
Here is the debug backtrace: with kamailio-dbg_1.5.0_i386.deb installed:
(gdb) bt #0 0xffffe424 in __kernel_vsyscall () #1 0xb7d694ac in sched_yield () from /lib/tls/i686/cmov/libc.so.6 #2 0x080a93fd in tcp_send (send_sock=0x8159d60, type=3, buf=0xb3992908 "SUBSCRIBE sip:aymeric2@mobipouce.com SIP/2.0\r\nRecord-Route: sip:91.121.81.212:5061;transport=tls;r2=on;lr=on\r\nRecord-Route: sip:91.121.81.212;r2=on;lr=on\r\nVia: SIP/2.0/TLS 91.121.81.212:5061;branc"..., len=645, to=0xb392f494, id=0) at fastlock.h:182 #3 0xb79ef679 in send_pr_buffer (rb=0xb392f480, buf=0xb3992908, len=645) at ../../forward.h:127 #4 0xb79f29ac in t_forward_nonack (t=0xb392f368, p_msg=0x81d02d8, proxy=0x0) at t_fwd.c:691 #5 0xb79ee784 in t_relay_to (p_msg=0x81d02d8, proxy=0x0, flags=<value optimized out>) at t_funcs.c:264 #6 0xb79fda11 in w_t_relay (p_msg=0x81d02d8, proxy=0x0, flags=0x0) at tm.c:1002 #7 0x080551ef in do_action (a=0x8172100, msg=0x81d02d8) at action.c:874 #8 0x080577df in run_action_list (a=0x8172100, msg=0x81d02d8) at action.c:145 #9 0x0808f11b in eval_expr (e=0x8172168, msg=0x81d02d8, val=0x0) at route.c:1171 #10 0x0808ebb0 in eval_expr (e=0x8172190, msg=0x81d02d8, val=0x0) at route.c:1488 #11 0x0808eb3f in eval_expr (e=0x81721b8, msg=0x81d02d8, val=0x0) at route.c:1493 #12 0x08055005 in do_action (a=0x81722d0, msg=0x81d02d8) at action.c:729 #13 0x080577df in run_action_list (a=0x8171928, msg=0x81d02d8) at action.c:145 #14 0x08055e49 in do_action (a=0x816ba50, msg=0x81d02d8) at action.c:120 #15 0x080577df in run_action_list (a=0x816ba50, msg=0x81d02d8) at action.c:145 #16 0x08056d0f in do_action (a=0x816bab8, msg=0x81d02d8) at action.c:746 #17 0x080577df in run_action_list (a=0x81618c0, msg=0x81d02d8) at action.c:145 #18 0x08057b93 in run_top_route (a=0x81618c0, msg=0x81d02d8) at action.c:120 #19 0x08083a0d in receive_msg ( buf=0x81341c0 "SUBSCRIBE sip:aymeric2@mobipouce.com SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.2.3:6010;rport;branch=z9hG4bK972183375\r\nFrom: "aymeric" sip:antisip@sip.antisip.com;tag=286101806\r\nTo: <sip:aymeric2@mobipouce."..., len=692, rcv_info=0xbfc9ad54) at receive.c:175 #20 0x080b3943 in udp_rcv_loop () at udp_server.c:460 #21 0x0806b294 in main (argc=-1211358212, argv=0xb7f61590) at main.c:774
One thing that didn't came up before is that it seems the message is containing TLS, not TCP. I don't have time to analyse it now deeper, but I may try to change the SRV to see how it differ.
Tks, Aymeric
On Thu, 28 Jan 2010, Daniel-Constantin Mierla wrote:
On 1/28/10 8:40 PM, Aymeric Moizard wrote:
On Thu, 28 Jan 2010, Andrei Pelinescu-Onciul wrote:
On Jan 28, 2010 at 14:56, Daniel-Constantin Mierla miconda@gmail.com wrote:
I am cc-ing sr-dev, since tcp code is from ser and Andrei may have more insights...
Is this kamailio 1.5 or kamailio 3.0 (looks like <3.0 to me)?
This is branches/1.5 With svn version 5949.
I thought it is 3.0.0, as all your other emails were related to this version.
Sorry, I though I did mentioned it in my initial mail (sent on kamailio mailing list) however, it waz not the case.
I would have asked on ser-users if it was 3.0 ;)
On another hand, if you run 1.x is better to use the last one, 1.5.3.
I'm pretty sure it's 1.5.3: changelog starts with: ===================== 2009-10-XX Kamailio v1.5.3 released =====================
Regards, Aymeric
Please include the version when you report a problem, otherwise we can hunt in difference places.
Cheers, Daniel
Here is the debug backtrace: with kamailio-dbg_1.5.0_i386.deb installed:
(gdb) bt #0 0xffffe424 in __kernel_vsyscall () #1 0xb7d694ac in sched_yield () from /lib/tls/i686/cmov/libc.so.6 #2 0x080a93fd in tcp_send (send_sock=0x8159d60, type=3, buf=0xb3992908 "SUBSCRIBE sip:aymeric2@mobipouce.com SIP/2.0\r\nRecord-Route: sip:91.121.81.212:5061;transport=tls;r2=on;lr=on\r\nRecord-Route: sip:91.121.81.212;r2=on;lr=on\r\nVia: SIP/2.0/TLS 91.121.81.212:5061;branc"..., len=645, to=0xb392f494, id=0) at fastlock.h:182 #3 0xb79ef679 in send_pr_buffer (rb=0xb392f480, buf=0xb3992908, len=645) at ../../forward.h:127 #4 0xb79f29ac in t_forward_nonack (t=0xb392f368, p_msg=0x81d02d8, proxy=0x0) at t_fwd.c:691 #5 0xb79ee784 in t_relay_to (p_msg=0x81d02d8, proxy=0x0, flags=<value optimized out>) at t_funcs.c:264 #6 0xb79fda11 in w_t_relay (p_msg=0x81d02d8, proxy=0x0, flags=0x0) at tm.c:1002 #7 0x080551ef in do_action (a=0x8172100, msg=0x81d02d8) at action.c:874 #8 0x080577df in run_action_list (a=0x8172100, msg=0x81d02d8) at action.c:145 #9 0x0808f11b in eval_expr (e=0x8172168, msg=0x81d02d8, val=0x0) at route.c:1171 #10 0x0808ebb0 in eval_expr (e=0x8172190, msg=0x81d02d8, val=0x0) at route.c:1488 #11 0x0808eb3f in eval_expr (e=0x81721b8, msg=0x81d02d8, val=0x0) at route.c:1493 #12 0x08055005 in do_action (a=0x81722d0, msg=0x81d02d8) at action.c:729 #13 0x080577df in run_action_list (a=0x8171928, msg=0x81d02d8) at action.c:145 #14 0x08055e49 in do_action (a=0x816ba50, msg=0x81d02d8) at action.c:120 #15 0x080577df in run_action_list (a=0x816ba50, msg=0x81d02d8) at action.c:145 #16 0x08056d0f in do_action (a=0x816bab8, msg=0x81d02d8) at action.c:746 #17 0x080577df in run_action_list (a=0x81618c0, msg=0x81d02d8) at action.c:145 #18 0x08057b93 in run_top_route (a=0x81618c0, msg=0x81d02d8) at action.c:120 #19 0x08083a0d in receive_msg ( buf=0x81341c0 "SUBSCRIBE sip:aymeric2@mobipouce.com SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.2.3:6010;rport;branch=z9hG4bK972183375\r\nFrom: "aymeric" sip:antisip@sip.antisip.com;tag=286101806\r\nTo: <sip:aymeric2@mobipouce."..., len=692, rcv_info=0xbfc9ad54) at receive.c:175 #20 0x080b3943 in udp_rcv_loop () at udp_server.c:460 #21 0x0806b294 in main (argc=-1211358212, argv=0xb7f61590) at main.c:774
One thing that didn't came up before is that it seems the message is containing TLS, not TCP. I don't have time to analyse it now deeper, but I may try to change the SRV to see how it differ.
Tks, Aymeric
-- Daniel-Constantin Mierla
On 1/28/10 9:17 PM, Aymeric Moizard wrote:
On Thu, 28 Jan 2010, Daniel-Constantin Mierla wrote:
On 1/28/10 8:40 PM, Aymeric Moizard wrote:
On Thu, 28 Jan 2010, Andrei Pelinescu-Onciul wrote:
On Jan 28, 2010 at 14:56, Daniel-Constantin Mierla miconda@gmail.com wrote:
I am cc-ing sr-dev, since tcp code is from ser and Andrei may have more insights...
Is this kamailio 1.5 or kamailio 3.0 (looks like <3.0 to me)?
This is branches/1.5 With svn version 5949.
I thought it is 3.0.0, as all your other emails were related to this version.
Sorry, I though I did mentioned it in my initial mail (sent on kamailio mailing list) however, it waz not the case.
I would have asked on ser-users if it was 3.0 ;)
kamailio 3.0.0 is on this mailing list, being latest stable version of kamailio. sr-users is mainly for mainstream sip-router users. serusers is for ser project users.
i do not care if kamailio and sr-users get mixed, though, i prefer kamailio to be discussed here, there are different default setting for it than in other 3.0 branches.
Cheers, Daniel
On another hand, if you run 1.x is better to use the last one, 1.5.3.
I'm pretty sure it's 1.5.3: changelog starts with: ===================== 2009-10-XX Kamailio v1.5.3 released =====================
Regards, Aymeric
Please include the version when you report a problem, otherwise we can hunt in difference places.
Cheers, Daniel
Here is the debug backtrace: with kamailio-dbg_1.5.0_i386.deb installed:
(gdb) bt #0 0xffffe424 in __kernel_vsyscall () #1 0xb7d694ac in sched_yield () from /lib/tls/i686/cmov/libc.so.6 #2 0x080a93fd in tcp_send (send_sock=0x8159d60, type=3, buf=0xb3992908 "SUBSCRIBE sip:aymeric2@mobipouce.com SIP/2.0\r\nRecord-Route: sip:91.121.81.212:5061;transport=tls;r2=on;lr=on\r\nRecord-Route: sip:91.121.81.212;r2=on;lr=on\r\nVia: SIP/2.0/TLS 91.121.81.212:5061;branc"..., len=645, to=0xb392f494, id=0) at fastlock.h:182 #3 0xb79ef679 in send_pr_buffer (rb=0xb392f480, buf=0xb3992908, len=645) at ../../forward.h:127 #4 0xb79f29ac in t_forward_nonack (t=0xb392f368, p_msg=0x81d02d8, proxy=0x0) at t_fwd.c:691 #5 0xb79ee784 in t_relay_to (p_msg=0x81d02d8, proxy=0x0, flags=<value optimized out>) at t_funcs.c:264 #6 0xb79fda11 in w_t_relay (p_msg=0x81d02d8, proxy=0x0, flags=0x0) at tm.c:1002 #7 0x080551ef in do_action (a=0x8172100, msg=0x81d02d8) at action.c:874 #8 0x080577df in run_action_list (a=0x8172100, msg=0x81d02d8) at action.c:145 #9 0x0808f11b in eval_expr (e=0x8172168, msg=0x81d02d8, val=0x0) at route.c:1171 #10 0x0808ebb0 in eval_expr (e=0x8172190, msg=0x81d02d8, val=0x0) at route.c:1488 #11 0x0808eb3f in eval_expr (e=0x81721b8, msg=0x81d02d8, val=0x0) at route.c:1493 #12 0x08055005 in do_action (a=0x81722d0, msg=0x81d02d8) at action.c:729 #13 0x080577df in run_action_list (a=0x8171928, msg=0x81d02d8) at action.c:145 #14 0x08055e49 in do_action (a=0x816ba50, msg=0x81d02d8) at action.c:120 #15 0x080577df in run_action_list (a=0x816ba50, msg=0x81d02d8) at action.c:145 #16 0x08056d0f in do_action (a=0x816bab8, msg=0x81d02d8) at action.c:746 #17 0x080577df in run_action_list (a=0x81618c0, msg=0x81d02d8) at action.c:145 #18 0x08057b93 in run_top_route (a=0x81618c0, msg=0x81d02d8) at action.c:120 #19 0x08083a0d in receive_msg ( buf=0x81341c0 "SUBSCRIBE sip:aymeric2@mobipouce.com SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.2.3:6010;rport;branch=z9hG4bK972183375\r\nFrom: "aymeric" sip:antisip@sip.antisip.com;tag=286101806\r\nTo: <sip:aymeric2@mobipouce."..., len=692, rcv_info=0xbfc9ad54) at receive.c:175 #20 0x080b3943 in udp_rcv_loop () at udp_server.c:460 #21 0x0806b294 in main (argc=-1211358212, argv=0xb7f61590) at main.c:774
One thing that didn't came up before is that it seems the message is containing TLS, not TCP. I don't have time to analyse it now deeper, but I may try to change the SRV to see how it differ.
Tks, Aymeric
-- Daniel-Constantin Mierla
sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
Hello Aymeric,
I was forwarding some registrations to your domain and didn't get the deadlock. I did get tcp_send failed many times and then used second dns record. I have two udp workers.
How fast did you get to lock?
I am using latest 1.5.3 here, what is your output of kamailio -V?
Thanks, Daniel
On 1/28/10 9:17 PM, Aymeric Moizard wrote:
On Thu, 28 Jan 2010, Daniel-Constantin Mierla wrote:
On 1/28/10 8:40 PM, Aymeric Moizard wrote:
On Thu, 28 Jan 2010, Andrei Pelinescu-Onciul wrote:
On Jan 28, 2010 at 14:56, Daniel-Constantin Mierla miconda@gmail.com wrote:
I am cc-ing sr-dev, since tcp code is from ser and Andrei may have more insights...
Is this kamailio 1.5 or kamailio 3.0 (looks like <3.0 to me)?
This is branches/1.5 With svn version 5949.
I thought it is 3.0.0, as all your other emails were related to this version.
Sorry, I though I did mentioned it in my initial mail (sent on kamailio mailing list) however, it waz not the case.
I would have asked on ser-users if it was 3.0 ;)
On another hand, if you run 1.x is better to use the last one, 1.5.3.
I'm pretty sure it's 1.5.3: changelog starts with: ===================== 2009-10-XX Kamailio v1.5.3 released =====================
Regards, Aymeric
Please include the version when you report a problem, otherwise we can hunt in difference places.
Cheers, Daniel
Here is the debug backtrace: with kamailio-dbg_1.5.0_i386.deb installed:
(gdb) bt #0 0xffffe424 in __kernel_vsyscall () #1 0xb7d694ac in sched_yield () from /lib/tls/i686/cmov/libc.so.6 #2 0x080a93fd in tcp_send (send_sock=0x8159d60, type=3, buf=0xb3992908 "SUBSCRIBE sip:aymeric2@mobipouce.com SIP/2.0\r\nRecord-Route: sip:91.121.81.212:5061;transport=tls;r2=on;lr=on\r\nRecord-Route: sip:91.121.81.212;r2=on;lr=on\r\nVia: SIP/2.0/TLS 91.121.81.212:5061;branc"..., len=645, to=0xb392f494, id=0) at fastlock.h:182 #3 0xb79ef679 in send_pr_buffer (rb=0xb392f480, buf=0xb3992908, len=645) at ../../forward.h:127 #4 0xb79f29ac in t_forward_nonack (t=0xb392f368, p_msg=0x81d02d8, proxy=0x0) at t_fwd.c:691 #5 0xb79ee784 in t_relay_to (p_msg=0x81d02d8, proxy=0x0, flags=<value optimized out>) at t_funcs.c:264 #6 0xb79fda11 in w_t_relay (p_msg=0x81d02d8, proxy=0x0, flags=0x0) at tm.c:1002 #7 0x080551ef in do_action (a=0x8172100, msg=0x81d02d8) at action.c:874 #8 0x080577df in run_action_list (a=0x8172100, msg=0x81d02d8) at action.c:145 #9 0x0808f11b in eval_expr (e=0x8172168, msg=0x81d02d8, val=0x0) at route.c:1171 #10 0x0808ebb0 in eval_expr (e=0x8172190, msg=0x81d02d8, val=0x0) at route.c:1488 #11 0x0808eb3f in eval_expr (e=0x81721b8, msg=0x81d02d8, val=0x0) at route.c:1493 #12 0x08055005 in do_action (a=0x81722d0, msg=0x81d02d8) at action.c:729 #13 0x080577df in run_action_list (a=0x8171928, msg=0x81d02d8) at action.c:145 #14 0x08055e49 in do_action (a=0x816ba50, msg=0x81d02d8) at action.c:120 #15 0x080577df in run_action_list (a=0x816ba50, msg=0x81d02d8) at action.c:145 #16 0x08056d0f in do_action (a=0x816bab8, msg=0x81d02d8) at action.c:746 #17 0x080577df in run_action_list (a=0x81618c0, msg=0x81d02d8) at action.c:145 #18 0x08057b93 in run_top_route (a=0x81618c0, msg=0x81d02d8) at action.c:120 #19 0x08083a0d in receive_msg ( buf=0x81341c0 "SUBSCRIBE sip:aymeric2@mobipouce.com SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.2.3:6010;rport;branch=z9hG4bK972183375\r\nFrom: "aymeric" sip:antisip@sip.antisip.com;tag=286101806\r\nTo: <sip:aymeric2@mobipouce."..., len=692, rcv_info=0xbfc9ad54) at receive.c:175 #20 0x080b3943 in udp_rcv_loop () at udp_server.c:460 #21 0x0806b294 in main (argc=-1211358212, argv=0xb7f61590) at main.c:774
One thing that didn't came up before is that it seems the message is containing TLS, not TCP. I don't have time to analyse it now deeper, but I may try to change the SRV to see how it differ.
Tks, Aymeric
-- Daniel-Constantin Mierla
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
-
Andrei Pelinescu-Onciul -
Aymeric Moizard -
Daniel-Constantin Mierla