Hi,
we have a load balancer which is handling a lot of SIP traffic all day. There's always 20-40 Mbit SIP traffic going through. From time to time we see in our logs messages like these:
Sep 16 09:46:28 ecker /usr/sbin/kamailio[25505]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7f2d9d6b3ce0,1321,0,46.237.225.126:5060,16): Operation not permitted(1) Sep 16 09:46:38 ecker /usr/sbin/kamailio[25194]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7efc982b8fc8,420,0,82.113.121.183:35794,16): Operation not permitted(1) Sep 16 09:46:40 ecker /usr/sbin/kamailio[25505]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7f2d9d6b3ce0,1338,0,5.158.137.9:55067,16): Operation not permitted(1) Sep 16 09:46:44 ecker /usr/sbin/kamailio[25183]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7efc982d9f48,450,0,178.165.131.197:37515,16): Operation not permitted(1) Sep 16 09:46:49 ecker /usr/sbin/kamailio[25643]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7f93fb624530,496,0,172.56.7.69:25643,16): Operation not permitted(1) Sep 16 09:46:55 ecker /usr/sbin/kamailio[25335]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7f41632cda98,598,0,80.215.234.139:3396,16): Operation not permitted(1) Sep 16 09:46:56 ecker /usr/sbin/kamailio[25345]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7f41632f4840,459,0,94.197.120.191:8225,16): Operation not permitted(1)
I know that these messages can be produced by iptables blocking the outbound traffic. But our outbound chain looks basically like this:
root@ecker:~# iptables-save | grep OUTPUT :OUTPUT DROP [0:0] -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -j ACCEPT -A OUTPUT -m state --state INVALID -j DROP -A OUTPUT -o lo -m state --state NEW -j ACCEPT -A OUTPUT -o lo -j ACCEPT -A OUTPUT -m state --state NEW -j ACCEPT
We don't have the nf_ct_sip module loaded, syslog doesn't say anything, and even clearing all iptables rules doesn't eliminate those errors.
Has anyone ever seen this? It looks like a load thing, because at weekends there are significantly less errors.
Thanks, Sebastian
Am 16.09.2015 um 10:00 schrieb Sebastian Damm:
Has anyone ever seen this? It looks like a load thing, because at weekends there are significantly less errors.
Could it be that you are trying to send from an ip address that is down at the time?
We see log messages like this on the standby machine that doesn't own the network interfaces. It'd be nice if the Kamailio dispatcher module could be told not do health checks temporarily.
-Sven
On Wed, Sep 16, 2015 at 10:21 AM, Sven Neuhaus neuhaus@tyntec.com wrote:
Could it be that you are trying to send from an ip address that is down at the time?
Interesting thought, but all interfaces on the machine are up at this time. And we are not manipulating the sending socket, so the address where the packet comes into the system is the same where it is sent out from.
Since I forgot to mention it: The system is running Debian Wheezy with it's standard kernel.
Best Regards Sebastian
On Wednesday 16 September 2015 10:00:53 Sebastian Damm wrote:
Has anyone ever seen this? It looks like a load thing, because at weekends there are significantly less errors.
You should look at the OS level, the error is from the kernel.
Are you runing out of sockets/files? It the connection tracker full?
BTW you accept related and new state, this makes no sense, you could just as well have no rules for the OUTPUT chain (which is much better for perfomance).
On Wed, Sep 16, 2015 at 10:44 AM, Daniel Tryba d.tryba@pocos.nl wrote:
You should look at the OS level, the error is from the kernel.
I know, but dmesg, syslog or kernel log don't say anything.
Are you runing out of sockets/files? It the connection tracker full?
The connection tracking table is monitored and never close to full. How could I check the sockets/files?
BTW you accept related and new state, this makes no sense, you could just as well have no rules for the OUTPUT chain (which is much better for perfomance).
I know. My old hand-written firewall was much smaller and almost stateless. But according to our administrators policy all firewalls should be generated by FWbuilder, which generates pretty ugly rules, and also implicitly injects the related rule. (I'm not really happy with that.)
Sebastian
On 9/16/15 4:00 AM, Sebastian Damm wrote:
Hi,
we have a load balancer which is handling a lot of SIP traffic all day. There's always 20-40 Mbit SIP traffic going through. From time to time we see in our logs messages like these:
Sep 16 09:46:28 ecker /usr/sbin/kamailio[25505]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7f2d9d6b3ce0,1321,0,46.237.225.126:5060 http://46.237.225.126:5060,16): Operation not permitted(1) Sep 16 09:46:38 ecker /usr/sbin/kamailio[25194]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7efc982b8fc8,420,0,82.113.121.183:35794 http://82.113.121.183:35794,16): Operation not permitted(1) Sep 16 09:46:40 ecker /usr/sbin/kamailio[25505]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7f2d9d6b3ce0,1338,0,5.158.137.9:55067 http://5.158.137.9:55067,16): Operation not permitted(1) Sep 16 09:46:44 ecker /usr/sbin/kamailio[25183]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7efc982d9f48,450,0,178.165.131.197:37515 http://178.165.131.197:37515,16): Operation not permitted(1) Sep 16 09:46:49 ecker /usr/sbin/kamailio[25643]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7f93fb624530,496,0,172.56.7.69:25643 http://172.56.7.69:25643,16): Operation not permitted(1) Sep 16 09:46:55 ecker /usr/sbin/kamailio[25335]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7f41632cda98,598,0,80.215.234.139:3396 http://80.215.234.139:3396,16): Operation not permitted(1) Sep 16 09:46:56 ecker /usr/sbin/kamailio[25345]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7f41632f4840,459,0,94.197.120.191:8225 http://94.197.120.191:8225,16): Operation not permitted(1)
I know that these messages can be produced by iptables blocking the outbound traffic. But our outbound chain looks basically like this:
root@ecker:~# iptables-save | grep OUTPUT :OUTPUT DROP [0:0] -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -j ACCEPT -A OUTPUT -m state --state INVALID -j DROP -A OUTPUT -o lo -m state --state NEW -j ACCEPT -A OUTPUT -o lo -j ACCEPT -A OUTPUT -m state --state NEW -j ACCEPT
We don't have the nf_ct_sip module loaded, syslog doesn't say anything, and even clearing all iptables rules doesn't eliminate those errors.
Has anyone ever seen this? It looks like a load thing, because at weekends there are significantly less errors.
Try to see if the transmit queue is high (third column) using netstat. You always want that to drop down to zero fast when it goes up. If it starts to grow, you definitely have a problem between the application and operating system.
# netstat -aupn | grep kama udp 0 0 23.239.16.141:50963 23.239.16.141:22555 ESTABLISHED 26101/kamailio udp 0 0 23.239.16.141:50195 198.58.110.117:22555 ESTABLISHED 26081/kamailio udp 0 0 23.239.16.141:42772 198.58.110.117:22555 ESTABLISHED 26122/kamailio udp 0 0 23.239.16.141:49046 23.239.16.141:22555 ESTABLISHED 26112/kamailio udp 0 0 23.239.16.141:49181 198.58.110.117:22555 ESTABLISHED 26118/kamailio udp 0 0 23.239.16.141:36898 23.239.16.141:22555 ESTABLISHED 26122/kamailio udp 0 0 23.239.16.141:59813 23.239.16.141:22555 ESTABLISHED 26120/kamailio udp 0 0 23.239.16.141:36016 23.239.16.141:22555 ESTABLISHED 26110/kamailio udp 0 0 23.239.16.141:47541 198.58.110.117:22555 ESTABLISHED 26103/kamailio udp 0 0 23.239.16.141:60727 23.239.16.141:22555 ESTABLISHED 26096/kamailio udp 0 0 23.239.16.141:52282 198.58.110.117:22555 ESTABLISHED 26120/kamailio udp 0 0 23.239.16.141:57275 198.58.110.117:22555 ESTABLISHED 26114/kamailio udp 0 0 23.239.16.141:51137 23.239.16.141:22555 ESTABLISHED 26114/kamailio udp 0 0 23.239.16.141:53574 23.239.16.141:22555 ESTABLISHED 26095/kamailio udp 0 0 23.239.16.141:5065 0.0.0.0:* 26081/kamailio
Thanks, Sebastian
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hi,
just to resolve this thread, we found the reason for the problem. It occurs, when we try sending out packets to a customer, which look identical to netfilter, at roughly the same time. Those could be for example forked calls to two extensions registered on the same device (a FRITZ Box for example). Then netfilter tries to insert the same packet into its conntrack table twice, causing a collision, leading to a rejection of one of the packets.
We played around with different kernels, without success. The errors kept on coming as long as the nf_conntrack module was loaded, even if there was no iptables rule using it.
The only solution right now seems to be a stateless firewall and unloading the module.
Best Regards, Sebastian
Hello,
this proves again my theory that best options one could get for contrack and selinux is to disable them completely ...
Anyhow, great that you reported back, I am sure it will help others over the time.
Cheers, Daniel
On 11/03/16 14:49, Sebastian Damm wrote:
Hi,
just to resolve this thread, we found the reason for the problem. It occurs, when we try sending out packets to a customer, which look identical to netfilter, at roughly the same time. Those could be for example forked calls to two extensions registered on the same device (a FRITZ Box for example). Then netfilter tries to insert the same packet into its conntrack table twice, causing a collision, leading to a rejection of one of the packets.
We played around with different kernels, without success. The errors kept on coming as long as the nf_conntrack module was loaded, even if there was no iptables rule using it.
The only solution right now seems to be a stateless firewall and unloading the module.
Best Regards, Sebastian
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-
Andres -
Daniel Tryba -
Daniel-Constantin Mierla -
Sebastian Damm -
Sven Neuhaus