Hello all,
Lately I faced an issue that Kamailio is not responding to some invite packets coming from SBC. When investigating I found that these packets exceeds MTU size. Could this be the issue? Knowing that in Kamailio configuration I have a condition on is_method("INVITE"). Could be that in case of packet exceeding MTU size it's not considered as invite packet by Kamailio?
I have Kamailio 5.2.5.
Regards, Ali Taher
Hello,
first: 5.2.5 is really old by now and not maintained, it is not easy anymore to get proper answers for questions related to it. You should upgrade to a recent release.
Kamailio has nothing to do with defragmentation of incoming UDP packets, it is the kernel/ip stack doing it. If it is done, then Kamailio received the entire UDP message.
You should run with debug=3 and see if Kamailio prints log messages related to processing the SIP request. If not, then Kamailio didn't receive it, either because it was not forwarded by kernel/ip stack or kamailio is busy doing something else (which should be somehow visbile from the log messages or run 'kamctl trap' to get the gdb backtraces for all kamailio processes and look into the stored file).
Cheers, Daniel
On 18.08.22 10:20, Ali Taher wrote:
Hello all,
Lately I faced an issue that Kamailio is not responding to some invite packets coming from SBC. When investigating I found that these packets exceeds MTU size.
Could this be the issue? Knowing that in Kamailio configuration I have a condition on is_method("INVITE"). Could be that in case of packet exceeding MTU size it’s not considered as invite packet by Kamailio?
I have Kamailio 5.2.5.
Regards,
Ali Taher
Kamailio - Users Mailing List - Non Commercial Discussions
- sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
Hi Ali,
Kamailio reassembles fragmented UDP just fine.
However, additional UDP fragments beyond the first packet don’t always get to Kamailio, since they don’t have a UDP header. UDP fragments can be dropped by stateful firewalls and/or NAT gateways for this reason.
This should not prevent the INVITE from being parsed; typically in real-world scenarios with a 1500 byte MTU, the first fragment captures all SIP headers, and fragmentation slices up the SDP payload. Fragmentation won’t adulterate the Request Line (first line), which contains the “INVITE” method verb. I suppose it is conceivable that a fragmentation boundary could occur in the middle of a SIP header and/or header value, causing the entire message to be discarded.
Your best bet for finding out what’s really going on is to look at Kamailio logs. An event like a parsing failure due to fragmentation should be logged.
— Alex
On Aug 18, 2022, at 4:20 AM, Ali Taher ataher@vanrise.com wrote:
Hello all,
Lately I faced an issue that Kamailio is not responding to some invite packets coming from SBC. When investigating I found that these packets exceeds MTU size. Could this be the issue? Knowing that in Kamailio configuration I have a condition on is_method("INVITE"). Could be that in case of packet exceeding MTU size it’s not considered as invite packet by Kamailio?
I have Kamailio 5.2.5.
Regards, Ali Taher
Kamailio - Users Mailing List - Non Commercial Discussions
- sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
Alex Balashov abalashov@evaristesys.com writes:
Hi Ali,
Kamailio reassembles fragmented UDP just fine.
Do you really mean that, or "operating systems reassemble fragmented UDP packets and hand the full packet to Kamailio"?
However, additional UDP fragments beyond the first packet don’t always get to Kamailio, since they don’t have a UDP header. UDP fragments can be dropped by stateful firewalls and/or NAT gateways for this reason.
Yes, correctly-behaving firewalls that want to block by port need to reassemble.
This should not prevent the INVITE from being parsed; typically in real-world scenarios with a 1500 byte MTU, the first fragment captures all SIP headers, and fragmentation slices up the SDP payload. Fragmentation won’t adulterate the Request Line (first line), which contains the “INVITE” method verb. I suppose it is conceivable that a fragmentation boundary could occur in the middle of a SIP header and/or header value, causing the entire message to be discarded.
I would expect that if there is say a 1500-byte fragment and a 700-byte fragment (to make things up) and a firewall drops the 700-byte one, then reassembly at the OS would fail and Kamailio would never see anything.
Or does Kamailio do some sort of raw socket listening and actually do fragment reassembly in user space? That would be very surprising to me.
On Aug 18, 2022, at 7:15 AM, Greg Troxel gdt@lexort.com wrote:
Alex Balashov abalashov@evaristesys.com writes:
Hi Ali,
Kamailio reassembles fragmented UDP just fine.
Do you really mean that, or "operating systems reassemble fragmented UDP packets and hand the full packet to Kamailio"?
No, it was shorthand for the latter. The intent was to emphasise that there’s nothing about Kamailio which is inherently confounded by fragmentation.
This should not prevent the INVITE from being parsed; typically in real-world scenarios with a 1500 byte MTU, the first fragment captures all SIP headers, and fragmentation slices up the SDP payload. Fragmentation won’t adulterate the Request Line (first line), which contains the “INVITE” method verb. I suppose it is conceivable that a fragmentation boundary could occur in the middle of a SIP header and/or header value, causing the entire message to be discarded.
I would expect that if there is say a 1500-byte fragment and a 700-byte fragment (to make things up) and a firewall drops the 700-byte one, then reassembly at the OS would fail and Kamailio would never see anything.
In principle, that’s right. Practically, this depends on the behaviour of various intermediaries. I have seen both behaviours. In the scenarios I have troubleshot, receiving only the first fragment on the other side of—for example—a NAT gateway is fairly common.
Or does Kamailio do some sort of raw socket listening and actually do fragment reassembly in user space? That would be very surprising to me.
No, it doesn’t. Sloppy phraseology on my part.
— Alex
Alex Balashov abalashov@evaristesys.com writes:
This should not prevent the INVITE from being parsed; typically in real-world scenarios with a 1500 byte MTU, the first fragment captures all SIP headers, and fragmentation slices up the SDP payload. Fragmentation won’t adulterate the Request Line (first line), which contains the “INVITE” method verb. I suppose it is conceivable that a fragmentation boundary could occur in the middle of a SIP header and/or header value, causing the entire message to be discarded.
I would expect that if there is say a 1500-byte fragment and a 700-byte fragment (to make things up) and a firewall drops the 700-byte one, then reassembly at the OS would fail and Kamailio would never see anything.
In principle, that’s right. Practically, this depends on the behaviour of various intermediaries. I have seen both behaviours. In the scenarios I have troubleshot, receiving only the first fragment on the other side of—for example—a NAT gateway is fairly common.
It is not surprising that a broken firewall or NAT causes only the first fragment to show up at an OS. It might even be more likely than not that a given firewall is broken :-(
It would surprise me if an OS delivered a UDP packet that contained the bytes in the first fragment only, and if so that's an OS bug. So I would expect 99.9% the symptom to be that the packets don't arrive.
(All of this is another reason to do TCP signalling, which is able to negotiate MTU and not end up with IP fragmentation, but I get it that people have to interoperate with what the peer will do.)
On Aug 18, 2022, at 7:33 AM, Greg Troxel gdt@lexort.com wrote:
Alex Balashov abalashov@evaristesys.com writes:
In principle, that’s right. Practically, this depends on the behaviour of various intermediaries. I have seen both behaviours. In the scenarios I have troubleshot, receiving only the first fragment on the other side of—for example—a NAT gateway is fairly common.
It is not surprising that a broken firewall or NAT causes only the first fragment to show up at an OS. It might even be more likely than not that a given firewall is broken :-(
It would surprise me if an OS delivered a UDP packet that contained the bytes in the first fragment only, and if so that's an OS bug. So I would expect 99.9% the symptom to be that the packets don't arrive.
Yeah, that’s right. A lot of the real-world UDP fragmentation issues revolve around operating through NATs or otherwise stateful firewalls, however.
For instance, as you may know, they are ubiquitous in the SDN architectures of the major public cloud providers.
(All of this is another reason to do TCP signalling, which is able to negotiate MTU and not end up with IP fragmentation, but I get it that people have to interoperate with what the peer will do.)
There are some traditional arguments against TCP signalling in high-capacity service provider cores, mostly having to do with overhead, resource consumption, attack vectors, end-to-end latency vs TCP congestion control, etc.
These have got less salient with increases in computing power, backbone bandwidth and overall public Internet reliability over the last two decades. Nevertheless, some of them are still persuasive in various scenarios.
On the other hand, signalling from the access layer/edge to customer endpoints across the public Internet perhaps should be exclusively TCP or TLS at this point. The near-universal NATification and CGNining of such endpoints combined with the overall trend toward increasing SIP and SDP payloads is a particularly insulting cocktail for UDP.
— Alex
-
Alex Balashov -
Ali Taher -
Daniel-Constantin Mierla -
Greg Troxel