To start off. Please forgive me, I'm not completely sure if this is an issue in Kamailio, or an issue in Linux that I'm overseeing. But I could really use a helping hand.
### Description
I have a setup with a Kamailio load-balancer and a secondary proxy that is handling all the complex work, let's call them balancer and core proxy. The core proxy is only listening on a private network, that's a separate VLAN with an rfc1918 IP address. The balancer is listening on that network too, but it has also a listening socket on a different VLAN with a public internet IP address.
``` Balancer: - Public IP : 150.0.0.51 (interface ens9, vlan 150) ---- - Private IP : 10.0.0.51 (interface ens3, vlan 10) | | Core: | - Private IP : 10.0.0.52 (interface ens3, vlan 10) <------ ````
When I'm sending an INVITE I see the balancer receiving the UDP packet on `150.0.0.51` and relaying it from the same address to `10.0.0.52`. On the core proxy (different server) I can see the packet coming in with tcpdump or ngrep. Kamailio seems to completely ignore this packet. Bumping up the debug level to the maximum still results in no logging at all when I send the SIP packet.
Now I can work around this issue by forcing the send socket to use the private IP on the balancer. But this introduces new problems in my environment, because I have actually many balancers and core proxies in a high availability setup and the return path to the user should always remain the same.
We can also work around this issue by exposing a public IP to the core proxy (and other internal proxies). But I really would like to keep only the load-balancer exposed to the internet. I have this working with Kamailio `5.1.1` in an existing environment, but now we are moving from Ubuntu `16.04` to Ubuntu `18.04` and updating Kamailio to `5.1.6` and it's not working anymore.
### Additional Information
* **Kamailio Version**: ``` Tested with 5.1.6 and 5.1.2 ```
* **Operating System**:
``` Ubuntu 18.04.1 LTS 4.15.0-33-generic #36-Ubuntu SMP Wed Aug 15 16:00:05 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux ```
### Logging
I can see a `REGISTER` packet like this coming in on the core proxy (with many retransmissions), here is the `ngrep` output captured on the core proxy on interface `ens3`:
Not working:
``` # U 2018/11/01 18:57:55.548408 150.0.0.51:5060 -> 10.0.0.52:5060 REGISTER sip:my.domain SIP/2.0. Via: SIP/2.0/UDP 150.0.0.51;branch=xxx. Via: SIP/2.0/UDP 10.x.x.x:34904;received=x.x.x.x;branch=xxx;rport=34904. From: "200" sip:200@my.domain;tag=xxx. To: "200" sip:200@my.domain. Call-ID: xxx. CSeq: 565 REGISTER. Max-Forwards: 69. User-Agent: snomD345/8.9.3.33. [...] ```
But when I force the source socket to be the private address:
Working:
``` # U 2018/11/01 19:03:42.914001 10.0.0.51:5060 -> 10.0.0.52:5060 REGISTER sip:my.domain SIP/2.0. Via: SIP/2.0/UDP 150.0.0.51;branch=xxx. Via: SIP/2.0/UDP 10.x.x.x:34904;received=x.x.x.x;branch=xxx;rport=34904. From: "200" sip:200@my.domain;tag=xxx. To: "200" sip:200@my.domain. Call-ID: xxx. CSeq: 587 REGISTER. Max-Forwards: 69. User-Agent: snomD345/8.9.3.33. [...] ```
Now Kamailio is processing the request normally for some reason.
Any help or suggestions will be appreciated.
Hi @ThomasLobker have you tried setting the `mhomed=1` in your kamailio.cfg?
https://www.kamailio.org/wiki/cookbooks/5.1.x/core#mhomed
I think this is what you are missing.
Hi @joelsdc thanks so much for your suggestion! I have tried that now. If I'm correct, this will have the same behaviour as setting the source socket by changing `$fs` yourself, but then instead it's choosing the right socket automatically for you.
Unfortunately this is not entirely fixing my problem. It does allow me to route traffic inbound, but any traffic in the other direction will have the same problem again. In my case, the packet needs to be routed to a specific host where the floating public IP happens to be at that moment.
It just looks like Kamailio is not accepting any traffic from a **private** address on a listening socket with a **public** address, and vice versa. This was not the case with Kamailio `5.1.1` on Ubuntu `16.04` with the same Kamailio configuration.
See here two screenshots from Wireshark. The capture was made on one interface on the internal proxy `10.x.x.x`. When it's receiving a SIP packet from another host in the same subnet, everything is working normally.
But when it's receiving a SIP packet from a public subnet, then it's just ignoring the packet completely.
Have you compared the routing tables on the existing environment's server to the new one? I would run `ip route show table all` on each just to make sure. Ubuntu started using netplan in 18.04 so you might find something by looking into the `netfilter-persistent` service (i.e., should it be starting).
@ThomasLobker are you set `mhomed=1` in kamailio coonfig. In your case this config is mandatory.
About not receiving messages. I experienced similar behaviour when in kamailio config is present string leading to create same lock. Try install on kamailio reference config and i expect you always can see received messages in kamailio log.
Closed #1703.
Thank you all for the good tips. I found the issue! Luckily not an issue with Kamailio after all. The issue is that Linux enables reverse path filtering by default. I tried to substitute Kamailio with a simple Python server, and I noticed that it wasn't showing me any output either. So I started digging around in the Linux settings and found that we needed to disable reverse path filtering.
The Python script used to substitute Kamailio: ```python import socket sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) sock.bind(('0.0.0.0', 5060))
while True: data, addr = sock.recvfrom(1024) print "Received message:", data ```
More information about reverse path filtering: https://www.tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.kernel.rpf.html
I guess we missed this setting during migration. If it helps anyone in the future, to disable remote path filtering (at your own risk, obviously):
```bash for interface in /proc/sys/net/ipv4/conf/*/rp_filter; do echo 1 > ${interface} done ```
To be clear, the `mhomed` setting is not mandatory and in our case not desired.
-
Joel Serrano -
rwha
-
sergey-safarov
-
Thomas Lobker