Wrong Record-Route and Via header fields when TCP is used

4 Jul 2019

Hello Daniel,
The traces and the simplified config file can be downloaded by
using the link below:

http://updates.xorcom.com/~xorcom/2019-jul-21.tar.gz

Regarding to an IP route from the local IP appearing in
Record-Route to the target IP. I don't have a special route for
it. But the net.ipv4.ip_forward is enabled in the system.
Anyway, I'm sending you IP addresses and the current routing table:

# ip a
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:ad:af:e9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.9.103/20 <http://192.168.9.103/20> brd
192.168.15.255 scope global dynamic ens32
       valid_lft 116999sec preferred_lft 116999sec
    inet6 fe80::20c:29ff:fead:afe9/64 scope link
       valid_lft forever preferred_lft forever
3: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP group default qlen 1000
    link/ether fe:d8:26:e7:21:dc brd ff:ff:ff:ff:ff:ff
    inet 10.28.80.1/24 <http://10.28.80.1/24> scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::501d:dbff:fe72:876e/64 scope link
       valid_lft forever preferred_lft forever
5: vethHT3VX5@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether fe:d8:26:e7:21:dc brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::fcd8:26ff:fee7:21dc/64 scope link
       valid_lft forever preferred_lft forever

# ip r
default via 192.168.0.1 dev ens32 proto dhcp src 192.168.9.103
metric 100
10.28.80.0/24 <http://10.28.80.0/24> dev lxdbr0 proto kernel scope
link src 10.28.80.1
192.168.0.0/20 <http://192.168.0.0/20> dev ens32 proto kernel
scope link src 192.168.9.103
192.168.0.1 dev ens32 proto dhcp scope link src 192.168.9.103
metric 100 

Thank you,
Leonid

On Sun, Jul 21, 2019 at 9:35 AM Daniel-Constantin Mierla
<miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

    Can you load debugger module and enable cfgtrace to see the
    actions executed in config file for such INVITE - reproduce
    and send the log messages. Might be an effect of something
    done there.

    Did I asked if you have IP route from the local IP appearing
    in Record-Route to the target IP address? Kamailio has the
    rule of trying to reuse first the local ip where the request
    was received also for sending out.

    On Thu, Jul 18, 2019 at 2:09 PM Leonid Fainshtein
    <leonid.fainshtein@xorcom.com
    <mailto:leonid.fainshtein@xorcom.com>> wrote:

        Yes, I have.

        Best regards,
        Leonid Fainshtein
        Xorcom Ltd

        On Thu, Jul 18, 2019 at 9:31 AM Daniel-Constantin Mierla
        <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

            Hello,
            traveling, so not much time available ...
            I looked over the logs when expected socket is not
            used, I couldn't spot any message about selecting the
            socket, so there is a route from them "wrong" socket
            to the destination.

            Do you have "mhomed=1" in kamailio.cfg?

            Cheers,
            Daniel

            On Wed, Jul 17, 2019 at 5:27 PM Leonid Fainshtein
            <leonid.fainshtein@xorcom.com
            <mailto:leonid.fainshtein@xorcom.com>> wrote:

                Dear Daniel,
                Did you have a chance to check the traces?

                Best regards,
                Leonid

                On Wed, Jul 10, 2019 at 9:15 AM Leonid Fainshtein
                <leonid.fainshtein@xorcom.com
                <mailto:leonid.fainshtein@xorcom.com>> wrote:

                    Hello Daniel,
                    The requested traces can be downloaded by
                    using the link below:

                    http://updates.xorcom.com/~xorcom/kam-tcp-problem.tar.gz

                    I don't use the force send socket option and
                    doesn't route out via
                    dispatcher in this particular call flow.
                    I found that the problem happens only when the
                    "listen" parameters are
                    defined in the Kamailio configuration.
                    Thus the server where I made the tests have
                    the following IPs configured:

                    2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP>
                    mtu 1500 qdisc fq_codel
                    state UP group default qlen 1000
                       link/ether 00:0c:29:ad:af:e9 brd
                    ff:ff:ff:ff:ff:ff
                       inet 192.168.9.103/20
                    <http://192.168.9.103/20> brd 192.168.15.255
                    scope global dynamic ens32
                    3: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP>
                    mtu 1500 qdisc noqueue
                    state UP group default qlen 1000
                       link/ether fe:d8:26:e7:21:dc brd
                    ff:ff:ff:ff:ff:ff
                       inet 10.28.80.1/24 <http://10.28.80.1/24>
                    scope global lxdbr0

                    The request is accepted on 10.28.80.1 and
                    forwarded from 192.168.9.103

                    If I define:
                    listen=udp:10.28.80.1:5060
                    <http://10.28.80.1:5060>
                    listen=tcp:10.28.80.1:5060
                    <http://10.28.80.1:5060>
                    listen=udp:192.168.9.103:5060
                    <http://192.168.9.103:5060>
                    listen=tcp:192.168.9.103:5060
                    <http://192.168.9.103:5060>

                    Then the problem occurs. Ref. files
                    syslog-bad.log and bad.cap.
                    If I remove all of the 'listen' parameters
                    then the forwarded INVITE
                    request is built properly. Ref. files
                    syslog-good.log and good.cap

                    Best regards,
                    Leonid Fainshtein
                    Xorcom Ltd

                    Best regards,
                    Leonid Fainshtein
                    Xorcom Ltd

                    On Tue, Jul 9, 2019 at 9:53 AM
                    Daniel-Constantin Mierla
                    <miconda@gmail.com <mailto:miconda@gmail.com>>
                    wrote:
                    >
                    > Hello,
                    >
                    > set debug = 3 in kamailio cfg and reproduce
                    this case. Send here all the
                    > log messages printed by kamailio from the
                    moment it receives the request
                    > till it sends it out.
                    >
                    > Some further questions:
                    >
                    >   - do you use any force send socket option?
                    >   - do you route out via dispatcher? If yes,
                    is the socket attribute set?
                    >
                    > Cheers,
                    > Daniel
                    >
                    > On 08.07.19 21:23, Leonid Fainshtein wrote:
                    > > Hello,
                    > > The source address is correct:
                    192.168.0.31. I see it in tcpdump and
                    > > also in sngrep.
                    > >
                    > > Thank you,
                    > > Leonid
                    > >
                    > > On Mon, Jul 8, 2019 at 9:02 PM
                    Daniel-Constantin Mierla
                    > > <miconda@gmail.com
                    <mailto:miconda@gmail.com>> wrote:
                    > >> Hello,
                    > >>
                    > >> when you look at the network traffic
                    9e.g., with ngrep, sngrep, ...)
                    > >> what is shown as source address for
                    outbound leg?
                    > >>
                    > >> Cheers,
                    > >> Daniel
                    > >>
                    > >> On 08.07.19 19:21, Leonid Fainshtein wrote:
                    > >>> I just found Daniel's response to a
                    similar question (ref.:
                    > >>>
                    https://lists.kamailio.org/pipermail/sr-users/2019-February/104853.html):
                    > >>>
                    > >>> "check the routing rules/table of the
                    operating systems, there should be
                    > >>> some differences between the two servers.
                    > >>> If you mhomed=1 and an unexpected
                    interface is used for routing out the
                    > >>> traffic, it means that the operating
                    system has internal routing rules that
                    > >>> allow going from that interface to the
                    target address."
                    > >>>
                    > >>> Don't see anything suspicious in my
                    server routing table:
                    > >>>
                    > >>> default via 192.168.0.1 dev eno1 proto
                    static
                    > >>> 10.159.65.0/24 <http://10.159.65.0/24>
                    dev lxdbr0 proto kernel scope link src 10.159.65.1
                    > >>> 172.200.4.0/24 <http://172.200.4.0/24>
                    dev eno1 proto kernel scope link src 172.200.4.1
                    > >>> 192.168.0.0/20 <http://192.168.0.0/20>
                    dev eno1 proto kernel scope link src 192.168.0.31
                    > >>>
                    > >>> The request is received on the lxdbr0
                    interface (10.159.65.1) and sent
                    > >>> out from the eno1 interface (192.168.0.31).
                    > >>> I even tried to delete the default route
                    but nothing helped. The
                    > >>> request is sent out with 10.159.65.1 in
                    the via and Record-Route
                    > >>> fields...
                    > >>>
                    > >>> Best regards,
                    > >>> Leonid
                    > >>>
                    > >>> On Thu, Jul 4, 2019 at 6:20 PM Leonid
                    Fainshtein
                    > >>> <leonid.fainshtein@xorcom.com
                    <mailto:leonid.fainshtein@xorcom.com>> wrote:
                    > >>>> Hi,
                    > >>>> Kamailio server has two legs that are
                    connected to different networks.
                    > >>>> I'm using Kamailio v.5.2.3 and the
                    "enable_double_rr" is implicitly set to "1".
                    > >>>> The leg "A" IP address is 10.159.65.1
                    > >>>> The leg "B" IP address is 192.168.0.31
                    > >>>> The call is initiated from 10.159.65.18
                    > >>>>
                    > >>>> According to the "rr" module
                    documentation, function record_route()
                    > >>>> should insert two "Record_Route" header
                    fields when a request is
                    > >>>> accepted on one leg is should go out
                    via the second leg. This works as
                    > >>>> expected in case of UDP protocol:
                    > >>>>
                    > >>>> INVITE sip:2000@192.168.6.106:5460
                    <http://sip:2000@192.168.6.106:5460>;transport=UDP
                    SIP/2.0
                    > >>>> Record-Route:
                    <sip:192.168.0.31;r2=on;lr;did=e2c.a191>
                    > >>>> Record-Route:
                    <sip:10.159.65.1;r2=on;lr;did=e2c.a191>
                    > >>>> Via: SIP/2.0/UDP
                    > >>>>
                    192.168.0.31;branch=z9hG4bKcfa5.d64ecbd87d5315b5993c4ccf16f86537.0
                    > >>>> Via: SIP/2.0/UDP
                    10.159.65.18:5060;rport=5060;branch=z9hG4bK3a9e9a4d
                    > >>>>
                    > >>>> But when the TCP protocol is used then
                    the outbound message looks like this:
                    > >>>>
                    > >>>> INVITE sip:2005@192.168.0.178:35058
                    <http://sip:2005@192.168.0.178:35058>;transport=tcp
                    SIP/2.0
                    > >>>> Record-Route:
                    <sip:10.159.65.1;transport=tcp;lr;did=bb6.7dc1>
                    > >>>> Via: SIP/2.0/TCP
                    > >>>>
                    10.159.65.1;branch=z9hG4bKc85a.14afc3867dd3220826f9b9940f78168f.0;i=3
                    > >>>> Via: SIP/2.0/TCP
                    10.159.65.18:5060;rport=58616;branch=z9hG4bK1469331f
                    > >>>>
                    > >>>> There are two problems there:
                    > >>>> a) only one Record-Route with leg is
                    inserted
                    > >>>> b) the added "Via" header field
                    contains the leg "A" IP address
                    > >>>> instead of expected leg "B" IP address
                    (192.168.0.31). In the LAN
                    > >>>> trace I see that in reality the message
                    was sent from leg "B".
                    > >>>>
                    > >>>> Is it a bug?
                    > >>>>
                    > >>>> Best regards,
                    > >>>> Leonid Fainshtein
                    > >>>
                    _______________________________________________
                    > >>> Kamailio (SER) - Users Mailing List
                    > >>> sr-users@lists.kamailio.org
                    <mailto:sr-users@lists.kamailio.org>
                    > >>>
                    https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
                    > >> --
                    > >> Daniel-Constantin Mierla --
                    www.asipto.com <http://www.asipto.com>
                    > >> www.twitter.com/miconda
                    <http://www.twitter.com/miconda> --
                    www.linkedin.com/in/miconda
                    <http://www.linkedin.com/in/miconda>
                    > >>
                    > --
                    > Daniel-Constantin Mierla -- www.asipto.com
                    <http://www.asipto.com>
                    > www.twitter.com/miconda
                    <http://www.twitter.com/miconda> --
                    www.linkedin.com/in/miconda
                    <http://www.linkedin.com/in/miconda>
                    >

            -- 
            Daniel-Constantin Mierla - http://www.asipto.com
            http://twitter.com/#!/miconda -
            http://www.linkedin.com/in/miconda

    -- 
    Daniel-Constantin Mierla - http://www.asipto.com
    http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

>>