Bastian,

According to mine stats all seems ok...

# kamctl stats shmem {   "jsonrpc":  "2.0",   "result": [ "shmem:fragments = 330", "shmem:free_size = 1889418024", "shmem:max_used_size = 258962624", "shmem:real_used_size = 258065624", "shmem:total_size = 2147483648", "shmem:used_size = 248809096"   ],   "id": 212042 }

pkg.stats   also not showing any signs of exhausting.

For me what is working is server can send packets, but not receive em back even. And it's happening only on 1 interface.

Le 30/08/2023 à 21:44, Bastian Triller a écrit :

I'm asking, because we kind of observed similar behaviour after overload. We've did some load testing and at some point SHM was exhausted and that didn't recover itself. One process was using nearly 100% CPU. And I guess that at least sl replies still works at that point.

On Wed, Aug 30, 2023 at 9:28 PM Ihor Olkhovskyi igorolhovskiy@gmail.com wrote:

Bastian,

I'm fine with retransmissions and so, but for me it's really
interesting, that server is not restoring it's state after a while
and only restart helps. And this is happening only with UDP on one
interface, others are working fine over UDP

Le 30/08/2023 à 17:19, Bastian Triller a écrit :

...

Maybe it's not beneficial to have a receive buffer that high. If
your presence server is stateful, it will send retransmissions if
packets get dropped. But it will also send retransmissions if
your proxy does not reply fast enough. So your proxy may be
damped by retransmissions. Did you check SHM/active transactions
on your proxy?

Regards,
Bastian

On Wed, Aug 30, 2023 at 3:56 PM Ihor Olkhovskyi
<igorolhovskiy@gmail.com> wrote:

    They are increasing, actually

    # ss -l -u -m src X.X.X.X/Y
    State       Recv-Q Send-Q                         Local
    Address:Port          Peer Address:Port
    UNCONN      25167616 0 X.X.X.X:sip                    *:*
    skmem:(r25167616,rb25165824,t0,tb65536,f2304,w0,o0,bl0,d514894)

    Le mer. 30 août 2023 à 15:04, Bastian Triller
    <bastian.triller@gmail.com> a écrit :

        Are drops increasing on that socket while it is happening?
        ss -l src <local_interface> -u sport 5060 -m

        On Tue, Aug 29, 2023 at 3:26 PM Ihor Olkhovskyi
        <igorolhovskiy@gmail.com> wrote:

            Just to add some info

            netstat -nlp
            Active Internet connections (only servers)
            Proto Recv-Q Send-Q Local Address     Foreign Address
                    State PID/Program name
            ...
            udp   25167616      0 <local_interface>:5060
            0.0.0.0:* 211759/kamailio
            ...

            So I see a huge Receive Queue on UDP for Kamailio
            which is not clearing.

            Le mar. 29 août 2023 à 14:29, Ihor Olkhovskyi
            <igorolhovskiy@gmail.com> a écrit :

                Hello,

                I've faced a bit strange issue, but a bit of
                preface. I have Kamailio as a proxy (TLS/WS <->
                UDP) and second Kamailio as a presence server. At
                some point presence server accepts around 5K
                PUBLISH within 1 minute and sending around the
                same amount of NOTIFY to proxy Kamailio.

                Proxy is "transforming" protocol to TLS, but at
                sime point I'm starting to get these type of errors

                tm [../../core/forward.h:292]: msg_send_buffer():
                tcp_send failed
                tm [t_fwd.c:1588]: t_send_branch(): sending
                request on branch 0 failed
                <script>: [RELAY] Relay to
                <sip:X.X.X.X:51571;transport=tls> failed!
                tm [../../core/forward.h:292]: msg_send_buffer():
                tcp_send failed
                tm [t_fwd.c:1588]: t_send_branch(): sending
                request on branch 0 failed

                Some of those messages are 100% valid as client
                can go away or so. Some are not, cause I'm sure
                client is alive and connected.

                But the problem comes later. At some moment proxy
                Kamailio just stops accept UDP traffic on this
                interface (where it also accepts all NOTIFY's),
                at the start of the "stopping accepting" Kamailio
                sends OPTIONS via DISPATCHER but not able to
                receive 200 OK.

                Over TLS on the same interface all is ok. On
                other (loopback) interface UDP is being processed
                fine, so I don't suspert some limit on open files
                here.

                Only restart of Kamailio proxy process helps in
                this case.

                I've tuned net.core.rmem_max and
                net.core.rmem_default to 25 Mb, so in theory
                buffer should not be the case.

                Is there some internal "interface buffer" in
                Kamailio that is not freed upon failure send or
                maybe I've missed somethig?

                Kamailio 5.6.4

                fork=yes
                children=12
                tcp_children=12

                enable_tls=yes

                tcp_accept_no_cl=yes
                tcp_max_connections=63536
                tls_max_connections=63536
                tcp_accept_aliases=no
                tcp_async=yes
                tcp_connect_timeout=10
                tcp_conn_wq_max=63536
                tcp_crlf_ping=yes
                tcp_delayed_ack=yes
                tcp_fd_cache=yes
                tcp_keepalive=yes
                tcp_keepcnt=3
                tcp_keepidle=30
                tcp_keepintvl=10
                tcp_linger2=30
                tcp_rd_buf_size=80000
                tcp_send_timeout=10
                tcp_wq_blk_size=2100
                tcp_wq_max=10485760
                open_files_limit=63536

                Sysctl

                # To increase the amount of memory available for
                socket input/output queues
                net.ipv4.tcp_rmem = 4096 25165824 25165824
                net.core.rmem_max = 25165824
                net.core.rmem_default = 25165824
                net.ipv4.tcp_wmem = 4096 65536 25165824
                net.core.wmem_max = 25165824
                net.core.wmem_default = 65536
                net.core.optmem_max = 25165824

                # To limit the maximum number of requests queued
                to a listen socket
                net.core.somaxconn = 128

                # Tells TCP to instead make decisions that would
                prefer lower latency.
                net.ipv4.tcp_low_latency=1

                # Optional (it will increase performance)
                net.core.netdev_max_backlog = 1000
                net.ipv4.tcp_max_syn_backlog = 128

                # Flush the routing table to make changes happen
                instantly.
                net.ipv4.route.flush=1
                -- 
                Best regards,
                Ihor (Igor)



            -- 
            Best regards,
            Ihor (Igor)
            __________________________________________________________
            Kamailio - Users Mailing List - Non Commercial
            Discussions
            To unsubscribe send an email to
            sr-users-leave@lists.kamailio.org
            Important: keep the mailing list in the recipients,
            do not reply only to the sender!
            Edit mailing list options or unsubscribe:

        __________________________________________________________
        Kamailio - Users Mailing List - Non Commercial Discussions
        To unsubscribe send an email to
        sr-users-leave@lists.kamailio.org
        Important: keep the mailing list in the recipients, do
        not reply only to the sender!
        Edit mailing list options or unsubscribe:



    -- 
    Best regards,
    Ihor (Igor)
    __________________________________________________________
    Kamailio - Users Mailing List - Non Commercial Discussions
    To unsubscribe send an email to sr-users-leave@lists.kamailio.org
    Important: keep the mailing list in the recipients, do not
    reply only to the sender!
    Edit mailing list options or unsubscribe:


__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
To unsubscribe send an email tosr-users-leave@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe: