21 Nov
2019
21 Nov
'19
5:46 p.m.
Hi
I have a kamailio 5.1.2 as load balancer and registration offloading,
but I have a problem with the max tcp connections that it can handle.
I suspect that is a linux limit, but I don't find the reason or config.
When that limit arrives, I can't connect to kamailio and I receive
"Connection reset by peer", but I can't view any error message in the
logs.
If I check the connections in kamailio, I view that it have "free" connections:
# kamctl kamcmd core.tcp_info
{
readers: 8
max_connections: 4096
max_tls_connections: 2048
opened_connections: 2655
opened_tls_connections: 0
write_queued_bytes: 0
}
I have this configs in kamailio.conf (related to tcp)
disable_tcp=no
tcp_connection_lifetime=3610
tcp_connect_timeout=5
tcp_crlf_ping=yes
tcp_accept_aliases=no
tcp_keepalive=yes
tcp_keepidle=5
tcp_rd_buf_size=65536
tcp_conn_wq_max=131072
mlock_pages=yes
shm_force_alloc=yes
tcp_max_connections=4096
The shm memory to 256 and the pkg memory to 32.
And, following this doc:
https://github.com/kamailio/kamailio/blob/master/doc/tutorials/tcp_tunning.…
I have setted this values:
net.ipv4.ip_local_port_range = 1024 65535
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 182757
Also, I had checked the limits for the main process pid:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size unlimited unlimited bytes
Max resident set unlimited unlimited bytes
Max processes unlimited unlimited processes
Max open files 1048576 1048576 files
Max locked memory 16777216 16777216 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 386297 386297 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
The service is running inside a lxc container, without any resource
limit, connected to the outside word throught macvlan interface.
Where can I find problem source?
Best regards
21 Nov
21 Nov
8:16 p.m.
I thought that the 5.1.2 is the latest release in this branch, sorry.
First I will upgrade to the latest version (5.1.9) and if the problem
persists, I'll resend the email.
Best regards
El jue., 21 nov. 2019 a las 14:46, Jose Fco. Irles Durá
(<josefu(a)gmail.com>) escribió:
Hi
I have a kamailio 5.1.2 as load balancer and registration offloading,
but I have a problem with the max tcp connections that it can handle.
I suspect that is a linux limit, but I don't find the reason or config.
When that limit arrives, I can't connect to kamailio and I receive
"Connection reset by peer", but I can't view any error message in the
logs.
If I check the connections in kamailio, I view that it have "free"
connections:
# kamctl kamcmd core.tcp_info
{
readers: 8
max_connections: 4096
max_tls_connections: 2048
opened_connections: 2655
opened_tls_connections: 0
write_queued_bytes: 0
}
I have this configs in kamailio.conf (related to tcp)
disable_tcp=no
tcp_connection_lifetime=3610
tcp_connect_timeout=5
tcp_crlf_ping=yes
tcp_accept_aliases=no
tcp_keepalive=yes
tcp_keepidle=5
tcp_rd_buf_size=65536
tcp_conn_wq_max=131072
mlock_pages=yes
shm_force_alloc=yes
tcp_max_connections=4096
The shm memory to 256 and the pkg memory to 32.
And, following this doc:
https://github.com/kamailio/kamailio/blob/master/doc/tutorials/tcp_tunning.…
I have setted this values:
net.ipv4.ip_local_port_range = 1024 65535
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 182757
Also, I had checked the limits for the main process pid:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size unlimited unlimited bytes
Max resident set unlimited unlimited bytes
Max processes unlimited unlimited processes
Max open files 1048576 1048576 files
Max locked memory 16777216 16777216 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 386297 386297 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
The service is running inside a lxc container, without any resource
limit, connected to the outside word throught macvlan interface.
Where can I find problem source?
Best regards
--
Jose Fco. Irles Durá
8:34 p.m.
On Thu, Nov 21, 2019 at 02:46:39PM +0100, Jose Fco. Irles Dur?? wrote:
Hi
I have a kamailio 5.1.2 as load balancer and registration offloading,
but I have a problem with the max tcp connections that it can handle.
I suspect that is a linux limit, but I don't find the reason or config.
When that limit arrives, I can't connect to kamailio and I receive
"Connection reset by peer", but I can't view any error message in the
logs.
I experienced the same with versions pre 5.1.9. It hasn't happened to me
with 5.1.9 yet, so I cannot say it was fixed. Additional changes I
applied where some sysctl tweaks:
net.core.somaxconn = 128
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_tw_buckets = 4096
net.ipv4.ip_local_port_range = 4096 65534
net.ipv4.tcp_syncookies = 0
Again, not really sure if these are related.
10:49 p.m.
Thanks for the info!
Tomorrow I'll test it with the 5.1.9 version.
Best regards
El jue., 21 nov. 2019 a las 17:35, Daniel Tryba (<d.tryba(a)pocos.nl>) escribió:
On Thu, Nov 21, 2019 at 02:46:39PM +0100, Jose Fco. Irles Dur?? wrote:
--
Jose Fco. Irles Durá
Hi
I have a kamailio 5.1.2 as load balancer and registration offloading,
but I have a problem with the max tcp connections that it can handle.
I suspect that is a linux limit, but I don't find the reason or config.
When that limit arrives, I can't connect to kamailio and I receive
"Connection reset by peer", but I can't view any error message in the
logs.
I experienced the same with versions pre 5.1.9. It hasn't happened to me
with 5.1.9 yet, so I cannot say it was fixed. Additional changes I
applied where some sysctl tweaks:
net.core.somaxconn = 128
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_tw_buckets = 4096
net.ipv4.ip_local_port_range = 4096 65534
net.ipv4.tcp_syncookies = 0
Again, not really sure if these are related.
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
26 Nov
26 Nov
1:32 p.m.
On Thu, Nov 21, 2019 at 07:49:28PM +0100, Jose Fco. Irles Dur?? wrote:
Thanks for the info!
Tomorrow I'll test it with the 5.1.9 version.
Well, the problem happened to me on 2 different loadbalancers (withing
24 hours where the loadbalancers had a near identical uptime) For about
35m no new connections can be established. Already established
connections work fine. I'm not seeing any queueing in to OS (netstat/ss)
After some time all works well again without doing anything to the machine/kamailio.
I could try to make a core dump if this happens again and the timing is more
appropriate for that.
kamcmd> core.tcp_info
{
readers: 4
max_connections: 4096
max_tls_connections: 2048
opened_connections: 617
opened_tls_connections: 0
write_queued_bytes: 0
}
tcp options in .cg:
tcp_max_connections=4096
tcp_connection_lifetime=3605
Only funny thing I see is that there are some TCP connections with a very high timeout:
{
id: 15453
type: TCP
state: CONN_OK
timeout: 268435455
lifetime: 3605
ref_count: 2
src_ip: a.b.c.d
src_port: 51196
dst_ip: e.f.g.h
dst_port: 5060
}
The timeout is either [0,3605] or 268435455. The high timeout ones
dissear quickly:
while sleep 1 ; do date; kamcmd core.tcp_list | grep 268435455; done
Tue Nov 26 10:29:21 CET 2019
Tue Nov 26 10:29:22 CET 2019
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
Tue Nov 26 10:29:23 CET 2019
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
Tue Nov 26 10:29:24 CET 2019
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
Tue Nov 26 10:29:25 CET 2019
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
Tue Nov 26 10:29:26 CET 2019
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
Tue Nov 26 10:29:27 CET 2019
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
timeout: 268435455
Tue Nov 26 10:29:28 CET 2019
timeout: 268435455
timeout: 268435455
Tue Nov 26 10:29:29 CET 2019
timeout: 268435455
timeout: 268435455
1:48 p.m.
On Tue, Nov 26, 2019 at 10:32:59AM +0100, Daniel Tryba wrote:
Well, the problem happened to me on 2 different
loadbalancers (withing
24 hours where the loadbalancers had a near identical uptime) For about
35m no new connections can be established. Already established
connections work fine. I'm not seeing any queueing in to OS (netstat/ss)
After some time all works well again without doing anything to the machine/kamailio.
I could try to make a core dump if this happens again and the timing is more
appropriate for that.
And then it happened again. Coredumps weren't reacted sadly enough. But
after restarting I still couldn't reconnect. I'm starting to think this
is an OS issue.
3:35 p.m.
In my case, from the upgrade I haven't issues (Kamailio has ~2700 tcp
connections)
But when I restarted the service (for the upgrade), tcp connections
grow until ~2850 and in that moment I can't create new connections.
I will continue investigating
El mar., 26 nov. 2019 a las 10:48, Daniel Tryba (<d.tryba(a)pocos.nl>) escribió:
On Tue, Nov 26, 2019 at 10:32:59AM +0100, Daniel Tryba wrote:
--
Jose Fco. Irles Durá
Well, the problem happened to me on 2 different
loadbalancers (withing
24 hours where the loadbalancers had a near identical uptime) For about
35m no new connections can be established. Already established
connections work fine. I'm not seeing any queueing in to OS (netstat/ss)
After some time all works well again without doing anything to the machine/kamailio.
I could try to make a core dump if this happens again and the timing is more
appropriate for that.
And then it happened again. Coredumps weren't reacted sadly enough. But
after restarting I still couldn't reconnect. I'm starting to think this
is an OS issue.
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users 
4:14 p.m.
That’s strange, I have 5000+ tcp connections and I can’t remember I did
anything special at the os level. I’m using 5.2 now, but it was the same
with 5.1
On Tue, 26 Nov 2019 at 11:37, Jose Fco. Irles Durá <josefu(a)gmail.com> wrote:
In my case, from the upgrade I haven't issues
(Kamailio has ~2700 tcp
connections)
But when I restarted the service (for the upgrade), tcp connections
grow until ~2850 and in that moment I can't create new connections.
I will continue investigating
El mar., 26 nov. 2019 a las 10:48, Daniel Tryba (<d.tryba(a)pocos.nl>)
escribió:
--
Jose Fco. Irles Durá
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Regards,
David Villasmil
email: david.villasmil.work(a)gmail.com
phone: +34669448337
On Tue, Nov 26, 2019 at 10:32:59AM +0100, Daniel Tryba wrote:
> Well, the problem happened to me on 2 different loadbalancers (withing
> 24 hours where the loadbalancers had a near identical uptime) For about
> 35m no new connections can be established. Already established
> connections work fine. I'm not seeing any queueing in to OS
(netstat/ss)
> After some time all works well again without
doing anything to the
machine/kamailio.
>
> I could try to make a core dump if this happens again and the timing
is more
appropriate for that.
And then it happened again. Coredumps weren't reacted sadly enough. But
after restarting I still couldn't reconnect. I'm starting to think this
is an OS issue.
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
7:14 p.m.
I will try to upgrade to the latest version and re test.
Best regards
El mar., 26 nov. 2019 a las 13:15, David Villasmil
(<david.villasmil.work(a)gmail.com>) escribió:
That’s strange, I have 5000+ tcp connections and I can’t remember I did anything special
at the os level. I’m using 5.2 now, but it was the same with 5.1
10 Dec
10 Dec
3:25 p.m.
I checked that the problem persist.
I have installed the version is 5.1.9, and the limit to 4096, but I
can't connect more than 2758.
I think that the problem is in the kernel, but i don't have any idea
how debug it. Any idea?
Best regards
El mar., 26 nov. 2019 a las 16:14, Jose Fco. Irles Durá
(<josefu(a)gmail.com>) escribió:
I will try to upgrade to the latest version and re test.
Best regards
El mar., 26 nov. 2019 a las 13:15, David Villasmil
(<david.villasmil.work(a)gmail.com>) escribió:
>
> That’s strange, I have 5000+ tcp connections and I can’t remember I did anything
special at the os level. I’m using 5.2 now, but it was the same with 5.1
>
--
Jose Fco. Irles Durá
4:34 p.m.
I know this is not recommended, but have you tried starting as root? If so
and you don’t hit a limit, then there’s something wrong in your limits for
the kamailio user. I have the same issue and it only works with root, I
haven’t been able to run it with kamailio for so many tcp connections.
On Tue, 10 Dec 2019 at 11:27, Jose Fco. Irles Durá <josefu(a)gmail.com> wrote:
I checked that the problem persist.
I have installed the version is 5.1.9, and the limit to 4096, but I
can't connect more than 2758.
I think that the problem is in the kernel, but i don't have any idea
how debug it. Any idea?
Best regards
El mar., 26 nov. 2019 a las 16:14, Jose Fco. Irles Durá
(<josefu(a)gmail.com>) escribió:
--
Regards,
David Villasmil
email: david.villasmil.work(a)gmail.com
phone: +34669448337
I will try to upgrade to the latest version and re test.
Best regards
El mar., 26 nov. 2019 a las 13:15, David Villasmil
(<david.villasmil.work(a)gmail.com>) escribió:
>
> That’s strange, I have 5000+ tcp connections and I can’t remember I
did
anything special at the os level. I’m using 5.2 now, but it was the
same with 5.1
>
--
Jose Fco. Irles Durá
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
4:39 p.m.
I will test it.
Thanks for the reply.
Best regards
El mar., 10 dic. 2019 a las 13:35, David Villasmil
(<david.villasmil.work(a)gmail.com>) escribió:
I know this is not recommended, but have you tried starting as root? If so and you don’t
hit a limit, then there’s something wrong in your limits for the kamailio user. I have the
same issue and it only works with root, I haven’t been able to run it with kamailio for so
many tcp connections.
On Tue, 10 Dec 2019 at 11:27, Jose Fco. Irles Durá <josefu(a)gmail.com> wrote:
--
Jose Fco. Irles Durá
I checked that the problem persist.
I have installed the version is 5.1.9, and the limit to 4096, but I
can't connect more than 2758.
I think that the problem is in the kernel, but i don't have any idea
how debug it. Any idea?
Best regards
El mar., 26 nov. 2019 a las 16:14, Jose Fco. Irles Durá
(<josefu(a)gmail.com>) escribió:
--
Regards,
David Villasmil
email: david.villasmil.work(a)gmail.com
phone: +34669448337
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
I will try to upgrade to the latest version and re test.
Best regards
El mar., 26 nov. 2019 a las 13:15, David Villasmil
(<david.villasmil.work(a)gmail.com>) escribió:
>
> That’s strange, I have 5000+ tcp connections and I can’t remember I did anything
special at the os level. I’m using 5.2 now, but it was the same with 5.1
>
--
Jose Fco. Irles Durá
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
11 Dec
11 Dec
4:25 p.m.
Today, running as root, kamailio has more connections. It seems that
the problem is some user limit, I will recheck the limits.
I will wait some days checking it.
Best regards
El mar., 10 dic. 2019 a las 13:39, Jose Fco. Irles Durá
(<josefu(a)gmail.com>) escribió:
I will test it.
Thanks for the reply.
Best regards
El mar., 10 dic. 2019 a las 13:35, David Villasmil
(<david.villasmil.work(a)gmail.com>) escribió:
--
Jose Fco. Irles Durá
I know this is not recommended, but have you tried starting as root? If so and you don’t
hit a limit, then there’s something wrong in your limits for the kamailio user. I have the
same issue and it only works with root, I haven’t been able to run it with kamailio for so
many tcp connections.
On Tue, 10 Dec 2019 at 11:27, Jose Fco. Irles Durá <josefu(a)gmail.com> wrote:
--
Jose Fco. Irles Durá
I checked that the problem persist.
I have installed the version is 5.1.9, and the limit to 4096, but I
can't connect more than 2758.
I think that the problem is in the kernel, but i don't have any idea
how debug it. Any idea?
Best regards
El mar., 26 nov. 2019 a las 16:14, Jose Fco. Irles Durá
(<josefu(a)gmail.com>) escribió:
--
Regards,
David Villasmil
email: david.villasmil.work(a)gmail.com
phone: +34669448337
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
I will try to upgrade to the latest version and re test.
Best regards
El mar., 26 nov. 2019 a las 13:15, David Villasmil
(<david.villasmil.work(a)gmail.com>) escribió:
>
> That’s strange, I have 5000+ tcp connections and I can’t remember I did anything
special at the os level. I’m using 5.2 now, but it was the same with 5.1
>
--
Jose Fco. Irles Durá
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users 
6 Apr
6 Apr
4:26 a.m.
Where did you guys get to with this?
Can you share details of the OS you were using?
I'm very curious if you are still seeing this behaviour, and if not, what
did you do to change it?
Thanks
Jay
On Wed, 11 Dec 2019 at 22:26, Jose Fco. Irles Durá <josefu(a)gmail.com> wrote:
Today, running as root, kamailio has more connections.
It seems that
the problem is some user limit, I will recheck the limits.
I will wait some days checking it.
Best regards
El mar., 10 dic. 2019 a las 13:39, Jose Fco. Irles Durá
(<josefu(a)gmail.com>) escribió:
--
Jose Fco. Irles Durá
--
Jose Fco. Irles Durá
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Sincerely
Jay
I will test it.
Thanks for the reply.
Best regards
El mar., 10 dic. 2019 a las 13:35, David Villasmil
(<david.villasmil.work(a)gmail.com>) escribió:
>
> I know this is not recommended, but have you tried starting as root?
If so
and you don’t hit a limit, then there’s something wrong in your
limits for the kamailio user. I have the same issue and it only works with
root, I haven’t been able to run it with kamailio for so many tcp
connections.
>
> On Tue, 10 Dec 2019 at 11:27, Jose Fco. Irles Durá <josefu(a)gmail.com>
wrote:
>>
>> I checked that the problem persist.
>>
>> I have installed the version is 5.1.9, and the limit to 4096, but I
>> can't connect more than 2758.
>>
>> I think that the problem is in the kernel, but i don't have any idea
>> how debug it. Any idea?
>>
>>
>> Best regards
>>
>>
>> El mar., 26 nov. 2019 a las 16:14, Jose Fco. Irles Durá
>> (<josefu(a)gmail.com>) escribió:
>> >
>> > I will try to upgrade to the latest version and re test.
>> >
>> > Best regards
>> >
>> > El mar., 26 nov. 2019 a las 13:15, David Villasmil
>> > (<david.villasmil.work(a)gmail.com>) escribió:
>> > >
>> > > That’s strange, I have 5000+ tcp connections and I can’t remember
I did anything special at the os level. I’m using 5.2 now, but it was the
same with 5.1
> >
--
Jose Fco. Irles Durá
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Regards,
David Villasmil
email: david.villasmil.work(a)gmail.com
phone: +34669448337
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
1677
days inactive
1814
days old
13 comments
4 participants
participants (4)
-
Daniel Tryba -
David Villasmil -
jay binks -
Jose Fco. Irles Durá