Hi Guys,
I recently tried to use Debian Stretch in production and it didn't went well. On load(500k-700k SIP messages per day) I get a problem that at some point there was a pike of CLOSE_WAIT connections(up to 2k of CLOSE_WAIT connections) and no new connections was possible, I tried to figure out where problem is, but I didn't found anything in system or Kamailio logs, Kamailio just stopped to receive traffic via TCP, but UDP continued to work. Test environment where was no load I didn't faced such issue.
Just curious is there anybody who are using Debian Stretch in production without issues?
Jurijs
Hello,
what is the kamailio version?
Were these bare tcp or also tls connections? Stretch comes with libssl 1.1 which is a major refactoring and wondering if that can be an effect.
Did you have to restart or the connections were ended after a while? Being just a temporary pike ...
What is the tcp connection lifetime value you use?
Might help a bit, you can increase the number of max tcp connections in kamailio (default is 2024).
- https://www.kamailio.org/wiki/cookbooks/5.0.x/core#tcp_max_connections
Cheers, Daniel
On 28.09.17 07:51, Jurijs Ivolga wrote:
Hi Guys,
I recently tried to use Debian Stretch in production and it didn't went well. On load(500k-700k SIP messages per day) I get a problem that at some point there was a pike of CLOSE_WAIT connections(up to 2k of CLOSE_WAIT connections) and no new connections was possible, I tried to figure out where problem is, but I didn't found anything in system or Kamailio logs, Kamailio just stopped to receive traffic via TCP, but UDP continued to work. Test environment where was no load I didn't faced such issue.
Just curious is there anybody who are using Debian Stretch in production without issues?
Jurijs
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi,
It was just SSL connections. I needed to restart kamailio, otherwise it just stopped to work.
kamailio 4.4.6 (x86_64/linux) 75f13d, I built from git.
TCP connection lifetime was 124 seconds.
tcp_max_connections=24000
I switched to debian 8 and now it works.
With kind regards,
Jurijs
On Thu, Sep 28, 2017 at 9:57 AM, Daniel-Constantin Mierla <miconda@gmail.com
wrote:
Hello,
what is the kamailio version?
Were these bare tcp or also tls connections? Stretch comes with libssl 1.1 which is a major refactoring and wondering if that can be an effect.
Did you have to restart or the connections were ended after a while? Being just a temporary pike ...
What is the tcp connection lifetime value you use?
Might help a bit, you can increase the number of max tcp connections in kamailio (default is 2024).
Cheers, Daniel
On 28.09.17 07:51, Jurijs Ivolga wrote:
Hi Guys,
I recently tried to use Debian Stretch in production and it didn't went well. On load(500k-700k SIP messages per day) I get a problem that at some point there was a pike of CLOSE_WAIT connections(up to 2k of CLOSE_WAIT connections) and no new connections was possible, I tried to figure out where problem is, but I didn't found anything in system or Kamailio logs, Kamailio just stopped to receive traffic via TCP, but UDP continued to work. Test environment where was no load I didn't faced such issue.
Just curious is there anybody who are using Debian Stretch in production without issues?
Jurijs
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlawww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio Advanced Training - www.asipto.com Kamailio World Conference - www.kamailioworld.com
Try adjust LimitNOFILE on systemd https://www.freedesktop.org/software/systemd/man/systemd.exec.html unit If this will help then create ticker and i will create PR Example https://freeswitch.org/stash/projects/FS/repos/freeswitch/browse/debian/freeswitch-systemd.freeswitch.service
чт, 28 сент. 2017 г. в 8:51, Jurijs Ivolga jurijs.ivolga@gmail.com:
Hi Guys,
I recently tried to use Debian Stretch in production and it didn't went well. On load(500k-700k SIP messages per day) I get a problem that at some point there was a pike of CLOSE_WAIT connections(up to 2k of CLOSE_WAIT connections) and no new connections was possible, I tried to figure out where problem is, but I didn't found anything in system or Kamailio logs, Kamailio just stopped to receive traffic via TCP, but UDP continued to work. Test environment where was no load I didn't faced such issue.
Just curious is there anybody who are using Debian Stretch in production without issues?
Jurijs _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi,
There no limits set...
core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 514946 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1048576 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
With kind regards,
Jurijs
On Thu, Sep 28, 2017 at 11:40 AM, Sergey Safarov s.safarov@gmail.com wrote:
Try adjust LimitNOFILE on systemd https://www.freedesktop.org/software/systemd/man/systemd.exec.html unit If this will help then create ticker and i will create PR Example https://freeswitch.org/stash/projects/FS/repos/freeswitch/browse/debian/freeswitch-systemd.freeswitch.service
чт, 28 сент. 2017 г. в 8:51, Jurijs Ivolga jurijs.ivolga@gmail.com:
Hi Guys,
I recently tried to use Debian Stretch in production and it didn't went well. On load(500k-700k SIP messages per day) I get a problem that at some point there was a pike of CLOSE_WAIT connections(up to 2k of CLOSE_WAIT connections) and no new connections was possible, I tried to figure out where problem is, but I didn't found anything in system or Kamailio logs, Kamailio just stopped to receive traffic via TCP, but UDP continued to work. Test environment where was no load I didn't faced such issue.
Just curious is there anybody who are using Debian Stretch in production without issues?
Jurijs _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Jurijs,
You have also DefaultLimitNOFILE in systemd that could be different of the values you see in console: https://www.freedesktop.org/software/systemd/man/systemd-system.conf.html
You should test to put LimitNOFILE in systemd unit file, just to be sure.
BTW, we had also some strange behaviors with TCP, Kamailio 5.0 and Debian Stretch. But with the Kamailio 4.4 integrated in Debian repository, it seems OK for now.
Regards.
-- Ludovic Gasc (GMLudo) Lead Developer Architect at ALLOcloud https://be.linkedin.com/in/ludovicgasc
2017-09-28 10:51 GMT+02:00 Jurijs Ivolga jurijs.ivolga@gmail.com:
Hi,
There no limits set...
core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 514946 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1048576 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
With kind regards,
Jurijs
On Thu, Sep 28, 2017 at 11:40 AM, Sergey Safarov s.safarov@gmail.com wrote:
Try adjust LimitNOFILE on systemd https://www.freedesktop.org/software/systemd/man/systemd.exec.html unit If this will help then create ticker and i will create PR Example https://freeswitch.org/stash/projects/FS/repos/freeswitch/browse/debian/freeswitch-systemd.freeswitch.service
чт, 28 сент. 2017 г. в 8:51, Jurijs Ivolga jurijs.ivolga@gmail.com:
Hi Guys,
I recently tried to use Debian Stretch in production and it didn't went well. On load(500k-700k SIP messages per day) I get a problem that at some point there was a pike of CLOSE_WAIT connections(up to 2k of CLOSE_WAIT connections) and no new connections was possible, I tried to figure out where problem is, but I didn't found anything in system or Kamailio logs, Kamailio just stopped to receive traffic via TCP, but UDP continued to work. Test environment where was no load I didn't faced such issue.
Just curious is there anybody who are using Debian Stretch in production without issues?
Jurijs _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi,
Ok, I'll confess, I'm running Kamailio in docker and with root user, so limits should be fine...
With kind regards,
Jurijs
On Thu, Sep 28, 2017 at 2:54 PM, Ludovic Gasc gmludo@gmail.com wrote:
Hi Jurijs,
You have also DefaultLimitNOFILE in systemd that could be different of the values you see in console: https://www.freedesktop.org/software/ systemd/man/systemd-system.conf.html
You should test to put LimitNOFILE in systemd unit file, just to be sure.
BTW, we had also some strange behaviors with TCP, Kamailio 5.0 and Debian Stretch. But with the Kamailio 4.4 integrated in Debian repository, it seems OK for now.
Regards.
-- Ludovic Gasc (GMLudo) Lead Developer Architect at ALLOcloud https://be.linkedin.com/in/ludovicgasc
2017-09-28 10:51 GMT+02:00 Jurijs Ivolga jurijs.ivolga@gmail.com:
Hi,
There no limits set...
core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 514946 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1048576 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
With kind regards,
Jurijs
On Thu, Sep 28, 2017 at 11:40 AM, Sergey Safarov s.safarov@gmail.com wrote:
Try adjust LimitNOFILE on systemd https://www.freedesktop.org/software/systemd/man/systemd.exec.html unit If this will help then create ticker and i will create PR Example https://freeswitch.org/stash/projects/FS/repos/freeswitch/browse/debian/freeswitch-systemd.freeswitch.service
чт, 28 сент. 2017 г. в 8:51, Jurijs Ivolga jurijs.ivolga@gmail.com:
Hi Guys,
I recently tried to use Debian Stretch in production and it didn't went well. On load(500k-700k SIP messages per day) I get a problem that at some point there was a pike of CLOSE_WAIT connections(up to 2k of CLOSE_WAIT connections) and no new connections was possible, I tried to figure out where problem is, but I didn't found anything in system or Kamailio logs, Kamailio just stopped to receive traffic via TCP, but UDP continued to work. Test environment where was no load I didn't faced such issue.
Just curious is there anybody who are using Debian Stretch in production without issues?
Jurijs _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
I had a similar problem and had to recompile Kamailio linked to openssl 1.0.X in debian stretch, since then, 0 issues with Tcp&ssl. I know Daniel did some changes so you might want to try out latest 5.0.3? I don't know if his patches got backported to 4.X...
This is the issue I had:
https://github.com/kamailio/kamailio/issues/1172
Not the same cause, but also related to openssl 1.1:
https://github.com/kamailio/kamailio/issues/1189
Joel.
On Thu, Sep 28, 2017 at 4:58 AM, Jurijs Ivolga jurijs.ivolga@gmail.com wrote:
Hi,
Ok, I'll confess, I'm running Kamailio in docker and with root user, so limits should be fine...
With kind regards,
Jurijs
On Thu, Sep 28, 2017 at 2:54 PM, Ludovic Gasc gmludo@gmail.com wrote:
Hi Jurijs,
You have also DefaultLimitNOFILE in systemd that could be different of the values you see in console: https://www.freedeskt op.org/software/systemd/man/systemd-system.conf.html
You should test to put LimitNOFILE in systemd unit file, just to be sure.
BTW, we had also some strange behaviors with TCP, Kamailio 5.0 and Debian Stretch. But with the Kamailio 4.4 integrated in Debian repository, it seems OK for now.
Regards.
-- Ludovic Gasc (GMLudo) Lead Developer Architect at ALLOcloud https://be.linkedin.com/in/ludovicgasc
2017-09-28 10:51 GMT+02:00 Jurijs Ivolga jurijs.ivolga@gmail.com:
Hi,
There no limits set...
core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 514946 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1048576 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
With kind regards,
Jurijs
On Thu, Sep 28, 2017 at 11:40 AM, Sergey Safarov s.safarov@gmail.com wrote:
Try adjust LimitNOFILE on systemd https://www.freedesktop.org/software/systemd/man/systemd.exec.html unit If this will help then create ticker and i will create PR Example https://freeswitch.org/stash/projects/FS/repos/freeswitch/browse/debian/freeswitch-systemd.freeswitch.service
чт, 28 сент. 2017 г. в 8:51, Jurijs Ivolga jurijs.ivolga@gmail.com:
Hi Guys,
I recently tried to use Debian Stretch in production and it didn't went well. On load(500k-700k SIP messages per day) I get a problem that at some point there was a pike of CLOSE_WAIT connections(up to 2k of CLOSE_WAIT connections) and no new connections was possible, I tried to figure out where problem is, but I didn't found anything in system or Kamailio logs, Kamailio just stopped to receive traffic via TCP, but UDP continued to work. Test environment where was no load I didn't faced such issue.
Just curious is there anybody who are using Debian Stretch in production without issues?
Jurijs _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users