I'm seeing this in the log and kamailio is crashing every hour or so: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33/var/log/kamailio/kamailio.log
Any ideas? Should I emergency upgrade to 5.3.1?
Hello,
that log message is irrelevant for actual crash. Did you get core files? Or other log messages before the one you pasted?
You should upgrade first to latest 5.2.x, either 5.2.5 or latest branch 5.2.
Cheers, Daniel
On 27.11.19 15:22, Daniel Greenwald wrote:
I'm seeing this in the log and kamailio is crashing every hour or so: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33/var/log/kamailio/kamailio.log
Any ideas? Should I emergency upgrade to 5.3.1?
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
I'm using ubuntu packages, I believe 5.2.3 is latest?
On Wed, Nov 27, 2019 at 9:34 AM Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
that log message is irrelevant for actual crash. Did you get core files? Or other log messages before the one you pasted?
You should upgrade first to latest 5.2.x, either 5.2.5 or latest branch 5.2.
Cheers, Daniel On 27.11.19 15:22, Daniel Greenwald wrote:
I'm seeing this in the log and kamailio is crashing every hour or so: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33/var/log/kamailio/kamailio.log
Any ideas? Should I emergency upgrade to 5.3.1?
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
If you use the repo listed at http://deb.kamailio.org/ , then it should be 5.2.5, and there is also option to install 5.2 branch debs built nightly.
Cheers, Daniel
On 27.11.19 15:37, Daniel Greenwald wrote:
I'm using ubuntu packages, I believe 5.2.3 is latest?
On Wed, Nov 27, 2019 at 9:34 AM Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
Hello, that log message is irrelevant for actual crash. Did you get core files? Or other log messages before the one you pasted? You should upgrade first to latest 5.2.x, either 5.2.5 or latest branch 5.2. Cheers, Daniel On 27.11.19 15:22, Daniel Greenwald wrote:I'm seeing this in the log and kamailio is crashing every hour or so: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33/var/log/kamailio/kamailio.log Any ideas? Should I emergency upgrade to 5.3.1? _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com <http://www.kamailioworld.com>
I'm sorry you are correct, the version we are running is 5.2.5. I've just run ulimit -c unlimited to hopefully get a core dump next time since I don't see one in /var/lib/systemd/coredump/. Not sure if there is anything I need to do to get a coredump. The log just shows:
2019-11-27T14:04:22.298010+00:00,ip-10-0-33-149: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33 2019-11-27T14:04:22.299092+00:00,ip-10-0-33-149: ALERT: <core> [main.c:761]: handle_sigs(): child process 12674 exited by a signal 11 2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core> [main.c:764]: handle_sigs(): core was generated 2019-11-27T14:04:22.299564+00:00,ip-10-0-33-149: INFO: <core> [main.c:787]: handle_sigs(): terminating due to SIGCHLD 2019-11-27T14:04:22.299892+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300298+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300569+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300820+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301070+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301315+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301564+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301823+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302056+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302277+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302525+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302773+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.303020+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received
On Wed, Nov 27, 2019 at 9:49 AM Daniel-Constantin Mierla miconda@gmail.com wrote:
If you use the repo listed at http://deb.kamailio.org/ , then it should be 5.2.5, and there is also option to install 5.2 branch debs built nightly.
Cheers, Daniel On 27.11.19 15:37, Daniel Greenwald wrote:
I'm using ubuntu packages, I believe 5.2.3 is latest?
On Wed, Nov 27, 2019 at 9:34 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
that log message is irrelevant for actual crash. Did you get core files? Or other log messages before the one you pasted?
You should upgrade first to latest 5.2.x, either 5.2.5 or latest branch 5.2.
Cheers, Daniel On 27.11.19 15:22, Daniel Greenwald wrote:
I'm seeing this in the log and kamailio is crashing every hour or so: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33/var/log/kamailio/kamailio.log
Any ideas? Should I emergency upgrade to 5.3.1?
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
This started happening after we enabled server certificate verification. Datadog is showing a steady memory usage increase whereas previously it was flatline. The changes we added were:
[server:default] method = TLSv1.2 verify_certificate = yes require_certificate = yes private_key = /etc/kamailio/kamailio.key certificate = /etc/kamailio/kamailio.crt ca_list = /etc/kamailio/trusted_roots.crt
Previously we had client verification enabled but not server:
[client:default] method = TLSv1.2 verify_certificate = yes require_certificate = yes ca_list = /etc/kamailio/trusted_roots.crt private_key = /etc/kamailio/kamailio.key certificate = /etc/kamailio/kamailio.crt
On Wed, Nov 27, 2019 at 10:03 AM Daniel Greenwald dgreenwald@gmail.com wrote:
I'm sorry you are correct, the version we are running is 5.2.5. I've just run ulimit -c unlimited to hopefully get a core dump next time since I don't see one in /var/lib/systemd/coredump/. Not sure if there is anything I need to do to get a coredump. The log just shows:
2019-11-27T14:04:22.298010+00:00,ip-10-0-33-149: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33 2019-11-27T14:04:22.299092+00:00,ip-10-0-33-149: ALERT: <core> [main.c:761]: handle_sigs(): child process 12674 exited by a signal 11 2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core> [main.c:764]: handle_sigs(): core was generated 2019-11-27T14:04:22.299564+00:00,ip-10-0-33-149: INFO: <core> [main.c:787]: handle_sigs(): terminating due to SIGCHLD 2019-11-27T14:04:22.299892+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300298+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300569+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300820+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301070+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301315+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301564+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301823+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302056+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302277+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302525+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302773+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.303020+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received
On Wed, Nov 27, 2019 at 9:49 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
If you use the repo listed at http://deb.kamailio.org/ , then it should be 5.2.5, and there is also option to install 5.2 branch debs built nightly.
Cheers, Daniel On 27.11.19 15:37, Daniel Greenwald wrote:
I'm using ubuntu packages, I believe 5.2.3 is latest?
On Wed, Nov 27, 2019 at 9:34 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
that log message is irrelevant for actual crash. Did you get core files? Or other log messages before the one you pasted?
You should upgrade first to latest 5.2.x, either 5.2.5 or latest branch 5.2.
Cheers, Daniel On 27.11.19 15:22, Daniel Greenwald wrote:
I'm seeing this in the log and kamailio is crashing every hour or so: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33/var/log/kamailio/kamailio.log
Any ideas? Should I emergency upgrade to 5.3.1?
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
You got a core file, logs show that:
2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core> [main.c:764]: handle_sigs(): core was generated
Locate it and send the full backtrace with gdb from it.
Cheers, Daniel
On 27.11.19 16:03, Daniel Greenwald wrote:
I'm sorry you are correct, the version we are running is 5.2.5. I've just run ulimit -c unlimited to hopefully get a core dump next time since I don't see one in /var/lib/systemd/coredump/. Not sure if there is anything I need to do to get a coredump. The log just shows:
2019-11-27T14:04:22.298010+00:00,ip-10-0-33-149: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33 2019-11-27T14:04:22.299092+00:00,ip-10-0-33-149: ALERT: <core> [main.c:761]: handle_sigs(): child process 12674 exited by a signal 11 2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core> [main.c:764]: handle_sigs(): core was generated 2019-11-27T14:04:22.299564+00:00,ip-10-0-33-149: INFO: <core> [main.c:787]: handle_sigs(): terminating due to SIGCHLD 2019-11-27T14:04:22.299892+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300298+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300569+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300820+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301070+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301315+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301564+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301823+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302056+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302277+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302525+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302773+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.303020+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received
On Wed, Nov 27, 2019 at 9:49 AM Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
If you use the repo listed at http://deb.kamailio.org/ , then it should be 5.2.5, and there is also option to install 5.2 branch debs built nightly. Cheers, Daniel On 27.11.19 15:37, Daniel Greenwald wrote:I'm using ubuntu packages, I believe 5.2.3 is latest? On Wed, Nov 27, 2019 at 9:34 AM Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, that log message is irrelevant for actual crash. Did you get core files? Or other log messages before the one you pasted? You should upgrade first to latest 5.2.x, either 5.2.5 or latest branch 5.2. Cheers, Daniel On 27.11.19 15:22, Daniel Greenwald wrote:I'm seeing this in the log and kamailio is crashing every hour or so: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33/var/log/kamailio/kamailio.log Any ideas? Should I emergency upgrade to 5.3.1? _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com <http://www.kamailioworld.com>-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com <http://www.kamailioworld.com>
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681 681 ../sysdeps/x86_64/multiarch/memcmp-sse4.S: No such file or directory. (gdb) bt #0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681 #1 0x00007fc864920b79 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #2 0x00007fc86492117a in lh_retrieve () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #3 0x00007fc864923651 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #4 0x00007fc8649240de in ERR_get_state () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #5 0x00007fc864924326 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #6 0x00007fc864c8a8e0 in SSL_get_error () from /lib/x86_64-linux-gnu/libssl.so.1.0.0 #7 0x00007fc843831cbc in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #8 0x00007fc84380d078 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #9 0x00007fc84380d317 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #10 0x00007fc84380e09c in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #11 0x00007fc843801fe7 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #12 0x00007fc8438030d8 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #13 0x00007fc843804e36 in mysql_real_query () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #14 0x00007fc843db0f76 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so #15 0x00007fc8433bbb79 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1 #16 0x00007fc8433bf95b in db_do_update () from /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1 #17 0x00007fc843db5a86 in db_mysql_update () from /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so #18 0x00007fc83f22a61b in update_dialog_dbinfo_unsafe () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so #19 0x00007fc83f22aad3 in update_dialog_dbinfo () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so #20 0x00007fc83f21a065 in dlg_onroute () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so #21 0x00007fc841017917 in run_rr_callbacks () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #22 0x00007fc84100c0dd in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #23 0x00007fc84100c7aa in loose_route () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #24 0x00007fc84100ffb5 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #25 0x0000000000480a22 in do_action () #26 0x000000000048db3f in run_actions () #27 0x000000000048e1fa in run_actions_safe () #28 0x0000000000450aed in rval_get_int () #29 0x00000000004551ed in rval_expr_eval_int () #30 0x00000000004804d0 in do_action () #31 0x000000000048db3f in run_actions () #32 0x000000000047d2a3 in do_action () #33 0x000000000048db3f in run_actions () #34 0x000000000048e306 in run_top_route () #35 0x00000000005b815c in receive_msg () #36 0x00000000004ab07f in udp_rcv_loop () #37 0x000000000042533c in main_loop () ---Type <return> to continue, or q <return> to quit--- #38 0x000000000042cb3a in main ()
On Wed, Nov 27, 2019 at 11:10 AM Daniel-Constantin Mierla miconda@gmail.com wrote:
You got a core file, logs show that:
2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core> [main.c:764]: handle_sigs(): core was generated
Locate it and send the full backtrace with gdb from it.
Cheers, Daniel On 27.11.19 16:03, Daniel Greenwald wrote:
I'm sorry you are correct, the version we are running is 5.2.5. I've just run ulimit -c unlimited to hopefully get a core dump next time since I don't see one in /var/lib/systemd/coredump/. Not sure if there is anything I need to do to get a coredump. The log just shows:
2019-11-27T14:04:22.298010+00:00,ip-10-0-33-149: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33 2019-11-27T14:04:22.299092+00:00,ip-10-0-33-149: ALERT: <core> [main.c:761]: handle_sigs(): child process 12674 exited by a signal 11 2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core> [main.c:764]: handle_sigs(): core was generated 2019-11-27T14:04:22.299564+00:00,ip-10-0-33-149: INFO: <core> [main.c:787]: handle_sigs(): terminating due to SIGCHLD 2019-11-27T14:04:22.299892+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300298+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300569+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300820+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301070+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301315+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301564+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301823+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302056+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302277+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302525+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302773+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.303020+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received
On Wed, Nov 27, 2019 at 9:49 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
If you use the repo listed at http://deb.kamailio.org/ , then it should be 5.2.5, and there is also option to install 5.2 branch debs built nightly.
Cheers, Daniel On 27.11.19 15:37, Daniel Greenwald wrote:
I'm using ubuntu packages, I believe 5.2.3 is latest?
On Wed, Nov 27, 2019 at 9:34 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
that log message is irrelevant for actual crash. Did you get core files? Or other log messages before the one you pasted?
You should upgrade first to latest 5.2.x, either 5.2.5 or latest branch 5.2.
Cheers, Daniel On 27.11.19 15:22, Daniel Greenwald wrote:
I'm seeing this in the log and kamailio is crashing every hour or so: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33/var/log/kamailio/kamailio.log
Any ideas? Should I emergency upgrade to 5.3.1?
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
The issue seems to come from libmysqlclient library, not to kamailio handling sip traffic over tls.
Is tls module loaded before the db_mysql module?
Have you made other updates to the operating system? Or only the change you mentioned about server requiring and verifying certificates?
What version of ubuntu do you run?
I see that it uses libssl 1.0, so it is no relation with the change in libssl 1.1 internal api.
Cheers, Daniel
On 27.11.19 17:38, Daniel Greenwald wrote:
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681 681 ../sysdeps/x86_64/multiarch/memcmp-sse4.S: No such file or directory. (gdb) bt #0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681 #1 0x00007fc864920b79 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #2 0x00007fc86492117a in lh_retrieve () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #3 0x00007fc864923651 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #4 0x00007fc8649240de in ERR_get_state () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #5 0x00007fc864924326 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #6 0x00007fc864c8a8e0 in SSL_get_error () from /lib/x86_64-linux-gnu/libssl.so.1.0.0 #7 0x00007fc843831cbc in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #8 0x00007fc84380d078 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #9 0x00007fc84380d317 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #10 0x00007fc84380e09c in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #11 0x00007fc843801fe7 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #12 0x00007fc8438030d8 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #13 0x00007fc843804e36 in mysql_real_query () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #14 0x00007fc843db0f76 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so #15 0x00007fc8433bbb79 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1 #16 0x00007fc8433bf95b in db_do_update () from /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1 #17 0x00007fc843db5a86 in db_mysql_update () from /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so #18 0x00007fc83f22a61b in update_dialog_dbinfo_unsafe () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so #19 0x00007fc83f22aad3 in update_dialog_dbinfo () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so #20 0x00007fc83f21a065 in dlg_onroute () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so #21 0x00007fc841017917 in run_rr_callbacks () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #22 0x00007fc84100c0dd in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #23 0x00007fc84100c7aa in loose_route () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #24 0x00007fc84100ffb5 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #25 0x0000000000480a22 in do_action () #26 0x000000000048db3f in run_actions () #27 0x000000000048e1fa in run_actions_safe () #28 0x0000000000450aed in rval_get_int () #29 0x00000000004551ed in rval_expr_eval_int () #30 0x00000000004804d0 in do_action () #31 0x000000000048db3f in run_actions () #32 0x000000000047d2a3 in do_action () #33 0x000000000048db3f in run_actions () #34 0x000000000048e306 in run_top_route () #35 0x00000000005b815c in receive_msg () #36 0x00000000004ab07f in udp_rcv_loop () #37 0x000000000042533c in main_loop () ---Type <return> to continue, or q <return> to quit--- #38 0x000000000042cb3a in main ()
On Wed, Nov 27, 2019 at 11:10 AM Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
You got a core file, logs show that: 2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core> [main.c:764]: handle_sigs(): core was generated Locate it and send the full backtrace with gdb from it. Cheers, Daniel On 27.11.19 16:03, Daniel Greenwald wrote:I'm sorry you are correct, the version we are running is 5.2.5. I've just run ulimit -c unlimited to hopefully get a core dump next time since I don't see one in /var/lib/systemd/coredump/. Not sure if there is anything I need to do to get a coredump. The log just shows: 2019-11-27T14:04:22.298010+00:00,ip-10-0-33-149: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33 2019-11-27T14:04:22.299092+00:00,ip-10-0-33-149: ALERT: <core> [main.c:761]: handle_sigs(): child process 12674 exited by a signal 11 2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core> [main.c:764]: handle_sigs(): core was generated 2019-11-27T14:04:22.299564+00:00,ip-10-0-33-149: INFO: <core> [main.c:787]: handle_sigs(): terminating due to SIGCHLD 2019-11-27T14:04:22.299892+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300298+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300569+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300820+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301070+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301315+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301564+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301823+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302056+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302277+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302525+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302773+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.303020+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received On Wed, Nov 27, 2019 at 9:49 AM Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: If you use the repo listed at http://deb.kamailio.org/ , then it should be 5.2.5, and there is also option to install 5.2 branch debs built nightly. Cheers, Daniel On 27.11.19 15:37, Daniel Greenwald wrote:I'm using ubuntu packages, I believe 5.2.3 is latest? On Wed, Nov 27, 2019 at 9:34 AM Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, that log message is irrelevant for actual crash. Did you get core files? Or other log messages before the one you pasted? You should upgrade first to latest 5.2.x, either 5.2.5 or latest branch 5.2. Cheers, Daniel On 27.11.19 15:22, Daniel Greenwald wrote:I'm seeing this in the log and kamailio is crashing every hour or so: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33/var/log/kamailio/kamailio.log Any ideas? Should I emergency upgrade to 5.3.1? _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com <http://www.kamailioworld.com>-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com <http://www.kamailioworld.com>-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com <http://www.kamailioworld.com>
yes tls is loaded before db_mysql, is that bad?
Yes apt updates were run so could have been updates to anything. OS version is 16.04.6 LTS (GNU/Linux 4.4.0-1098-aws x86_64) Updates we last run a couple of weeks ago.
Upon further digging I think the tls change was released a couple weeks ago so yes probably unrelated.
On Wed, Nov 27, 2019 at 12:03 PM Daniel-Constantin Mierla miconda@gmail.com wrote:
The issue seems to come from libmysqlclient library, not to kamailio handling sip traffic over tls.
Is tls module loaded before the db_mysql module?
Have you made other updates to the operating system? Or only the change you mentioned about server requiring and verifying certificates?
What version of ubuntu do you run?
I see that it uses libssl 1.0, so it is no relation with the change in libssl 1.1 internal api.
Cheers, Daniel On 27.11.19 17:38, Daniel Greenwald wrote:
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681 681 ../sysdeps/x86_64/multiarch/memcmp-sse4.S: No such file or directory. (gdb) bt #0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681 #1 0x00007fc864920b79 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #2 0x00007fc86492117a in lh_retrieve () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #3 0x00007fc864923651 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #4 0x00007fc8649240de in ERR_get_state () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #5 0x00007fc864924326 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #6 0x00007fc864c8a8e0 in SSL_get_error () from /lib/x86_64-linux-gnu/libssl.so.1.0.0 #7 0x00007fc843831cbc in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #8 0x00007fc84380d078 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #9 0x00007fc84380d317 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #10 0x00007fc84380e09c in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #11 0x00007fc843801fe7 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #12 0x00007fc8438030d8 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #13 0x00007fc843804e36 in mysql_real_query () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #14 0x00007fc843db0f76 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so #15 0x00007fc8433bbb79 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1 #16 0x00007fc8433bf95b in db_do_update () from /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1 #17 0x00007fc843db5a86 in db_mysql_update () from /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so #18 0x00007fc83f22a61b in update_dialog_dbinfo_unsafe () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so #19 0x00007fc83f22aad3 in update_dialog_dbinfo () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so #20 0x00007fc83f21a065 in dlg_onroute () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so #21 0x00007fc841017917 in run_rr_callbacks () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #22 0x00007fc84100c0dd in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #23 0x00007fc84100c7aa in loose_route () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #24 0x00007fc84100ffb5 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #25 0x0000000000480a22 in do_action () #26 0x000000000048db3f in run_actions () #27 0x000000000048e1fa in run_actions_safe () #28 0x0000000000450aed in rval_get_int () #29 0x00000000004551ed in rval_expr_eval_int () #30 0x00000000004804d0 in do_action () #31 0x000000000048db3f in run_actions () #32 0x000000000047d2a3 in do_action () #33 0x000000000048db3f in run_actions () #34 0x000000000048e306 in run_top_route () #35 0x00000000005b815c in receive_msg () #36 0x00000000004ab07f in udp_rcv_loop () #37 0x000000000042533c in main_loop () ---Type <return> to continue, or q <return> to quit--- #38 0x000000000042cb3a in main ()
On Wed, Nov 27, 2019 at 11:10 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
You got a core file, logs show that:
2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core> [main.c:764]: handle_sigs(): core was generated
Locate it and send the full backtrace with gdb from it.
Cheers, Daniel On 27.11.19 16:03, Daniel Greenwald wrote:
I'm sorry you are correct, the version we are running is 5.2.5. I've just run ulimit -c unlimited to hopefully get a core dump next time since I don't see one in /var/lib/systemd/coredump/. Not sure if there is anything I need to do to get a coredump. The log just shows:
2019-11-27T14:04:22.298010+00:00,ip-10-0-33-149: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33 2019-11-27T14:04:22.299092+00:00,ip-10-0-33-149: ALERT: <core> [main.c:761]: handle_sigs(): child process 12674 exited by a signal 11 2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core> [main.c:764]: handle_sigs(): core was generated 2019-11-27T14:04:22.299564+00:00,ip-10-0-33-149: INFO: <core> [main.c:787]: handle_sigs(): terminating due to SIGCHLD 2019-11-27T14:04:22.299892+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300298+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300569+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300820+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301070+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301315+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301564+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301823+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302056+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302277+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302525+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302773+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.303020+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received
On Wed, Nov 27, 2019 at 9:49 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
If you use the repo listed at http://deb.kamailio.org/ , then it should be 5.2.5, and there is also option to install 5.2 branch debs built nightly.
Cheers, Daniel On 27.11.19 15:37, Daniel Greenwald wrote:
I'm using ubuntu packages, I believe 5.2.3 is latest?
On Wed, Nov 27, 2019 at 9:34 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
that log message is irrelevant for actual crash. Did you get core files? Or other log messages before the one you pasted?
You should upgrade first to latest 5.2.x, either 5.2.5 or latest branch 5.2.
Cheers, Daniel On 27.11.19 15:22, Daniel Greenwald wrote:
I'm seeing this in the log and kamailio is crashing every hour or so: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33/var/log/kamailio/kamailio.log
Any ideas? Should I emergency upgrade to 5.3.1?
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
On 27.11.19 18:15, Daniel Greenwald wrote:
yes tls is loaded before db_mysql, is that bad?
Not, tls before any other modules potentially using libssl is the right order.
Yes apt updates were run so could have been updates to anything. OS version is 16.04.6 LTS (GNU/Linux 4.4.0-1098-aws x86_64) Updates we last run a couple of weeks ago.
Upon further digging I think the tls change was released a couple weeks ago so yes probably unrelated.
Install the packages with debug symbols for kamailio, libmysqlclient and libssl/libcrypto, then grab again the "bt full", it should show more details in the backtrace that may help to identify the issue.
Cheers, Daniel
On Wed, Nov 27, 2019 at 12:03 PM Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
The issue seems to come from libmysqlclient library, not to kamailio handling sip traffic over tls. Is tls module loaded before the db_mysql module? Have you made other updates to the operating system? Or only the change you mentioned about server requiring and verifying certificates? What version of ubuntu do you run? I see that it uses libssl 1.0, so it is no relation with the change in libssl 1.1 internal api. Cheers, Daniel On 27.11.19 17:38, Daniel Greenwald wrote:Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681 681 ../sysdeps/x86_64/multiarch/memcmp-sse4.S: No such file or directory. (gdb) bt #0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681 #1 0x00007fc864920b79 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #2 0x00007fc86492117a in lh_retrieve () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #3 0x00007fc864923651 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #4 0x00007fc8649240de in ERR_get_state () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #5 0x00007fc864924326 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #6 0x00007fc864c8a8e0 in SSL_get_error () from /lib/x86_64-linux-gnu/libssl.so.1.0.0 #7 0x00007fc843831cbc in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #8 0x00007fc84380d078 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #9 0x00007fc84380d317 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #10 0x00007fc84380e09c in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #11 0x00007fc843801fe7 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #12 0x00007fc8438030d8 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #13 0x00007fc843804e36 in mysql_real_query () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #14 0x00007fc843db0f76 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so #15 0x00007fc8433bbb79 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1 #16 0x00007fc8433bf95b in db_do_update () from /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1 #17 0x00007fc843db5a86 in db_mysql_update () from /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so #18 0x00007fc83f22a61b in update_dialog_dbinfo_unsafe () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so #19 0x00007fc83f22aad3 in update_dialog_dbinfo () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so #20 0x00007fc83f21a065 in dlg_onroute () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so #21 0x00007fc841017917 in run_rr_callbacks () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #22 0x00007fc84100c0dd in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #23 0x00007fc84100c7aa in loose_route () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #24 0x00007fc84100ffb5 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #25 0x0000000000480a22 in do_action () #26 0x000000000048db3f in run_actions () #27 0x000000000048e1fa in run_actions_safe () #28 0x0000000000450aed in rval_get_int () #29 0x00000000004551ed in rval_expr_eval_int () #30 0x00000000004804d0 in do_action () #31 0x000000000048db3f in run_actions () #32 0x000000000047d2a3 in do_action () #33 0x000000000048db3f in run_actions () #34 0x000000000048e306 in run_top_route () #35 0x00000000005b815c in receive_msg () #36 0x00000000004ab07f in udp_rcv_loop () #37 0x000000000042533c in main_loop () ---Type <return> to continue, or q <return> to quit--- #38 0x000000000042cb3a in main () On Wed, Nov 27, 2019 at 11:10 AM Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: You got a core file, logs show that: 2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core> [main.c:764]: handle_sigs(): core was generated Locate it and send the full backtrace with gdb from it. Cheers, Daniel On 27.11.19 16:03, Daniel Greenwald wrote:I'm sorry you are correct, the version we are running is 5.2.5. I've just run ulimit -c unlimited to hopefully get a core dump next time since I don't see one in /var/lib/systemd/coredump/. Not sure if there is anything I need to do to get a coredump. The log just shows: 2019-11-27T14:04:22.298010+00:00,ip-10-0-33-149: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33 2019-11-27T14:04:22.299092+00:00,ip-10-0-33-149: ALERT: <core> [main.c:761]: handle_sigs(): child process 12674 exited by a signal 11 2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core> [main.c:764]: handle_sigs(): core was generated 2019-11-27T14:04:22.299564+00:00,ip-10-0-33-149: INFO: <core> [main.c:787]: handle_sigs(): terminating due to SIGCHLD 2019-11-27T14:04:22.299892+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300298+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300569+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300820+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301070+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301315+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301564+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301823+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302056+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302277+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302525+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302773+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.303020+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received On Wed, Nov 27, 2019 at 9:49 AM Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: If you use the repo listed at http://deb.kamailio.org/ , then it should be 5.2.5, and there is also option to install 5.2 branch debs built nightly. Cheers, Daniel On 27.11.19 15:37, Daniel Greenwald wrote:I'm using ubuntu packages, I believe 5.2.3 is latest? On Wed, Nov 27, 2019 at 9:34 AM Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, that log message is irrelevant for actual crash. Did you get core files? Or other log messages before the one you pasted? You should upgrade first to latest 5.2.x, either 5.2.5 or latest branch 5.2. Cheers, Daniel On 27.11.19 15:22, Daniel Greenwald wrote:I'm seeing this in the log and kamailio is crashing every hour or so: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33/var/log/kamailio/kamailio.log Any ideas? Should I emergency upgrade to 5.3.1? _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com <http://www.kamailioworld.com>-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com <http://www.kamailioworld.com>-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com <http://www.kamailioworld.com>-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com <http://www.kamailioworld.com>
We noticed mysql did an upgrade to 5.7
On Wed, Nov 27, 2019 at 12:19 PM Daniel-Constantin Mierla miconda@gmail.com wrote:
On 27.11.19 18:15, Daniel Greenwald wrote:
yes tls is loaded before db_mysql, is that bad?
Not, tls before any other modules potentially using libssl is the right order.
Yes apt updates were run so could have been updates to anything. OS version is 16.04.6 LTS (GNU/Linux 4.4.0-1098-aws x86_64) Updates we last run a couple of weeks ago.
Upon further digging I think the tls change was released a couple weeks ago so yes probably unrelated.
Install the packages with debug symbols for kamailio, libmysqlclient and libssl/libcrypto, then grab again the "bt full", it should show more details in the backtrace that may help to identify the issue.
Cheers, Daniel
On Wed, Nov 27, 2019 at 12:03 PM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
The issue seems to come from libmysqlclient library, not to kamailio handling sip traffic over tls.
Is tls module loaded before the db_mysql module?
Have you made other updates to the operating system? Or only the change you mentioned about server requiring and verifying certificates?
What version of ubuntu do you run?
I see that it uses libssl 1.0, so it is no relation with the change in libssl 1.1 internal api.
Cheers, Daniel On 27.11.19 17:38, Daniel Greenwald wrote:
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681 681 ../sysdeps/x86_64/multiarch/memcmp-sse4.S: No such file or directory. (gdb) bt #0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681 #1 0x00007fc864920b79 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #2 0x00007fc86492117a in lh_retrieve () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #3 0x00007fc864923651 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #4 0x00007fc8649240de in ERR_get_state () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #5 0x00007fc864924326 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #6 0x00007fc864c8a8e0 in SSL_get_error () from /lib/x86_64-linux-gnu/libssl.so.1.0.0 #7 0x00007fc843831cbc in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #8 0x00007fc84380d078 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #9 0x00007fc84380d317 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #10 0x00007fc84380e09c in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #11 0x00007fc843801fe7 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #12 0x00007fc8438030d8 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #13 0x00007fc843804e36 in mysql_real_query () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #14 0x00007fc843db0f76 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so #15 0x00007fc8433bbb79 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1 #16 0x00007fc8433bf95b in db_do_update () from /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1 #17 0x00007fc843db5a86 in db_mysql_update () from /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so #18 0x00007fc83f22a61b in update_dialog_dbinfo_unsafe () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so #19 0x00007fc83f22aad3 in update_dialog_dbinfo () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so #20 0x00007fc83f21a065 in dlg_onroute () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so #21 0x00007fc841017917 in run_rr_callbacks () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #22 0x00007fc84100c0dd in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #23 0x00007fc84100c7aa in loose_route () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #24 0x00007fc84100ffb5 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #25 0x0000000000480a22 in do_action () #26 0x000000000048db3f in run_actions () #27 0x000000000048e1fa in run_actions_safe () #28 0x0000000000450aed in rval_get_int () #29 0x00000000004551ed in rval_expr_eval_int () #30 0x00000000004804d0 in do_action () #31 0x000000000048db3f in run_actions () #32 0x000000000047d2a3 in do_action () #33 0x000000000048db3f in run_actions () #34 0x000000000048e306 in run_top_route () #35 0x00000000005b815c in receive_msg () #36 0x00000000004ab07f in udp_rcv_loop () #37 0x000000000042533c in main_loop () ---Type <return> to continue, or q <return> to quit--- #38 0x000000000042cb3a in main ()
On Wed, Nov 27, 2019 at 11:10 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
You got a core file, logs show that:
2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core> [main.c:764]: handle_sigs(): core was generated
Locate it and send the full backtrace with gdb from it.
Cheers, Daniel On 27.11.19 16:03, Daniel Greenwald wrote:
I'm sorry you are correct, the version we are running is 5.2.5. I've just run ulimit -c unlimited to hopefully get a core dump next time since I don't see one in /var/lib/systemd/coredump/. Not sure if there is anything I need to do to get a coredump. The log just shows:
2019-11-27T14:04:22.298010+00:00,ip-10-0-33-149: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33 2019-11-27T14:04:22.299092+00:00,ip-10-0-33-149: ALERT: <core> [main.c:761]: handle_sigs(): child process 12674 exited by a signal 11 2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core> [main.c:764]: handle_sigs(): core was generated 2019-11-27T14:04:22.299564+00:00,ip-10-0-33-149: INFO: <core> [main.c:787]: handle_sigs(): terminating due to SIGCHLD 2019-11-27T14:04:22.299892+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300298+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300569+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300820+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301070+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301315+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301564+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301823+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302056+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302277+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302525+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302773+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.303020+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received
On Wed, Nov 27, 2019 at 9:49 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
If you use the repo listed at http://deb.kamailio.org/ , then it should be 5.2.5, and there is also option to install 5.2 branch debs built nightly.
Cheers, Daniel On 27.11.19 15:37, Daniel Greenwald wrote:
I'm using ubuntu packages, I believe 5.2.3 is latest?
On Wed, Nov 27, 2019 at 9:34 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
that log message is irrelevant for actual crash. Did you get core files? Or other log messages before the one you pasted?
You should upgrade first to latest 5.2.x, either 5.2.5 or latest branch 5.2.
Cheers, Daniel On 27.11.19 15:22, Daniel Greenwald wrote:
I'm seeing this in the log and kamailio is crashing every hour or so: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33/var/log/kamailio/kamailio.log
Any ideas? Should I emergency upgrade to 5.3.1?
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
How do I install the symbol packages?
On Wed, Nov 27, 2019 at 12:23 PM Daniel Greenwald dgreenwald@gmail.com wrote:
We noticed mysql did an upgrade to 5.7
On Wed, Nov 27, 2019 at 12:19 PM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
On 27.11.19 18:15, Daniel Greenwald wrote:
yes tls is loaded before db_mysql, is that bad?
Not, tls before any other modules potentially using libssl is the right order.
Yes apt updates were run so could have been updates to anything. OS version is 16.04.6 LTS (GNU/Linux 4.4.0-1098-aws x86_64) Updates we last run a couple of weeks ago.
Upon further digging I think the tls change was released a couple weeks ago so yes probably unrelated.
Install the packages with debug symbols for kamailio, libmysqlclient and libssl/libcrypto, then grab again the "bt full", it should show more details in the backtrace that may help to identify the issue.
Cheers, Daniel
On Wed, Nov 27, 2019 at 12:03 PM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
The issue seems to come from libmysqlclient library, not to kamailio handling sip traffic over tls.
Is tls module loaded before the db_mysql module?
Have you made other updates to the operating system? Or only the change you mentioned about server requiring and verifying certificates?
What version of ubuntu do you run?
I see that it uses libssl 1.0, so it is no relation with the change in libssl 1.1 internal api.
Cheers, Daniel On 27.11.19 17:38, Daniel Greenwald wrote:
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681 681 ../sysdeps/x86_64/multiarch/memcmp-sse4.S: No such file or directory. (gdb) bt #0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681 #1 0x00007fc864920b79 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #2 0x00007fc86492117a in lh_retrieve () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #3 0x00007fc864923651 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #4 0x00007fc8649240de in ERR_get_state () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #5 0x00007fc864924326 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #6 0x00007fc864c8a8e0 in SSL_get_error () from /lib/x86_64-linux-gnu/libssl.so.1.0.0 #7 0x00007fc843831cbc in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #8 0x00007fc84380d078 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #9 0x00007fc84380d317 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #10 0x00007fc84380e09c in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #11 0x00007fc843801fe7 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #12 0x00007fc8438030d8 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #13 0x00007fc843804e36 in mysql_real_query () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 #14 0x00007fc843db0f76 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so #15 0x00007fc8433bbb79 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1 #16 0x00007fc8433bf95b in db_do_update () from /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1 #17 0x00007fc843db5a86 in db_mysql_update () from /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so #18 0x00007fc83f22a61b in update_dialog_dbinfo_unsafe () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so #19 0x00007fc83f22aad3 in update_dialog_dbinfo () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so #20 0x00007fc83f21a065 in dlg_onroute () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so #21 0x00007fc841017917 in run_rr_callbacks () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #22 0x00007fc84100c0dd in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #23 0x00007fc84100c7aa in loose_route () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #24 0x00007fc84100ffb5 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so #25 0x0000000000480a22 in do_action () #26 0x000000000048db3f in run_actions () #27 0x000000000048e1fa in run_actions_safe () #28 0x0000000000450aed in rval_get_int () #29 0x00000000004551ed in rval_expr_eval_int () #30 0x00000000004804d0 in do_action () #31 0x000000000048db3f in run_actions () #32 0x000000000047d2a3 in do_action () #33 0x000000000048db3f in run_actions () #34 0x000000000048e306 in run_top_route () #35 0x00000000005b815c in receive_msg () #36 0x00000000004ab07f in udp_rcv_loop () #37 0x000000000042533c in main_loop () ---Type <return> to continue, or q <return> to quit--- #38 0x000000000042cb3a in main ()
On Wed, Nov 27, 2019 at 11:10 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
You got a core file, logs show that:
2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core> [main.c:764]: handle_sigs(): core was generated
Locate it and send the full backtrace with gdb from it.
Cheers, Daniel On 27.11.19 16:03, Daniel Greenwald wrote:
I'm sorry you are correct, the version we are running is 5.2.5. I've just run ulimit -c unlimited to hopefully get a core dump next time since I don't see one in /var/lib/systemd/coredump/. Not sure if there is anything I need to do to get a coredump. The log just shows:
2019-11-27T14:04:22.298010+00:00,ip-10-0-33-149: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33 2019-11-27T14:04:22.299092+00:00,ip-10-0-33-149: ALERT: <core> [main.c:761]: handle_sigs(): child process 12674 exited by a signal 11 2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core> [main.c:764]: handle_sigs(): core was generated 2019-11-27T14:04:22.299564+00:00,ip-10-0-33-149: INFO: <core> [main.c:787]: handle_sigs(): terminating due to SIGCHLD 2019-11-27T14:04:22.299892+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300298+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300569+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.300820+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301070+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301315+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301564+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.301823+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302056+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302277+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302525+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.302773+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received 2019-11-27T14:04:22.303020+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received
On Wed, Nov 27, 2019 at 9:49 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
If you use the repo listed at http://deb.kamailio.org/ , then it should be 5.2.5, and there is also option to install 5.2 branch debs built nightly.
Cheers, Daniel On 27.11.19 15:37, Daniel Greenwald wrote:
I'm using ubuntu packages, I believe 5.2.3 is latest?
On Wed, Nov 27, 2019 at 9:34 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
that log message is irrelevant for actual crash. Did you get core files? Or other log messages before the one you pasted?
You should upgrade first to latest 5.2.x, either 5.2.5 or latest branch 5.2.
Cheers, Daniel On 27.11.19 15:22, Daniel Greenwald wrote:
I'm seeing this in the log and kamailio is crashing every hour or so: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33/var/log/kamailio/kamailio.log
Any ideas? Should I emergency upgrade to 5.3.1?
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
-
Daniel Greenwald -
Daniel-Constantin Mierla