27 Nov
2019
27 Nov
'19
8:23 p.m.
We noticed mysql did an upgrade to 5.7
On Wed, Nov 27, 2019 at 12:19 PM Daniel-Constantin Mierla <miconda(a)gmail.com>
wrote:
On 27.11.19 18:15, Daniel Greenwald wrote:
yes tls is loaded before db_mysql, is that bad?
Not, tls before any other modules potentially using libssl is the right
order.
Yes apt updates were run so could have been updates to anything. OS
version is 16.04.6 LTS (GNU/Linux 4.4.0-1098-aws x86_64) Updates we last
run a couple of weeks ago.
Upon further digging I think the tls change was released a couple weeks
ago so yes probably unrelated.
Install the packages with debug symbols for kamailio, libmysqlclient and
libssl/libcrypto, then grab again the "bt full", it should show more
details in the backtrace that may help to identify the issue.
Cheers,
Daniel
On Wed, Nov 27, 2019 at 12:03 PM Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
The issue seems to come from libmysqlclient
library, not to kamailio
handling sip traffic over tls.
Is tls module loaded before the db_mysql module?
Have you made other updates to the operating system? Or only the change
you mentioned about server requiring and verifying certificates?
What version of ubuntu do you run?
I see that it uses libssl 1.0, so it is no relation with the change in
libssl 1.1 internal api.
Cheers,
Daniel
On 27.11.19 17:38, Daniel Greenwald wrote:
Core was generated by `/usr/sbin/kamailio -P
/var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681
681 ../sysdeps/x86_64/multiarch/memcmp-sse4.S: No such file or
directory.
(gdb) bt
#0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681
#1 0x00007fc864920b79 in ?? () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#2 0x00007fc86492117a in lh_retrieve () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#3 0x00007fc864923651 in ?? () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#4 0x00007fc8649240de in ERR_get_state () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#5 0x00007fc864924326 in ?? () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#6 0x00007fc864c8a8e0 in SSL_get_error () from
/lib/x86_64-linux-gnu/libssl.so.1.0.0
#7 0x00007fc843831cbc in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#8 0x00007fc84380d078 in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#9 0x00007fc84380d317 in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#10 0x00007fc84380e09c in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#11 0x00007fc843801fe7 in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#12 0x00007fc8438030d8 in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#13 0x00007fc843804e36 in mysql_real_query () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#14 0x00007fc843db0f76 in ?? () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so
#15 0x00007fc8433bbb79 in ?? () from
/usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1
#16 0x00007fc8433bf95b in db_do_update () from
/usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1
#17 0x00007fc843db5a86 in db_mysql_update () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so
#18 0x00007fc83f22a61b in update_dialog_dbinfo_unsafe () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so
#19 0x00007fc83f22aad3 in update_dialog_dbinfo () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so
#20 0x00007fc83f21a065 in dlg_onroute () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so
#21 0x00007fc841017917 in run_rr_callbacks () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so
#22 0x00007fc84100c0dd in ?? () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so
#23 0x00007fc84100c7aa in loose_route () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so
#24 0x00007fc84100ffb5 in ?? () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so
#25 0x0000000000480a22 in do_action ()
#26 0x000000000048db3f in run_actions ()
#27 0x000000000048e1fa in run_actions_safe ()
#28 0x0000000000450aed in rval_get_int ()
#29 0x00000000004551ed in rval_expr_eval_int ()
#30 0x00000000004804d0 in do_action ()
#31 0x000000000048db3f in run_actions ()
#32 0x000000000047d2a3 in do_action ()
#33 0x000000000048db3f in run_actions ()
#34 0x000000000048e306 in run_top_route ()
#35 0x00000000005b815c in receive_msg ()
#36 0x00000000004ab07f in udp_rcv_loop ()
#37 0x000000000042533c in main_loop ()
---Type <return> to continue, or q <return> to quit---
#38 0x000000000042cb3a in main ()
On Wed, Nov 27, 2019 at 11:10 AM Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda --
www.linkedin.com/in/miconda
Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
You got a core file, logs show that:
2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core>
[main.c:764]: handle_sigs(): core was generated
Locate it and send the full backtrace with gdb from it.
Cheers,
Daniel
On 27.11.19 16:03, Daniel Greenwald wrote:
I'm sorry you are correct, the version we are running is 5.2.5. I've
just run ulimit -c unlimited to hopefully get a core dump next time since I
don't see one in /var/lib/systemd/coredump/. Not sure if there is anything
I need to do to get a coredump.
The log just shows:
2019-11-27T14:04:22.298010+00:00,ip-10-0-33-149: CRITICAL: <core>
[core/pass_fd.c:277]: receive_fd(): EOF on 33
2019-11-27T14:04:22.299092+00:00,ip-10-0-33-149: ALERT: <core>
[main.c:761]: handle_sigs(): child process 12674 exited by a signal 11
2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core>
[main.c:764]: handle_sigs(): core was generated
2019-11-27T14:04:22.299564+00:00,ip-10-0-33-149: INFO: <core>
[main.c:787]: handle_sigs(): terminating due to SIGCHLD
2019-11-27T14:04:22.299892+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.300298+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.300569+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.300820+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.301070+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.301315+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.301564+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.301823+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.302056+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.302277+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.302525+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.302773+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.303020+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received
On Wed, Nov 27, 2019 at 9:49 AM Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda --
www.linkedin.com/in/miconda
Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
-- If you use the repo listed at
http://deb.kamailio.org/ , then it
should be 5.2.5, and there is also option to install 5.2 branch debs built
nightly.
Cheers,
Daniel
On 27.11.19 15:37, Daniel Greenwald wrote:
I'm using ubuntu packages, I believe 5.2.3 is latest?
On Wed, Nov 27, 2019 at 9:34 AM Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
> Hello,
>
> that log message is irrelevant for actual crash. Did you get core
> files? Or other log messages before the one you pasted?
>
> You should upgrade first to latest 5.2.x, either 5.2.5 or latest
> branch 5.2.
>
> Cheers,
> Daniel
> On 27.11.19 15:22, Daniel Greenwald wrote:
>
> I'm seeing this in the log and kamailio is crashing every hour or so:
> CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on
> 33/var/log/kamailio/kamailio.log
>
> Any ideas? Should I emergency upgrade to 5.3.1?
>
> _______________________________________________
> Kamailio (SER) - Users Mailing
Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
> --
> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda --
www.linkedin.com/in/miconda
> Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
>
> --
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda --
www.linkedin.com/in/miconda
Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda --
www.linkedin.com/in/miconda
Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
--