### Description
Kamailio suddenly stops working (this is a production server so debug logging is disabled), when we restart it a core is generated and a segfault is logged.
### Troubleshooting
The only thing I have seen is some reference to IPv6 in the backtrace. But no idea where to start looking. I'm pretty sure we will need more information, but I don't know how to read or understand backtraces, please let me know what you see in it so I can dig deeper into this problem.
#### Reproduction
I don't know how to reproduce yet, first I have to understand the cause.
#### Debugging Data
Backtrace: ``` [New LWP 12052] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/sbc.cfg -'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007efcf3b5df69 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 #0 0x00007efcf3b5df69 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #1 0x00007efcf3b5e234 in OPENSSL_cleanup () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #2 0x00007efcf7721910 in __run_exit_handlers (status=0, listp=0x7efcf7a855d8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:83 atfct = <optimized out> onfct = <optimized out> cxafct = <optimized out> f = <optimized out> #3 0x00007efcf772196a in __GI_exit (status=<optimized out>) at exit.c:105 No locals. #4 0x000055f0a9483d54 in handle_sigs () at main.c:699 chld = 845005216 chld_status = 845005248 any_chld_stopped = 32766 memlog = 0 __func__ = "handle_sigs" #5 0x000055f0a948ea43 in main_loop () at main.c:1758 i = 8 pid = 12091 si = 0x0 si_desc = "udp receiver child=7 sock=[2602:FF37:0:1:0:0:C601:377A]:5060\000\367v\215\200\262+\363\372~\000\000\025\067\236\003\000\000\000\000\300\017H\251\360U\000\000\340\305]2\376\177", '\000' <repeats 18 times>, "\020\303]2\376\177\000\000a\202g\251\360U\000" nrprocs = 8 woneinit = 1 __func__ = "main_loop" #6 0x000055f0a9494fb9 in main (argc=13, argv=0x7ffe325dc5e8) at main.c:2646 cfg_stream = 0x55f0aa7fb010 c = -1 r = 0 tmp = 0x7ffe325dceb6 "" tmp_len = -133383536 port = 32508 proto = 845006016 options = 0x55f0a97d80e8 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 3679304743 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x0 p = 0xffffffff <error: Cannot access memory at address 0xffffffff> st = {st_dev = 19, st_ino = 14769, st_nlink = 2, st_mode = 16832, st_uid = 109, st_gid = 113, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1498523393, tv_nsec = 731915583}, st_mtim = {tv_sec = 1498533865, tv_nsec = 274210743}, st_ctim = {tv_sec = 1498533865, tv_nsec = 274210743}, __glibc_reserved = {0, 0, 0}} __func__ = "main" ```
Info locals: ``` [New LWP 12052] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/sbc.cfg -'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007efcf3b5df69 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. ```
List: ``` [New LWP 12052] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/sbc.cfg -'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007efcf3b5df69 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 ```
#### Log Messages
We have regular logging with no issues, then suddenly we get:
``` Jun 28 01:57:51 kam-1 sbc[12075]: WARNING: <script>: [DISPATCHER] - Destination down: OPTIONS sip:A.A.A.A:5060 (<null>) Jun 28 01:57:51 kam-1 sbc[12075]: WARNING: <script>: [DISPATCHER] - Destination down: OPTIONS sip:B.B.B.B:5060 (<null>) Jun 28 01:58:40 kam-1 sbc[12076]: NOTICE: <script>: Node unavailable, trying next sip:A.A.A.A:5060 - M=INVITE R=sip:+12345678900@A.A.A.A:5060 ID=334f3vqDtj Jun 28 01:58:40 kam-1 sbc[12076]: NOTICE: <script>: Forwarding request - M=INVITE R=sip:+12345678900@A.A.A.A:5060 ID=334f3vqDtj Jun 28 01:58:55 kam-1 sbc[12075]: WARNING: dialog [dlg_handlers.c:1574]: dlg_ontimeout(): timeout for dlg with CallID '00g4gNy5KZ' and tags 'FHmrD-CM3' 'SZ3a5Sg58c5pB' Jun 28 01:58:55 kam-1 sbc[12075]: WARNING: dialog [dlg_handlers.c:1574]: dlg_ontimeout(): timeout for dlg with CallID 'Kj7uLRzJfd' and tags 'wRSVjfGaT' 'yQHvjNaZ5Z3rc' Jun 28 01:58:55 kam-1 sbc[12075]: WARNING: dialog [dlg_handlers.c:1574]: dlg_ontimeout(): timeout for dlg with CallID 'S5QUIXtS6-' and tags 'FV4ySEouz' 'eS7F910ycNg8H' Jun 28 01:59:10 kam-1 sbc[12075]: ERROR: <script>: No available gateways - M=INVITE R=sip:+12345678900@B.B.B.B:5060 ID=334f3vqDtj Jun 28 02:01:45 kam-1 sbc[12080]: NOTICE: dialog [dlg_hash.c:248]: dlg_clean_run(): dialog in delete state is too old (0x7efaf3904008 ref 1) Jun 28 02:03:15 kam-1 sbc[12080]: NOTICE: dialog [dlg_hash.c:248]: dlg_clean_run(): dialog in delete state is too old (0x7efaf40f0f50 ref 1) Jun 28 02:04:45 kam-1 sbc[12080]: NOTICE: dialog [dlg_hash.c:248]: dlg_clean_run(): dialog in delete state is too old (0x7efaf4d92108 ref 1) Jun 28 02:55:59 kam-1 sbc[12071]: INFO: <core> [main.c:823]: sig_usr(): signal 15 received Jun 28 02:55:59 kam-1 sbc[12052]: NOTICE: <core> [main.c:696]: handle_sigs(): Thank you for flying kamailio!!! ```
NOTE: The process stops logging at 02:04:45, then at 02:55:59 we manually restart kamailio and a core is generated.
Also, in `dmesg` we have:
``` [138953.744199] kamailio[12052]: segfault at 7efaf32f51b8 ip 00007efcf3b5df69 sp 00007ffe325dc0c0 error 4 in libcrypto.so.1.1[7efcf39f6000+265000] ```
#### SIP Traffic
I don't have sip traffic, If required I can start capturing it.
### Additional Information
``` # kamailio -v version: kamailio 5.0.2 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 6.3.0 # ```
``` # dpkg -l | grep kamailio ii kamailio 5.0.2+stretch amd64 very fast, dynamic and configurable SIP server ii kamailio-dbg:amd64 5.0.2+stretch amd64 very fast and configurable SIP server [debug symbols] ii kamailio-extra-modules:amd64 5.0.2+stretch amd64 Extra modules for the Kamailio SIP Server ii kamailio-tls-modules:amd64 5.0.2+stretch amd64 TLS support for the Kamailio SIP server (authentication, transport) # ```
``` # dpkg -l |grep libssl ii libssl1.0.2:amd64 1.0.2l-2 amd64 Secure Sockets Layer toolkit - shared libraries ii libssl1.1:amd64 1.1.0f-3 amd64 Secure Sockets Layer toolkit - shared libraries# ```
* **Operating System**:
``` # uname -a Linux kam-1 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u1 (2017-06-18) x86_64 GNU/Linux ```
``` # lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 9.0 (stretch) Release: 9.0 Codename: stretch # ```