[SR-Users] SCSCF hangs randomly

Hi All, We currently experience issues where scscf hangs randomly throughout the day. It started hanging once in a month, the issue has gotten worse over time and now it can hang more than 5 times in a day. I have no prior kamailio experience or experience debugging c applications. Kamailio version: 4.4.0 Running ps shows scscf processes are running. Service status also shows scscf is running but it does not respond to sip requests. Testing with sipsack hangs and print timeout errors. root@NGVI-PCSCF2:/var/log # service scscf status * Status of Kamailio SIP Server: * scscf is running sipsak -vv -s sip:NGVI-PCSCF2:6060 received ICMP message received ICMP message** timeout after 500 ms** Below is the extract from log files Apr 5 12:04:46 NGVI-PCSCF2 /usr/sbin/kamailio[5102]: {1 1 ACK isbc505ramjjjr3g23cs02r53cms0uxbg2es(a)10.18.5.64} WARNING: tm [t_lookup.c:1476]: t_unref(): WARNING: script writer didn't release transaction Apr 5 12:04:46 NGVI-PCSCF2 /usr/sbin/kamailio[5104]: {1 3 REGISTER _A9LGREyW2miE7LhPJzv7g..} ERROR: ims_registrar_scscf [save.c:1034]: update_contacts(): Unable to remove contact <sip:+2347020246874@154.66.2.182:5060;rinstance=28bf37d2cce6e4a1;transport=udp> Apr 5 12:04:47 NGVI-PCSCF2 /usr/sbin/kamailio[5131]: ERROR: rr [rr_mod.c:300]: w_record_route(): Double attempt to record-route Apr 5 12:04:47 NGVI-PCSCF2 /usr/sbin/kamailio[5144]: ERROR: rr [rr_mod.c:300]: w_record_route(): Double attempt to record-route Apr 5 12:04:47 NGVI-PCSCF2 /usr/sbin/kamailio[5108]: {1 1 ACK r8ug398d118i8iaaunrouggisntkigkn(a)10.18.5.64} WARNING: tm [t_lookup.c:1476]: t_unref(): WARNING: script writer didn't release transaction Apr 5 12:04:48 NGVI-PCSCF2 /usr/sbin/kamailio[5190]: CRITICAL: <core> [pass_fd.c:275]: receive_fd(): EOF on 23 Apr 5 12:04:48 NGVI-PCSCF2 /usr/sbin/kamailio[5062]: ALERT: <core> [main.c:741]: handle_sigs(): child process 5103 exited by a signal 11 Apr 5 12:04:48 NGVI-PCSCF2 /usr/sbin/kamailio[5062]: ALERT: <core> [main.c:744]: handle_sigs(): core was generated Apr 5 12:06:12 NGVI-PCSCF2 kamailio: WARNING: <core> [daemonize.c:348]: daemonize(): pid file contains old pid, replacing pid Apr 5 12:06:13 NGVI-PCSCF2 /usr/sbin/kamailio[15060]: WARNING: interconnectroute [ported_cache.c:113]: initPortedNumberCount(): Requested records[726049]: Total allocated[871258]: Fat 20% :Total MemUsed [26 MB divided between primary[0x7f22b8667d60] and secondary[0x7f22b93b3390]] pr 5 04:38:24 NGVI-PCSCF2 /usr/sbin/kamailio[14018]: message repeated 107 times: [ ERROR: ims_usrloc_scscf [subscribe.c:430]: free_subscriber(): record not found in hash table] Apr 5 12:50:38 NGVI-PCSCF2 /usr/sbin/kamailio[2604]: WARNING: interconnectroute [icncache_eventprocessor.c:76]: icnRefreshProcess(): total size 2147483648 qm_real_used 700707576 qm_used 387064680 1st_frag 0x7fd5f6d6a690 last_frag 0x7fd676d30fd0 Apr 5 13:00:25 NGVI-PCSCF2 /usr/sbin/kamailio[2437]: ERROR: ims_usrloc_scscf [subscribe.c:430]: free_subscriber(): record not found in hash table Apr 5 13:00:26 NGVI-PCSCF2 /usr/sbin/kamailio[2437]: message repeated 125 times: [ ERROR: ims_usrloc_scscf [subscribe.c:430]: free_subscriber(): record not found in hash table] Apr 5 13:01:25 NGVI-PCSCF2 /usr/sbin/kamailio[2437]: CRITICAL: <core> [main.c:650]: sig_alarm_abort(): shutdown timeout triggered, dying... Apr 5 13:02:30 NGVI-PCSCF2 kamailio: WARNING: <core> [daemonize.c:348]: daemonize(): pid file contains old pid, replacing pid Apr 5 13:02:30 NGVI-PCSCF2 /usr/sbin/kamailio[27629]: WARNING: interconnectroute [ported_cache.c:113]: initPortedNumberCount(): Requested records[726049]: Total allocated[871258]: Fat 20% :Total MemUsed [26 MB divided between primary[0x7f3093a3fd60] and secondary[0x7f309478b390]] Thank you in advance. Regards, Solly Thoobe