SCSCF hangs randomly - sr-users

5 Apr 2023

Hi All,

We currently experience issues where scscf hangs randomly throughout the day. It started
hanging once in a month, the issue has gotten worse over time and now it can hang more
than 5 times in a day.

I have no prior kamailio experience or experience debugging c applications.

Kamailio version: 4.4.0

Running ps shows scscf processes are running.
Service status also shows scscf is running but it does not respond to sip requests.

Testing with sipsack hangs and print timeout errors.

root@NGVI-PCSCF2:/var/log # service scscf status
 * Status of Kamailio SIP Server:                                                         

 * scscf is running

sipsak -vv -s sip:NGVI-PCSCF2:6060

received ICMP message
received ICMP message** timeout after 500 ms**

Below is the extract from log files

Apr  5 12:04:46 NGVI-PCSCF2 /usr/sbin/kamailio[5102]: {1 1 ACK
isbc505ramjjjr3g23cs02r53cms0uxbg2es(a)10.18.5.64} WARNING: tm [t_lookup.c:1476]: t_unref():
WARNING: script writer didn't release transaction
Apr  5 12:04:46 NGVI-PCSCF2 /usr/sbin/kamailio[5104]: {1 3 REGISTER
_A9LGREyW2miE7LhPJzv7g..} ERROR: ims_registrar_scscf [save.c:1034]: update_contacts():
Unable to remove contact
<sip:+2347020246874@154.66.2.182:5060;rinstance=28bf37d2cce6e4a1;transport=udp>
Apr  5 12:04:47 NGVI-PCSCF2 /usr/sbin/kamailio[5131]: ERROR: rr [rr_mod.c:300]:
w_record_route(): Double attempt to record-route
Apr  5 12:04:47 NGVI-PCSCF2 /usr/sbin/kamailio[5144]: ERROR: rr [rr_mod.c:300]:
w_record_route(): Double attempt to record-route
Apr  5 12:04:47 NGVI-PCSCF2 /usr/sbin/kamailio[5108]: {1 1 ACK
r8ug398d118i8iaaunrouggisntkigkn(a)10.18.5.64} WARNING: tm [t_lookup.c:1476]: t_unref():
WARNING: script writer didn't release transaction
Apr  5 12:04:48 NGVI-PCSCF2 /usr/sbin/kamailio[5190]: CRITICAL: <core>
[pass_fd.c:275]: receive_fd(): EOF on 23
Apr  5 12:04:48 NGVI-PCSCF2 /usr/sbin/kamailio[5062]: ALERT: <core> [main.c:741]:
handle_sigs(): child process 5103 exited by a signal 11
Apr  5 12:04:48 NGVI-PCSCF2 /usr/sbin/kamailio[5062]: ALERT: <core> [main.c:744]:
handle_sigs(): core was generated
Apr  5 12:06:12 NGVI-PCSCF2 kamailio: WARNING: <core> [daemonize.c:348]:
daemonize(): pid file contains old pid, replacing pid
Apr  5 12:06:13 NGVI-PCSCF2 /usr/sbin/kamailio[15060]: WARNING: interconnectroute
[ported_cache.c:113]: initPortedNumberCount(): Requested records[726049]: Total
allocated[871258]: Fat 20% :Total MemUsed [26 MB divided between primary[0x7f22b8667d60]
and secondary[0x7f22b93b3390]]

pr  5 04:38:24 NGVI-PCSCF2 /usr/sbin/kamailio[14018]: message repeated 107 times: [ ERROR:
ims_usrloc_scscf [subscribe.c:430]: free_subscriber(): record not found in hash table]
Apr  5 12:50:38 NGVI-PCSCF2 /usr/sbin/kamailio[2604]: WARNING: interconnectroute
[icncache_eventprocessor.c:76]: icnRefreshProcess(): total size 2147483648 qm_real_used
700707576 qm_used 387064680 1st_frag 0x7fd5f6d6a690 last_frag 0x7fd676d30fd0
Apr  5 13:00:25 NGVI-PCSCF2 /usr/sbin/kamailio[2437]: ERROR: ims_usrloc_scscf
[subscribe.c:430]: free_subscriber(): record not found in hash table
Apr  5 13:00:26 NGVI-PCSCF2 /usr/sbin/kamailio[2437]: message repeated 125 times: [ ERROR:
ims_usrloc_scscf [subscribe.c:430]: free_subscriber(): record not found in hash table]
Apr  5 13:01:25 NGVI-PCSCF2 /usr/sbin/kamailio[2437]: CRITICAL: <core> [main.c:650]:
sig_alarm_abort(): shutdown timeout triggered, dying...
Apr  5 13:02:30 NGVI-PCSCF2 kamailio: WARNING: <core> [daemonize.c:348]:
daemonize(): pid file contains old pid, replacing pid
Apr  5 13:02:30 NGVI-PCSCF2 /usr/sbin/kamailio[27629]: WARNING: interconnectroute
[ported_cache.c:113]: initPortedNumberCount(): Requested records[726049]: Total
allocated[871258]: Fat 20% :Total MemUsed [26 MB divided between primary[0x7f3093a3fd60]
and secondary[0x7f309478b390]]

Thank you in advance.

Regards,
Solly Thoobe