Carsten,
I pulled all latest changes (including your commit 72bbf9280879475dcdbda59f00b75a274885abaa), built new RPMS using the spec file, upgraded my current packages (5.1.0) with the new ones (5.2.0) and restarted the service. I got core dump again. It seems you have to take care of AVP_Vendor_Specific_Application_Id as well.
Log file:
2018-01-19T10:53:14.082895+02:00 linux-o12d systemd-coredump[25526]: Process 25523 (kamailio) of user 479 dumped core. 2018-01-19T10:53:14.425810+02:00 linux-o12d kamailio[25613]: 78(25694) DEBUG: cdp [peermanager.c:263]: peer_timer(): peer_timer(): taking care of peers... 2018-01-19T10:53:14.425875+02:00 linux-o12d kamailio[25613]: 78(25694) DEBUG: cdp [peermanager.c:280]: peer_timer(): peer_timer(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State 0 2018-01-19T10:53:14.425892+02:00 linux-o12d kamailio[25613]: 78(25694) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Closed Event Start 2018-01-19T10:53:14.425908+02:00 linux-o12d kamailio[25613]: 78(25694) INFO: cdp [peerstatemachine.c:525]: I_Snd_Conn_Req(): I_Snd_Conn_Req(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org 2018-01-19T10:53:14.425929+02:00 linux-o12d kamailio[25613]: 78(25694) INFO: cdp [receiver.c:869]: peer_connect(): peer_connect(): Trying to connect to 10.82.10.85 port 3868 2018-01-19T10:53:14.426048+02:00 linux-o12d kamailio[25613]: 78(25694) INFO: cdp [receiver.c:937]: peer_connect(): peer_connect(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org:3868 connected 2018-01-19T10:53:14.426087+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [receiver.c:697]: receive_loop(): select_recv(): There is something on the fd exchange pipe 2018-01-19T10:53:14.426106+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [receiver.c:706]: receive_loop(): select_recv(): fd exchange pipe says fd [20] for peer 0x7fe39d89fc98:[hss.epc.mnc019.mcc425.3gppnetwork.org] 2018-01-19T10:53:14.426203+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Wait_Conn_Ack Event I_Rcv_Conn_Ack 2018-01-19T10:53:14.426230+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [diameter_msg.c:184]: AAANewMessage(): AAANewMessage: param session received null and it's a request!! 2018-01-19T10:53:14.426310+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [diameter_msg.c:81]: AAABuildMsgBuffer(): AAABuildMsgBuffer(): len=204 2018-01-19T10:53:14.426336+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [receiver.c:994]: peer_send_msg(): peer_send_msg(): Pipe push [0x7fe39d907a88] 2018-01-19T10:53:14.426353+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [receiver.c:751]: receive_loop(): select_recv(): There is something on the send pipe 2018-01-19T10:53:14.426368+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [receiver.c:764]: receive_loop(): select_recv(): Send pipe says [0x7fe39d907a88] 8 2018-01-19T10:53:14.426383+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [diameter_msg.c:410]: AAAFreeMessage(): AAAFreeMessage: Freeing message (0x7fe39d907a88) 257 2018-01-19T10:53:14.426610+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [receiver.c:574]: do_receive(): receive_loop(): [hss.epc.mnc019.mcc425.3gppnetwork.org] Recv Version 1 Length 360 2018-01-19T10:53:14.426695+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [receiver.c:1088]: receive_message(): receive_message(): [hss.epc.mnc019.mcc425.3gppnetwork.org] Recv msg 257 2018-01-19T10:53:14.426722+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Wait_I_CEA Event I_Rcv_CEA 2018-01-19T10:53:14.426739+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:698]: count_Supported_Vendor_Id_AVPS(): Found 4 Supported_Vendor AVPS76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 0 of maximum 20 2018-01-19T10:53:14.426755+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-19T10:53:14.426770+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 1 of maximum 20 2018-01-19T10:53:14.426785+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-19T10:53:14.426799+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 2 of maximum 20 2018-01-19T10:53:14.426814+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-19T10:53:14.426830+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 3 of maximum 20 2018-01-19T10:53:14.426845+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-19T10:53:14.426860+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 4 of maximum 20 2018-01-19T10:53:14.426875+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 5 of maximum 20 2018-01-19T10:53:14.426890+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-19T10:53:14.426904+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 6 of maximum 20 2018-01-19T10:53:14.426919+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-19T10:53:14.426934+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 7 of maximum 20 2018-01-19T10:53:14.426949+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-19T10:53:14.426974+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 8 of maximum 20 2018-01-19T10:53:14.426990+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-19T10:53:14.427005+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 9 of maximum 20 2018-01-19T10:53:14.427020+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 10 of maximum 20 2018-01-19T10:53:14.427035+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-19T10:53:14.427049+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 11 of maximum 20 2018-01-19T10:53:14.427064+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-19T10:53:14.427079+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 12 of maximum 20 2018-01-19T10:53:14.427093+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-19T10:53:14.427108+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 13 of maximum 20 2018-01-19T10:53:14.427122+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-19T10:53:14.427137+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 14 of maximum 20 2018-01-19T10:53:14.427152+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 15 of maximum 20 2018-01-19T10:53:14.427166+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:763]: save_peer_applications(): Found Supported Vendor for Application 1: 5535 2018-01-19T10:53:14.427181+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 16 of maximum 20 2018-01-19T10:53:14.427196+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:763]: save_peer_applications(): Found Supported Vendor for Application 1: 10415 2018-01-19T10:53:14.427210+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 17 of maximum 20 2018-01-19T10:53:14.427225+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:763]: save_peer_applications(): Found Supported Vendor for Application 1: 13019 2018-01-19T10:53:14.427244+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 18 of maximum 20 2018-01-19T10:53:14.427259+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:763]: save_peer_applications(): Found Supported Vendor for Application 1: 58637 2018-01-19T10:53:14.427274+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 19 of maximum 20 2018-01-19T10:53:14.427289+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 20 of maximum 20 2018-01-19T10:53:14.427304+02:00 linux-o12d kamailio[25613]: 76(25692) CRITICAL: <core> [core/mem/q_malloc.c:145]: qm_debug_frag(): BUG: qm: prev. fragm. tail overwritten(28af01000000, 0)[0x7fe39d908e10:0x7fe39d908e48]! Memory allocator was called from cdp: diameter_avp. c:365. Fragment marked by cdp: diameter_avp.c:142. 2018-01-19T10:53:14.480944+02:00 linux-o12d kamailio[25613]: 0(25613) ALERT: <core> [main.c:746]: handle_sigs(): child process 25692 exited by a signal 6 2018-01-19T10:53:14.481049+02:00 linux-o12d kamailio[25613]: 0(25613) ALERT: <core> [main.c:749]: handle_sigs(): core was generated 2018-01-19T10:53:14.481079+02:00 linux-o12d kamailio[25613]: 0(25613) INFO: <core> [main.c:771]: handle_sigs(): terminating due to SIGCHLD 2018-01-19T10:53:14.481115+02:00 linux-o12d kamailio[25613]: 0(25613) DEBUG: <core> [main.c:773]: handle_sigs(): terminating due to SIGCHLD 2018-01-19T10:53:14.481159+02:00 linux-o12d kamailio[25613]: 1(25617) INFO: <core> [main.c:826]: sig_usr(): signal 15 received
This time I have debug symbols:
(gdb) bt full #0 0x00007feb551c78c7 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 resultvar = 0 pid = 25344 selftid = 25344 #1 0x00007feb551c8c9a in __GI_abort () at abort.c:78 save_stage = 1 act = {__sigaction_handler = {sa_handler = 0x28af01000000, sa_sigaction = 0x28af01000000}, sa_mask = {__val = {140726850332656, 0, 140648729949568, 6765256, 140648577228624, 140648574046208, 140648567419080, 140648574276992, 21474836483, 140648574046208, 9017383233312, 4, 0, 234976, 4, 8193552}}, sa_flags = 1437865728, sa_restorer = 0x0} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x0000000000673a7b in qm_debug_frag (qm=0x7feb4c4b5000, f=0x7feb4c7bde10, file=0x7feb4be6f852 "cdp: diameter_avp.c", line=365) at core/mem/q_malloc.c:147 __FUNCTION__ = "qm_debug_frag" #3 0x00000000006768bc in qm_free (qmp=0x7feb4c4b5000, p=0x7feb4c7bde48, file=0x7feb4be6f852 "cdp: diameter_avp.c", func=0x7feb4be71338 <__FUNCTION__.7007> "AAAFreeAVP", line=365, mname=0x7feb4be6f680 "cdp") at core/mem/q_malloc.c:496 qm = 0x7feb4c4b5000 f = 0x7feb4c7bde10 size = 140726850332944 next = 0x7ffd85eca9a0 prev = 0x6756b8 <qm_malloc+1406> __FUNCTION__ = "qm_free" #4 0x0000000000680002 in qm_shm_free (qmp=0x7feb4c4b5000, p=0x7feb4c7bde48, file=0x7feb4be6f852 "cdp: diameter_avp.c", func=0x7feb4be71338 <__FUNCTION__.7007> "AAAFreeAVP", line=365, mname=0x7feb4be6f680 "cdp") at core/mem/q_malloc.c:1255 No locals. #5 0x00007feb4be4b111 in AAAFreeAVP (avp=0x7ffd85eca9e0) at diameter_avp.c:365 __FUNCTION__ = "AAAFreeAVP" #6 0x00007feb4be1cbb4 in AAAFreeAVPList (avpList=0x7ffd85ecaa40) at diameter_msg.c:396 avp_t = 0x7feb4c7bde48 avp = 0x7feb4c7bdee8 #7 0x00007feb4bdea79c in save_peer_applications (p=0x7feb4c754c98, msg=0x7feb4c7bca88) at peerstatemachine.c:785 total_cnt = 20 supported_vendor_id_avp_cnt = 4 avp = 0x7feb4c7bdbb0 avp_vendor = 0x7feb4c7bde48 avp2 = 0x0 group = {head = 0x7feb4c7bde48, tail = 0x7feb4c7bdee8} id = 16777216 vendor = 10415 __FUNCTION__ = "save_peer_applications" #8 0x00007feb4bdea810 in Process_CEA (p=0x7feb4c754c98, cea=0x7feb4c7bca88) at peerstatemachine.c:803 avp = 0x7feb4c7bd218 #9 0x00007feb4bde39e1 in sm_process (p=0x7feb4c754c98, event=I_Rcv_CEA, msg=0x7feb4c7bca88, peer_locked=0, sock=20) at peerstatemachine.c:166 result_code = 1273380608 next_event = 32747 msg_received = 0 __FUNCTION__ = "sm_process" #10 0x00007feb4be3b76e in receive_message (msg=0x7feb4c7bca88, sp=0x7feb54e43c88) at receiver.c:1128 avp1 = 0x4be66ee0 avp2 = 0x7feb4c7bdc50 __FUNCTION__ = "receive_message" #11 0x00007feb4be311ca in do_receive (sp=0x7feb54e43c88) at receiver.c:593 cnt = 340 n = 340 version = 1 dst = 0x7feb4c7bd4f4 "" dmsg = 0x7feb4c7bca88 __FUNCTION__ = "do_receive" #12 0x00007feb4be34fb1 in receive_loop (original_peer=0x7feb4c754c98) at receiver.c:800 rfds = {__fds_bits = {1048576, 0 <repeats 15 times>}} efds = {__fds_bits = {0 <repeats 16 times>}} tv = {tv_sec = 0, tv_usec = 999996} n = 1 max = 21 cnt = 1 msg = 0x0 sp = 0x7feb54e43c88 sp2 = 0x7feb54e43c88 p = 0x7feb4c754c98 fd = 20 fd_exchange_pipe_local = 16 __FUNCTION__ = "receive_loop" #13 0x00007feb4be2e6fb in receiver_process (p=0x7feb4c754c98) at receiver.c:459 ---Type <return> to continue, or q <return> to quit--- __FUNCTION__ = "receiver_process" #14 0x00007feb4bddfc4b in diameter_peer_start (blocking=0) at diameter_peer.c:289 pid = 0 k = -1 p = 0x7feb4c754c98 __FUNCTION__ = "diameter_peer_start" #15 0x00007feb4bdd1e1e in cdp_child_init (rank=0) at cdp_mod.c:243 __FUNCTION__ = "cdp_child_init" #16 0x00000000004e6312 in init_mod_child (m=0x7feb54dfc9e8, rank=0) at core/sr_module.c:939 __FUNCTION__ = "init_mod_child" #17 0x00000000004e5fb4 in init_mod_child (m=0x7feb54dfda10, rank=0) at core/sr_module.c:935 __FUNCTION__ = "init_mod_child" #18 0x00000000004e5fb4 in init_mod_child (m=0x7feb54dfddd0, rank=0) at core/sr_module.c:935 __FUNCTION__ = "init_mod_child" #19 0x00000000004e5fb4 in init_mod_child (m=0x7feb54dff050, rank=0) at core/sr_module.c:935 __FUNCTION__ = "init_mod_child" #20 0x00000000004e5fb4 in init_mod_child (m=0x7feb54dffe70, rank=0) at core/sr_module.c:935 __FUNCTION__ = "init_mod_child" #21 0x00000000004e66e6 in init_child (rank=0) at core/sr_module.c:966 No locals. #22 0x0000000000426955 in main_loop () at main.c:1708 i = 8 pid = 25334 si = 0x0 si_desc = "sctp receiver child=7 sock=192.168.151.1:5060", '\000' <repeats 19 times>, "\360\270\354\205\375\177\000\000g-\036U\353\177\000\000\000\353v\000\000\000\000\000x\261\343T\353\177\000\000 \267\354\205\375\177\000\000P\266\354\205\375\177\000\000\004\000\000\000\000\000\000\000\000PKL\353\177\000" nrprocs = 8 woneinit = 1 __FUNCTION__ = "main_loop" #23 0x000000000042d24e in main (argc=10, argv=0x7ffd85ecb9d8) at main.c:2646 cfg_stream = 0x2264010 c = -1 r = 0 tmp = 0x7ffd85eccf01 "" tmp_len = 32765 port = -2048083952 proto = 32765 options = 0x74d6d0 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2568094531 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 2 n_lst = 0x7feb55b41700 p = 0x7ffd85ecba30 "\a\317\354\205\375\177" st = {st_dev = 20, st_ino = 729, st_nlink = 2, st_mode = 16832, st_uid = 479, st_gid = 2, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1516189324, tv_nsec = 279961062}, st_mtim = {tv_sec = 1516351702, tv_nsec = 739657044}, st_ctim = {tv_sec = 1516351702, tv_nsec = 739657044}, __glibc_reserved = {0, 0, 0}} __FUNCTION__ = "main" (gdb) info locals cfg_stream = 0x2264010 c = -1 r = 0 tmp = 0x7ffd85eccf01 "" tmp_len = 32765 port = -2048083952 proto = 32765 options = 0x74d6d0 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2568094531 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 2 n_lst = 0x7feb55b41700 p = 0x7ffd85ecba30 "\a\317\354\205\375\177" st = {st_dev = 20, st_ino = 729, st_nlink = 2, st_mode = 16832, st_uid = 479, st_gid = 2, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1516189324, tv_nsec = 279961062}, st_mtim = {tv_sec = 1516351702, tv_nsec = 739657044}, st_ctim = {tv_sec = 1516351702, tv_nsec = 739657044}, __glibc_reserved = {0, 0, 0}} __FUNCTION__ = "main" (gdb) list 50 in ../sysdeps/unix/sysv/linux/raise.c
Regards.
On 18.01.2018 21:14, Carsten Bock wrote:
Hi Tsvetan,
i've just pushed a safety check to master. Can you check, if this avoids the crash? If yes, can you share the output of the log? If I am correct, it's due to a miscalculation of available apps for the peer.
Thanks, Carsten
2018-01-16 9:49 GMT+01:00 Tsvetan Filev tsvetan.filev@inno-networks.com:
Hi.
I get a coredump which is caused by bad memory handling during the diameter capability exchange process. Here is part of the log file: =================================== 2018-01-16T09:16:39.890992+02:00 linux-o12d kamailio[13298]: 87(13388) INFO: cdp [worker.c:332]: worker_process(): [0] Worker process started... 2018-01-16T09:16:40.296393+02:00 linux-o12d systemd-coredump[13196]: Process 13193 (kamailio) of user 479 dumped core. 2018-01-16T09:16:40.871483+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peermanager.c:263]: peer_timer(): peer_timer(): taking care of peers... 2018-01-16T09:16:40.871545+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peermanager.c:280]: peer_timer(): peer_timer(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State 0 2018-01-16T09:16:40.871596+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Closed Event Start 2018-01-16T09:16:40.871612+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [peerstatemachine.c:525]: I_Snd_Conn_Req(): I_Snd_Conn_Req(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org 2018-01-16T09:16:40.871636+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [receiver.c:869]: peer_connect(): peer_connect(): Trying to connect to 10.82.10.85 port 3868 2018-01-16T09:16:40.871782+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [receiver.c:937]: peer_connect(): peer_connect(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org:3868 connected 2018-01-16T09:16:40.871813+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:697]: receive_loop(): select_recv(): There is something on the fd exchange pipe 2018-01-16T09:16:40.871828+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:706]: receive_loop(): select_recv(): fd exchange pipe says fd [22] for peer 0x7f0626b91c98:[hss.epc.mnc019.mcc425.3gppnetwork.org] 2018-01-16T09:16:40.871910+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Wait_Conn_Ack Event I_Rcv_Conn_Ack 2018-01-16T09:16:40.871933+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:184]: AAANewMessage(): AAANewMessage: param session received null and it's a request!! 2018-01-16T09:16:40.872011+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:81]: AAABuildMsgBuffer(): AAABuildMsgBuffer(): len=204 2018-01-16T09:16:40.872041+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:994]: peer_send_msg(): peer_send_msg(): Pipe push [0x7f0626c02548] 2018-01-16T09:16:40.872064+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:751]: receive_loop(): select_recv(): There is something on the send pipe 2018-01-16T09:16:40.872084+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:764]: receive_loop(): select_recv(): Send pipe says [0x7f0626c02548] 8 2018-01-16T09:16:40.872104+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:410]: AAAFreeMessage(): AAAFreeMessage: Freeing message (0x7f0626c02548) 257 2018-01-16T09:16:40.872277+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:574]: do_receive(): receive_loop(): [hss.epc.mnc019.mcc425.3gppnetwork.org] Recv Version 1 Length 360 2018-01-16T09:16:40.872322+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:1088]: receive_message(): receive_message(): [hss.epc.mnc019.mcc425.3gppnetwork.org] Recv msg 257 2018-01-16T09:16:40.872345+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Wait_I_CEA Event I_Rcv_CEA 2018-01-16T09:16:40.872372+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:692]: count_Supported_Vendor_Id_AVPS(): Found 4 Supported_Vendor AVPS92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872389+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872405+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872420+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872438+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872453+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872468+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872486+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872504+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872523+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872539+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872554+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872570+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 5535 2018-01-16T09:16:40.872586+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 10415 2018-01-16T09:16:40.872601+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 13019 2018-01-16T09:16:40.872616+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 58637 2018-01-16T09:16:40.872634+02:00 linux-o12d kamailio[13298]: 92(13393) CRITICAL: <core> [core/mem/q_malloc.c:145]: qm_debug_frag(): BUG: qm: prev. fragm. tail overwritten(28af01000000, 0)[0x7f0626c038d0:0x7f0626c03908]! Memory allocator was called from cdp: diameter_avp.c:365. Fragment marked by cdp: diameter_avp.c:142. 2018-01-16T09:16:41.054292+02:00 linux-o12d kamailio[13298]: 0(13298) ALERT: <core> [main.c:746]: handle_sigs(): child process 13393 exited by a signal 6 2018-01-16T09:16:41.054403+02:00 linux-o12d kamailio[13298]: 0(13298) ALERT: <core> [main.c:749]: handle_sigs(): core was generated 2018-01-16T09:16:41.054422+02:00 linux-o12d kamailio[13298]: 0(13298) INFO: <core> [main.c:771]: handle_sigs(): terminating due to SIGCHLD 2018-01-16T09:16:41.054438+02:00 linux-o12d kamailio[13298]: 0(13298) DEBUG: <core> [main.c:773]: handle_sigs(): terminating due to SIGCHLD 2018-01-16T09:16:41.054464+02:00 linux-o12d kamailio[13298]: 2(13303) INFO: <core> [main.c:826]: sig_usr(): signal 15 received ... ===================================
Attached is wireshark trace. Here is my DiameterPeer.xml:
===================================
<?xml version="1.0" encoding="UTF-8"?>
<DiameterPeer FQDN="ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org" Realm="epc.mnc019.mcc425.3gppnetwork.org" Vendor_Id="10415" Product_Name="CDiameterPeer" AcceptUnknownPeers="1" DropUnknownOnDisconnect="1" Tc="30" Workers="4" QueueLength="8" TransactionTimeout="5" SessionsHashSize="128" DefaultAuthSessionTimeout="3600" MaxAuthSessionTimeout="3600">
<Peer FQDN="hss.epc.mnc019.mcc425.3gppnetwork.org"Realm="epc.mnc019.mcc425.3gppnetwork.org" port="3868" /> <Acceptor port="3869" bind="10.82.10.56" /> <Auth id="16777216" vendor="10415" /> <!--3GPP CxDX --> <DefaultRoute FQDN="ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org" metric="10" />
</DiameterPeer> ===================================
It looks like buffer overflow to me but I'm not sure.
kamailio version is:
kamailio -v version: kamailio 5.1.0 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 05:30:36 Jan 15 2018 with gcc 4.8.5 ===================================
OS is openSUSE Leap 42.3. Kernel: Linux linux-o12d 4.4.104-39-default #1 SMP Thu Jan 4 08:11:03 UTC 2018 (7db1912) x86_64 x86_64 x86_64 GNU/Linux
Do you have any idea what might be wrong ? It could be bad config but still it should say something in the log without a crash.
Regards.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users