Hi.
I get a coredump which is caused by bad memory handling during the diameter capability exchange process. Here is part of the log file: =================================== 2018-01-16T09:16:39.890992+02:00 linux-o12d kamailio[13298]: 87(13388) INFO: cdp [worker.c:332]: worker_process(): [0] Worker process started... 2018-01-16T09:16:40.296393+02:00 linux-o12d systemd-coredump[13196]: Process 13193 (kamailio) of user 479 dumped core. 2018-01-16T09:16:40.871483+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peermanager.c:263]: peer_timer(): peer_timer(): taking care of peers... 2018-01-16T09:16:40.871545+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peermanager.c:280]: peer_timer(): peer_timer(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State 0 2018-01-16T09:16:40.871596+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Closed Event Start 2018-01-16T09:16:40.871612+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [peerstatemachine.c:525]: I_Snd_Conn_Req(): I_Snd_Conn_Req(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org 2018-01-16T09:16:40.871636+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [receiver.c:869]: peer_connect(): peer_connect(): Trying to connect to 10.82.10.85 port 3868 2018-01-16T09:16:40.871782+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [receiver.c:937]: peer_connect(): peer_connect(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org:3868 connected 2018-01-16T09:16:40.871813+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:697]: receive_loop(): select_recv(): There is something on the fd exchange pipe 2018-01-16T09:16:40.871828+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:706]: receive_loop(): select_recv(): fd exchange pipe says fd [22] for peer 0x7f0626b91c98:[hss.epc.mnc019.mcc425.3gppnetwork.org] 2018-01-16T09:16:40.871910+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Wait_Conn_Ack Event I_Rcv_Conn_Ack 2018-01-16T09:16:40.871933+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:184]: AAANewMessage(): AAANewMessage: param session received null and it's a request!! 2018-01-16T09:16:40.872011+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:81]: AAABuildMsgBuffer(): AAABuildMsgBuffer(): len=204 2018-01-16T09:16:40.872041+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:994]: peer_send_msg(): peer_send_msg(): Pipe push [0x7f0626c02548] 2018-01-16T09:16:40.872064+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:751]: receive_loop(): select_recv(): There is something on the send pipe 2018-01-16T09:16:40.872084+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:764]: receive_loop(): select_recv(): Send pipe says [0x7f0626c02548] 8 2018-01-16T09:16:40.872104+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:410]: AAAFreeMessage(): AAAFreeMessage: Freeing message (0x7f0626c02548) 257 2018-01-16T09:16:40.872277+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:574]: do_receive(): receive_loop(): [hss.epc.mnc019.mcc425.3gppnetwork.org] Recv Version 1 Length 360 2018-01-16T09:16:40.872322+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:1088]: receive_message(): receive_message(): [hss.epc.mnc019.mcc425.3gppnetwork.org] Recv msg 257 2018-01-16T09:16:40.872345+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Wait_I_CEA Event I_Rcv_CEA 2018-01-16T09:16:40.872372+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:692]: count_Supported_Vendor_Id_AVPS(): Found 4 Supported_Vendor AVPS92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872389+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872405+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872420+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872438+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872453+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872468+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872486+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872504+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872523+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872539+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872554+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872570+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 5535 2018-01-16T09:16:40.872586+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 10415 2018-01-16T09:16:40.872601+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 13019 2018-01-16T09:16:40.872616+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 58637 2018-01-16T09:16:40.872634+02:00 linux-o12d kamailio[13298]: 92(13393) CRITICAL: <core> [core/mem/q_malloc.c:145]: qm_debug_frag(): BUG: qm: prev. fragm. tail overwritten(28af01000000, 0)[0x7f0626c038d0:0x7f0626c03908]! Memory allocator was called from cdp: diameter_avp.c:365. Fragment marked by cdp: diameter_avp.c:142. 2018-01-16T09:16:41.054292+02:00 linux-o12d kamailio[13298]: 0(13298) ALERT: <core> [main.c:746]: handle_sigs(): child process 13393 exited by a signal 6 2018-01-16T09:16:41.054403+02:00 linux-o12d kamailio[13298]: 0(13298) ALERT: <core> [main.c:749]: handle_sigs(): core was generated 2018-01-16T09:16:41.054422+02:00 linux-o12d kamailio[13298]: 0(13298) INFO: <core> [main.c:771]: handle_sigs(): terminating due to SIGCHLD 2018-01-16T09:16:41.054438+02:00 linux-o12d kamailio[13298]: 0(13298) DEBUG: <core> [main.c:773]: handle_sigs(): terminating due to SIGCHLD 2018-01-16T09:16:41.054464+02:00 linux-o12d kamailio[13298]: 2(13303) INFO: <core> [main.c:826]: sig_usr(): signal 15 received ... ===================================
Attached is wireshark trace. Here is my DiameterPeer.xml:
=================================== <?xml version="1.0" encoding="UTF-8"?> <DiameterPeer FQDN="ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org" Realm="epc.mnc019.mcc425.3gppnetwork.org" Vendor_Id="10415" Product_Name="CDiameterPeer" AcceptUnknownPeers="1" DropUnknownOnDisconnect="1" Tc="30" Workers="4" QueueLength="8" TransactionTimeout="5" SessionsHashSize="128" DefaultAuthSessionTimeout="3600" MaxAuthSessionTimeout="3600">
<Peer FQDN="hss.epc.mnc019.mcc425.3gppnetwork.org" Realm="epc.mnc019.mcc425.3gppnetwork.org" port="3868" /> <Acceptor port="3869" bind="10.82.10.56" /> <Auth id="16777216" vendor="10415" /> <!--3GPP CxDX --> <DefaultRoute FQDN="ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org" metric="10" /> </DiameterPeer> ===================================
It looks like buffer overflow to me but I'm not sure.
kamailio version is: =================================== kamailio -v version: kamailio 5.1.0 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 05:30:36 Jan 15 2018 with gcc 4.8.5 ===================================
OS is openSUSE Leap 42.3. Kernel: Linux linux-o12d 4.4.104-39-default #1 SMP Thu Jan 4 08:11:03 UTC 2018 (7db1912) x86_64 x86_64 x86_64 GNU/Linux
Do you have any idea what might be wrong ? It could be bad config but still it should say something in the log without a crash.
Regards.
Hey,
looks like an error with memory allocation. If you haven't done yet, it would be nice if you could create an issue on GitHub. Do you still have the coredump? If so please create an trace with gdb and attache it to the issue.
Greetings Rick
Am 16.01.2018 09:50 schrieb "Tsvetan Filev" <tsvetan.filev@inno-networks.com
:
Hi.
I get a coredump which is caused by bad memory handling during the diameter capability exchange process. Here is part of the log file: =================================== 2018-01-16T09:16:39.890992+02:00 linux-o12d kamailio[13298]: 87(13388) INFO: cdp [worker.c:332]: worker_process(): [0] Worker process started... 2018-01-16T09:16:40.296393+02:00 linux-o12d systemd-coredump[13196]: Process 13193 (kamailio) of user 479 dumped core. 2018-01-16T09:16:40.871483+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peermanager.c:263]: peer_timer(): peer_timer(): taking care of peers... 2018-01-16T09:16:40.871545+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peermanager.c:280]: peer_timer(): peer_timer(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State 0 2018-01-16T09:16:40.871596+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Closed Event Start 2018-01-16T09:16:40.871612+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [peerstatemachine.c:525]: I_Snd_Conn_Req(): I_Snd_Conn_Req(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org 2018-01-16T09:16:40.871636+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [receiver.c:869]: peer_connect(): peer_connect(): Trying to connect to 10.82.10.85 port 3868 2018-01-16T09:16:40.871782+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [receiver.c:937]: peer_connect(): peer_connect(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org:3868 connected 2018-01-16T09:16:40.871813+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:697]: receive_loop(): select_recv(): There is something on the fd exchange pipe 2018-01-16T09:16:40.871828+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:706]: receive_loop(): select_recv(): fd exchange pipe says fd [22] for peer 0x7f0626b91c98:[hss.epc.mnc019 .mcc425.3gppnetwork.org] 2018-01-16T09:16:40.871910+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Wait_Conn_Ack Event I_Rcv_Conn_Ack 2018-01-16T09:16:40.871933+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:184]: AAANewMessage(): AAANewMessage: param session received null and it's a request!! 2018-01-16T09:16:40.872011+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:81]: AAABuildMsgBuffer(): AAABuildMsgBuffer(): len=204 2018-01-16T09:16:40.872041+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:994]: peer_send_msg(): peer_send_msg(): Pipe push [0x7f0626c02548] 2018-01-16T09:16:40.872064+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:751]: receive_loop(): select_recv(): There is something on the send pipe 2018-01-16T09:16:40.872084+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:764]: receive_loop(): select_recv(): Send pipe says [0x7f0626c02548] 8 2018-01-16T09:16:40.872104+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:410]: AAAFreeMessage(): AAAFreeMessage: Freeing message (0x7f0626c02548) 257 2018-01-16T09:16:40.872277+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:574]: do_receive(): receive_loop(): [ hss.epc.mnc019.mcc425.3gppnetwork.org] Recv Version 1 Length 360 2018-01-16T09:16:40.872322+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:1088]: receive_message(): receive_message(): [ hss.epc.mnc019.mcc425.3gppnetwork.org] Recv msg 257 2018-01-16T09:16:40.872345+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Wait_I_CEA Event I_Rcv_CEA 2018-01-16T09:16:40.872372+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:692]: count_Supported_Vendor_Id_AVPS(): Found 4 Supported_Vendor AVPS92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872389+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872405+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872420+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872438+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872453+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872468+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872486+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872504+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872523+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872539+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872554+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872570+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 5535 2018-01-16T09:16:40.872586+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 10415 2018-01-16T09:16:40.872601+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 13019 2018-01-16T09:16:40.872616+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 58637 2018-01-16T09:16:40.872634+02:00 linux-o12d kamailio[13298]: 92(13393) CRITICAL: <core> [core/mem/q_malloc.c:145]: qm_debug_frag(): BUG: qm: prev. fragm. tail overwritten(28af01000000, 0)[0x7f0626c038d0:0x7f0626c03908]! Memory allocator was called from cdp: diameter_avp.c:365. Fragment marked by cdp: diameter_avp.c:142. 2018-01-16T09:16:41.054292+02:00 linux-o12d kamailio[13298]: 0(13298) ALERT: <core> [main.c:746]: handle_sigs(): child process 13393 exited by a signal 6 2018-01-16T09:16:41.054403+02:00 linux-o12d kamailio[13298]: 0(13298) ALERT: <core> [main.c:749]: handle_sigs(): core was generated 2018-01-16T09:16:41.054422+02:00 linux-o12d kamailio[13298]: 0(13298) INFO: <core> [main.c:771]: handle_sigs(): terminating due to SIGCHLD 2018-01-16T09:16:41.054438+02:00 linux-o12d kamailio[13298]: 0(13298) DEBUG: <core> [main.c:773]: handle_sigs(): terminating due to SIGCHLD 2018-01-16T09:16:41.054464+02:00 linux-o12d kamailio[13298]: 2(13303) INFO: <core> [main.c:826]: sig_usr(): signal 15 received ... ===================================
Attached is wireshark trace. Here is my DiameterPeer.xml:
=================================== <?xml version="1.0" encoding="UTF-8"?> <DiameterPeer FQDN="ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org" Realm="epc.mnc019.mcc425.3gppnetwork.org" Vendor_Id="10415" Product_Name="CDiameterPeer" AcceptUnknownPeers="1" DropUnknownOnDisconnect="1" Tc="30" Workers="4" QueueLength="8" TransactionTimeout="5" SessionsHashSize="128" DefaultAuthSessionTimeout="3600" MaxAuthSessionTimeout="3600">
<Peer FQDN="hss.epc.mnc019.mcc425.3gppnetwork.org" Realm=" epc.mnc019.mcc425.3gppnetwork.org" port="3868" /> <Acceptor port="3869" bind="10.82.10.56" /> <Auth id="16777216" vendor="10415" /> <!--3GPP CxDX --> <DefaultRoute FQDN="ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org" metric="10" /> </DiameterPeer> ===================================
It looks like buffer overflow to me but I'm not sure.
kamailio version is: =================================== kamailio -v version: kamailio 5.1.0 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 05:30:36 Jan 15 2018 with gcc 4.8.5 ===================================
OS is openSUSE Leap 42.3. Kernel: Linux linux-o12d 4.4.104-39-default #1 SMP Thu Jan 4 08:11:03 UTC 2018 (7db1912) x86_64 x86_64 x86_64 GNU/Linux
Do you have any idea what might be wrong ? It could be bad config but still it should say something in the log without a crash.
Regards.
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Rick.
Yes I have coredumps and backtraces but I installed the rpms from here https://www.kamailio.org/wiki/packages/rpms (http://download.opensuse.org/repositories/home:/kamailio:/v5.1.x-rpms/openSU... ) so I'm missing the debug symbols. Can I find the kamailio debug rpm (kamailio-debuginfo-5.1.0-21.7.x86_64) somewhere ?
Once I have the symbols I will create an issue on GitHub.
Here is what I get for now:
# gdb kamailio core.kamailio.479.e6ada9851c9848a98237890cfddfafe3.6560.1516177422000000
GNU gdb (GDB; openSUSE Leap 42.3) 8.0.1 Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-suse-linux". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://bugs.opensuse.org/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from kamailio...(no debugging symbols found)...done. [New LWP 6560] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamai'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f637dfb38c7 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 55 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. Missing separate debuginfos, use: zypper install kamailio-debuginfo-5.1.0-21.7.x86_64 (gdb) bt full #0 0x00007f637dfb38c7 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 resultvar = 0 pid = 6560 selftid = 6560 #1 0x00007f637dfb4c9a in __GI_abort () at abort.c:78 save_stage = 1 act = {__sigaction_handler = {sa_handler = 0x28af01000000, sa_sigaction = 0x28af01000000}, sa_mask = {__val = {140724448160064, 0, 140065300084096, 6721723, 140065147406864, 140065144188928, 140065137572536, 140065144419712, 21474836483, 140065144188928, 9019276028016, 4, 0, 234976, 4, 8177296}}, sa_flags = 2123556608, sa_restorer = 0x0} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x000000000066906e in qm_debug_frag () No symbol table info available. #3 0x000000000066beaf in qm_free () No symbol table info available. #4 0x00000000006755f5 in qm_shm_free () No symbol table info available. #5 0x00007f6374c38a1d in AAAFreeAVP () from /usr/lib64/kamailio/modules/cdp.so No symbol table info available. #6 0x00007f6374c144ac in AAAFreeAVPList () from /usr/lib64/kamailio/modules/cdp.so No symbol table info available. #7 0x00007f6374beb0f9 in save_peer_applications () from /usr/lib64/kamailio/modules/cdp.so No symbol table info available. #8 0x00007f6374beb16d in Process_CEA () from /usr/lib64/kamailio/modules/cdp.so No symbol table info available. #9 0x00007f6374be4a22 in sm_process () from /usr/lib64/kamailio/modules/cdp.so No symbol table info available. #10 0x00007f6374bd15d6 in receive_message () from /usr/lib64/kamailio/modules/cdp.so No symbol table info available. #11 0x00007f6374bc7032 in do_receive () from /usr/lib64/kamailio/modules/cdp.so No symbol table info available. #12 0x00007f6374bcae19 in receive_loop () from /usr/lib64/kamailio/modules/cdp.so No symbol table info available. #13 0x00007f6374bc4563 in receiver_process () from /usr/lib64/kamailio/modules/cdp.so No symbol table info available. #14 0x00007f6374be0c8c in diameter_peer_start () from /usr/lib64/kamailio/modules/cdp.so No symbol table info available. #15 0x00007f6374bd2e5f in cdp_child_init () from /usr/lib64/kamailio/modules/cdp.so No symbol table info available. #16 0x000000000054b705 in init_mod_child () No symbol table info available. #17 0x000000000054b422 in init_mod_child () No symbol table info available. #18 0x000000000054b422 in init_mod_child () No symbol table info available. #19 0x000000000054b422 in init_mod_child () No symbol table info available. #20 0x000000000054b422 in init_mod_child () No symbol table info available. #21 0x000000000054ba3d in init_child () No symbol table info available. #22 0x0000000000426925 in main_loop () No symbol table info available. #23 0x000000000042d21e in main () No symbol table info available. (gdb) info locals save_stage = 1 \act = {__sigaction_handler = {sa_handler = 0x28af01000000, sa_sigaction = 0x28af01000000}, sa_mask = {__val = {140724448160064, 0, 140065300084096, 6721723, 140065147406864, 140065144188928, 140065137572536, 140065144419712, 21474836483, 140065144188928, 9019276028016, 4, 0, 234976, 4, 8177296}}, sa_flags = 2123556608, sa_restorer = 0x0} sigs = {__val = {32, 0 <repeats 15 times>}} (gdb) list 50 in ../sysdeps/unix/sysv/linux/raise.c
Regards.
On 16.01.2018 19:51, Rick Barenthin wrote:
Hey,
looks like an error with memory allocation. If you haven't done yet, it would be nice if you could create an issue on GitHub. Do you still have the coredump? If so please create an trace with gdb and attache it to the issue.
Greetings Rick
Am 16.01.2018 09:50 schrieb "Tsvetan Filev" <tsvetan.filev@inno-networks.com mailto:tsvetan.filev@inno-networks.com>:
Hi. I get a coredump which is caused by bad memory handling during the diameter capability exchange process. Here is part of the log file: =================================== 2018-01-16T09:16:39.890992+02:00 linux-o12d kamailio[13298]: 87(13388) INFO: cdp [worker.c:332]: worker_process(): [0] Worker process started... 2018-01-16T09:16:40.296393+02:00 linux-o12d systemd-coredump[13196]: Process 13193 (kamailio) of user 479 dumped core. 2018-01-16T09:16:40.871483+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peermanager.c:263]: peer_timer(): peer_timer(): taking care of peers... 2018-01-16T09:16:40.871545+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peermanager.c:280]: peer_timer(): peer_timer(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org <http://hss.epc.mnc019.mcc425.3gppnetwork.org> State 0 2018-01-16T09:16:40.871596+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org <http://hss.epc.mnc019.mcc425.3gppnetwork.org> State Closed Event Start 2018-01-16T09:16:40.871612+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [peerstatemachine.c:525]: I_Snd_Conn_Req(): I_Snd_Conn_Req(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org <http://hss.epc.mnc019.mcc425.3gppnetwork.org> 2018-01-16T09:16:40.871636+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [receiver.c:869]: peer_connect(): peer_connect(): Trying to connect to 10.82.10.85 port 3868 2018-01-16T09:16:40.871782+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [receiver.c:937]: peer_connect(): peer_connect(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org:3868 <http://hss.epc.mnc019.mcc425.3gppnetwork.org:3868> connected 2018-01-16T09:16:40.871813+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:697]: receive_loop(): select_recv(): There is something on the fd exchange pipe 2018-01-16T09:16:40.871828+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:706]: receive_loop(): select_recv(): fd exchange pipe says fd [22] for peer 0x7f0626b91c98:[hss.epc.mnc019.mcc425.3gppnetwork.org <http://hss.epc.mnc019.mcc425.3gppnetwork.org>] 2018-01-16T09:16:40.871910+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org <http://hss.epc.mnc019.mcc425.3gppnetwork.org> State Wait_Conn_Ack Event I_Rcv_Conn_Ack 2018-01-16T09:16:40.871933+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:184]: AAANewMessage(): AAANewMessage: param session received null and it's a request!! 2018-01-16T09:16:40.872011+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:81]: AAABuildMsgBuffer(): AAABuildMsgBuffer(): len=204 2018-01-16T09:16:40.872041+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:994]: peer_send_msg(): peer_send_msg(): Pipe push [0x7f0626c02548] 2018-01-16T09:16:40.872064+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:751]: receive_loop(): select_recv(): There is something on the send pipe 2018-01-16T09:16:40.872084+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:764]: receive_loop(): select_recv(): Send pipe says [0x7f0626c02548] 8 2018-01-16T09:16:40.872104+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:410]: AAAFreeMessage(): AAAFreeMessage: Freeing message (0x7f0626c02548) 257 2018-01-16T09:16:40.872277+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:574]: do_receive(): receive_loop(): [hss.epc.mnc019.mcc425.3gppnetwork.org <http://hss.epc.mnc019.mcc425.3gppnetwork.org>] Recv Version 1 Length 360 2018-01-16T09:16:40.872322+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:1088]: receive_message(): receive_message(): [hss.epc.mnc019.mcc425.3gppnetwork.org <http://hss.epc.mnc019.mcc425.3gppnetwork.org>] Recv msg 257 2018-01-16T09:16:40.872345+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org <http://hss.epc.mnc019.mcc425.3gppnetwork.org> State Wait_I_CEA Event I_Rcv_CEA 2018-01-16T09:16:40.872372+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:692]: count_Supported_Vendor_Id_AVPS(): Found 4 Supported_Vendor AVPS92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872389+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872405+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872420+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872438+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872453+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872468+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872486+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872504+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872523+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872539+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872554+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872570+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 5535 2018-01-16T09:16:40.872586+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 10415 2018-01-16T09:16:40.872601+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 13019 2018-01-16T09:16:40.872616+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 58637 2018-01-16T09:16:40.872634+02:00 linux-o12d kamailio[13298]: 92(13393) CRITICAL: <core> [core/mem/q_malloc.c:145]: qm_debug_frag(): BUG: qm: prev. fragm. tail overwritten(28af01000000, 0)[0x7f0626c038d0:0x7f0626c03908]! Memory allocator was called from cdp: diameter_avp.c:365. Fragment marked by cdp: diameter_avp.c:142. 2018-01-16T09:16:41.054292+02:00 linux-o12d kamailio[13298]: 0(13298) ALERT: <core> [main.c:746]: handle_sigs(): child process 13393 exited by a signal 6 2018-01-16T09:16:41.054403+02:00 linux-o12d kamailio[13298]: 0(13298) ALERT: <core> [main.c:749]: handle_sigs(): core was generated 2018-01-16T09:16:41.054422+02:00 linux-o12d kamailio[13298]: 0(13298) INFO: <core> [main.c:771]: handle_sigs(): terminating due to SIGCHLD 2018-01-16T09:16:41.054438+02:00 linux-o12d kamailio[13298]: 0(13298) DEBUG: <core> [main.c:773]: handle_sigs(): terminating due to SIGCHLD 2018-01-16T09:16:41.054464+02:00 linux-o12d kamailio[13298]: 2(13303) INFO: <core> [main.c:826]: sig_usr(): signal 15 received ... =================================== Attached is wireshark trace. Here is my DiameterPeer.xml: =================================== <?xml version="1.0" encoding="UTF-8"?> <DiameterPeer FQDN="ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org <http://ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org>" Realm="epc.mnc019.mcc425.3gppnetwork.org <http://epc.mnc019.mcc425.3gppnetwork.org>" Vendor_Id="10415" Product_Name="CDiameterPeer" AcceptUnknownPeers="1" DropUnknownOnDisconnect="1" Tc="30" Workers="4" QueueLength="8" TransactionTimeout="5" SessionsHashSize="128" DefaultAuthSessionTimeout="3600" MaxAuthSessionTimeout="3600"> <Peer FQDN="hss.epc.mnc019.mcc425.3gppnetwork.org <http://hss.epc.mnc019.mcc425.3gppnetwork.org>" Realm="epc.mnc019.mcc425.3gppnetwork.org <http://epc.mnc019.mcc425.3gppnetwork.org>" port="3868" /> <Acceptor port="3869" bind="10.82.10.56" /> <Auth id="16777216" vendor="10415" /> <!--3GPP CxDX --> <DefaultRoute FQDN="ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org <http://ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org>" metric="10" /> </DiameterPeer> =================================== It looks like buffer overflow to me but I'm not sure. kamailio version is: =================================== kamailio -v version: kamailio 5.1.0 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 05:30:36 Jan 15 2018 with gcc 4.8.5 =================================== OS is openSUSE Leap 42.3. Kernel: Linux linux-o12d 4.4.104-39-default #1 SMP Thu Jan 4 08:11:03 UTC 2018 (7db1912) x86_64 x86_64 x86_64 GNU/Linux Do you have any idea what might be wrong ? It could be bad config but still it should say something in the log without a crash. Regards. _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi,
The log clearly states, what went wrong and where. There was insufficient memory allocated at some point. I'm traveling right now, but I will check either tonight from the Hotel or tomorrow.
Thanks for reporting, Carsten
Am 17.01.2018 09:51 schrieb "Tsvetan Filev" <tsvetan.filev@inno-networks.com
:
Hi Rick.
Yes I have coredumps and backtraces but I installed the rpms from here https://www.kamailio.org/wiki/packages/rpms (http://download.opensuse.org/ repositories/home:/kamailio:/v5.1.x-rpms/openSUSE_Leap_42.3/ ) so I'm missing the debug symbols. Can I find the kamailio debug rpm (kamailio-debuginfo-5.1.0-21.7.x86_64) somewhere ?
Once I have the symbols I will create an issue on GitHub.
Here is what I get for now:
# gdb kamailio core.kamailio.479.e6ada9851c9848a98237890cfddfaf e3.6560.1516177422000000
GNU gdb (GDB; openSUSE Leap 42.3) 8.0.1 Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl. html http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-suse-linux". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://bugs.opensuse.org/ http://bugs.opensuse.org/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/ http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from kamailio...(no debugging symbols found)...done. [New LWP 6560] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamai'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f637dfb38c7 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 55 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. Missing separate debuginfos, use: zypper install kamailio-debuginfo-5.1.0-21.7.x86_64 (gdb) bt full #0 0x00007f637dfb38c7 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 resultvar = 0 pid = 6560 selftid = 6560 #1 0x00007f637dfb4c9a in __GI_abort () at abort.c:78 save_stage = 1 act = {__sigaction_handler = {sa_handler = 0x28af01000000, sa_sigaction = 0x28af01000000}, sa_mask = {__val = {140724448160064, 0, 140065300084096, 6721723, 140065147406864, 140065144188928, 140065137572536, 140065144419712, 21474836483, 140065144188928, 9019276028016, 4, 0, 234976, 4, 8177296}}, sa_flags = 2123556608, sa_restorer = 0x0} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x000000000066906e in qm_debug_frag () No symbol table info available. #3 0x000000000066beaf in qm_free () No symbol table info available. #4 0x00000000006755f5 in qm_shm_free () No symbol table info available. #5 0x00007f6374c38a1d in AAAFreeAVP () from /usr/lib64/kamailio/modules/ cdp.so No symbol table info available. #6 0x00007f6374c144ac in AAAFreeAVPList () from /usr/lib64/kamailio/modules/cdp.so No symbol table info available. #7 0x00007f6374beb0f9 in save_peer_applications () from /usr/lib64/kamailio/modules/cdp.so No symbol table info available. #8 0x00007f6374beb16d in Process_CEA () from /usr/lib64/kamailio/modules/ cdp.so No symbol table info available. #9 0x00007f6374be4a22 in sm_process () from /usr/lib64/kamailio/modules/ cdp.so No symbol table info available. #10 0x00007f6374bd15d6 in receive_message () from /usr/lib64/kamailio/modules/cdp.so No symbol table info available. #11 0x00007f6374bc7032 in do_receive () from /usr/lib64/kamailio/modules/ cdp.so No symbol table info available. #12 0x00007f6374bcae19 in receive_loop () from /usr/lib64/kamailio/modules/ cdp.so No symbol table info available. #13 0x00007f6374bc4563 in receiver_process () from /usr/lib64/kamailio/modules/cdp.so No symbol table info available. #14 0x00007f6374be0c8c in diameter_peer_start () from /usr/lib64/kamailio/modules/cdp.so No symbol table info available. #15 0x00007f6374bd2e5f in cdp_child_init () from /usr/lib64/kamailio/modules/cdp.so No symbol table info available. #16 0x000000000054b705 in init_mod_child () No symbol table info available. #17 0x000000000054b422 in init_mod_child () No symbol table info available. #18 0x000000000054b422 in init_mod_child () No symbol table info available. #19 0x000000000054b422 in init_mod_child () No symbol table info available. #20 0x000000000054b422 in init_mod_child () No symbol table info available. #21 0x000000000054ba3d in init_child () No symbol table info available. #22 0x0000000000426925 in main_loop () No symbol table info available. #23 0x000000000042d21e in main () No symbol table info available. (gdb) info locals save_stage = 1 \act = {__sigaction_handler = {sa_handler = 0x28af01000000, sa_sigaction = 0x28af01000000}, sa_mask = {__val = {140724448160064, 0, 140065300084096, 6721723, 140065147406864, 140065144188928, 140065137572536, 140065144419712, 21474836483, 140065144188928, 9019276028016, 4, 0, 234976, 4, 8177296}}, sa_flags = 2123556608, sa_restorer = 0x0} sigs = {__val = {32, 0 <repeats 15 times>}} (gdb) list 50 in ../sysdeps/unix/sysv/linux/raise.c
Regards.
On 16.01.2018 19:51, Rick Barenthin wrote:
Hey,
looks like an error with memory allocation. If you haven't done yet, it would be nice if you could create an issue on GitHub. Do you still have the coredump? If so please create an trace with gdb and attache it to the issue.
Greetings Rick
Am 16.01.2018 09:50 schrieb "Tsvetan Filev" <tsvetan.filev@inno-networks. com>:
Hi.
I get a coredump which is caused by bad memory handling during the diameter capability exchange process. Here is part of the log file: =================================== 2018-01-16T09:16:39.890992+02:00 linux-o12d kamailio[13298]: 87(13388) INFO: cdp [worker.c:332]: worker_process(): [0] Worker process started... 2018-01-16T09:16:40.296393+02:00 linux-o12d systemd-coredump[13196]: Process 13193 (kamailio) of user 479 dumped core. 2018-01-16T09:16:40.871483+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peermanager.c:263]: peer_timer(): peer_timer(): taking care of peers... 2018-01-16T09:16:40.871545+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peermanager.c:280]: peer_timer(): peer_timer(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State 0 2018-01-16T09:16:40.871596+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Closed Event Start 2018-01-16T09:16:40.871612+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [peerstatemachine.c:525]: I_Snd_Conn_Req(): I_Snd_Conn_Req(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org 2018-01-16T09:16:40.871636+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [receiver.c:869]: peer_connect(): peer_connect(): Trying to connect to 10.82.10.85 port 3868 2018-01-16T09:16:40.871782+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [receiver.c:937]: peer_connect(): peer_connect(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org:3868 connected 2018-01-16T09:16:40.871813+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:697]: receive_loop(): select_recv(): There is something on the fd exchange pipe 2018-01-16T09:16:40.871828+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:706]: receive_loop(): select_recv(): fd exchange pipe says fd [22] for peer 0x7f0626b91c98:[hss.epc.mnc019 .mcc425.3gppnetwork.org] 2018-01-16T09:16:40.871910+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Wait_Conn_Ack Event I_Rcv_Conn_Ack 2018-01-16T09:16:40.871933+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:184]: AAANewMessage(): AAANewMessage: param session received null and it's a request!! 2018-01-16T09:16:40.872011+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:81]: AAABuildMsgBuffer(): AAABuildMsgBuffer(): len=204 2018-01-16T09:16:40.872041+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:994]: peer_send_msg(): peer_send_msg(): Pipe push [0x7f0626c02548] 2018-01-16T09:16:40.872064+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:751]: receive_loop(): select_recv(): There is something on the send pipe 2018-01-16T09:16:40.872084+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:764]: receive_loop(): select_recv(): Send pipe says [0x7f0626c02548] 8 2018-01-16T09:16:40.872104+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:410]: AAAFreeMessage(): AAAFreeMessage: Freeing message (0x7f0626c02548) 257 2018-01-16T09:16:40.872277+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:574]: do_receive(): receive_loop(): [ hss.epc.mnc019.mcc425.3gppnetwork.org] Recv Version 1 Length 360 2018-01-16T09:16:40.872322+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:1088]: receive_message(): receive_message(): [ hss.epc.mnc019.mcc425.3gppnetwork.org] Recv msg 257 2018-01-16T09:16:40.872345+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Wait_I_CEA Event I_Rcv_CEA 2018-01-16T09:16:40.872372+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:692]: count_Supported_Vendor_Id_AVPS(): Found 4 Supported_Vendor AVPS92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872389+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872405+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872420+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872438+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872453+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872468+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872486+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872504+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872523+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872539+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872554+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872570+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 5535 2018-01-16T09:16:40.872586+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 10415 2018-01-16T09:16:40.872601+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 13019 2018-01-16T09:16:40.872616+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 58637 2018-01-16T09:16:40.872634+02:00 linux-o12d kamailio[13298]: 92(13393) CRITICAL: <core> [core/mem/q_malloc.c:145]: qm_debug_frag(): BUG: qm: prev. fragm. tail overwritten(28af01000000, 0)[0x7f0626c038d0:0x7f0626c03908]! Memory allocator was called from cdp: diameter_avp.c:365. Fragment marked by cdp: diameter_avp.c:142. 2018-01-16T09:16:41.054292 <54292>+02:00 linux-o12d kamailio[13298]: 0(13298) ALERT: <core> [main.c:746]: handle_sigs(): child process 13393 exited by a signal 6 2018-01-16T09:16:41.054403 <54403>+02:00 linux-o12d kamailio[13298]: 0(13298) ALERT: <core> [main.c:749]: handle_sigs(): core was generated 2018-01-16T09:16:41.054422 <54422>+02:00 linux-o12d kamailio[13298]: 0(13298) INFO: <core> [main.c:771]: handle_sigs(): terminating due to SIGCHLD 2018-01-16T09:16:41.054438 <54438>+02:00 linux-o12d kamailio[13298]: 0(13298) DEBUG: <core> [main.c:773]: handle_sigs(): terminating due to SIGCHLD 2018-01-16T09:16:41.054464 <54464>+02:00 linux-o12d kamailio[13298]: 2(13303) INFO: <core> [main.c:826]: sig_usr(): signal 15 received ... ===================================
Attached is wireshark trace. Here is my DiameterPeer.xml:
===================================
<?xml version="1.0" encoding="UTF-8"?>
<DiameterPeer FQDN="ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org" Realm="epc.mnc019.mcc425.3gppnetwork.org" Vendor_Id="10415" Product_Name="CDiameterPeer" AcceptUnknownPeers="1" DropUnknownOnDisconnect="1" Tc="30" Workers="4" QueueLength="8" TransactionTimeout="5" SessionsHashSize="128" DefaultAuthSessionTimeout="3600" MaxAuthSessionTimeout="3600">
<Peer FQDN="hss.epc.mnc019.mcc425.3gppnetwork.org" Realm="epc.mnc019.mcc425.3gppnetwork.org" port="3868" /> <Acceptor port="3869" bind="10.82.10.56" /> <Auth id="16777216" vendor="10415" /> <!--3GPP CxDX --> <DefaultRoute FQDN="ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org" metric="10" />
</DiameterPeer> ===================================
It looks like buffer overflow to me but I'm not sure.
kamailio version is:
kamailio -v version: kamailio 5.1.0 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 05:30:36 Jan 15 2018 with gcc 4.8.5 ===================================
OS is openSUSE Leap 42.3. Kernel: Linux linux-o12d 4.4.104-39-default #1 SMP Thu Jan 4 08:11:03 UTC 2018 (7db1912) x86_64 x86_64 x86_64 GNU/Linux
Do you have any idea what might be wrong ? It could be bad config but still it should say something in the log without a crash.
Regards.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Tsvetan,
i've just pushed a safety check to master. Can you check, if this avoids the crash? If yes, can you share the output of the log? If I am correct, it's due to a miscalculation of available apps for the peer.
Thanks, Carsten
2018-01-16 9:49 GMT+01:00 Tsvetan Filev tsvetan.filev@inno-networks.com:
Hi.
I get a coredump which is caused by bad memory handling during the diameter capability exchange process. Here is part of the log file: =================================== 2018-01-16T09:16:39.890992+02:00 linux-o12d kamailio[13298]: 87(13388) INFO: cdp [worker.c:332]: worker_process(): [0] Worker process started... 2018-01-16T09:16:40.296393+02:00 linux-o12d systemd-coredump[13196]: Process 13193 (kamailio) of user 479 dumped core. 2018-01-16T09:16:40.871483+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peermanager.c:263]: peer_timer(): peer_timer(): taking care of peers... 2018-01-16T09:16:40.871545+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peermanager.c:280]: peer_timer(): peer_timer(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State 0 2018-01-16T09:16:40.871596+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Closed Event Start 2018-01-16T09:16:40.871612+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [peerstatemachine.c:525]: I_Snd_Conn_Req(): I_Snd_Conn_Req(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org 2018-01-16T09:16:40.871636+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [receiver.c:869]: peer_connect(): peer_connect(): Trying to connect to 10.82.10.85 port 3868 2018-01-16T09:16:40.871782+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [receiver.c:937]: peer_connect(): peer_connect(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org:3868 connected 2018-01-16T09:16:40.871813+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:697]: receive_loop(): select_recv(): There is something on the fd exchange pipe 2018-01-16T09:16:40.871828+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:706]: receive_loop(): select_recv(): fd exchange pipe says fd [22] for peer 0x7f0626b91c98:[hss.epc.mnc019.mcc425.3gppnetwork.org] 2018-01-16T09:16:40.871910+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Wait_Conn_Ack Event I_Rcv_Conn_Ack 2018-01-16T09:16:40.871933+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:184]: AAANewMessage(): AAANewMessage: param session received null and it's a request!! 2018-01-16T09:16:40.872011+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:81]: AAABuildMsgBuffer(): AAABuildMsgBuffer(): len=204 2018-01-16T09:16:40.872041+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:994]: peer_send_msg(): peer_send_msg(): Pipe push [0x7f0626c02548] 2018-01-16T09:16:40.872064+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:751]: receive_loop(): select_recv(): There is something on the send pipe 2018-01-16T09:16:40.872084+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:764]: receive_loop(): select_recv(): Send pipe says [0x7f0626c02548] 8 2018-01-16T09:16:40.872104+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:410]: AAAFreeMessage(): AAAFreeMessage: Freeing message (0x7f0626c02548) 257 2018-01-16T09:16:40.872277+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:574]: do_receive(): receive_loop(): [hss.epc.mnc019.mcc425.3gppnetwork.org] Recv Version 1 Length 360 2018-01-16T09:16:40.872322+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:1088]: receive_message(): receive_message(): [hss.epc.mnc019.mcc425.3gppnetwork.org] Recv msg 257 2018-01-16T09:16:40.872345+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Wait_I_CEA Event I_Rcv_CEA 2018-01-16T09:16:40.872372+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:692]: count_Supported_Vendor_Id_AVPS(): Found 4 Supported_Vendor AVPS92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872389+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872405+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872420+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872438+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872453+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872468+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872486+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872504+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872523+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872539+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872554+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872570+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 5535 2018-01-16T09:16:40.872586+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 10415 2018-01-16T09:16:40.872601+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 13019 2018-01-16T09:16:40.872616+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 58637 2018-01-16T09:16:40.872634+02:00 linux-o12d kamailio[13298]: 92(13393) CRITICAL: <core> [core/mem/q_malloc.c:145]: qm_debug_frag(): BUG: qm: prev. fragm. tail overwritten(28af01000000, 0)[0x7f0626c038d0:0x7f0626c03908]! Memory allocator was called from cdp: diameter_avp.c:365. Fragment marked by cdp: diameter_avp.c:142. 2018-01-16T09:16:41.054292+02:00 linux-o12d kamailio[13298]: 0(13298) ALERT: <core> [main.c:746]: handle_sigs(): child process 13393 exited by a signal 6 2018-01-16T09:16:41.054403+02:00 linux-o12d kamailio[13298]: 0(13298) ALERT: <core> [main.c:749]: handle_sigs(): core was generated 2018-01-16T09:16:41.054422+02:00 linux-o12d kamailio[13298]: 0(13298) INFO: <core> [main.c:771]: handle_sigs(): terminating due to SIGCHLD 2018-01-16T09:16:41.054438+02:00 linux-o12d kamailio[13298]: 0(13298) DEBUG: <core> [main.c:773]: handle_sigs(): terminating due to SIGCHLD 2018-01-16T09:16:41.054464+02:00 linux-o12d kamailio[13298]: 2(13303) INFO: <core> [main.c:826]: sig_usr(): signal 15 received ... ===================================
Attached is wireshark trace. Here is my DiameterPeer.xml:
===================================
<?xml version="1.0" encoding="UTF-8"?>
<DiameterPeer FQDN="ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org" Realm="epc.mnc019.mcc425.3gppnetwork.org" Vendor_Id="10415" Product_Name="CDiameterPeer" AcceptUnknownPeers="1" DropUnknownOnDisconnect="1" Tc="30" Workers="4" QueueLength="8" TransactionTimeout="5" SessionsHashSize="128" DefaultAuthSessionTimeout="3600" MaxAuthSessionTimeout="3600">
<Peer FQDN="hss.epc.mnc019.mcc425.3gppnetwork.org"Realm="epc.mnc019.mcc425.3gppnetwork.org" port="3868" /> <Acceptor port="3869" bind="10.82.10.56" /> <Auth id="16777216" vendor="10415" /> <!--3GPP CxDX --> <DefaultRoute FQDN="ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org" metric="10" />
</DiameterPeer> ===================================
It looks like buffer overflow to me but I'm not sure.
kamailio version is:
kamailio -v version: kamailio 5.1.0 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 05:30:36 Jan 15 2018 with gcc 4.8.5 ===================================
OS is openSUSE Leap 42.3. Kernel: Linux linux-o12d 4.4.104-39-default #1 SMP Thu Jan 4 08:11:03 UTC 2018 (7db1912) x86_64 x86_64 x86_64 GNU/Linux
Do you have any idea what might be wrong ? It could be bad config but still it should say something in the log without a crash.
Regards.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Carsten,
I pulled all latest changes (including your commit 72bbf9280879475dcdbda59f00b75a274885abaa), built new RPMS using the spec file, upgraded my current packages (5.1.0) with the new ones (5.2.0) and restarted the service. I got core dump again. It seems you have to take care of AVP_Vendor_Specific_Application_Id as well.
Log file:
2018-01-19T10:53:14.082895+02:00 linux-o12d systemd-coredump[25526]: Process 25523 (kamailio) of user 479 dumped core. 2018-01-19T10:53:14.425810+02:00 linux-o12d kamailio[25613]: 78(25694) DEBUG: cdp [peermanager.c:263]: peer_timer(): peer_timer(): taking care of peers... 2018-01-19T10:53:14.425875+02:00 linux-o12d kamailio[25613]: 78(25694) DEBUG: cdp [peermanager.c:280]: peer_timer(): peer_timer(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State 0 2018-01-19T10:53:14.425892+02:00 linux-o12d kamailio[25613]: 78(25694) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Closed Event Start 2018-01-19T10:53:14.425908+02:00 linux-o12d kamailio[25613]: 78(25694) INFO: cdp [peerstatemachine.c:525]: I_Snd_Conn_Req(): I_Snd_Conn_Req(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org 2018-01-19T10:53:14.425929+02:00 linux-o12d kamailio[25613]: 78(25694) INFO: cdp [receiver.c:869]: peer_connect(): peer_connect(): Trying to connect to 10.82.10.85 port 3868 2018-01-19T10:53:14.426048+02:00 linux-o12d kamailio[25613]: 78(25694) INFO: cdp [receiver.c:937]: peer_connect(): peer_connect(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org:3868 connected 2018-01-19T10:53:14.426087+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [receiver.c:697]: receive_loop(): select_recv(): There is something on the fd exchange pipe 2018-01-19T10:53:14.426106+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [receiver.c:706]: receive_loop(): select_recv(): fd exchange pipe says fd [20] for peer 0x7fe39d89fc98:[hss.epc.mnc019.mcc425.3gppnetwork.org] 2018-01-19T10:53:14.426203+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Wait_Conn_Ack Event I_Rcv_Conn_Ack 2018-01-19T10:53:14.426230+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [diameter_msg.c:184]: AAANewMessage(): AAANewMessage: param session received null and it's a request!! 2018-01-19T10:53:14.426310+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [diameter_msg.c:81]: AAABuildMsgBuffer(): AAABuildMsgBuffer(): len=204 2018-01-19T10:53:14.426336+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [receiver.c:994]: peer_send_msg(): peer_send_msg(): Pipe push [0x7fe39d907a88] 2018-01-19T10:53:14.426353+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [receiver.c:751]: receive_loop(): select_recv(): There is something on the send pipe 2018-01-19T10:53:14.426368+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [receiver.c:764]: receive_loop(): select_recv(): Send pipe says [0x7fe39d907a88] 8 2018-01-19T10:53:14.426383+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [diameter_msg.c:410]: AAAFreeMessage(): AAAFreeMessage: Freeing message (0x7fe39d907a88) 257 2018-01-19T10:53:14.426610+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [receiver.c:574]: do_receive(): receive_loop(): [hss.epc.mnc019.mcc425.3gppnetwork.org] Recv Version 1 Length 360 2018-01-19T10:53:14.426695+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [receiver.c:1088]: receive_message(): receive_message(): [hss.epc.mnc019.mcc425.3gppnetwork.org] Recv msg 257 2018-01-19T10:53:14.426722+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Wait_I_CEA Event I_Rcv_CEA 2018-01-19T10:53:14.426739+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:698]: count_Supported_Vendor_Id_AVPS(): Found 4 Supported_Vendor AVPS76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 0 of maximum 20 2018-01-19T10:53:14.426755+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-19T10:53:14.426770+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 1 of maximum 20 2018-01-19T10:53:14.426785+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-19T10:53:14.426799+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 2 of maximum 20 2018-01-19T10:53:14.426814+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-19T10:53:14.426830+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 3 of maximum 20 2018-01-19T10:53:14.426845+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-19T10:53:14.426860+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 4 of maximum 20 2018-01-19T10:53:14.426875+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 5 of maximum 20 2018-01-19T10:53:14.426890+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-19T10:53:14.426904+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 6 of maximum 20 2018-01-19T10:53:14.426919+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-19T10:53:14.426934+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 7 of maximum 20 2018-01-19T10:53:14.426949+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-19T10:53:14.426974+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 8 of maximum 20 2018-01-19T10:53:14.426990+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-19T10:53:14.427005+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 9 of maximum 20 2018-01-19T10:53:14.427020+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 10 of maximum 20 2018-01-19T10:53:14.427035+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-19T10:53:14.427049+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 11 of maximum 20 2018-01-19T10:53:14.427064+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-19T10:53:14.427079+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 12 of maximum 20 2018-01-19T10:53:14.427093+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-19T10:53:14.427108+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 13 of maximum 20 2018-01-19T10:53:14.427122+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:750]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-19T10:53:14.427137+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 14 of maximum 20 2018-01-19T10:53:14.427152+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 15 of maximum 20 2018-01-19T10:53:14.427166+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:763]: save_peer_applications(): Found Supported Vendor for Application 1: 5535 2018-01-19T10:53:14.427181+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 16 of maximum 20 2018-01-19T10:53:14.427196+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:763]: save_peer_applications(): Found Supported Vendor for Application 1: 10415 2018-01-19T10:53:14.427210+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 17 of maximum 20 2018-01-19T10:53:14.427225+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:763]: save_peer_applications(): Found Supported Vendor for Application 1: 13019 2018-01-19T10:53:14.427244+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 18 of maximum 20 2018-01-19T10:53:14.427259+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:763]: save_peer_applications(): Found Supported Vendor for Application 1: 58637 2018-01-19T10:53:14.427274+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 19 of maximum 20 2018-01-19T10:53:14.427289+02:00 linux-o12d kamailio[25613]: 76(25692) DEBUG: cdp [peerstatemachine.c:681]: add_peer_application(): Application 20 of maximum 20 2018-01-19T10:53:14.427304+02:00 linux-o12d kamailio[25613]: 76(25692) CRITICAL: <core> [core/mem/q_malloc.c:145]: qm_debug_frag(): BUG: qm: prev. fragm. tail overwritten(28af01000000, 0)[0x7fe39d908e10:0x7fe39d908e48]! Memory allocator was called from cdp: diameter_avp. c:365. Fragment marked by cdp: diameter_avp.c:142. 2018-01-19T10:53:14.480944+02:00 linux-o12d kamailio[25613]: 0(25613) ALERT: <core> [main.c:746]: handle_sigs(): child process 25692 exited by a signal 6 2018-01-19T10:53:14.481049+02:00 linux-o12d kamailio[25613]: 0(25613) ALERT: <core> [main.c:749]: handle_sigs(): core was generated 2018-01-19T10:53:14.481079+02:00 linux-o12d kamailio[25613]: 0(25613) INFO: <core> [main.c:771]: handle_sigs(): terminating due to SIGCHLD 2018-01-19T10:53:14.481115+02:00 linux-o12d kamailio[25613]: 0(25613) DEBUG: <core> [main.c:773]: handle_sigs(): terminating due to SIGCHLD 2018-01-19T10:53:14.481159+02:00 linux-o12d kamailio[25613]: 1(25617) INFO: <core> [main.c:826]: sig_usr(): signal 15 received
This time I have debug symbols:
(gdb) bt full #0 0x00007feb551c78c7 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 resultvar = 0 pid = 25344 selftid = 25344 #1 0x00007feb551c8c9a in __GI_abort () at abort.c:78 save_stage = 1 act = {__sigaction_handler = {sa_handler = 0x28af01000000, sa_sigaction = 0x28af01000000}, sa_mask = {__val = {140726850332656, 0, 140648729949568, 6765256, 140648577228624, 140648574046208, 140648567419080, 140648574276992, 21474836483, 140648574046208, 9017383233312, 4, 0, 234976, 4, 8193552}}, sa_flags = 1437865728, sa_restorer = 0x0} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x0000000000673a7b in qm_debug_frag (qm=0x7feb4c4b5000, f=0x7feb4c7bde10, file=0x7feb4be6f852 "cdp: diameter_avp.c", line=365) at core/mem/q_malloc.c:147 __FUNCTION__ = "qm_debug_frag" #3 0x00000000006768bc in qm_free (qmp=0x7feb4c4b5000, p=0x7feb4c7bde48, file=0x7feb4be6f852 "cdp: diameter_avp.c", func=0x7feb4be71338 <__FUNCTION__.7007> "AAAFreeAVP", line=365, mname=0x7feb4be6f680 "cdp") at core/mem/q_malloc.c:496 qm = 0x7feb4c4b5000 f = 0x7feb4c7bde10 size = 140726850332944 next = 0x7ffd85eca9a0 prev = 0x6756b8 <qm_malloc+1406> __FUNCTION__ = "qm_free" #4 0x0000000000680002 in qm_shm_free (qmp=0x7feb4c4b5000, p=0x7feb4c7bde48, file=0x7feb4be6f852 "cdp: diameter_avp.c", func=0x7feb4be71338 <__FUNCTION__.7007> "AAAFreeAVP", line=365, mname=0x7feb4be6f680 "cdp") at core/mem/q_malloc.c:1255 No locals. #5 0x00007feb4be4b111 in AAAFreeAVP (avp=0x7ffd85eca9e0) at diameter_avp.c:365 __FUNCTION__ = "AAAFreeAVP" #6 0x00007feb4be1cbb4 in AAAFreeAVPList (avpList=0x7ffd85ecaa40) at diameter_msg.c:396 avp_t = 0x7feb4c7bde48 avp = 0x7feb4c7bdee8 #7 0x00007feb4bdea79c in save_peer_applications (p=0x7feb4c754c98, msg=0x7feb4c7bca88) at peerstatemachine.c:785 total_cnt = 20 supported_vendor_id_avp_cnt = 4 avp = 0x7feb4c7bdbb0 avp_vendor = 0x7feb4c7bde48 avp2 = 0x0 group = {head = 0x7feb4c7bde48, tail = 0x7feb4c7bdee8} id = 16777216 vendor = 10415 __FUNCTION__ = "save_peer_applications" #8 0x00007feb4bdea810 in Process_CEA (p=0x7feb4c754c98, cea=0x7feb4c7bca88) at peerstatemachine.c:803 avp = 0x7feb4c7bd218 #9 0x00007feb4bde39e1 in sm_process (p=0x7feb4c754c98, event=I_Rcv_CEA, msg=0x7feb4c7bca88, peer_locked=0, sock=20) at peerstatemachine.c:166 result_code = 1273380608 next_event = 32747 msg_received = 0 __FUNCTION__ = "sm_process" #10 0x00007feb4be3b76e in receive_message (msg=0x7feb4c7bca88, sp=0x7feb54e43c88) at receiver.c:1128 avp1 = 0x4be66ee0 avp2 = 0x7feb4c7bdc50 __FUNCTION__ = "receive_message" #11 0x00007feb4be311ca in do_receive (sp=0x7feb54e43c88) at receiver.c:593 cnt = 340 n = 340 version = 1 dst = 0x7feb4c7bd4f4 "" dmsg = 0x7feb4c7bca88 __FUNCTION__ = "do_receive" #12 0x00007feb4be34fb1 in receive_loop (original_peer=0x7feb4c754c98) at receiver.c:800 rfds = {__fds_bits = {1048576, 0 <repeats 15 times>}} efds = {__fds_bits = {0 <repeats 16 times>}} tv = {tv_sec = 0, tv_usec = 999996} n = 1 max = 21 cnt = 1 msg = 0x0 sp = 0x7feb54e43c88 sp2 = 0x7feb54e43c88 p = 0x7feb4c754c98 fd = 20 fd_exchange_pipe_local = 16 __FUNCTION__ = "receive_loop" #13 0x00007feb4be2e6fb in receiver_process (p=0x7feb4c754c98) at receiver.c:459 ---Type <return> to continue, or q <return> to quit--- __FUNCTION__ = "receiver_process" #14 0x00007feb4bddfc4b in diameter_peer_start (blocking=0) at diameter_peer.c:289 pid = 0 k = -1 p = 0x7feb4c754c98 __FUNCTION__ = "diameter_peer_start" #15 0x00007feb4bdd1e1e in cdp_child_init (rank=0) at cdp_mod.c:243 __FUNCTION__ = "cdp_child_init" #16 0x00000000004e6312 in init_mod_child (m=0x7feb54dfc9e8, rank=0) at core/sr_module.c:939 __FUNCTION__ = "init_mod_child" #17 0x00000000004e5fb4 in init_mod_child (m=0x7feb54dfda10, rank=0) at core/sr_module.c:935 __FUNCTION__ = "init_mod_child" #18 0x00000000004e5fb4 in init_mod_child (m=0x7feb54dfddd0, rank=0) at core/sr_module.c:935 __FUNCTION__ = "init_mod_child" #19 0x00000000004e5fb4 in init_mod_child (m=0x7feb54dff050, rank=0) at core/sr_module.c:935 __FUNCTION__ = "init_mod_child" #20 0x00000000004e5fb4 in init_mod_child (m=0x7feb54dffe70, rank=0) at core/sr_module.c:935 __FUNCTION__ = "init_mod_child" #21 0x00000000004e66e6 in init_child (rank=0) at core/sr_module.c:966 No locals. #22 0x0000000000426955 in main_loop () at main.c:1708 i = 8 pid = 25334 si = 0x0 si_desc = "sctp receiver child=7 sock=192.168.151.1:5060", '\000' <repeats 19 times>, "\360\270\354\205\375\177\000\000g-\036U\353\177\000\000\000\353v\000\000\000\000\000x\261\343T\353\177\000\000 \267\354\205\375\177\000\000P\266\354\205\375\177\000\000\004\000\000\000\000\000\000\000\000PKL\353\177\000" nrprocs = 8 woneinit = 1 __FUNCTION__ = "main_loop" #23 0x000000000042d24e in main (argc=10, argv=0x7ffd85ecb9d8) at main.c:2646 cfg_stream = 0x2264010 c = -1 r = 0 tmp = 0x7ffd85eccf01 "" tmp_len = 32765 port = -2048083952 proto = 32765 options = 0x74d6d0 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2568094531 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 2 n_lst = 0x7feb55b41700 p = 0x7ffd85ecba30 "\a\317\354\205\375\177" st = {st_dev = 20, st_ino = 729, st_nlink = 2, st_mode = 16832, st_uid = 479, st_gid = 2, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1516189324, tv_nsec = 279961062}, st_mtim = {tv_sec = 1516351702, tv_nsec = 739657044}, st_ctim = {tv_sec = 1516351702, tv_nsec = 739657044}, __glibc_reserved = {0, 0, 0}} __FUNCTION__ = "main" (gdb) info locals cfg_stream = 0x2264010 c = -1 r = 0 tmp = 0x7ffd85eccf01 "" tmp_len = 32765 port = -2048083952 proto = 32765 options = 0x74d6d0 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2568094531 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 2 n_lst = 0x7feb55b41700 p = 0x7ffd85ecba30 "\a\317\354\205\375\177" st = {st_dev = 20, st_ino = 729, st_nlink = 2, st_mode = 16832, st_uid = 479, st_gid = 2, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1516189324, tv_nsec = 279961062}, st_mtim = {tv_sec = 1516351702, tv_nsec = 739657044}, st_ctim = {tv_sec = 1516351702, tv_nsec = 739657044}, __glibc_reserved = {0, 0, 0}} __FUNCTION__ = "main" (gdb) list 50 in ../sysdeps/unix/sysv/linux/raise.c
Regards.
On 18.01.2018 21:14, Carsten Bock wrote:
Hi Tsvetan,
i've just pushed a safety check to master. Can you check, if this avoids the crash? If yes, can you share the output of the log? If I am correct, it's due to a miscalculation of available apps for the peer.
Thanks, Carsten
2018-01-16 9:49 GMT+01:00 Tsvetan Filev tsvetan.filev@inno-networks.com:
Hi.
I get a coredump which is caused by bad memory handling during the diameter capability exchange process. Here is part of the log file: =================================== 2018-01-16T09:16:39.890992+02:00 linux-o12d kamailio[13298]: 87(13388) INFO: cdp [worker.c:332]: worker_process(): [0] Worker process started... 2018-01-16T09:16:40.296393+02:00 linux-o12d systemd-coredump[13196]: Process 13193 (kamailio) of user 479 dumped core. 2018-01-16T09:16:40.871483+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peermanager.c:263]: peer_timer(): peer_timer(): taking care of peers... 2018-01-16T09:16:40.871545+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peermanager.c:280]: peer_timer(): peer_timer(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State 0 2018-01-16T09:16:40.871596+02:00 linux-o12d kamailio[13298]: 94(13395) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Closed Event Start 2018-01-16T09:16:40.871612+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [peerstatemachine.c:525]: I_Snd_Conn_Req(): I_Snd_Conn_Req(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org 2018-01-16T09:16:40.871636+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [receiver.c:869]: peer_connect(): peer_connect(): Trying to connect to 10.82.10.85 port 3868 2018-01-16T09:16:40.871782+02:00 linux-o12d kamailio[13298]: 94(13395) INFO: cdp [receiver.c:937]: peer_connect(): peer_connect(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org:3868 connected 2018-01-16T09:16:40.871813+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:697]: receive_loop(): select_recv(): There is something on the fd exchange pipe 2018-01-16T09:16:40.871828+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:706]: receive_loop(): select_recv(): fd exchange pipe says fd [22] for peer 0x7f0626b91c98:[hss.epc.mnc019.mcc425.3gppnetwork.org] 2018-01-16T09:16:40.871910+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Wait_Conn_Ack Event I_Rcv_Conn_Ack 2018-01-16T09:16:40.871933+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:184]: AAANewMessage(): AAANewMessage: param session received null and it's a request!! 2018-01-16T09:16:40.872011+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:81]: AAABuildMsgBuffer(): AAABuildMsgBuffer(): len=204 2018-01-16T09:16:40.872041+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:994]: peer_send_msg(): peer_send_msg(): Pipe push [0x7f0626c02548] 2018-01-16T09:16:40.872064+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:751]: receive_loop(): select_recv(): There is something on the send pipe 2018-01-16T09:16:40.872084+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:764]: receive_loop(): select_recv(): Send pipe says [0x7f0626c02548] 8 2018-01-16T09:16:40.872104+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [diameter_msg.c:410]: AAAFreeMessage(): AAAFreeMessage: Freeing message (0x7f0626c02548) 257 2018-01-16T09:16:40.872277+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:574]: do_receive(): receive_loop(): [hss.epc.mnc019.mcc425.3gppnetwork.org] Recv Version 1 Length 360 2018-01-16T09:16:40.872322+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [receiver.c:1088]: receive_message(): receive_message(): [hss.epc.mnc019.mcc425.3gppnetwork.org] Recv msg 257 2018-01-16T09:16:40.872345+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org State Wait_I_CEA Event I_Rcv_CEA 2018-01-16T09:16:40.872372+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:692]: count_Supported_Vendor_Id_AVPS(): Found 4 Supported_Vendor AVPS92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872389+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872405+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872420+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872438+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872453+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872468+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872486+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872504+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 5535 2018-01-16T09:16:40.872523+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 10415 2018-01-16T09:16:40.872539+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 13019 2018-01-16T09:16:40.872554+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found Supported Vendor for Application 0: 58637 2018-01-16T09:16:40.872570+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 5535 2018-01-16T09:16:40.872586+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 10415 2018-01-16T09:16:40.872601+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 13019 2018-01-16T09:16:40.872616+02:00 linux-o12d kamailio[13298]: 92(13393) DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found Supported Vendor for Application 1: 58637 2018-01-16T09:16:40.872634+02:00 linux-o12d kamailio[13298]: 92(13393) CRITICAL: <core> [core/mem/q_malloc.c:145]: qm_debug_frag(): BUG: qm: prev. fragm. tail overwritten(28af01000000, 0)[0x7f0626c038d0:0x7f0626c03908]! Memory allocator was called from cdp: diameter_avp.c:365. Fragment marked by cdp: diameter_avp.c:142. 2018-01-16T09:16:41.054292+02:00 linux-o12d kamailio[13298]: 0(13298) ALERT: <core> [main.c:746]: handle_sigs(): child process 13393 exited by a signal 6 2018-01-16T09:16:41.054403+02:00 linux-o12d kamailio[13298]: 0(13298) ALERT: <core> [main.c:749]: handle_sigs(): core was generated 2018-01-16T09:16:41.054422+02:00 linux-o12d kamailio[13298]: 0(13298) INFO: <core> [main.c:771]: handle_sigs(): terminating due to SIGCHLD 2018-01-16T09:16:41.054438+02:00 linux-o12d kamailio[13298]: 0(13298) DEBUG: <core> [main.c:773]: handle_sigs(): terminating due to SIGCHLD 2018-01-16T09:16:41.054464+02:00 linux-o12d kamailio[13298]: 2(13303) INFO: <core> [main.c:826]: sig_usr(): signal 15 received ... ===================================
Attached is wireshark trace. Here is my DiameterPeer.xml:
===================================
<?xml version="1.0" encoding="UTF-8"?>
<DiameterPeer FQDN="ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org" Realm="epc.mnc019.mcc425.3gppnetwork.org" Vendor_Id="10415" Product_Name="CDiameterPeer" AcceptUnknownPeers="1" DropUnknownOnDisconnect="1" Tc="30" Workers="4" QueueLength="8" TransactionTimeout="5" SessionsHashSize="128" DefaultAuthSessionTimeout="3600" MaxAuthSessionTimeout="3600">
<Peer FQDN="hss.epc.mnc019.mcc425.3gppnetwork.org"Realm="epc.mnc019.mcc425.3gppnetwork.org" port="3868" /> <Acceptor port="3869" bind="10.82.10.56" /> <Auth id="16777216" vendor="10415" /> <!--3GPP CxDX --> <DefaultRoute FQDN="ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org" metric="10" />
</DiameterPeer> ===================================
It looks like buffer overflow to me but I'm not sure.
kamailio version is:
kamailio -v version: kamailio 5.1.0 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 05:30:36 Jan 15 2018 with gcc 4.8.5 ===================================
OS is openSUSE Leap 42.3. Kernel: Linux linux-o12d 4.4.104-39-default #1 SMP Thu Jan 4 08:11:03 UTC 2018 (7db1912) x86_64 x86_64 x86_64 GNU/Linux
Do you have any idea what might be wrong ? It could be bad config but still it should say something in the log without a crash.
Regards.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-
Carsten Bock -
Rick Barenthin -
Tsvetan Filev