Hey,
On 12.05.2011 12:37, Anton Roman wrote:
we got a core in dialog module. We are using kamailio 3.1.2. Below you can find a full backtrace from the dump and the Kamailio compilation options. Please, if you need further information don't hesitate to ask me for it. I can't precise the situation when it is generated because we have a quite high load in this server.
The call path seems to be like this:
transaction timer fires -> tm module walking through callback list finds unref_dlg() -> tm module calls unref_dlg() -> boom.
I wonder why unref_dlg() was registered as a tm callback in the first place -- the dialog module shouldn't do that. Are you using any custom modules that would possibly do such registrations?
As to the reason of the segfault, the dialog structure or hash table may already be gone when unref_dlg() is called. Can you go to stack #0 and tell us what the value of each of the following data structures is (use "p <data structure> in gdb):
*dlg d_table d_table->entries
Cheers,
--Timo
(gdb) bt full #0 unref_dlg (dlg=0x7f08a9f67da8, cnt=1) at dlg_hash.c:598 d_entry = (struct dlg_entry *) 0x7f10304b8b68 #1 0x00007f08ce92fa02 in run_trans_callbacks_internal (cb_lst=0x7f08aa203e98, type=32768, trans=0x7f08aa203e28, params=0x7fff49059a10) at t_hooks.c:290 cbp = (struct tm_callback *) 0x7f08a9f6e7e0 backup_from = (avp_list_t *) 0x8b3330 backup_to = (avp_list_t *) 0x8b3338 backup_dom_from = (avp_list_t *) 0x8b3340 backup_dom_to = (avp_list_t *) 0x8b3348 backup_uri_from = (avp_list_t *) 0x8b3320 backup_uri_to = (avp_list_t *) 0x8b3328 #2 0x00007f08ce92fc56 in run_trans_callbacks (type=32768, trans=<value optimized out>, req=0x1, rpl=0x7f10304b8b68, code=-868566200) at t_hooks.c:317 params = {req = 0x0, rpl = 0x0, param = 0x7f08a9f6e7f0, code = 0, flags = 0, branch = 0, t_rbuf = 0x0, dst = 0x0, send_buf = { s = 0x0, len = 0}} #3 0x00007f08ce915b36 in free_cell (dead_cell=0x7f08aa203e28) at h_table.c:136 b = <value optimized out> i = <value optimized out> rpl = <value optimized out> tt = <value optimized out> foo = <value optimized out> cbs = <value optimized out> ---Type <return> to continue, or q <return> to quit--- __FUNCTION__ = "free_cell" #4 0x00007f08ce9319f1 in wait_handler (ti=<value optimized out>, wait_tl=<value optimized out>, data=<value optimized out>) at timer.c:645 p_cell = (struct cell *) 0x7f08aa203e28 #5 0x0000000000513d8f in timer_main () at timer.c:894 No locals. #6 0x000000000046501b in main_loop () at main.c:1618 i = 4 pid = <value optimized out> si = (struct socket_info *) 0x0 si_desc = "udp receiver child=3 sock=XXX.XXX.XXX.XX:XXXX\000\000\000\210�\231\000\000\000\000\000\031", '\0' <repeats 15 times>, "\001\000\000\000\000\000\000\000�\215\213", '\0' <repeats 13 times>, "\004", '\0' <repeats 15 times>, "\b\236\005I�\177\000\000\227%J\000\000\000\000" #7 0x0000000000467873 in main (argc=<value optimized out>, argv=0x7fff49059e08) at main.c:2398 cfg_stream = (FILE *) 0x12e1010 c = <value optimized out> r = <value optimized out> tmp = 0x7fff4905ae90 "" tmp_len = 32520 port = <value optimized out> proto = <value optimized out> ret = <value optimized out> seed = 1235801225 ---Type <return> to continue, or q <return> to quit--- rfd = 4 debug_save = <value optimized out> debug_flag = 0 dont_fork_cnt = 0 n_lst = <value optimized out> p = <value optimized out> (gdb) (gdb) quit kamailio2:/var/kamailio# kamailio -V version: kamailio 3.1.2 (x86_64/linux) eb24c1-dirty flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: eb24c1 -dirty compiled on 09:35:52 Apr 28 2011 with gcc 4.3.2