We have a SER 0.9.4-rc3 (i386/linux)) system running on RedHat ES4 that crashes about once every three days. I've done a cvs update, performed a re-make of all modules as well as just mysql.so and reinitialized the mysql database using the script provided. Nothing seems to fix the problem. Using gdb to examine a core dump seems to indicate SER fails when receiving REGISTER messages but I cannot figure out why. In each of the three core dumps the REGISTER message is from a different client.
Below is the gdb output from the latest core dump. Can anyone shed some light on what might be happening?
Thanks,Steve
----- start gdb output ----
[blairs@voicesrvr tmp]$ sudo gdb /usr/local/sbin/ser /tmp/core.18868 GNU gdb Red Hat Linux (6.3.0.0-0.31rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1".
Core was generated by `/usr/local/sbin/ser -P /var/run/ser.pid -dddd -E -w /tmp -f /usr/local/etc/ser/'. Program terminated with signal 6, Aborted. Reading symbols from /lib/libdl.so.2...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libresolv.so.2...done. Loaded symbols for /lib/libresolv.so.2 Reading symbols from /lib/tls/libc.so.6...done. Loaded symbols for /lib/tls/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /usr/local/lib/ser/modules/sl.so...done. Loaded symbols for /usr/local/lib/ser/modules/sl.so Reading symbols from /usr/local/lib/ser/modules/tm.so...done. Loaded symbols for /usr/local/lib/ser/modules/tm.so Reading symbols from /usr/local/lib/ser/modules/acc.so...done. Loaded symbols for /usr/local/lib/ser/modules/acc.so Reading symbols from /usr/local/lib/ser/modules/rr.so...done. Loaded symbols for /usr/local/lib/ser/modules/rr.so Reading symbols from /usr/local/lib/ser/modules/maxfwd.so...done. Loaded symbols for /usr/local/lib/ser/modules/maxfwd.so Reading symbols from /usr/local/lib/ser/modules/mysql.so...done. Loaded symbols for /usr/local/lib/ser/modules/mysql.so Reading symbols from /usr/lib/mysql/libmysqlclient.so.14...done. Loaded symbols for /usr/lib/mysql/libmysqlclient.so.14 Reading symbols from /usr/lib/libz.so.1...done. Loaded symbols for /usr/lib/libz.so.1 Reading symbols from /lib/libcrypt.so.1...done. Loaded symbols for /lib/libcrypt.so.1 Reading symbols from /lib/libnsl.so.1...done. Loaded symbols for /lib/libnsl.so.1 Reading symbols from /lib/tls/libm.so.6...done. Loaded symbols for /lib/tls/libm.so.6 Reading symbols from /usr/local/lib/ser/modules/usrloc.so...done. Loaded symbols for /usr/local/lib/ser/modules/usrloc.so Reading symbols from /usr/local/lib/ser/modules/registrar.so...done. Loaded symbols for /usr/local/lib/ser/modules/registrar.so Reading symbols from /usr/local/lib/ser/modules/auth.so...done. Loaded symbols for /usr/local/lib/ser/modules/auth.so Reading symbols from /usr/local/lib/ser/modules/auth_db.so...done. Loaded symbols for /usr/local/lib/ser/modules/auth_db.so Reading symbols from /usr/local/lib/ser/modules/textops.so...done. Loaded symbols for /usr/local/lib/ser/modules/textops.so Reading symbols from /usr/local/lib/ser/modules/uri.so...done. Loaded symbols for /usr/local/lib/ser/modules/uri.so Reading symbols from /usr/local/lib/ser/modules/uri_db.so...done. Loaded symbols for /usr/local/lib/ser/modules/uri_db.so Reading symbols from /usr/local/lib/ser/modules/print.so...done. Loaded symbols for /usr/local/lib/ser/modules/print.so Reading symbols from /usr/local/lib/ser/modules/speeddial.so...done. Loaded symbols for /usr/local/lib/ser/modules/speeddial.so Reading symbols from /usr/local/lib/ser/modules/options.so...done. Loaded symbols for /usr/local/lib/ser/modules/options.so Reading symbols from /usr/local/lib/ser/modules/avpops.so...done. Loaded symbols for /usr/local/lib/ser/modules/avpops.so Reading symbols from /usr/local/lib/ser/modules/permissions.so...done. Loaded symbols for /usr/local/lib/ser/modules/permissions.so Reading symbols from /usr/local/lib/ser/modules/flatstore.so...done. Loaded symbols for /usr/local/lib/ser/modules/flatstore.so Reading symbols from /usr/local/lib/ser/modules/group.so...done. Loaded symbols for /usr/local/lib/ser/modules/group.so Reading symbols from /usr/local/lib/ser/modules/xlog.so...done. Loaded symbols for /usr/local/lib/ser/modules/xlog.so Reading symbols from /lib/libnss_files.so.2...done. Loaded symbols for /lib/libnss_files.so.2 Reading symbols from /lib/libnss_dns.so.2...done. Loaded symbols for /lib/libnss_dns.so.2 #0 0x006857a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 (gdb) bt #0 0x006857a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x006c57d5 in raise () from /lib/tls/libc.so.6 #2 0x006c7149 in abort () from /lib/tls/libc.so.6 #3 0x006bedb1 in __assert_fail () from /lib/tls/libc.so.6 #4 0x003907c8 in t_retransmit_reply (t=0xb6189db8) at t_reply.c:855 #5 0x00388103 in t_newtran (p_msg=0x813afc8) at t_lookup.c:1096 #6 0x003800ba in t_relay_to (p_msg=0x813afc8, proxy=0x8116858, proto=1, replicate=1) at t_funcs.c:224 #7 0x00382ad3 in t_replicate (p_msg=0x0, proxy=0x0, proto=0) at t_fwd.c:513 #8 0x0038b09e in w_t_replicate (p_msg=0x6, proxy=0x0, _foo=0x8116a98 "5060") at tm.c:839 #9 0x0804fc81 in do_action (a=0x8116b28, msg=0x813afc8) at action.c:610 #10 0x0805099d in run_actions (a=0x8116b28, msg=0x813afc8) at action.c:718 #11 0x0804ecd3 in do_action (a=0x8116b58, msg=0x813afc8) at action.c:603 #12 0x0805099d in run_actions (a=0x8116b58, msg=0x813afc8) at action.c:718 #13 0x0804ecd3 in do_action (a=0x8116b88, msg=0x813afc8) at action.c:603 #14 0x0805099d in run_actions (a=0x8116b88, msg=0x813afc8) at action.c:718 #15 0x0804ecd3 in do_action (a=0x8116be8, msg=0x813afc8) at action.c:603 #16 0x0805099d in run_actions (a=0x8116be8, msg=0x813afc8) at action.c:718 #17 0x0804ecd3 in do_action (a=0x8128af8, msg=0x813afc8) at action.c:603 #18 0x0805099d in run_actions (a=0x8128af8, msg=0x813afc8) at action.c:718 #19 0x0806cfd2 in receive_msg ( buf=0x80d61e0 "REGISTER sip:net.isc.upenn.edu SIP/2.0\r\nVia: SIP/2.0/UDP 128.91.56.10:5060;branch=z9hG4bK52d3bc34\r\nFrom: sip:68007@net.isc.upenn.edu\r\nTo: sip:68007@net.isc.upenn.edu\r\nCall-ID: 00042744-ab3d0002-14a230"..., len=579, rcv_info=0xbff43360) at receive.c:165 #20 0x0808421c in udp_rcv_loop () at udp_server.c:472 #21 0x0805cd0f in main_loop () at main.c:1056 #22 0x0805e36b in main (argc=9, argv=0xbff43524) at main.c:1592
--- end gdb output ----
Steve,
On Wed, 10 Aug 2005, Steve Blair wrote:
We have a SER 0.9.4-rc3 (i386/linux)) system running on RedHat ES4 that crashes about once every three days. I've done a cvs update, performed a re-make of all modules as well as just mysql.so and reinitialized the mysql database using the script provided. Nothing seems to fix the problem. Using gdb to examine a core dump seems to indicate SER fails when receiving REGISTER messages but I cannot figure out why. In each of the three core dumps the REGISTER message is from a different client.
I experience similar problems with rel_0_9_0 SER on RedHat 9. It could work for a week without problems, then it could crash 2 times within a day on reception of REGISTER requests from the same agents that worked fine before. The server load is very low, (one to ten registered clients; one to six concurrent calls).
I thought my problems could be caused by oracle module I use or by some modifications to SER sources, but looks like they are very similar to yours. Do you use any non-standard modules or SER patches as I do?
A bit more details to compare: NAT-Pass evaluation (proprietary NAT traversal solution) is installed on the same PC with SER. Some requests go directly to SER, but some go via NAT-Pass.
P.S. I successfully stress-tested my SER with hundreds of thousands REGISTER requests via sipsak. So, I suspect the problem could be in a particular request field formatting or in particular sequence of REGISTERs.
P.P.S. I have no time right now to investigate the problem in deep. But certainly I'll try to catch it in near future.
On Fri, 12 Aug 2005, Dmitry Semyonov wrote:
On Wed, 10 Aug 2005, Steve Blair wrote:
We have a SER 0.9.4-rc3 (i386/linux)) system running on RedHat ES4 that crashes about once every three days.
I experience similar problems with rel_0_9_0 SER on RedHat 9.
Mine crashes were caused by my bugs in experimental oracle module. Hopefully fixed now.
On Aug 10, 2005 at 07:07, Steve Blair blairs@isc.upenn.edu wrote:
We have a SER 0.9.4-rc3 (i386/linux)) system running on RedHat ES4 that crashes about once every three days. I've done a cvs update, performed a re-make of all modules as well as just mysql.so and reinitialized the mysql database using the script provided. Nothing seems to fix the problem. Using gdb to examine a core dump seems to indicate SER fails when receiving REGISTER messages but I cannot figure out why. In each of the three core dumps the REGISTER message is from a different client.
Below is the gdb output from the latest core dump. Can anyone shed some light on what might be happening?
There was a hard to trigger race condition in tm. When this race happens (a transaction retransmission buffer destination address is not intialized before a new retransmission comes), an assert() in the code caused the proxy to crash.
This should be fixed on the latest cvs (0.9.4-rc6 for stable).
Andrei
-
Andrei Pelinescu-Onciul -
Dmitry Semyonov -
Steve Blair