Hi,
you mean that after first register the proxy dies? if so, it seams like a mem corruption to me...any logs? the memory debugger messages are very useful.
regards, bogdan
Helge Waastad wrote:
Hi, this is the output after compiling with the extra options.
Anyway, that really killed the daemon.. One register, and it dumps.
br hw
(gdb) bt #0 0x002727a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00338d8c in sched_yield () from /lib/tls/libc.so.6 #2 0x00e4b100 in lock_udomain (_d=0xb618fe58) at ../../fastlock.h:166 #3 0x00e4db9a in mem_timer_udomain (_d=0xb618fe58) at udomain.c:677 #4 0x00e47f05 in synchronize_all_udomains () at dlist.c:488 #5 0x00e50a15 in destroy () at ul_mod.c:285 #6 0x08085dce in destroy_modules () #7 0x080638ba in cleanup () #8 0x080644f6 in handle_sigs () #9 0x08065215 in main_loop () #10 0x08065835 in main ()
Mvh, Helge Waastad Senior Engineer Smartnet tlf: 67830017
-----Original Message----- From: Bogdan-Andrei Iancu [mailto:bogdan@voice-system.ro] Sent: Tue 3/7/2006 9:17 PM To: Helge Waastad Cc: Andreas Granig; users@openser.org Subject: Re: [Users] Daemons and killing processes...
Hi Helge,
I suspect a problem related to memory - the remaining process may cycle with no control trying to print the mem status (which may be broken).
to see if it;s the case, compile mem debug support (remove F_MALLOC and add DBG_QM_MALLOC and recompile everything)
regards, bogdan
Helge Waastad wrote:
Hi, I've been waiting fo 12 min now, and are preparing to put then kettle on to fix me a cup of coffee...I guess I have time enough :-)
...is there a "debugging" manual avaliable? I guess it would be nice having feedback on the dev list already including the backtrace?
BTW, the output for the hanging process is:
(gdb) bt #0 0x080a0ab9 in fm_status () #1 0x080663e2 in sig_usr () #2 <signal handler called> #3 0x007787a0 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #4 0x008594d1 in recvfrom () from /lib/tls/libc.so.6 #5 0x08098a87 in udp_rcv_loop () #6 0x08066df2 in main_loop () #7 0x08067585 in main ()
tir, 07,.03.2006 kl. 18.56 +0100, skrev Andreas Granig:
Helge Waastad wrote:
Hi, I have somewhat trouble to get a core dump from the init script. I get core dumps when I run it in shell, but not as init script.
I'll try to be creative later this evening.
The backtrace for the processes looks like this:
#0 0x0808eb0d in fm_status (qm=0x811d940) at mem/f_malloc.c:515 #1 0x08065fee in sig_usr (signo=1077141592) at main.c:565 #2 <signal handler called> #3 0x4010d534 in recvfrom () from /lib/libc.so.6 #4 0x08089433 in udp_rcv_loop () at udp_server.c:415 #5 0x08063de4 in main_loop () at main.c:919 #6 0x0806522e in main (argc=1, argv=0xbfbc54e4) at main.c:1472
But it shuts down after quite some (long) time...
Andy
Hi, bt is as follows:
(gdb) bt #0 0x08098ebf in qm_free () #1 0x007aaf97 in update_contacts (_m=0x814f308, _r=0xb61920d8, _c=0x8139548) at save.c:627 #2 0x007ab67d in save_real (_m=0x814f308, _t=0xb618fe88, doreply=1) at save.c:662 #3 0x0805087b in do_action () #4 0x08051a0c in do_action () #5 0x08051bd2 in run_actions () #6 0x0807534f in receive_msg () #7 0x08092ca3 in udp_rcv_loop () #8 0x08065056 in main_loop () #9 0x08065835 in main () (gdb)
I'm not sure how to get even more memory debugs.
br hw
fre, 10,.03.2006 kl. 11.44 +0200, skrev Bogdan-Andrei Iancu:
Hi,
you mean that after first register the proxy dies? if so, it seams like a mem corruption to me...any logs? the memory debugger messages are very useful.
regards, bogdan
Helge Waastad wrote:
Hi, this is the output after compiling with the extra options.
Anyway, that really killed the daemon.. One register, and it dumps.
br hw
(gdb) bt #0 0x002727a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00338d8c in sched_yield () from /lib/tls/libc.so.6 #2 0x00e4b100 in lock_udomain (_d=0xb618fe58) at ../../fastlock.h:166 #3 0x00e4db9a in mem_timer_udomain (_d=0xb618fe58) at udomain.c:677 #4 0x00e47f05 in synchronize_all_udomains () at dlist.c:488 #5 0x00e50a15 in destroy () at ul_mod.c:285 #6 0x08085dce in destroy_modules () #7 0x080638ba in cleanup () #8 0x080644f6 in handle_sigs () #9 0x08065215 in main_loop () #10 0x08065835 in main ()
Mvh, Helge Waastad Senior Engineer Smartnet tlf: 67830017
-----Original Message----- From: Bogdan-Andrei Iancu [mailto:bogdan@voice-system.ro] Sent: Tue 3/7/2006 9:17 PM To: Helge Waastad Cc: Andreas Granig; users@openser.org Subject: Re: [Users] Daemons and killing processes...
Hi Helge,
I suspect a problem related to memory - the remaining process may cycle with no control trying to print the mem status (which may be broken).
to see if it;s the case, compile mem debug support (remove F_MALLOC and add DBG_QM_MALLOC and recompile everything)
regards, bogdan
Helge Waastad wrote:
Hi, I've been waiting fo 12 min now, and are preparing to put then kettle on to fix me a cup of coffee...I guess I have time enough :-)
...is there a "debugging" manual avaliable? I guess it would be nice having feedback on the dev list already including the backtrace?
BTW, the output for the hanging process is:
(gdb) bt #0 0x080a0ab9 in fm_status () #1 0x080663e2 in sig_usr () #2 <signal handler called> #3 0x007787a0 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #4 0x008594d1 in recvfrom () from /lib/tls/libc.so.6 #5 0x08098a87 in udp_rcv_loop () #6 0x08066df2 in main_loop () #7 0x08067585 in main ()
tir, 07,.03.2006 kl. 18.56 +0100, skrev Andreas Granig:
Helge Waastad wrote:
Hi, I have somewhat trouble to get a core dump from the init script. I get core dumps when I run it in shell, but not as init script.
I'll try to be creative later this evening.
The backtrace for the processes looks like this:
#0 0x0808eb0d in fm_status (qm=0x811d940) at mem/f_malloc.c:515 #1 0x08065fee in sig_usr (signo=1077141592) at main.c:565 #2 <signal handler called> #3 0x4010d534 in recvfrom () from /lib/libc.so.6 #4 0x08089433 in udp_rcv_loop () at udp_server.c:415 #5 0x08063de4 in main_loop () at main.c:919 #6 0x0806522e in main (argc=1, argv=0xbfbc54e4) at main.c:1472
But it shuts down after quite some (long) time...
Andy
Hi Helge,
one you compiled the mem debug support (remove F_MALLOC and add DBG_QM_MALLOC) if you set debug level to a high level (9 for ex),you will get as output a lot of info msgs about the memory ops.
regards, bogdan
Helge Waastad wrote:
Hi, bt is as follows:
(gdb) bt #0 0x08098ebf in qm_free () #1 0x007aaf97 in update_contacts (_m=0x814f308, _r=0xb61920d8, _c=0x8139548) at save.c:627 #2 0x007ab67d in save_real (_m=0x814f308, _t=0xb618fe88, doreply=1) at save.c:662 #3 0x0805087b in do_action () #4 0x08051a0c in do_action () #5 0x08051bd2 in run_actions () #6 0x0807534f in receive_msg () #7 0x08092ca3 in udp_rcv_loop () #8 0x08065056 in main_loop () #9 0x08065835 in main () (gdb)
I'm not sure how to get even more memory debugs.
br hw
fre, 10,.03.2006 kl. 11.44 +0200, skrev Bogdan-Andrei Iancu:
Hi,
you mean that after first register the proxy dies? if so, it seams like a mem corruption to me...any logs? the memory debugger messages are very useful.
regards, bogdan
Helge Waastad wrote:
Hi, this is the output after compiling with the extra options.
Anyway, that really killed the daemon.. One register, and it dumps.
br hw
(gdb) bt #0 0x002727a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00338d8c in sched_yield () from /lib/tls/libc.so.6 #2 0x00e4b100 in lock_udomain (_d=0xb618fe58) at ../../fastlock.h:166 #3 0x00e4db9a in mem_timer_udomain (_d=0xb618fe58) at udomain.c:677 #4 0x00e47f05 in synchronize_all_udomains () at dlist.c:488 #5 0x00e50a15 in destroy () at ul_mod.c:285 #6 0x08085dce in destroy_modules () #7 0x080638ba in cleanup () #8 0x080644f6 in handle_sigs () #9 0x08065215 in main_loop () #10 0x08065835 in main ()
Mvh, Helge Waastad Senior Engineer Smartnet tlf: 67830017
-----Original Message----- From: Bogdan-Andrei Iancu [mailto:bogdan@voice-system.ro] Sent: Tue 3/7/2006 9:17 PM To: Helge Waastad Cc: Andreas Granig; users@openser.org Subject: Re: [Users] Daemons and killing processes...
Hi Helge,
I suspect a problem related to memory - the remaining process may cycle with no control trying to print the mem status (which may be broken).
to see if it;s the case, compile mem debug support (remove F_MALLOC and add DBG_QM_MALLOC and recompile everything)
regards, bogdan
Helge Waastad wrote:
Hi, I've been waiting fo 12 min now, and are preparing to put then kettle on to fix me a cup of coffee...I guess I have time enough :-)
...is there a "debugging" manual avaliable? I guess it would be nice having feedback on the dev list already including the backtrace?
BTW, the output for the hanging process is:
(gdb) bt #0 0x080a0ab9 in fm_status () #1 0x080663e2 in sig_usr () #2 <signal handler called> #3 0x007787a0 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #4 0x008594d1 in recvfrom () from /lib/tls/libc.so.6 #5 0x08098a87 in udp_rcv_loop () #6 0x08066df2 in main_loop () #7 0x08067585 in main ()
tir, 07,.03.2006 kl. 18.56 +0100, skrev Andreas Granig:
Helge Waastad wrote:
Hi, I have somewhat trouble to get a core dump from the init script. I get core dumps when I run it in shell, but not as init script.
I'll try to be creative later this evening.
The backtrace for the processes looks like this:
#0 0x0808eb0d in fm_status (qm=0x811d940) at mem/f_malloc.c:515 #1 0x08065fee in sig_usr (signo=1077141592) at main.c:565 #2 <signal handler called> #3 0x4010d534 in recvfrom () from /lib/libc.so.6 #4 0x08089433 in udp_rcv_loop () at udp_server.c:415 #5 0x08063de4 in main_loop () at main.c:919 #6 0x0806522e in main (argc=1, argv=0xbfbc54e4) at main.c:1472
But it shuts down after quite some (long) time...
Andy
Hi, I do not know if this is sufficient enough, but anyway:
(I belive there is a memory problem, cause it's crashing the moment a REGISTER comes, a replication from another OpenSER....)
br hw
Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: SIP Request: Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: method: <REGISTER> Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: uri: sip:smartnet.no Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: version: <SIP/2.0> Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: flags=2 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: Found param type 232, <branch> = <z9hG4bKc68c.bdbe9a8.0>; state=16 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: end of header reached, state=5 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: Via found, flags=2 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: this is the first via Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: After parse_msg... Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: preparing to run routing scripts... Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: flags=80 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: Found param type 232, <branch> = <z9hG4bKc68c.61bc4535.0>; state=16 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: end of header reached, state=5 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: Via found, flags=80 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: this is the second via Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: Found param type 232, <branch> = <z9hG4bK78F48B37>; state=16 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: end of header reached, state=5 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: Via found, flags=80 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: get_hdr_field: cseq <CSeq>: <135> <REGISTER> Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: DEBUG:parse_to:end of header reached, state=9 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: DEBUG: get_hdr_field: <To> [44]; uri=[sip:67512391@smartnet.no] Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: DEBUG: to body ["Helge Waastad" sip:67512391@smartnet.no^M ] Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: DEBUG: get_hdr_body : content_length=0 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: subst_run: running. r=0 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: subst_run: matched (16, 133): [sip:67512391@80.203.98.197:59086;transport=udp;methods="INVITE, MESSAGE, INFO, SUBSCRIBE, OPTIONS, BYE, CANCEL, NOTIFY, ACK, REFER"] Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: avpops:ops_subst: subst to 1 avps Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: REGISTER 195.18.134.147:5080 -> 195.18.134.150:5080 Replication (no-nat) ("Helge Waastad" sip:67512391@80.203.98.197:59086;transport=udp) Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: flags=ffffffffffffffff Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: found end of header Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: flags=8000000 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: flags=ffffffffffffffff Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: build_contact(): Created Contact HF: Contact: sip:67512391@80.203.98.197:59086;transport=udp;q=0;expires=300^M Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22201]: ERROR: receive_fd: EOF on 11 Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22201]: DBG: handle_ser_child: dead child 2, pid 22188 (shutting down?) Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22201]: DBG: io_watch_del (0x80f0d60, 11, -1, 0x0) fd_no=10 called Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22171]: child process 22188 exited by a signal 11 Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22171]: core was generated Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22171]: INFO: terminating due to SIGCHLD Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22191]: INFO: signal 15 received Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: INFO: signal 15 received Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22191]: Memory status (pkg): Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: Memory status (pkg): Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: qm_status (0x8122b00): Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: heap size= 1048576 Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: used= 129088, used +overhead=177312, free=871264 Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: max used (+overhead)= 181344 Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: dumping all alloc'ed. fragments:
fre, 10,.03.2006 kl. 21.21 +0200, skrev Bogdan-Andrei Iancu:
Hi Helge,
one you compiled the mem debug support (remove F_MALLOC and add DBG_QM_MALLOC) if you set debug level to a high level (9 for ex),you will get as output a lot of info msgs about the memory ops.
regards, bogdan
Helge Waastad wrote:
Hi, bt is as follows:
(gdb) bt #0 0x08098ebf in qm_free () #1 0x007aaf97 in update_contacts (_m=0x814f308, _r=0xb61920d8, _c=0x8139548) at save.c:627 #2 0x007ab67d in save_real (_m=0x814f308, _t=0xb618fe88, doreply=1) at save.c:662 #3 0x0805087b in do_action () #4 0x08051a0c in do_action () #5 0x08051bd2 in run_actions () #6 0x0807534f in receive_msg () #7 0x08092ca3 in udp_rcv_loop () #8 0x08065056 in main_loop () #9 0x08065835 in main () (gdb)
I'm not sure how to get even more memory debugs.
br hw
fre, 10,.03.2006 kl. 11.44 +0200, skrev Bogdan-Andrei Iancu:
Hi,
you mean that after first register the proxy dies? if so, it seams like a mem corruption to me...any logs? the memory debugger messages are very useful.
regards, bogdan
Helge Waastad wrote:
Hi, this is the output after compiling with the extra options.
Anyway, that really killed the daemon.. One register, and it dumps.
br hw
(gdb) bt #0 0x002727a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00338d8c in sched_yield () from /lib/tls/libc.so.6 #2 0x00e4b100 in lock_udomain (_d=0xb618fe58) at ../../fastlock.h:166 #3 0x00e4db9a in mem_timer_udomain (_d=0xb618fe58) at udomain.c:677 #4 0x00e47f05 in synchronize_all_udomains () at dlist.c:488 #5 0x00e50a15 in destroy () at ul_mod.c:285 #6 0x08085dce in destroy_modules () #7 0x080638ba in cleanup () #8 0x080644f6 in handle_sigs () #9 0x08065215 in main_loop () #10 0x08065835 in main ()
Mvh, Helge Waastad Senior Engineer Smartnet tlf: 67830017
-----Original Message----- From: Bogdan-Andrei Iancu [mailto:bogdan@voice-system.ro] Sent: Tue 3/7/2006 9:17 PM To: Helge Waastad Cc: Andreas Granig; users@openser.org Subject: Re: [Users] Daemons and killing processes...
Hi Helge,
I suspect a problem related to memory - the remaining process may cycle with no control trying to print the mem status (which may be broken).
to see if it;s the case, compile mem debug support (remove F_MALLOC and add DBG_QM_MALLOC and recompile everything)
regards, bogdan
Helge Waastad wrote:
Hi, I've been waiting fo 12 min now, and are preparing to put then kettle on to fix me a cup of coffee...I guess I have time enough :-)
...is there a "debugging" manual avaliable? I guess it would be nice having feedback on the dev list already including the backtrace?
BTW, the output for the hanging process is:
(gdb) bt #0 0x080a0ab9 in fm_status () #1 0x080663e2 in sig_usr () #2 <signal handler called> #3 0x007787a0 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #4 0x008594d1 in recvfrom () from /lib/tls/libc.so.6 #5 0x08098a87 in udp_rcv_loop () #6 0x08066df2 in main_loop () #7 0x08067585 in main ()
tir, 07,.03.2006 kl. 18.56 +0100, skrev Andreas Granig:
Helge Waastad wrote:
>Hi, >I have somewhat trouble to get a core dump from the init script. >I get core dumps when I run it in shell, but not as init script. > >I'll try to be creative later this evening. > > > > The backtrace for the processes looks like this:
#0 0x0808eb0d in fm_status (qm=0x811d940) at mem/f_malloc.c:515 #1 0x08065fee in sig_usr (signo=1077141592) at main.c:565 #2 <signal handler called> #3 0x4010d534 in recvfrom () from /lib/libc.so.6 #4 0x08089433 in udp_rcv_loop () at udp_server.c:415 #5 0x08063de4 in main_loop () at main.c:919 #6 0x0806522e in main (argc=1, argv=0xbfbc54e4) at main.c:1472
But it shuts down after quite some (long) time...
Andy
Hi Helge,
I manage to find the bug - please update and see if the problem is solved.
regards, bogdan
Helge Waastad wrote:
Hi, I do not know if this is sufficient enough, but anyway:
(I belive there is a memory problem, cause it's crashing the moment a REGISTER comes, a replication from another OpenSER....)
br hw
Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: SIP Request: Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: method: <REGISTER> Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: uri: sip:smartnet.no Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: version: <SIP/2.0> Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: flags=2 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: Found param type 232, <branch> = <z9hG4bKc68c.bdbe9a8.0>; state=16 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: end of header reached, state=5 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: Via found, flags=2 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: this is the first via Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: After parse_msg... Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: preparing to run routing scripts... Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: flags=80 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: Found param type 232, <branch> = <z9hG4bKc68c.61bc4535.0>; state=16 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: end of header reached, state=5 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: Via found, flags=80 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: this is the second via Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: Found param type 232, <branch> = <z9hG4bK78F48B37>; state=16 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: end of header reached, state=5 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: Via found, flags=80 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: get_hdr_field: cseq <CSeq>: <135> <REGISTER> Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: DEBUG:parse_to:end of header reached, state=9 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: DEBUG: get_hdr_field: <To> [44]; uri=[sip:67512391@smartnet.no] Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: DEBUG: to body ["Helge Waastad" sip:67512391@smartnet.no^M ] Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: DEBUG: get_hdr_body : content_length=0 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: subst_run: running. r=0 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: subst_run: matched (16, 133): [sip:67512391@80.203.98.197:59086;transport=udp;methods="INVITE, MESSAGE, INFO, SUBSCRIBE, OPTIONS, BYE, CANCEL, NOTIFY, ACK, REFER"] Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: avpops:ops_subst: subst to 1 avps Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: REGISTER 195.18.134.147:5080 -> 195.18.134.150:5080 Replication (no-nat) ("Helge Waastad" sip:67512391@80.203.98.197:59086;transport=udp) Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: flags=ffffffffffffffff Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: found end of header Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: flags=8000000 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: flags=ffffffffffffffff Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: build_contact(): Created Contact HF: Contact: sip:67512391@80.203.98.197:59086;transport=udp;q=0;expires=300^M Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22201]: ERROR: receive_fd: EOF on 11 Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22201]: DBG: handle_ser_child: dead child 2, pid 22188 (shutting down?) Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22201]: DBG: io_watch_del (0x80f0d60, 11, -1, 0x0) fd_no=10 called Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22171]: child process 22188 exited by a signal 11 Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22171]: core was generated Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22171]: INFO: terminating due to SIGCHLD Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22191]: INFO: signal 15 received Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: INFO: signal 15 received Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22191]: Memory status (pkg): Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: Memory status (pkg): Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: qm_status (0x8122b00): Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: heap size= 1048576 Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: used= 129088, used +overhead=177312, free=871264 Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: max used (+overhead)= 181344 Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: dumping all alloc'ed. fragments:
fre, 10,.03.2006 kl. 21.21 +0200, skrev Bogdan-Andrei Iancu:
Hi Helge,
one you compiled the mem debug support (remove F_MALLOC and add DBG_QM_MALLOC) if you set debug level to a high level (9 for ex),you will get as output a lot of info msgs about the memory ops.
regards, bogdan
Helge Waastad wrote:
Hi, bt is as follows:
(gdb) bt #0 0x08098ebf in qm_free () #1 0x007aaf97 in update_contacts (_m=0x814f308, _r=0xb61920d8, _c=0x8139548) at save.c:627 #2 0x007ab67d in save_real (_m=0x814f308, _t=0xb618fe88, doreply=1) at save.c:662 #3 0x0805087b in do_action () #4 0x08051a0c in do_action () #5 0x08051bd2 in run_actions () #6 0x0807534f in receive_msg () #7 0x08092ca3 in udp_rcv_loop () #8 0x08065056 in main_loop () #9 0x08065835 in main () (gdb)
I'm not sure how to get even more memory debugs.
br hw
fre, 10,.03.2006 kl. 11.44 +0200, skrev Bogdan-Andrei Iancu:
Hi,
you mean that after first register the proxy dies? if so, it seams like a mem corruption to me...any logs? the memory debugger messages are very useful.
regards, bogdan
Helge Waastad wrote:
Hi, this is the output after compiling with the extra options.
Anyway, that really killed the daemon.. One register, and it dumps.
br hw
(gdb) bt #0 0x002727a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00338d8c in sched_yield () from /lib/tls/libc.so.6 #2 0x00e4b100 in lock_udomain (_d=0xb618fe58) at ../../fastlock.h:166 #3 0x00e4db9a in mem_timer_udomain (_d=0xb618fe58) at udomain.c:677 #4 0x00e47f05 in synchronize_all_udomains () at dlist.c:488 #5 0x00e50a15 in destroy () at ul_mod.c:285 #6 0x08085dce in destroy_modules () #7 0x080638ba in cleanup () #8 0x080644f6 in handle_sigs () #9 0x08065215 in main_loop () #10 0x08065835 in main ()
Mvh, Helge Waastad Senior Engineer Smartnet tlf: 67830017
-----Original Message----- From: Bogdan-Andrei Iancu [mailto:bogdan@voice-system.ro] Sent: Tue 3/7/2006 9:17 PM To: Helge Waastad Cc: Andreas Granig; users@openser.org Subject: Re: [Users] Daemons and killing processes...
Hi Helge,
I suspect a problem related to memory - the remaining process may cycle with no control trying to print the mem status (which may be broken).
to see if it;s the case, compile mem debug support (remove F_MALLOC and add DBG_QM_MALLOC and recompile everything)
regards, bogdan
Helge Waastad wrote:
Hi, I've been waiting fo 12 min now, and are preparing to put then kettle on to fix me a cup of coffee...I guess I have time enough :-)
...is there a "debugging" manual avaliable? I guess it would be nice having feedback on the dev list already including the backtrace?
BTW, the output for the hanging process is:
(gdb) bt #0 0x080a0ab9 in fm_status () #1 0x080663e2 in sig_usr () #2 <signal handler called> #3 0x007787a0 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #4 0x008594d1 in recvfrom () from /lib/tls/libc.so.6 #5 0x08098a87 in udp_rcv_loop () #6 0x08066df2 in main_loop () #7 0x08067585 in main ()
tir, 07,.03.2006 kl. 18.56 +0100, skrev Andreas Granig:
>Helge Waastad wrote: > > > > > > >>Hi, >>I have somewhat trouble to get a core dump from the init script. >>I get core dumps when I run it in shell, but not as init script. >> >>I'll try to be creative later this evening. >> >> >> >> >> >> >The backtrace for the processes looks like this: > >#0 0x0808eb0d in fm_status (qm=0x811d940) at mem/f_malloc.c:515 >#1 0x08065fee in sig_usr (signo=1077141592) at main.c:565 >#2 <signal handler called> >#3 0x4010d534 in recvfrom () from /lib/libc.so.6 >#4 0x08089433 in udp_rcv_loop () at udp_server.c:415 >#5 0x08063de4 in main_loop () at main.c:919 >#6 0x0806522e in main (argc=1, argv=0xbfbc54e4) at main.c:1472 > >But it shuts down after quite some (long) time... > >Andy > > > > > >
Hi, a little late but just now I got to update by cvs.
everythings OK now.
Thanks,
br hw
man, 13,.03.2006 kl. 16.59 +0200, skrev Bogdan-Andrei Iancu:
Hi Helge,
I manage to find the bug - please update and see if the problem is solved.
regards, bogdan
Helge Waastad wrote:
Hi, I do not know if this is sufficient enough, but anyway:
(I belive there is a memory problem, cause it's crashing the moment a REGISTER comes, a replication from another OpenSER....)
br hw
Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: SIP Request: Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: method: <REGISTER> Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: uri: sip:smartnet.no Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: version: <SIP/2.0> Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: flags=2 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: Found param type 232, <branch> = <z9hG4bKc68c.bdbe9a8.0>; state=16 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: end of header reached, state=5 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: Via found, flags=2 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: this is the first via Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: After parse_msg... Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: preparing to run routing scripts... Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: flags=80 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: Found param type 232, <branch> = <z9hG4bKc68c.61bc4535.0>; state=16 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: end of header reached, state=5 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: Via found, flags=80 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: this is the second via Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: Found param type 232, <branch> = <z9hG4bK78F48B37>; state=16 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: end of header reached, state=5 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: Via found, flags=80 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: get_hdr_field: cseq <CSeq>: <135> <REGISTER> Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: DEBUG:parse_to:end of header reached, state=9 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: DEBUG: get_hdr_field: <To> [44]; uri=[sip:67512391@smartnet.no] Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: DEBUG: to body ["Helge Waastad" sip:67512391@smartnet.no^M ] Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: DEBUG: get_hdr_body : content_length=0 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: subst_run: running. r=0 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: subst_run: matched (16, 133): [sip:67512391@80.203.98.197:59086;transport=udp;methods="INVITE, MESSAGE, INFO, SUBSCRIBE, OPTIONS, BYE, CANCEL, NOTIFY, ACK, REFER"] Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: avpops:ops_subst: subst to 1 avps Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: REGISTER 195.18.134.147:5080 -> 195.18.134.150:5080 Replication (no-nat) ("Helge Waastad" sip:67512391@80.203.98.197:59086;transport=udp) Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: flags=ffffffffffffffff Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: found end of header Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: flags=8000000 Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: parse_headers: flags=ffffffffffffffff Mar 10 22:49:06 proxy-02 /usr/sbin/openser[22188]: build_contact(): Created Contact HF: Contact: sip:67512391@80.203.98.197:59086;transport=udp;q=0;expires=300^M Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22201]: ERROR: receive_fd: EOF on 11 Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22201]: DBG: handle_ser_child: dead child 2, pid 22188 (shutting down?) Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22201]: DBG: io_watch_del (0x80f0d60, 11, -1, 0x0) fd_no=10 called Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22171]: child process 22188 exited by a signal 11 Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22171]: core was generated Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22171]: INFO: terminating due to SIGCHLD Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22191]: INFO: signal 15 received Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: INFO: signal 15 received Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22191]: Memory status (pkg): Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: Memory status (pkg): Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: qm_status (0x8122b00): Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: heap size= 1048576 Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: used= 129088, used +overhead=177312, free=871264 Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: max used (+overhead)= 181344 Mar 10 22:49:07 proxy-02 /usr/sbin/openser[22190]: dumping all alloc'ed. fragments:
fre, 10,.03.2006 kl. 21.21 +0200, skrev Bogdan-Andrei Iancu:
Hi Helge,
one you compiled the mem debug support (remove F_MALLOC and add DBG_QM_MALLOC) if you set debug level to a high level (9 for ex),you will get as output a lot of info msgs about the memory ops.
regards, bogdan
Helge Waastad wrote:
Hi, bt is as follows:
(gdb) bt #0 0x08098ebf in qm_free () #1 0x007aaf97 in update_contacts (_m=0x814f308, _r=0xb61920d8, _c=0x8139548) at save.c:627 #2 0x007ab67d in save_real (_m=0x814f308, _t=0xb618fe88, doreply=1) at save.c:662 #3 0x0805087b in do_action () #4 0x08051a0c in do_action () #5 0x08051bd2 in run_actions () #6 0x0807534f in receive_msg () #7 0x08092ca3 in udp_rcv_loop () #8 0x08065056 in main_loop () #9 0x08065835 in main () (gdb)
I'm not sure how to get even more memory debugs.
br hw
fre, 10,.03.2006 kl. 11.44 +0200, skrev Bogdan-Andrei Iancu:
Hi,
you mean that after first register the proxy dies? if so, it seams like a mem corruption to me...any logs? the memory debugger messages are very useful.
regards, bogdan
Helge Waastad wrote:
Hi, this is the output after compiling with the extra options.
Anyway, that really killed the daemon.. One register, and it dumps.
br hw
(gdb) bt #0 0x002727a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00338d8c in sched_yield () from /lib/tls/libc.so.6 #2 0x00e4b100 in lock_udomain (_d=0xb618fe58) at ../../fastlock.h:166 #3 0x00e4db9a in mem_timer_udomain (_d=0xb618fe58) at udomain.c:677 #4 0x00e47f05 in synchronize_all_udomains () at dlist.c:488 #5 0x00e50a15 in destroy () at ul_mod.c:285 #6 0x08085dce in destroy_modules () #7 0x080638ba in cleanup () #8 0x080644f6 in handle_sigs () #9 0x08065215 in main_loop () #10 0x08065835 in main ()
Mvh, Helge Waastad Senior Engineer Smartnet tlf: 67830017
-----Original Message----- From: Bogdan-Andrei Iancu [mailto:bogdan@voice-system.ro] Sent: Tue 3/7/2006 9:17 PM To: Helge Waastad Cc: Andreas Granig; users@openser.org Subject: Re: [Users] Daemons and killing processes...
Hi Helge,
I suspect a problem related to memory - the remaining process may cycle with no control trying to print the mem status (which may be broken).
to see if it;s the case, compile mem debug support (remove F_MALLOC and add DBG_QM_MALLOC and recompile everything)
regards, bogdan
Helge Waastad wrote:
>Hi, >I've been waiting fo 12 min now, and are preparing to put then kettle on >to fix me a cup of coffee...I guess I have time enough :-) > >...is there a "debugging" manual avaliable? I guess it would be nice >having feedback on the dev list already including the backtrace? > > >BTW, the output for the hanging process is: > >(gdb) bt >#0 0x080a0ab9 in fm_status () >#1 0x080663e2 in sig_usr () >#2 <signal handler called> >#3 0x007787a0 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 >#4 0x008594d1 in recvfrom () from /lib/tls/libc.so.6 >#5 0x08098a87 in udp_rcv_loop () >#6 0x08066df2 in main_loop () >#7 0x08067585 in main () > > > > >tir, 07,.03.2006 kl. 18.56 +0100, skrev Andreas Granig: > > > > > > >>Helge Waastad wrote: >> >> >> >> >> >> >>>Hi, >>>I have somewhat trouble to get a core dump from the init script. >>>I get core dumps when I run it in shell, but not as init script. >>> >>>I'll try to be creative later this evening. >>> >>> >>> >>> >>> >>> >>The backtrace for the processes looks like this: >> >>#0 0x0808eb0d in fm_status (qm=0x811d940) at mem/f_malloc.c:515 >>#1 0x08065fee in sig_usr (signo=1077141592) at main.c:565 >>#2 <signal handler called> >>#3 0x4010d534 in recvfrom () from /lib/libc.so.6 >>#4 0x08089433 in udp_rcv_loop () at udp_server.c:415 >>#5 0x08063de4 in main_loop () at main.c:919 >>#6 0x0806522e in main (argc=1, argv=0xbfbc54e4) at main.c:1472 >> >>But it shuts down after quite some (long) time... >> >>Andy >> >> >> >> >> >>
-
Bogdan-Andrei Iancu -
Helge Waastad