Hey all,
I am in the process of installing SER for the first time and doing some initial configuration and testing. However, I am having some problems and hoping that someone on this list may have seen this before and can point me in the right direction for a solution.
I have downloaded the Solaris9 Sparc64 package of SER version 0.8.11. It's been installed and starts up and runs okay. I've also configured it to use MySQL for authentication, and setup serweb to integrate with the MySQL server as well. So far so good.. everything seems to work. I can add a user using "serctl" and then login with that user via "serweb". very cool!
The problem occurs when I try to connect to the SER server from my SIP phone. I get the following errors:
Nov 20 13:11:02 ultra60 ./ser[3942]: ERROR: build_req_buf_from_sip_req: out of memory Nov 20 13:11:02 ultra60 ./ser[3942]: ERROR: print_uac_request: no pkg_mem Nov 20 13:11:02 ultra60 ./ser[3942]: ERROR: t_forward_nonack: failure to add branches Nov 20 13:11:32 ultra60 ./ser[3942]: ERROR: build_req_buf_from_sip_req: out of memory Nov 20 13:11:32 ultra60 ./ser[3942]: ERROR: print_uac_request: no pkg_mem Nov 20 13:11:32 ultra60 ./ser[3942]: ERROR: t_forward_nonack: failure to add branches
Has anyone seen this in the past? Or know of any reason that this might happen? Maybe even a possible solution? :)
Any help would be appreciated. Thanks!!
Darren Nay - dnay@libertyisp.com
On Nov 20, 2003 at 12:01, Darren Nay dnay@libertyisp.com wrote:
Hey all,
[...]
Nov 20 13:11:02 ultra60 ./ser[3942]: ERROR: build_req_buf_from_sip_req: out of memory
[...]
Has anyone seen this in the past? Or know of any reason that this might happen? Maybe even a possible solution? :)
This happens when ser runs out of shared memory :-) Are you making lots of calls simultaneoulsy? Do you have a lot of users?
By default ser will use only 32Mb of shared memory which should be enough for most test/small scenarios. To make it use more start it with -m size_in_M (e.g ser -m 256 -f ser.cfg ...).
Andrei
Hmm.. That's peculiar. :) We only have one user on the server right now and we are only trying with one single phone (one call at a time).
Maybe we have a loop somewhere sucking up the memory? :(
Oh, another note. Prior to installing mysql and configuring SER to use mysql for authentication the SIP phone registered with SER without any problems. After the mysql config it's getting this error. Does implementing MySQL take extra memory? If so, how much extra? and how will it affect the perfomance of the SER server?
Thanks for your help! That does help me, I will continue to play with it and see if I can figure it out as well.
Thanks again! Darren Nay
----- Original Message ----- From: "Andrei Pelinescu-Onciul" pelinescu-onciul@fokus.fraunhofer.de To: "Darren Nay" dnay@libertyisp.com Cc: serusers@lists.iptel.org Sent: Thursday, November 20, 2003 12:42 PM Subject: Re: [Serusers] Memory errors
This happens when ser runs out of shared memory :-) Are you making lots of calls simultaneoulsy? Do you have a lot of users?
By default ser will use only 32Mb of shared memory which should be enough for most test/small scenarios. To make it use more start it with -m size_in_M (e.g ser -m 256 -f ser.cfg ...).
Andrei
I think it is not shared memory, but private memory what is exhausted because the logs you provided shows functions that do not use shared memory.
When this happen, can you send SIGUSR1 to the first ser process ? It should print some memory statistics. Can you then send them to us ?
To me it looks like a memory leak. How quickly is the memory exhausted ? Immediately after you start the server or after some time ? Can you reproduce it ?
Jan.
On 20-11 12:54, Darren Nay wrote:
Hmm.. That's peculiar. :) We only have one user on the server right now and we are only trying with one single phone (one call at a time).
Maybe we have a loop somewhere sucking up the memory? :(
Oh, another note. Prior to installing mysql and configuring SER to use mysql for authentication the SIP phone registered with SER without any problems. After the mysql config it's getting this error. Does implementing MySQL take extra memory? If so, how much extra? and how will it affect the perfomance of the SER server?
Thanks for your help! That does help me, I will continue to play with it and see if I can figure it out as well.
Thanks again! Darren Nay
----- Original Message ----- From: "Andrei Pelinescu-Onciul" pelinescu-onciul@fokus.fraunhofer.de To: "Darren Nay" dnay@libertyisp.com Cc: serusers@lists.iptel.org Sent: Thursday, November 20, 2003 12:42 PM Subject: Re: [Serusers] Memory errors
This happens when ser runs out of shared memory :-) Are you making lots of calls simultaneoulsy? Do you have a lot of users?
By default ser will use only 32Mb of shared memory which should be enough for most test/small scenarios. To make it use more start it with -m size_in_M (e.g ser -m 256 -f ser.cfg ...).
Andrei
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
Thanks for your followup Jan.
I did a SIGUSR1 and here were the results. The first time it output the mem statistics. Then I did another SIGUSR1 again and it output them and then the SER server died.. Here was the output
First SIGUSR1 ---------------- 0(5066) Memory status (pkg): 0(5066) fm_status (8e8f0): 0(5066) heap size= 1047440 0(5066) dumping free list: 0(5066) hash = 1 fragments no.: 4, bucket size: 8 - 8 (first 8) 0(5066) hash = 2 fragments no.: 1, bucket size: 16 - 16 (first 16) 0(5066) hash = 6 fragments no.: 1, bucket size: 48 - 48 (first 48) 0(5066) hash = 20 fragments no.: 1, bucket size: 160 - 160 (first 160) 0(5066) hash = 265 fragments no.: 1, bucket size: 524288 - 1048576 (first 1042112) 0(5066) TOTAL: 8 free fragments = 1042368 free bytes 0(5066) ----------------------------- 0(5066) Memory status (shm): 0(5066) fm_status (ef000000): 0(5066) heap size= 268434320 0(5066) dumping free list: 0(5066) hash = 273 fragments no.: 1, bucket size: 134217728 - 268435456 (first 266841440) 0(5066) TOTAL: 1 free fragments = 266841440 free bytes 0(5066) ----------------------------- 0(5066) ERROR: udp_rcv_loop:recvfrom:[4] Interrupted system call
Second SIGUSR1 ------------------- 2(5068) ERROR: fifo_server fgets failed: Error 0 2(5068) INFO: signal 15 received 2(5068) Memory status (pkg): 2(5068) fm_status (8e8f0): 2(5068) heap size= 1047440 2(5068) dumping free list: 2(5068) hash = 1 fragments no.: 4, bucket size: 8 - 8 (first 8) 2(5068) hash = 2 fragments no.: 1, bucket size: 16 - 16 (first 16) 2(5068) hash = 4 fragments no.: 1, bucket size: 32 - 32 (first 32) 2(5068) hash = 6 fragments no.: 1, bucket size: 48 - 48 (first 48) 2(5068) hash = 20 fragments no.: 1, bucket size: 160 - 160 (first 160) 2(5068) hash = 265 fragments no.: 1, bucket size: 524288 - 1048576 (first 1042112) 2(5068) TOTAL: 9 free fragments = 1042400 free bytes 2(5068) ----------------------------- 1(5067) INFO: signal 15 received 1(5067) Memory status (pkg): 1(5067) fm_status (8e8f0): 1(5067) heap size= 1047440 1(5067) dumping free list: 1(5067) hash = 1 fragments no.: 4, bucket size: 8 - 8 (first 8) 1(5067) hash = 2 fragments no.: 1, bucket size: 16 - 16 (first 16) 1(5067) hash = 4 fragments no.: 1, bucket size: 32 - 32 (first 32) 1(5067) hash = 6 fragments no.: 1, bucket size: 48 - 48 (first 48) 1(5067) hash = 20 fragments no.: 1, bucket size: 160 - 160 (first 160) 1(5067) hash = 265 fragments no.: 1, bucket size: 524288 - 1048576 (first 1042112) 1(5067) User Signal 1 TOTAL: 9 free fragments = 1042400 free bytes 1(5067) ----------------------------- root:/usr/local/sbin # root:/usr/local/sbin #
----- Original Message ----- From: "Jan Janak" jan@iptel.org To: "Darren Nay" dnay@libertyisp.com Cc: "Andrei Pelinescu-Onciul" pelinescu-onciul@fokus.fraunhofer.de; serusers@lists.iptel.org Sent: Thursday, November 20, 2003 1:06 PM Subject: Re: [Serusers] Memory errors
I think it is not shared memory, but private memory what is exhausted because the logs you provided shows functions that do not use shared memory.
When this happen, can you send SIGUSR1 to the first ser process ? It should print some memory statistics. Can you then send them to us ?
To me it looks like a memory leak. How quickly is the memory exhausted ? Immediately after you start the server or after some time ? Can you reproduce it ?
Jan.
On 20-11 12:54, Darren Nay wrote:
Hmm.. That's peculiar. :) We only have one user on the server right now
and
we are only trying with one single phone (one call at a time).
Maybe we have a loop somewhere sucking up the memory? :(
Oh, another note. Prior to installing mysql and configuring SER to use mysql for authentication the SIP phone registered with SER without any problems. After the mysql config it's getting this error. Does implementing MySQL take extra memory? If so, how much extra? and how
will
it affect the perfomance of the SER server?
Thanks for your help! That does help me, I will continue to play with
it
and see if I can figure it out as well.
Thanks again! Darren Nay
----- Original Message ----- From: "Andrei Pelinescu-Onciul" pelinescu-onciul@fokus.fraunhofer.de To: "Darren Nay" dnay@libertyisp.com Cc: serusers@lists.iptel.org Sent: Thursday, November 20, 2003 12:42 PM Subject: Re: [Serusers] Memory errors
This happens when ser runs out of shared memory :-) Are you making
lots
of calls simultaneoulsy? Do you have a lot of users?
By default ser will use only 32Mb of shared memory which should be enough for most test/small scenarios. To make it use more start it
with
-m size_in_M (e.g ser -m 256 -f ser.cfg ...).
Andrei
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
It looks like you have a lot of free memory. Did you print the statistics at the same time when the server was complaining about insufficient memory ?
Still pending questions from my previous email: How quickly is the memory exhausted ? Immediately after you start the server ? Can you reproduce it ?
Jan.
On 20-11 13:26, Darren Nay wrote:
Thanks for your followup Jan.
I did a SIGUSR1 and here were the results. The first time it output the mem statistics. Then I did another SIGUSR1 again and it output them and then the SER server died.. Here was the output
First SIGUSR1
0(5066) Memory status (pkg): 0(5066) fm_status (8e8f0): 0(5066) heap size= 1047440 0(5066) dumping free list: 0(5066) hash = 1 fragments no.: 4, bucket size: 8 - 8 (first 8) 0(5066) hash = 2 fragments no.: 1, bucket size: 16 - 16 (first 16) 0(5066) hash = 6 fragments no.: 1, bucket size: 48 - 48 (first 48) 0(5066) hash = 20 fragments no.: 1, bucket size: 160 - 160 (first 160) 0(5066) hash = 265 fragments no.: 1, bucket size: 524288 - 1048576 (first 1042112) 0(5066) TOTAL: 8 free fragments = 1042368 free bytes 0(5066) ----------------------------- 0(5066) Memory status (shm): 0(5066) fm_status (ef000000): 0(5066) heap size= 268434320 0(5066) dumping free list: 0(5066) hash = 273 fragments no.: 1, bucket size: 134217728 - 268435456 (first 266841440) 0(5066) TOTAL: 1 free fragments = 266841440 free bytes 0(5066) ----------------------------- 0(5066) ERROR: udp_rcv_loop:recvfrom:[4] Interrupted system call
Second SIGUSR1
2(5068) ERROR: fifo_server fgets failed: Error 0 2(5068) INFO: signal 15 received 2(5068) Memory status (pkg): 2(5068) fm_status (8e8f0): 2(5068) heap size= 1047440 2(5068) dumping free list: 2(5068) hash = 1 fragments no.: 4, bucket size: 8 - 8 (first 8) 2(5068) hash = 2 fragments no.: 1, bucket size: 16 - 16 (first 16) 2(5068) hash = 4 fragments no.: 1, bucket size: 32 - 32 (first 32) 2(5068) hash = 6 fragments no.: 1, bucket size: 48 - 48 (first 48) 2(5068) hash = 20 fragments no.: 1, bucket size: 160 - 160 (first 160) 2(5068) hash = 265 fragments no.: 1, bucket size: 524288 - 1048576 (first 1042112) 2(5068) TOTAL: 9 free fragments = 1042400 free bytes 2(5068) ----------------------------- 1(5067) INFO: signal 15 received 1(5067) Memory status (pkg): 1(5067) fm_status (8e8f0): 1(5067) heap size= 1047440 1(5067) dumping free list: 1(5067) hash = 1 fragments no.: 4, bucket size: 8 - 8 (first 8) 1(5067) hash = 2 fragments no.: 1, bucket size: 16 - 16 (first 16) 1(5067) hash = 4 fragments no.: 1, bucket size: 32 - 32 (first 32) 1(5067) hash = 6 fragments no.: 1, bucket size: 48 - 48 (first 48) 1(5067) hash = 20 fragments no.: 1, bucket size: 160 - 160 (first 160) 1(5067) hash = 265 fragments no.: 1, bucket size: 524288 - 1048576 (first 1042112) 1(5067) User Signal 1 TOTAL: 9 free fragments = 1042400 free bytes 1(5067) ----------------------------- root:/usr/local/sbin # root:/usr/local/sbin #
----- Original Message ----- From: "Jan Janak" jan@iptel.org To: "Darren Nay" dnay@libertyisp.com Cc: "Andrei Pelinescu-Onciul" pelinescu-onciul@fokus.fraunhofer.de; serusers@lists.iptel.org Sent: Thursday, November 20, 2003 1:06 PM Subject: Re: [Serusers] Memory errors
I think it is not shared memory, but private memory what is exhausted because the logs you provided shows functions that do not use shared memory.
When this happen, can you send SIGUSR1 to the first ser process ? It should print some memory statistics. Can you then send them to us ?
To me it looks like a memory leak. How quickly is the memory exhausted ? Immediately after you start the server or after some time ? Can you reproduce it ?
Jan.
On 20-11 12:54, Darren Nay wrote:
Hmm.. That's peculiar. :) We only have one user on the server right now
and
we are only trying with one single phone (one call at a time).
Maybe we have a loop somewhere sucking up the memory? :(
Oh, another note. Prior to installing mysql and configuring SER to use mysql for authentication the SIP phone registered with SER without any problems. After the mysql config it's getting this error. Does implementing MySQL take extra memory? If so, how much extra? and how
will
it affect the perfomance of the SER server?
Thanks for your help! That does help me, I will continue to play with
it
and see if I can figure it out as well.
Thanks again! Darren Nay
----- Original Message ----- From: "Andrei Pelinescu-Onciul" pelinescu-onciul@fokus.fraunhofer.de To: "Darren Nay" dnay@libertyisp.com Cc: serusers@lists.iptel.org Sent: Thursday, November 20, 2003 12:42 PM Subject: Re: [Serusers] Memory errors
This happens when ser runs out of shared memory :-) Are you making
lots
of calls simultaneoulsy? Do you have a lot of users?
By default ser will use only 32Mb of shared memory which should be enough for most test/small scenarios. To make it use more start it
with
-m size_in_M (e.g ser -m 256 -f ser.cfg ...).
Andrei
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
Darren,
Did you ever get this figured out? I am running in to this problem when I enable domain.so processing. It is curious, because when I first tried it I got this error:
if (!mf_process_maxfwd_header("10")) { sl_send_reply("483", "too many hops"); };
when I take this test out I get the memory overflow. when I leave it in I get a 483 at the UA when registering.
This started happening with domain processing. ngrep only shows one packet, but if I put an xlog() trace at the top of the route {}; logic I see recursion. Somehow the route loop is being entered over and over again.
If I comment out the 'loadmodule domain.so', and remove the #modparam("domain", "db_mode", 1) #modparam("domain", "domain_table", "domain") #modparam("domain", "domain_col", "did") #modparam("domain","db_url", "X"); #modparam("usrloc|group", "use_domain", 1) lines, modify the www_challenge and www_authorize accordingly, and change if (!uri==myself) { # if(!is_from_local()) { then everything works. Were you trying to use the domain stuff?
---greg
Darren Nay wrote:
Thanks for your followup Jan.
I did a SIGUSR1 and here were the results. The first time it output the mem statistics. Then I did another SIGUSR1 again and it output them and then the SER server died.. Here was the output
First SIGUSR1
0(5066) Memory status (pkg): 0(5066) fm_status (8e8f0): 0(5066) heap size= 1047440 0(5066) dumping free list: 0(5066) hash = 1 fragments no.: 4, bucket size: 8 - 8 (first 8) 0(5066) hash = 2 fragments no.: 1, bucket size: 16 - 16 (first 16) 0(5066) hash = 6 fragments no.: 1, bucket size: 48 - 48 (first 48) 0(5066) hash = 20 fragments no.: 1, bucket size: 160 - 160 (first 160) 0(5066) hash = 265 fragments no.: 1, bucket size: 524288 - 1048576 (first 1042112) 0(5066) TOTAL: 8 free fragments = 1042368 free bytes 0(5066) ----------------------------- 0(5066) Memory status (shm): 0(5066) fm_status (ef000000): 0(5066) heap size= 268434320 0(5066) dumping free list: 0(5066) hash = 273 fragments no.: 1, bucket size: 134217728 - 268435456 (first 266841440) 0(5066) TOTAL: 1 free fragments = 266841440 free bytes 0(5066) ----------------------------- 0(5066) ERROR: udp_rcv_loop:recvfrom:[4] Interrupted system call
Second SIGUSR1
2(5068) ERROR: fifo_server fgets failed: Error 0 2(5068) INFO: signal 15 received 2(5068) Memory status (pkg): 2(5068) fm_status (8e8f0): 2(5068) heap size= 1047440 2(5068) dumping free list: 2(5068) hash = 1 fragments no.: 4, bucket size: 8 - 8 (first 8) 2(5068) hash = 2 fragments no.: 1, bucket size: 16 - 16 (first 16) 2(5068) hash = 4 fragments no.: 1, bucket size: 32 - 32 (first 32) 2(5068) hash = 6 fragments no.: 1, bucket size: 48 - 48 (first 48) 2(5068) hash = 20 fragments no.: 1, bucket size: 160 - 160 (first 160) 2(5068) hash = 265 fragments no.: 1, bucket size: 524288 - 1048576 (first 1042112) 2(5068) TOTAL: 9 free fragments = 1042400 free bytes 2(5068) ----------------------------- 1(5067) INFO: signal 15 received 1(5067) Memory status (pkg): 1(5067) fm_status (8e8f0): 1(5067) heap size= 1047440 1(5067) dumping free list: 1(5067) hash = 1 fragments no.: 4, bucket size: 8 - 8 (first 8) 1(5067) hash = 2 fragments no.: 1, bucket size: 16 - 16 (first 16) 1(5067) hash = 4 fragments no.: 1, bucket size: 32 - 32 (first 32) 1(5067) hash = 6 fragments no.: 1, bucket size: 48 - 48 (first 48) 1(5067) hash = 20 fragments no.: 1, bucket size: 160 - 160 (first 160) 1(5067) hash = 265 fragments no.: 1, bucket size: 524288 - 1048576 (first 1042112) 1(5067) User Signal 1 TOTAL: 9 free fragments = 1042400 free bytes 1(5067) ----------------------------- root:/usr/local/sbin # root:/usr/local/sbin #
----- Original Message ----- From: "Jan Janak" jan@iptel.org To: "Darren Nay" dnay@libertyisp.com Cc: "Andrei Pelinescu-Onciul" pelinescu-onciul@fokus.fraunhofer.de; serusers@lists.iptel.org Sent: Thursday, November 20, 2003 1:06 PM Subject: Re: [Serusers] Memory errors
I think it is not shared memory, but private memory what is exhausted because the logs you provided shows functions that do not use shared memory.
When this happen, can you send SIGUSR1 to the first ser process ? It should print some memory statistics. Can you then send them to us ?
To me it looks like a memory leak. How quickly is the memory exhausted ? Immediately after you start the server or after some time ? Can you reproduce it ?
Jan.
On 20-11 12:54, Darren Nay wrote:
Hmm.. That's peculiar. :) We only have one user on the server right now
and
we are only trying with one single phone (one call at a time).
Maybe we have a loop somewhere sucking up the memory? :(
Oh, another note. Prior to installing mysql and configuring SER to use mysql for authentication the SIP phone registered with SER without any problems. After the mysql config it's getting this error. Does implementing MySQL take extra memory? If so, how much extra? and how
will
it affect the perfomance of the SER server?
Thanks for your help! That does help me, I will continue to play with
it
and see if I can figure it out as well.
Thanks again! Darren Nay
----- Original Message ----- From: "Andrei Pelinescu-Onciul" pelinescu-onciul@fokus.fraunhofer.de To: "Darren Nay" dnay@libertyisp.com Cc: serusers@lists.iptel.org Sent: Thursday, November 20, 2003 12:42 PM Subject: Re: [Serusers] Memory errors
This happens when ser runs out of shared memory :-) Are you making
lots
of calls simultaneoulsy? Do you have a lot of users?
By default ser will use only 32Mb of shared memory which should be enough for most test/small scenarios. To make it use more start it
with
-m size_in_M (e.g ser -m 256 -f ser.cfg ...).
Andrei
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
On 25-11 07:52, Greg Fausak wrote:
Darren,
Did you ever get this figured out? I am running in to this problem when I enable domain.so processing. It is curious, because when I first tried it I got this error:
if (!mf_process_maxfwd_header("10")) { sl_send_reply("483", "too many hops");};
when I take this test out I get the memory overflow. when I leave it in I get a 483 at the UA when registering.
That sounds like an infinite loop -- ser tries to send a message to itself over and over again. If you remove the condition then it will keep sending the message until the memory is exhausted. If you keep the condition then it will return the error after 10 loops.
This started happening with domain processing.
Probably one of the domains isn't properly recognized as belonging to your server, but DNS resolves to the server. That will result in an infinite loop.
ngrep only shows one packet, but if I put an xlog() trace at the top of the route {}; logic I see recursion. Somehow the route loop is being entered over and over again.
Yes. ngrep doesn't show the looping packets because it can listen on one interface only and that's usually the network card. Looping packets are sent over loopback so ngrep will not show them.
If I comment out the 'loadmodule domain.so', and remove the #modparam("domain", "db_mode", 1) #modparam("domain", "domain_table", "domain") #modparam("domain", "domain_col", "did") #modparam("domain","db_url", "X"); #modparam("usrloc|group", "use_domain", 1) lines, modify the www_challenge and www_authorize accordingly, and change if (!uri==myself) { # if(!is_from_local()) { then everything works. Were you trying to use the domain stuff?
I guess the condition is the problem. Probably is_from_local() doesn't match one of domains that are matched properly by (uri==myself) test.
To find out which domain is the troublemaker you can run ngrep listening on the loopback (use -d lo) option. Look at the Request-URI and make sure you have the domain from the Request-URI in your domain table.
Jan.
-
Andrei Pelinescu-Onciul -
Darren Nay -
Greg Fausak -
Jan Janak