Some more update.. got the same problem today.. and when it happend, ser blocked. and I got ALOT of these error messages. Anybody know why ?
-Atle
VOIP /usr/local/sbin/ser[1677]: ERROR: new_t: out of mem: VOIP /usr/local/sbin/ser[1677]: ERROR: t_newtran: new_t failed VOIP /usr/local/sbin/ser[1677]: ERROR: sl_reply_error used: I'm terribly sorry, server error occured (2/SL)
* Atle Samuelsen clona@camaro.no [040621 07:35]:
Hi Guys and Girls.
I ran into a problem a couple of days ago, when trying to restart ser. The scenario was that ser was sending out ALOT of 500 "I'm sorry... ". I found it then best to restart ser, to see if it stoped. When I then was going to start ser again (via ser's rc scripts) It did'nt want to start.
Here's the messages in /var/log/messages. (First it looked like ser started normaly)
VOIP /usr/local/sbin/ser[1302]: mod_init(): Database connection opened successfuly VOIP ser: ser startup succeeded VOIP /usr/local/sbin/ser[1302]: convert_row(): No memory left VOIP /usr/local/sbin/ser[1302]: convert_rows(): Error while converting row #6256 VOIP /usr/local/sbin/ser[1302]: convert_result(): Error while converting rows VOIP /usr/local/sbin/ser[1302]: get_result(): Error while converting result VOIP /usr/local/sbin/ser[1302]: preload_udomain(): Error while doing db_query VOIP /usr/local/sbin/ser[1302]: register_udomain(): Error while preloading domain 'location' VOIP /usr/local/sbin/ser[1302]: domain_fixup(): Error while registering domain VOIP /usr/local/sbin/ser[1302]: ERROR: fix_expr : fix_actions error
To get ser upandRunning again, I had to delete all the entries in mysql::ser::location.
Have you guys ever seen anything like this before?
- Atle
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
Hi,
Atle Samuelsen wrote:
Some more update.. got the same problem today.. and when it happend, ser blocked. and I got ALOT of these error messages. Anybody know why ?
Are there any subscribers with a lot of location entries? I had this problem some months ago with Ser 0.8.10 IIRC, and the reason was a user accidently registering very often (~500 times or so) with different IPs because of a DHCP issue. After deleting his userlocs it started up again...
Andy
PS: I currently watch the number of locations per user with an external script, but in general a user should only be allowed to register one location here, so maybe I should think about some sort of check when registering a client. Are there any out-of-the-box solutions or is this another task for an external script or extra module?
On Jun 21, 2004 at 19:47, Atle Samuelsen clona@cyberhouse.no wrote:
Some more update.. got the same problem today.. and when it happend, ser blocked. and I got ALOT of these error messages. Anybody know why ?
Ser runs out of "local" memory. The localtions are first loaded in local (per process memory) and copied and inserted in the usrloc shared memory list. (probably you have a lot of entries)
By default ser uses maximum 1Mb of local memory/process. Try to increases it to a bigger value. Change PKG_MEM_POOL_SIZE in config.h and recompile.
Andrei
Andrei Pelinescu-Onciul wrote:
On Jun 21, 2004 at 19:47, Atle Samuelsen clona@cyberhouse.no wrote:
Some more update.. got the same problem today.. and when it happend, ser blocked. and I got ALOT of these error messages. Anybody know why ?
Ser runs out of "local" memory. The localtions are first loaded in local (per process memory) and copied and inserted in the usrloc shared memory list. (probably you have a lot of entries)
By default ser uses maximum 1Mb of local memory/process. Try to increases it to a bigger value. Change PKG_MEM_POOL_SIZE in config.h and recompile.
D'oh, same problem again here now (Ser 0.8.12)!
It started with log entries like this:
Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: build_req_buf_from_sip_req: out of memory Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: print_uac_request: no pkg_mem Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: t_forward_nonack: failure to add branches
I thought that maybe a restart could help, but SER didn't come up again, shouting:
Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_row(): No memory left Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_rows(): Error while converting row #6346 Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_result(): Error while converting rows Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: get_result(): Error while converting result Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: preload_udomain(): Error while doing db_query Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: register_udomain(): Error while preloading domain 'location' Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: domain_fixup(): Error while registering domain
After deleting 100 entries from location it came up again. Hope that the increase of PKG_MEM_POOL_SIZE to 2MB helps.
So it seems that after about 6300 entries or so in location Ser stops working with the default configuration!? I'd be interested in how many users do other Ser-Servers out there serve? Are there similiar problems?
Regards, Andy
On Jun 24, 2004 at 12:54, Andreas Granig a.granig@inode.at wrote:
Andrei Pelinescu-Onciul wrote:
On Jun 21, 2004 at 19:47, Atle Samuelsen clona@cyberhouse.no wrote:
Some more update.. got the same problem today.. and when it happend, ser blocked. and I got ALOT of these error messages. Anybody know why ?
Ser runs out of "local" memory. The localtions are first loaded in local (per process memory) and copied and inserted in the usrloc shared memory list. (probably you have a lot of entries)
By default ser uses maximum 1Mb of local memory/process. Try to increases it to a bigger value. Change PKG_MEM_POOL_SIZE in config.h and recompile.
D'oh, same problem again here now (Ser 0.8.12)!
It started with log entries like this:
[...]
After deleting 100 entries from location it came up again. Hope that the increase of PKG_MEM_POOL_SIZE to 2MB helps.
You'll be ok until you reach twice the location table size you have right now.
So it seems that after about 6300 entries or so in location Ser stops working with the default configuration!? I'd be interested in how many users do other Ser-Servers out there serve? Are there similiar problems?
Yes there are and you are right they usually start at ~6000 entries.
We will either increase the default temp. memory size or we will switch to shared memory for loading the locations tables on startup. The ideea behind the local memory is to have some kind of private per process, very fast to allocate memory space, used mainly for temporary stuff. However in the usrloc case since the tables are loaded on startup we don't care so much about the performance and we could switch to shared memory (which is by default 32Mb and can be changed with the command line switch -m).
Andrei
Thanks Andrei :-)
- Atle
* Andrei Pelinescu-Onciul pelinescu-onciul@fokus.fraunhofer.de [040624 14:14]:
On Jun 24, 2004 at 12:54, Andreas Granig a.granig@inode.at wrote:
Andrei Pelinescu-Onciul wrote:
On Jun 21, 2004 at 19:47, Atle Samuelsen clona@cyberhouse.no wrote:
Some more update.. got the same problem today.. and when it happend, ser blocked. and I got ALOT of these error messages. Anybody know why ?
Ser runs out of "local" memory. The localtions are first loaded in local (per process memory) and copied and inserted in the usrloc shared memory list. (probably you have a lot of entries)
By default ser uses maximum 1Mb of local memory/process. Try to increases it to a bigger value. Change PKG_MEM_POOL_SIZE in config.h and recompile.
D'oh, same problem again here now (Ser 0.8.12)!
It started with log entries like this:
[...]
After deleting 100 entries from location it came up again. Hope that the increase of PKG_MEM_POOL_SIZE to 2MB helps.
You'll be ok until you reach twice the location table size you have right now.
So it seems that after about 6300 entries or so in location Ser stops working with the default configuration!? I'd be interested in how many users do other Ser-Servers out there serve? Are there similiar problems?
Yes there are and you are right they usually start at ~6000 entries.
We will either increase the default temp. memory size or we will switch to shared memory for loading the locations tables on startup. The ideea behind the local memory is to have some kind of private per process, very fast to allocate memory space, used mainly for temporary stuff. However in the usrloc case since the tables are loaded on startup we don't care so much about the performance and we could switch to shared memory (which is by default 32Mb and can be changed with the command line switch -m).
Andrei
Under normal operation, SER can handle significantly more registrations, by orders of magnitude. To me, it sounds much more like a result of a memory leak. Are you using stock SER or some modified SER? In any case, the way to catch such leaks is to set a talkative memory log level (memlog=2 in ser.cfg), recompile ser with memory debugging informaiton (uncomment DBG_QM_MALLOC, comment F_MALLOC in Makefile.defs and do recompile completely) and watch log output on SER shutdown for excessive occurent of unallocated memory fragement. I would be very thankful if you could report your observations back to us.
-jiri
At 12:54 PM 6/24/2004, Andreas Granig wrote:
Andrei Pelinescu-Onciul wrote:
On Jun 21, 2004 at 19:47, Atle Samuelsen clona@cyberhouse.no wrote:
Some more update.. got the same problem today.. and when it happend, ser blocked. and I got ALOT of these error messages. Anybody know why ?
Ser runs out of "local" memory. The localtions are first loaded in local (per process memory) and copied and inserted in the usrloc shared memory list. (probably you have a lot of entries) By default ser uses maximum 1Mb of local memory/process. Try to increases it to a bigger value. Change PKG_MEM_POOL_SIZE in config.h and recompile.
D'oh, same problem again here now (Ser 0.8.12)!
It started with log entries like this:
Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: build_req_buf_from_sip_req: out of memory Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: print_uac_request: no pkg_mem Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: t_forward_nonack: failure to add branches
I thought that maybe a restart could help, but SER didn't come up again, shouting:
Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_row(): No memory left Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_rows(): Error while converting row #6346 Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_result(): Error while converting rows Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: get_result(): Error while converting result Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: preload_udomain(): Error while doing db_query Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: register_udomain(): Error while preloading domain 'location' Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: domain_fixup(): Error while registering domain
After deleting 100 entries from location it came up again. Hope that the increase of PKG_MEM_POOL_SIZE to 2MB helps.
So it seems that after about 6300 entries or so in location Ser stops working with the default configuration!? I'd be interested in how many users do other Ser-Servers out there serve? Are there similiar problems?
Regards, Andy
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
-- Jiri Kuthan http://iptel.org/~jiri/
On Jun 25, 2004 at 00:30, Jiri Kuthan jiri@iptel.org wrote:
Under normal operation, SER can handle significantly more registrations, by orders of magnitude. To me, it sounds much more like a result of a memory leak.
Jiri, note that this happens on startup. In fact it runs out of mem. when it tries to alloc. a chunk of memory big enough to hold all the entries in location. This is not a memory leak and it really happens if PKG_MEM_POOL_SIZE is 1Mb and you have >6000 entries in location (it depends of course on your average entry size, but the critical point is in general between 6000-7000).
Are you using stock SER or some modified SER? In any case, the way to catch such leaks is to set a talkative memory log level (memlog=2 in ser.cfg), recompile ser with memory debugging informaiton (uncomment DBG_QM_MALLOC, comment F_MALLOC in Makefile.defs and do recompile completely) and watch log output on SER shutdown for excessive occurent of unallocated memory fragement. I would be very thankful if you could report your observations back to us.
-jiri
At 12:54 PM 6/24/2004, Andreas Granig wrote:
Andrei Pelinescu-Onciul wrote:
On Jun 21, 2004 at 19:47, Atle Samuelsen clona@cyberhouse.no wrote:
Some more update.. got the same problem today.. and when it happend, ser blocked. and I got ALOT of these error messages. Anybody know why ?
Ser runs out of "local" memory. The localtions are first loaded in local (per process memory) and copied and inserted in the usrloc shared memory list. (probably you have a lot of entries) By default ser uses maximum 1Mb of local memory/process. Try to increases it to a bigger value. Change PKG_MEM_POOL_SIZE in config.h and recompile.
D'oh, same problem again here now (Ser 0.8.12)!
It started with log entries like this:
Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: build_req_buf_from_sip_req: out of memory Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: print_uac_request: no pkg_mem Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: t_forward_nonack: failure to add branches
This could be a mem. leak (but this is unrelated to the startup problem).
I thought that maybe a restart could help, but SER didn't come up again, shouting:
Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_row(): No memory left Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_rows(): Error while converting row #6346 Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_result(): Error while converting rows Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: get_result(): Error while converting result Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: preload_udomain(): Error while doing db_query Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: register_udomain(): Error while preloading domain 'location' Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: domain_fixup(): Error while registering domain
After deleting 100 entries from location it came up again. Hope that the increase of PKG_MEM_POOL_SIZE to 2MB helps.
So it seems that after about 6300 entries or so in location Ser stops working with the default configuration!? I'd be interested in how many users do other Ser-Servers out there serve? Are there similiar problems?
Regards, Andy
Andrei
At 12:43 AM 6/25/2004, Andrei Pelinescu-Onciul wrote:
On Jun 25, 2004 at 00:30, Jiri Kuthan jiri@iptel.org wrote:
Under normal operation, SER can handle significantly more registrations, by orders of magnitude. To me, it sounds much more like a result of a memory leak.
Jiri, note that this happens on startup. In fact it runs out of mem.
oh thanks -- I misread. -jiri
when it tries to alloc. a chunk of memory big enough to hold all the entries in location. This is not a memory leak and it really happens if PKG_MEM_POOL_SIZE is 1Mb and you have >6000 entries in location (it depends of course on your average entry size, but the critical point is in general between 6000-7000).
Are you using stock SER or some modified SER? In any case, the way to catch such leaks is to set a talkative memory log level (memlog=2 in ser.cfg), recompile ser with memory debugging informaiton (uncomment DBG_QM_MALLOC, comment F_MALLOC in Makefile.defs and do recompile completely) and watch log output on SER shutdown for excessive occurent of unallocated memory fragement. I would be very thankful if you could report your observations back to us.
-jiri
At 12:54 PM 6/24/2004, Andreas Granig wrote:
Andrei Pelinescu-Onciul wrote:
On Jun 21, 2004 at 19:47, Atle Samuelsen clona@cyberhouse.no wrote:
Some more update.. got the same problem today.. and when it happend, ser blocked. and I got ALOT of these error messages. Anybody know why ?
Ser runs out of "local" memory. The localtions are first loaded in local (per process memory) and copied and inserted in the usrloc shared memory list. (probably you have a lot of entries) By default ser uses maximum 1Mb of local memory/process. Try to increases it to a bigger value. Change PKG_MEM_POOL_SIZE in config.h and recompile.
D'oh, same problem again here now (Ser 0.8.12)!
It started with log entries like this:
Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: build_req_buf_from_sip_req: out of memory Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: print_uac_request: no pkg_mem Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: t_forward_nonack: failure to add branches
This could be a mem. leak (but this is unrelated to the startup problem).
I thought that maybe a restart could help, but SER didn't come up again, shouting:
Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_row(): No memory left Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_rows(): Error while converting row #6346 Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_result(): Error while converting rows Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: get_result(): Error while converting result Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: preload_udomain(): Error while doing db_query Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: register_udomain(): Error while preloading domain 'location' Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: domain_fixup(): Error while registering domain
After deleting 100 entries from location it came up again. Hope that the increase of PKG_MEM_POOL_SIZE to 2MB helps.
So it seems that after about 6300 entries or so in location Ser stops working with the default configuration!? I'd be interested in how many users do other Ser-Servers out there serve? Are there similiar problems?
Regards, Andy
Andrei
-- Jiri Kuthan http://iptel.org/~jiri/
Andrei Pelinescu-Onciul wrote:
It started with log entries like this:
Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: build_req_buf_from_sip_req: out of memory Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: print_uac_request: no pkg_mem Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: t_forward_nonack: failure to add branches
This could be a mem. leak (but this is unrelated to the startup problem).
I use a stock SER (0.8.12) without modifications. I just execute some external C code by exec_dset and exec_msg, but this shouldn't matter.
Since this problem occured on our public server, I just wanted to get the service up and running again and thus didn't think about dumping the database for later analysis on the test system. However, I'll try to reproduce this effect and send some logs...
Regards, Andy
Hi Andy
* Andreas Granig a.granig@inode.at [040625 02:19]:
Andrei Pelinescu-Onciul wrote:
It started with log entries like this:
Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: build_req_buf_from_sip_req: out of memory Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: print_uac_request: no pkg_mem Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: t_forward_nonack: failure to add branches
This could be a mem. leak (but this is unrelated to the startup problem).
I use a stock SER (0.8.12) without modifications. I just execute some external C code by exec_dset and exec_msg, but this shouldn't matter.
Since this problem occured on our public server, I just wanted to get the service up and running again and thus didn't think about dumping the database for later analysis on the test system. However, I'll try to reproduce this effect and send some logs...
I would check with Andrei and see what he says.
-Atle
-
Andreas Granig -
Andrei Pelinescu-Onciul -
Atle Samuelsen -
Atle Samuelsen -
Jiri Kuthan