Hello,

there is no indication yet where is the leak, so you have to upgrade core and all modules you are using.

If you configured with 4GB of memory, then there is a leak. You say you have your own customized modules, you should investigate your changes first to see if there are leaks.

There are some notes on how to compile with memory debugger and troubleshoot leaks at:

  - http://kamailio.org/wiki/tutorials/troubleshooting/memory

Cheers,
Daniel

On 06/05/15 10:40, Bui Minh Yen wrote:
Hello Daniel,

Thank you so much for your answer
I want to ask you something before upgrade kamailio, currently we are config shared memory size is 4GB, and normal kamailio only use 12-14 MB. We are changed some module for our purpose like using cnxcc (using for prepaid) and alias_db module for call forwarding feature. Like you said we should upgrade to kamailio-4.1.8, so we have to upgrade all modules and core or only some modules which related to this error? If only some modules, could you tell me what modules?

Thanks so much for your help!  

On Wed, May 6, 2015 at 2:34 PM, Daniel-Constantin Mierla <miconda@gmail.com> wrote:
Hello,

upgrade to kamailio 4.1.8, because there are fixes since 4.1.5. Same config and database structure are used in all 4.1.x releases, therefore you just need to deploy the new version and restart.

Also, be sure you have allocated sufficient shared memory (-m command line parameter), by default is 32MB, once that is filled, kamailio throws out of memory errors.

Cheers,
Daniel


On 06/05/15 04:27, Bui Minh Yen wrote:
Hi everyone,

I am using Kamailio-4.1.5. Sometime kamailio was crash because memory leak.

Log when out of mem

May  2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: tm [t_lookup.c:1472]: t_newtran(): ERROR: t_newtran: new_t failed
May  2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: tm [t_lookup.c:1332]: new_t(): ERROR: new_t: out of mem:
May  2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: tm [t_lookup.c:1472]: t_newtran(): ERROR: t_newtran: new_t failed
May  2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: usrloc [ucontact.c:103]: new_ucontact(): no more shm memory
May  2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: usrloc [urecord.c:155]: mem_insert_ucontact(): failed to create new contact
May  2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: usrloc [urecord.c:547]: insert_ucontact(): failed to insert contact
May  2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: registrar [save.c:708]: update_contacts(): failed to insert contact
May  2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: tm [t_lookup.c:1332]: new_t(): ERROR: new_t: out of mem:
May  2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: tm [t_lookup.c:1472]: t_newtran(): ERROR: t_newtran: new_t failed
May  2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: usrloc [ucontact.c:103]: new_ucontact(): no more shm memory
May  2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: usrloc [urecord.c:155]: mem_insert_ucontact(): failed to create new contact
May  2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: usrloc [urecord.c:547]: insert_ucontact(): failed to insert contact
May  2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: registrar [save.c:521]: insert_contacts(): failed to insert contact

Log when kamailio was crash

May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30818]: ALERT: <core> [main.c:775]: handle_sigs(): child process 30821 exited by a signal 11
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30818]: ALERT: <core> [main.c:778]: handle_sigs(): core was not generated
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30818]: INFO: <core> [main.c:790]: handle_sigs(): INFO: terminating due to SIGCHLD
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: <core> [main.c:852]: sig_usr(): Memory status (pkg):
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: (0x7f6af5968010):
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: heap size= 749731840
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: used= 1110960, used+overhead=1663384, free=748068456
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: max used (+overhead)= 1825240
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: dumping all alloc'ed. fragments:
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:      0. N  address=0x7f6af599d528 frag=0x7f6af599d4f8 size=1024 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:           alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:      1. N  address=0x7f6af599d988 frag=0x7f6af599d958 size=256 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30828]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:           alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:      2. N  address=0x7f6af599dae8 frag=0x7f6af599dab8 size=176 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:           alloc'd from <core>: counters.c: cnt_hash_add(339)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:      3. N  address=0x7f6af599dbf8 frag=0x7f6af599dbc8 size=240 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:           alloc'd from <core>: counters.c: cnt_hash_add(339)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:      4. N  address=0x7f6af599dd48 frag=0x7f6af599dd18 size=128 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:           alloc'd from <core>: counters.c: init_counters(129)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:      5. N  address=0x7f6af599de28 frag=0x7f6af599ddf8 size=16 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:           alloc'd from <core>: modparam.c: set_mod_param_regex(124)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:      6. N  address=0x7f6af599de98 frag=0x7f6af599de68 size=128 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:           alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:      7. N  address=0x7f6af599df78 frag=0x7f6af599df48 size=48 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:           alloc'd from <core>: route.c: route_add(159)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:      8. N  address=0x7f6af599e008 frag=0x7f6af599dfd8 size=16 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:           alloc'd from <core>: route.c: init_rlist(181)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:      9. N  address=0x7f6af599e078 frag=0x7f6af599e048 size=128 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:           alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:     10. N  address=0x7f6af599e158 frag=0x7f6af599e128 size=48 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:           alloc'd from <core>: route.c: route_add(159)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:     11. N  address=0x7f6af599e1e8 frag=0x7f6af599e1b8 size=16 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:           alloc'd from <core>: route.c: init_rlist(181)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:     12. N  address=0x7f6af599e258 frag=0x7f6af599e228 size=128 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:           alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:     13. N  address=0x7f6af599e338 frag=0x7f6af599e308 size=48 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:           alloc'd from <core>: route.c: route_add(159)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:     14. N  address=0x7f6af599e3c8 frag=0x7f6af599e398 size=16 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:           alloc'd from <core>: route.c: init_rlist(181)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:     15. N  address=0x7f6af599e438 frag=0x7f6af599e408 size=128 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:           alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: <core> [main.c:852]: sig_usr(): Memory status (pkg):
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: qm_status: (0x7f6af5968010):
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: qm_status: heap size= 749731840
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: qm_status: used= 1110768, used+overhead=1663192, free=748068648
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: qm_status: max used (+overhead)= 1825240
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: qm_status: dumping all alloc'ed. fragments:
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: qm_status:      0. N  address=0x7f6af599d528 frag=0x7f6af599d4f8 size=1024 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: qm_status:           alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: qm_status:      1. N  address=0x7f6af599d988 frag=0x7f6af599d958 size=256 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: qm_status:           alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30827]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30828]: NOTICE: <core> [main.c:852]: sig_usr(): Memory status (pkg):
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: <core> [main.c:852]: sig_usr(): Memory status (pkg):
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status: (0x7f6af5968010):
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status: heap size= 749731840
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status: used= 1110768, used+overhead=1663192, free=748068648
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status: max used (+overhead)= 1825240
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status: dumping all alloc'ed. fragments:
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30828]: NOTICE: qm_status: (0x7f6af5968010):
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status:      0. N  address=0x7f6af599d528 frag=0x7f6af599d4f8 size=1024 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status:           alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: <core> [main.c:852]: sig_usr(): Memory status (pkg):
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: qm_status: (0x7f6af5968010):
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: qm_status: heap size= 749731840
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: qm_status: used= 1110768, used+overhead=1663192, free=748068648
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: qm_status: max used (+overhead)= 1825240
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: qm_status: dumping all alloc'ed. fragments:
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: qm_status:      0. N  address=0x7f6af599d528 frag=0x7f6af599d4f8 size=1024 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status:      1. N  address=0x7f6af599d988 frag=0x7f6af599d958 size=256 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: qm_status:           alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status:           alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30834]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status:      2. N  address=0x7f6af599dae8 frag=0x7f6af599dab8 size=176 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status:           alloc'd from <core>: counters.c: cnt_hash_add(339)
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: qm_status:      1. N  address=0x7f6af599d988 frag=0x7f6af599d958 size=256 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30834]: NOTICE: <core> [main.c:852]: sig_usr(): Memory status (pkg):
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status:     16. N  address=0x7f6af599e518 frag=0x7f6af599e4e8 size=48 used=1
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30834]: NOTICE: qm_status: (0x7f6af5968010):
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30822]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received
May  2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30822]: NOTICE: <core> [main.c:852]: sig_usr(): Memory status (pkg):

Could you help me to fix this problem!

Thanks so much!

--

---------------------------

Bui Thi Minh Yen



_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users



-- 
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio World Conference, May 27-29, 2015
Berlin, Germany - http://www.kamailioworld.com



--

---------------------------

Bui Thi Minh Yen

SSDC, VNPT Technology

124 Hoang Quoc Viet | Mobile:  (+84) 973110551 | Skype:  minhyenbt


-- 
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio World Conference, May 27-29, 2015
Berlin, Germany - http://www.kamailioworld.com