at some point i have started to get these kind of warning when i stop my sip proxy: --text follows this line-- Oct 1 10:23:01 siika /usr/sbin/sip-proxy[16970]: WARNING: <core> [mem/f_malloc.c:512]: WARNING:fm_free: free(0) called
any idea where they come from or how i could found out where they come from?
-- juha
Hello,
I moved the check for *free(0) before the check for range (min and max address of pkg/shm) - 0 is out of range always and the range check was done only for mem debugging mode.
What version are you using? Being fm_free, means it is without mem debug on, so it should be same behavior as before. This memory manager does not keep track of locations for malloc/free, you have to recompile with MEMDBG=1 and then you should see the location details.
Cheers, Daniel
On 10/1/12 9:27 AM, Juha Heinanen wrote:
at some point i have started to get these kind of warning when i stop my sip proxy: --text follows this line-- Oct 1 10:23:01 siika /usr/sbin/sip-proxy[16970]: WARNING: <core> [mem/f_malloc.c:512]: WARNING:fm_free: free(0) called
any idea where they come from or how i could found out where they come from?
-- juha
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Daniel-Constantin Mierla writes:
What version are you using? Being fm_free, means it is without mem debug on, so it should be same behavior as before. This memory manager does not keep track of locations for malloc/free, you have to recompile with MEMDBG=1 and then you should see the location details.
i see those warnings at least with latest master. i'll compile with MEMDBG=1 to find out more.
-- juha
Daniel-Constantin Mierla writes:
What version are you using? Being fm_free, means it is without mem debug on, so it should be same behavior as before. This memory manager does not keep track of locations for malloc/free, you have to recompile with MEMDBG=1 and then you should see the location details.
i recompiled with these options:
/usr/bin/make cfg CC=cc CFLAGS="-Wall -g" MEMDBG=1 QUIET=1 FMSTATS=1 KMSTATS=1 SCTP=1 STUN=1 ...
but don't get any more info to syslog than earlier:
Oct 2 14:49:15 siika /usr/sbin/sip-proxy[26672]: WARNING: <core> [mem/f_malloc.c:512]: WARNING:fm_free: free(0) called
-- juha
On 10/2/12 1:53 PM, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
What version are you using? Being fm_free, means it is without mem debug on, so it should be same behavior as before. This memory manager does not keep track of locations for malloc/free, you have to recompile with MEMDBG=1 and then you should see the location details.
i recompiled with these options:
/usr/bin/make cfg CC=cc CFLAGS="-Wall -g" MEMDBG=1 QUIET=1 FMSTATS=1 KMSTATS=1 SCTP=1 STUN=1 ...
but don't get any more info to syslog than earlier:
Oct 2 14:49:15 siika /usr/sbin/sip-proxy[26672]: WARNING: <core> [mem/f_malloc.c:512]: WARNING:fm_free: free(0) called
Interesting, after compiling in the source tree, can you paste the output of:
./kamailio -I
(that's an upper case i).
Cheers, Daniel
-- juha
Daniel-Constantin Mierla writes:
Interesting, after compiling in the source tree, can you paste the output of:
./kamailio -I
(that's an upper case i).
given in source dir after compilation:
/usr/src/trunk-src/openxg-sip-proxy$ ./sip-proxy -I Print out of sip-proxy internals Version: sip-proxy 3.4.0-dev4 (x86_64/linux) Default config: /etc/sip-proxy/sip-proxy.cfg Default paths to modules: /usr/lib/sip-proxy/modules:/usr/lib/sip-proxy/modules_s:/usr/lib/sip-proxy/modules_k Compile flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, USE_STUN, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES MAX_RECV_BUFFER_SIZE=262144 MAX_LISTEN=16 MAX_URI_SIZE=1024 BUF_SIZE=65535 DEFAULT PKG_SIZE=4MB DEFAULT SHM_SIZE=32MB ADAPTIVE_WAIT_LOOPS=1024 TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt, select Source code revision ID: unknown Compiled with: cc 4.7.1 Compiled on: 14:40:42 Oct 2 2012 Thank you for flying sip-proxy!
for some reason MEMDBG does not show up in the flags although for sure it was on the make line:
/usr/bin/make cfg CC=cc CFLAGS="-Wall -g" MEMDBG=1 QUIET=1 FMSTATS=1 KMSTATS=1 MAIN_NAME=sip-proxy SCTP=1 STUN=1 LIBDIR=lib ...
-- juha
I just tried 'make cfg MEMDBG=1' and the -DDBG_QM_MALLOC is there (you can check config.mak).
Do a make proper before.
Cheers, Daniel
On 10/2/12 2:25 PM, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
Interesting, after compiling in the source tree, can you paste the output of:
./kamailio -I
(that's an upper case i).
given in source dir after compilation:
/usr/src/trunk-src/openxg-sip-proxy$ ./sip-proxy -I Print out of sip-proxy internals Version: sip-proxy 3.4.0-dev4 (x86_64/linux) Default config: /etc/sip-proxy/sip-proxy.cfg Default paths to modules: /usr/lib/sip-proxy/modules:/usr/lib/sip-proxy/modules_s:/usr/lib/sip-proxy/modules_k Compile flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, USE_STUN, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES MAX_RECV_BUFFER_SIZE=262144 MAX_LISTEN=16 MAX_URI_SIZE=1024 BUF_SIZE=65535 DEFAULT PKG_SIZE=4MB DEFAULT SHM_SIZE=32MB ADAPTIVE_WAIT_LOOPS=1024 TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt, select Source code revision ID: unknown Compiled with: cc 4.7.1 Compiled on: 14:40:42 Oct 2 2012 Thank you for flying sip-proxy!
for some reason MEMDBG does not show up in the flags although for sure it was on the make line:
/usr/bin/make cfg CC=cc CFLAGS="-Wall -g" MEMDBG=1 QUIET=1 FMSTATS=1 KMSTATS=1 MAIN_NAME=sip-proxy SCTP=1 STUN=1 LIBDIR=lib ...
-- juha
Daniel-Constantin Mierla writes:
I just tried 'make cfg MEMDBG=1' and the -DDBG_QM_MALLOC is there (you can check config.mak).
i added directly to -DDBG_QM_MALLOC to CFLAGS:
/usr/bin/make cfg CC=cc CFLAGS="-Wall -DDBG_QM_MALLOC -g" MEMDBG=1 QUIET=1 FMSTATS=1 KMSTATS=1 MAIN_NAME=sip-proxy SCTP=1 STUN=1
after that -DDBG_QM_MALLOC shows up with -I:
# /usr/sbin/sip-proxy -I Print out of sip-proxy internals Version: sip-proxy 3.4.0-dev4 (x86_64/linux) Default config: /etc/sip-proxy/sip-proxy.cfg Default paths to modules: /usr/lib/sip-proxy/modules:/usr/lib/sip-proxy/modules_s:/usr/lib/sip-proxy/modules_k Compile flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, USE_STUN, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
when i then stop the proxy at debug level 2, i get the same free(0) WARNINGs to syslog and no additional info. when i set debug level to 3 and then stop the proxy, i get lots of mem DEBUGs to syslog, but NOT the WARNINGs. does this mean that the problem does not appear at debug level 3?
-- juha
On 10/2/12 3:06 PM, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
I just tried 'make cfg MEMDBG=1' and the -DDBG_QM_MALLOC is there (you can check config.mak).
i added directly to -DDBG_QM_MALLOC to CFLAGS:
/usr/bin/make cfg CC=cc CFLAGS="-Wall -DDBG_QM_MALLOC -g" MEMDBG=1 QUIET=1 FMSTATS=1 KMSTATS=1 MAIN_NAME=sip-proxy SCTP=1 STUN=1
This is not the right way, because when DBG_QM_MALLOC is activated, the F_MALLOC has to dissapear.
It is what MEMDBG=1 does, adds some compile flags and removes others.
Have you tried make proper?
Cheers, Daniel
after that -DDBG_QM_MALLOC shows up with -I:
# /usr/sbin/sip-proxy -I Print out of sip-proxy internals Version: sip-proxy 3.4.0-dev4 (x86_64/linux) Default config: /etc/sip-proxy/sip-proxy.cfg Default paths to modules: /usr/lib/sip-proxy/modules:/usr/lib/sip-proxy/modules_s:/usr/lib/sip-proxy/modules_k Compile flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, USE_STUN, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
when i then stop the proxy at debug level 2, i get the same free(0) WARNINGs to syslog and no additional info. when i set debug level to 3 and then stop the proxy, i get lots of mem DEBUGs to syslog, but NOT the WARNINGs. does this mean that the problem does not appear at debug level 3?
-- juha
Daniel-Constantin Mierla writes:
It is what MEMDBG=1 does, adds some compile flags and removes others.
is there something i would need to remove?
Have you tried make proper?
i made each time debian package from scratch, i.e., my script copies source tree from git to an empty directory and then issues command to build the package. i thus don't see why i would need to issue 'make proper'. the result is that MEMDBG=1 does not set DBG_QM_MALLOC.
when i include -DDBG_QM_MALLOC to CFLAGS, perhaps i would need to turn off manually some other flags:
Compile flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, USE_STUN, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
-- juha
I think debian has its own rules file for generating the make command.
Better change the Makefile.defs and set MEMDBG=1 inside it.
Cheers, Daniel
On 10/2/12 3:31 PM, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
It is what MEMDBG=1 does, adds some compile flags and removes others.
is there something i would need to remove?
Have you tried make proper?
i made each time debian package from scratch, i.e., my script copies source tree from git to an empty directory and then issues command to build the package. i thus don't see why i would need to issue 'make proper'. the result is that MEMDBG=1 does not set DBG_QM_MALLOC.
when i include -DDBG_QM_MALLOC to CFLAGS, perhaps i would need to turn off manually some other flags:
Compile flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, USE_STUN, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
-- juha
Daniel-Constantin Mierla writes:
Better change the Makefile.defs and set MEMDBG=1 inside it.
ok, i added this to my debian/rules:
# Turn on memory debuging sed -i -e 's/MEMDBG ?= 0/MEMDBG ?= 1/' Makefile.defs
after build, i checked in source dir that the change took effect:
/usr/src/trunk-src/openxg-sip-proxy$ egrep MEMDBG Makefile.defs MEMDBG ?= 1 ifeq ($(MEMDBG), 1)
now also -I shows DBG_QM_MALLOC flag:
/usr/src/trunk-src/openxg-sip-proxy$ ./sip-proxy -I Print out of sip-proxy internals Version: sip-proxy 3.4.0-dev4 (x86_64/linux) Default config: /etc/sip-proxy/sip-proxy.cfg Default paths to modules: /usr/lib/sip-proxy/modules:/usr/lib/sip-proxy/modules_s:/usr/lib/sip-proxy/modules_k Compile flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, USE_STUN, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
still, when i stop sip-proxy at debug level 2, i only get the WARNINGs:
Oct 2 17:51:12 siika /usr/sbin/sip-proxy[30707]: WARNING: <core> [mem/q_malloc.c:441]: WARNING:qm_free: free(0) called
and at debug level 3 i don't get any WARNINGs. does debug level 3 replace WARNINGs with DEBUG messages instead of adding DEBUG messages on top of WARNINGs?
-- juha
You have to set memdbg (or memlog) parameters to lower value than debug.
Cheers, Daniel
On 2 Oct 2012, at 16:57, Juha Heinanen jh@tutpro.com wrote:
Daniel-Constantin Mierla writes:
Better change the Makefile.defs and set MEMDBG=1 inside it.
ok, i added this to my debian/rules:
# Turn on memory debuging sed -i -e 's/MEMDBG ?= 0/MEMDBG ?= 1/' Makefile.defs
after build, i checked in source dir that the change took effect:
/usr/src/trunk-src/openxg-sip-proxy$ egrep MEMDBG Makefile.defs MEMDBG ?= 1 ifeq ($(MEMDBG), 1)
now also -I shows DBG_QM_MALLOC flag:
/usr/src/trunk-src/openxg-sip-proxy$ ./sip-proxy -I Print out of sip-proxy internals Version: sip-proxy 3.4.0-dev4 (x86_64/linux) Default config: /etc/sip-proxy/sip-proxy.cfg Default paths to modules: /usr/lib/sip-proxy/modules:/usr/lib/sip-proxy/modules_s:/usr/lib/sip-proxy/modules_k Compile flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, USE_STUN, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
still, when i stop sip-proxy at debug level 2, i only get the WARNINGs:
Oct 2 17:51:12 siika /usr/sbin/sip-proxy[30707]: WARNING: <core> [mem/q_malloc.c:441]: WARNING:qm_free: free(0) called
and at debug level 3 i don't get any WARNINGs. does debug level 3 replace WARNINGs with DEBUG messages instead of adding DEBUG messages on top of WARNINGs?
-- juha
Quite strange I would say.
No, debug parameter value does not replace warning level with debug - just to comment on your previous message related to it.
But now do you get the messages related to alloc and free operations?
It should be first the message from mem/q_malloc.c, line 437, after it is the condition on free(0).
Cheers, Daniel
On 10/2/12 6:04 PM, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
You have to set memdbg (or memlog) parameters to lower value than debug.
now i have
memdbg=1 memlog=1 debug=2
and when i stop sip-proxy i still don't get any free(0) warnings. i do get the warnings if i have not set memdbg and memlog.
-- juha
Daniel-Constantin Mierla writes:
But now do you get the messages related to alloc and free operations?
yes.
It should be first the message from mem/q_malloc.c, line 437, after it is the condition on free(0).
i see lots of these kind of entries, but no free(0) entries:
Oct 2 21:56:02 siika /usr/sbin/sip-proxy[15833]: NOTICE: <core> [mem/q_malloc.c:437]: qm_free(0x7f31b9db0000, 0x7f31ba93a4b0), called from <core>: usr_avp.c: destroy_avp_list_unsafe(634) Oct 2 21:56:02 siika /usr/sbin/sip-proxy[15833]: NOTICE: <core> [mem/q_malloc.c:468]: qm_free: freeing frag. 0x7f31ba93a480 alloc'ed from <core>: usr_avp.c: create_avp(187) ...
Oct 2 21:56:06 siika /usr/sbin/sip-proxy[15847]: NOTICE: <core> [mem/q_malloc.c:437]: qm_free(0x7f31c0c19010, 0x7f31c0d284f8), called from nathelper: nathelper.c: nh_timer(1986)
Oct 2 21:56:06 siika /usr/sbin/sip-proxy[15847]: NOTICE: <core> [mem/q_malloc.c:468]: qm_free: freeing frag. 0x7f31c0d284c8 alloc'ed from nathelper: nathelper.c: nh_timer(1871)
free(0) at close shows up only when i have not set memdbg/memlog vars.
-- juha
Strange, unless it is related to mem debug logs.
Anyhow, I pushed a patch to master branch to print the file and line for free(0) when compiled with memdbg=1.
Try it to see if you get the details of free(0) location.
Cheers, Daniel
On 10/2/12 9:18 PM, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
But now do you get the messages related to alloc and free operations?
yes.
It should be first the message from mem/q_malloc.c, line 437, after it is the condition on free(0).
i see lots of these kind of entries, but no free(0) entries:
Oct 2 21:56:02 siika /usr/sbin/sip-proxy[15833]: NOTICE: <core> [mem/q_malloc.c:437]: qm_free(0x7f31b9db0000, 0x7f31ba93a4b0), called from <core>: usr_avp.c: destroy_avp_list_unsafe(634) Oct 2 21:56:02 siika /usr/sbin/sip-proxy[15833]: NOTICE: <core> [mem/q_malloc.c:468]: qm_free: freeing frag. 0x7f31ba93a480 alloc'ed from <core>: usr_avp.c: create_avp(187) ...
Oct 2 21:56:06 siika /usr/sbin/sip-proxy[15847]: NOTICE: <core> [mem/q_malloc.c:437]: qm_free(0x7f31c0c19010, 0x7f31c0d284f8), called from nathelper: nathelper.c: nh_timer(1986)
Oct 2 21:56:06 siika /usr/sbin/sip-proxy[15847]: NOTICE: <core> [mem/q_malloc.c:468]: qm_free: freeing frag. 0x7f31c0d284c8 alloc'ed from nathelper: nathelper.c: nh_timer(1871)
free(0) at close shows up only when i have not set memdbg/memlog vars.
-- juha
Daniel-Constantin Mierla writes:
Anyhow, I pushed a patch to master branch to print the file and line for free(0) when compiled with memdbg=1.
Try it to see if you get the details of free(0) location.
daniel,
thanks for the patch. i now got bunch of these at shutdown:
Oct 2 22:43:54 siika /usr/sbin/sip-proxy[5996]: WARNING: <core> [mem/q_malloc.c:442]: WARNING:qm_free: free(0) called from tls: tls_init.c: ser_free(298)
-- juha
This is actually the reason I moved the check for free(0) before the check of the memory address range. Someone reported on irc that libssl does free(0) causing core dump due to an internal abort() executed in qm_free(p), if p is less than minimum memory address (which was the case when p=0).
I think we can make it MDBG(...) in f_malloc.c, so it is printed only if one wants memory debug messages -- this memory manager is intended for production/stable releases, although in the past we did some releases with q_malloc.
In q_malloc.c maybe it should stay the same, being for troubleshooting reasons, to spot if we have free(0) inside our code.
You can go and update in f_malloc and backport, if it is a solution that is ok for you.
Cheers, Daniel
On 10/2/12 9:45 PM, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
Anyhow, I pushed a patch to master branch to print the file and line for free(0) when compiled with memdbg=1.
Try it to see if you get the details of free(0) location.
daniel,
thanks for the patch. i now got bunch of these at shutdown:
Oct 2 22:43:54 siika /usr/sbin/sip-proxy[5996]: WARNING: <core> [mem/q_malloc.c:442]: WARNING:qm_free: free(0) called from tls: tls_init.c: ser_free(298)
-- juha
Daniel-Constantin Mierla writes:
You can go and update in f_malloc and backport, if it is a solution that is ok for you.
i committed the LOG -> MDBG change to master. i still don't understand though, why it is only me who is seeing those warnings, since it is highly unlikely that others don't use tls.
also, i don't see those warnings on another host that runs kamailio 3.3 on debian i386 squeeze, but see the warnings on a host running kamailio master on debian amd64 wheezy.
-- juha
On 10/3/12 8:35 AM, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
You can go and update in f_malloc and backport, if it is a solution that is ok for you.
i committed the LOG -> MDBG change to master. i still don't understand though, why it is only me who is seeing those warnings, since it is highly unlikely that others don't use tls.
also, i don't see those warnings on another host that runs kamailio 3.3 on debian i386 squeeze, but see the warnings on a host running kamailio master on debian amd64 wheezy.
As said, these *free(0) are coming from inside libssl, so it might be just a matter of libssl version.
Cheers, Daniel
-
Daniel-Constantin Mierla -
Juha Heinanen