Hello,
I am wondering whether we are using the right defaults for code downloaded and compiled from the master branch in the git repository. I mean the code in that branch is almost always considered unstable and when people try to compile and use it they mostly do this for debugging purposes (well, at least developers, ordinary users should not be using the code).
If you compile the code from the git repository, using compilation defaults and a very simple configuration file, it is likely that you end up having lots of memory related messages in syslog and and binary optimized for speed which can be difficult to debug with gdb.
We have DBG_QM_MALLOC enabled by default in the git repository which I think is good, however, the value of memdbg is set to L_DBG and the value of debug is set to L_WARN by default. This means that unless you explicitly configure memdbg in your configuration file, you'll see all the memory debugging messages and it is a *a lot* of text.
You can disable memory debugging messages by setting the value of memdbg higher than debug, but I pretty much always forget to do it. Also the default configuration files are not consistent in this regard. The simple one disables it by setting debug to 2 (L_DBG is 3 so memory debugging is disabled), but if you want to investigate a problem and set debug to 3 to do that then you end up having lots of traffic in syslog again and you have to use memdbg explicitly in the configuration file. The complex one, sip-router-oob.cfg, uses the defaults so you get all the messages.
How about changing the default for memdbg to a value that is higher than 3 (the debugging value)? Then we would always have memory debugging messages disabled and people (both users and developers) can easily turn them on by configuring a lower value in the configuration file.
The second default which I believe is not entirely correct for the code stored in the master branch in the git repository is the compilation mode. The makefile system generates code that is optimized for speed by default. We use -O9 which turns on pretty much all performance optimizations in gcc. That includes variables stored in CPU registers, however, code that uses such optimizations is difficult to debug because gdb cannot display values of variables stored in registers properly.
Here again, I pretty much always forget to add mode=debug when configuring the build and often have to recompile the code after trying to use gdb to debug something. Shouldn't we use mode=debug by default for the code in git master branch?
Or perhaps it is just me having these issues? Maybe others have secret techniques to share how to avoid such dumb problems?
-- Jan
On Thu, Oct 15, 2009 at 2:06 AM, Jan Janak jan@ryngle.com wrote:
We have DBG_QM_MALLOC enabled by default in the git repository which I think is good, however, the value of memdbg is set to L_DBG and the value of debug is set to L_WARN by default. This means that unless you explicitly configure memdbg in your configuration file, you'll see all the memory debugging messages and it is a *a lot* of text.
You can disable memory debugging messages by setting the value of memdbg higher than debug, but I pretty much always forget to do it.
Here I forgot to mention that you also have to set memlog or mem_summary in the configuration file if you do now want to see the long memory statistics dump. Even with just two modules and a very simple configuration file the statistics is pretty dump is pretty big.
-- Jan
On Oct 15, 2009 at 03:14, Jan Janak jan@ryngle.com wrote:
On Thu, Oct 15, 2009 at 2:06 AM, Jan Janak jan@ryngle.com wrote:
We have DBG_QM_MALLOC enabled by default in the git repository which I think is good, however, the value of memdbg is set to L_DBG and the value of debug is set to L_WARN by default. This means that unless you explicitly configure memdbg in your configuration file, you'll see all the memory debugging messages and it is a *a lot* of text.
You can disable memory debugging messages by setting the value of memdbg higher than debug, but I pretty much always forget to do it.
Here I forgot to mention that you also have to set memlog or mem_summary in the configuration file if you do now want to see the long memory statistics dump. Even with just two modules and a very simple configuration file the statistics is pretty dump is pretty big.
Try mem_summary set to 2 (2 should be the default once we test it a little more).
It's good to have at least mem_summary=2 information, maybe that way more people will cleanup on mod_destroy and we will be able to spot memleaks easier (some code in the core, like the fixups part, needs also some work).
Andrei
On Oct 15, 2009 at 02:06, Jan Janak jan@ryngle.com wrote: [...]
We have DBG_QM_MALLOC enabled by default in the git repository which I think is good, however, the value of memdbg is set to L_DBG and the value of debug is set to L_WARN by default. This means that unless you explicitly configure memdbg in your configuration file, you'll see all the memory debugging messages and it is a *a lot* of text.
No, that means by _default_ you don't see any malloc debug message (debug=L_WARN and memdbg=L_DBG => memdbg>debug => you don't see them unless you change debug to L_DBG or bigger).
You can disable memory debugging messages by setting the value of memdbg higher than debug, but I pretty much always forget to do it. Also the default configuration files are not consistent in this regard. The simple one disables it by setting debug to 2 (L_DBG is 3 so memory debugging is disabled), but if you want to investigate a problem and set debug to 3 to do that then you end up having lots of traffic in syslog again and you have to use memdbg explicitly in the configuration file. The complex one, sip-router-oob.cfg, uses the defaults so you get all the messages.
In this case debug=L_WARN=0, so you won't see them.
How about changing the default for memdbg to a value that is higher than 3 (the debugging value)? Then we would always have memory debugging messages disabled and people (both users and developers) can easily turn them on by configuring a lower value in the configuration file.
Agreed.
The second default which I believe is not entirely correct for the code stored in the master branch in the git repository is the compilation mode. The makefile system generates code that is optimized for speed by default. We use -O9 which turns on pretty much all performance optimizations in gcc. That includes variables stored in CPU registers, however, code that uses such optimizations is difficult to debug because gdb cannot display values of variables stored in registers properly.
Here again, I pretty much always forget to add mode=debug when configuring the build and often have to recompile the code after trying to use gdb to debug something. Shouldn't we use mode=debug by default for the code in git master branch?
The problem with turning optimization off is that then you won't see any warnings related to features which are enabled only at higher optimizations levels (e.g. pointer aliasing, arrays overflows) and even more importantly you won't see some bugs (e.g. forgetting a compiler barrier won't have any ill effects since no vars are optimized in registers). I think it's better to find all the problems as early as possible and not wait for a release to turn on the optimizations.
Or perhaps it is just me having these issues? Maybe others have secret techniques to share how to avoid such dumb problems?
In some cases optimizations debugging is harder, but statistically that number of cases is small (we can figure most backtraces even with all the optimizations). I would rather have a fully optimized version and if I cannot figure a crash, turn on debugging, rather then having a version which works perfectly with debugging but sometimes crashes when optimized.
Andrei
On Donnerstag, 15. Oktober 2009, Andrei Pelinescu-Onciul wrote:
Or perhaps it is just me having these issues? Maybe others have secret techniques to share how to avoid such dumb problems?
In some cases optimizations debugging is harder, but statistically that number of cases is small (we can figure most backtraces even with all the optimizations). I would rather have a fully optimized version and if I cannot figure a crash, turn on debugging, rather then having a version which works perfectly with debugging but sometimes crashes when optimized.
Andrei,
this is a good point. In kamailio times the default for the code was also "- O9" (which is the same as -O3, IMHO), and debugging enabled. I don't remember any case at the moment where i need to change this because of debugging, so i think this setting is fine.
Regards,
Henning
On Thu, Oct 15, 2009 at 11:45 AM, Henning Westerholt henning.westerholt@1und1.de wrote:
On Donnerstag, 15. Oktober 2009, Andrei Pelinescu-Onciul wrote:
Or perhaps it is just me having these issues? Maybe others have secret techniques to share how to avoid such dumb problems?
In some cases optimizations debugging is harder, but statistically that number of cases is small (we can figure most backtraces even with all the optimizations). I would rather have a fully optimized version and if I cannot figure a crash, turn on debugging, rather then having a version which works perfectly with debugging but sometimes crashes when optimized.
Andrei,
this is a good point. In kamailio times the default for the code was also "-O9" (which is the same as -O3, IMHO), and debugging enabled. I don't remember any case at the moment where i need to change this because of debugging, so i think this setting is fine.
I agree, this is a good point and I no longer think we should change the compilation mode. I did not realize that we would not be able to catch bugs like that in debug mode.
-- Jan
-
Andrei Pelinescu-Onciul -
Henning Westerholt -
Jan Janak