### Description
We noticed a PKG memory leak when using DMQ module's function _dmq_bcast_message_ .
### Troubleshooting
#### Reproduction
Issue can be reproduced with configuration below.
* 3 Kamailio nodes each one listening on dedicated port 5070 for DMQ on their private IP address. * A dns name _[KAMAILIO_DNS]_ resolving these 3 nodes:
``` # dig +short A [KAMAILIO_DNS] 10.234.97.169 10.234.97.170 10.234.92.126 ```
DMQ module configured as below (example for node 1) ``` loadmodule "dmq.so" ... modparam("dmq", "server_address", "sip:10.234.97.169:5070") modparam("dmq", "notification_address", "sip:[KAMAILIO_DNS]:5070") modparam("dmq", "ping_interval", 10) modparam("dmq", "multi_notify", 1) ```
Each time an INVITE comes in, we use function below to broadcast custom message to all of the nodes:
``` ... if(is_method("INVITE")) { dmq_bcast_message("invitenotification", "invite received", "text/plain"); } ... ```
This custom message is handled as below:
``` request_route { ... if(is_method("KDMQ")) { if($Rp == 5070) route(DMQHANDLE); else drop; } ... } ...
route[DMQHANDLE] { if($tU == "invitenotification") { # Doing some additional stuff here, but not the cause of the leak send_reply("200", "OK"); } else { dmq_handle_message(); } } ```
Using **sipp** to send lots of messages, we can easily notice the free PKG memory decreasing over time (using `kamcmd pkg.stats`)
#### Debugging Data
Processes impacted by the leak are the ones processing SIP messages. In case of TCP, it is one of the tcp receivers. When doing a dump of such processes with command below:
``` kamcmd cfg.set_now_int core mem_dump_pkg [PID] ```
Results are full of records like these ones:
``` ALERT: qm_status: alloc'd from core: core/parser/parse_from.c: parse_from_header(63) ALERT: qm_status: alloc'd from core: core/parser/parse_addr_spec.c: parse_to_param(285) ```
We are sure that the leak is generated by this DMQ broadcast function: we isolated it and noticed no leaks when removing it from a basic script configuration.
We tried different configurations: * disabling multi_notify => memory leak still present * using an IP address instead of the DNS name => memory leak still present
Valgrind did not help us to find out more precisely where the issue in the code could be located.
### Possible Solutions
Would a _free_ of _msg->from->parsed_ be missing somewhere in DMQ code? Behavior looks very similar to https://github.com/kamailio/kamailio/pull/59
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` version: kamailio 5.3.7 (x86_64/linux) c16b3f-dirty flags: , EXTRA_DEBUGUSE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, TIMER_DEBUG, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ```
* **Operating System**:
``` Debian 9.13 under docker Linux 4.9.0-9-amd64 ```
Could not spot any potential issue upon quick look at the code.
To use valgrind to track private memory issues, you have to compile Kamailio with system memory, like:
``` make MEMPKG=sys include_modules="..." cfg make all ... ```
Try to see if you can get any leads with valgrind.
Thanks for having a look into it. We did compile with system memory and use valgrind following the tips on dedicated page https://www.kamailio.org/wiki/tutorials/troubleshooting/memory
Unfortunately this gave no more clues (though it is possible I used it badly because it was the first time I used such tools).
We removed totally DMQ module from our Kamailios anyhow. We replaced it with a solution based on mqueue/http_async_client modules.
This issue can be closed as I am no more able to help in investigating further. Thank you.
OK. Maybe it will surface again with more details.
Closed #2600.