4 Feb
2019
4 Feb
'19
2:26 p.m.
### Description
Kamailio 5.1 on Debian
- Enabled xavp_rcd in registrar
- Shared memory usage keeps growing over time
### Troubleshooting
- `mod.stats core all `show a large amount of allocated memory for xavp_new_value that
keeps growing over time
` xavp_new_value(94): 48815768`
#### Reproduction
We have the same symptom on multiple servers
#### Log Messages
Nothing shown in the log yet.
### Possible Solutions
Using reg_fetch_contacts to grab expiry time is an alternative that works.
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
```
version: kamailio 5.1.4 (x86_64/linux)
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST,
DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled on 10:59:52 Jul 3 2018 with gcc 6.3.0
```
* **Operating System**:
```
Debian 9.7 x86_64
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1834
4 Feb
4 Feb
6:20 p.m.
Are you using any async functions (t_suspend(), async or evapi modules ...)?
What is the registrar functions you use in the config after which you expect this leak
happens? Are they used for a normal classic route block, or is inside an event_route
block?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1834#issuecomment-460287374
5 Feb
5 Feb
10:30 a.m.
No async functions. Just a standard save() - one with reply sent and one for replicated
registrations with no reply sent.
All used in a classic route block .
I have an event_route for registry expirations, but nothing else. Changed one server to
not use the rcd xavp and instead use reg_fetch_contacts and that stopped the memory
growth.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1834#issuecomment-460539182
12:05 p.m.
Can you give the relevant snippet of config that results in leak? Like fro `save()` to the
end of processing for REGISTER, or where you use the xavp_rcd. I want to make a basic
config to try to reproduce.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1834#issuecomment-460561775
12:48 p.m.
```
$var(rc) = save("location", 0x04);
if (!$var(rc)) {
xlog("L_ERR", "-- REGISTRAR: failed to save
registration: $tu => UA: $ua, IP: $si:$sp, Caller-ID: $ci, CSeq: $cs, Expiry:
$hdr(Expires), Contact $ct, Return code: $var(rc)\n");
sl_reply_error();
exit;
}
if ($hdr(Expires) == "") {
$var(expire) = $sel(contact.expires);
} else {
$var(expire) = $hdr(Expires);
}
$var(regtime) = $xavp(ulrcd[0]=>expires);
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1834#issuecomment-460574714
13 Feb
13 Feb
3:37 p.m.
Can you test with the latest version of the branch? There was a related fix with commit
adc4493fa6861895bdcf8b459e5fbc76e80d0f1f . It will be useful to know if that was for your
case or there is something else.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1834#issuecomment-463183830
3:43 p.m.
I will try, but probably not until next week sometime. Please stand by.
We downgraded our production servers to an older solution.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1834#issuecomment-463185617
3:58 p.m.
@miconda i don't think its related. the issue on
[adc4493](https://github.com/kamailio/kamailio/commit/adc4493fa6861895bdcf8b…
was to prevent crashing after leak that led to shm exhaustion.
in our case it was a instant shm exhaustion not a slow leak. the slow leak could be
related to the growth in xavp_rcd_helper or lookup_helper.
i still need to proof this but my theory is that different processes accessed the contact
xvap or the ulrcd in shm during xavp_clone and created a circular reference. something
like a register and a notify (that calls lookup) at the same time. the server worked
stable for 2 weeks, no shm growth, suddenly shm was exhausted, and this unprotected
xavp_clone made sip workers died and server became unresponsive.
btw, is there a way to create/generate/force a core file with the contents of the shm mem
? the core files we got (from sip workers) could not access the shm mem to see if the shm
xavp's contents could proof my theory
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1834#issuecomment-463189815
21 Feb
21 Feb
6:11 p.m.
@lazedo - the core file contains the shared memory, but you need to know where to look for
xavps. They are linked either in the transaction structure, or registrar/location
structure, or from global variables, a matter of what it is done at the moment of
generating the crash.
You can use `abort()` (iirc, from cfgutils module )inside kamailio.cfg to trigger a crash
at a specific line, so you know what that worker did just before crashing.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1834#issuecomment-466035674
10:22 p.m.
That's interesting. So if a device keeps re-registering using the same call ID the
structure stays open and the xavp's keep hanging to the registration structure? Could
that be it? Just a wild guess... We have a short time between re-registrations.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1834#issuecomment-466131349
19 Sep
19 Sep
11:44 a.m.
New subject: [kamailio/kamailio] registrar: memory leak in xavp_rcd (#1834)
Is this issue fixed in latest kamailio version 5.2.4? i'm asking this because before i
upgrade my system that was using kamailio-5.2.1 i had a similar issue reported here, but
after upgrade kamailio to version 5.2.4 its gone. So i suppose that its resolved. if you
guys confirm that this issue is resolved in 5.2.4 then i guess that this issue can be
closed, right?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1834#issuecomment-533031800
24 Sep
24 Sep
11:20 a.m.
New subject: [kamailio/kamailio] registrar: memory leak in xavp_rcd (#1834)
There were a memory leak present in the xavp_clone_level_nodata function. From a look to
the code it looks that it was used also in the registrar lookup function if the xavp_rcd
parameter was set.
This was fixed in 0c93efec739 and also backported to 5.2 and 5.1 branch. This fix is
included in 5.2.4, which matches the comment from @sebra.
Close this issue, if you still see it please re-open.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1834#issuecomment-534448728
11:20 a.m.
New subject: [kamailio/kamailio] registrar: memory leak in xavp_rcd (#1834)
Closed #1834.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1834#event-2657116708
1889
days inactive
2121
days old
12 comments
5 participants
participants (5)
-
Daniel-Constantin Mierla -
Henning Westerholt
-
José Seabra
-
lazedo
-
Olle E. Johansson