7 Jan
2022
7 Jan
'22
1:33 p.m.
Hello all and happy new year,
I have some new information to share regarding this issue. I believe the
previous metrics I sent to the list might not be indicative of the way the
problem manifests. Here's what I believe so far:
- Issue is exacerbated (or manifests) during moderate-to-high cps, or
grows linearly with total traffic processed since last restart
- shm stats show a lot of memory consumed by sip_msg_shm_clone
- also reproduced this time on bullseye with kamailio 5.5.3
Here's some more meaningful stats taken at more appropriate times (i.e.
after more traffic has been processed) than the previous ones. These two
kamailio instances have identical configuration and traffic patterns:
- https://pastebin.com/gHa803kB for kamailio 5.5.3 showing high
sip_msg_shm_clone on debian bullseye
- https://pastebin.com/JbcZbbSQ for kamailio 5.4.6 on debian buster
There is still DMQ use for these instances despite the version mismatch.
Unfortunately I can't migrate all DMQ nodes to 5.5.x at this time, not
unless I can have assurances that it is DMQ that causes this issue with shm
memory exhaustion...
After shmem was exhausted on 5.5.3, it stopped processing traffic. I issued
a kamctl trap at that time but I'm assuming the backtrace won't show much
except for the inability to allocate shm? If you think the backtrace at
that point would be useful in any way, let me know and I'll try to share it
privately. In case it isn't useful, what other debugging information can be
gathered to dissect this issue? Thanks!
BR,
George
On Wed, 30 Jun 2021 at 19:20, Daniel-Constantin Mierla <miconda(a)gmail.com>
wrote:
Hello,
for the sake of completion: the autoexpire should clean the items if they
are not used during the expiration interval. If you want to get them
deleted after first expiration interval always, see the updateexpire
attribute for htable modparam.
Also, note that replication should be done only between Kamailio instances
with same major version, because there can be internal differences between
major versions that can lead to unexpected behaviour. In other words, if
you replicate, doing between two kamailio with version 5.5.x or between two
kamailio with version 5.4.x, but not between a kamailkio 5.5.x and a
kamailio 5.4.x.
The total amount of used memory in the stats file for 5.5 does not seem to
be high as a rough estimation. The highest by module is in htable, but it
is around 20MB. Maybe you took the stats too early, quickly after a restart?
Cheers,
Daniel
On 30.06.21 17:20, George Diamantopoulos wrote:
Hello Daniel,
Thanks for the feedback. I think I might have been too quick to blame
htable for this behaviour. In fact, version 5.4 seems to consume more
memory than 5.5 (175129776 bytes vs 20581096), which makes sense since it
has been running for longer (I missed the extra digit previously).
So I'm not sure htable is to blame. On the other hand, I don't see any
other modules using up too much of shmem either, so maybe memory stats
can't provide the answer here?
To answer your question, though, I do use DMQ and both tables that use it
have autoexpire set to the same value on both 5.4 and 5.5:
/etc/kamailio# grep dmq kamailio-module-params.cfg
modparam("dmq", "server_address", "sip:172.30.43.1:5090")
modparam("dmq", "notification_address", "sip:
dmq.services.mydomain.com:5090")
modparam("dmq", "multi_notify", 1)
modparam("htable", "enable_dmq", 1)
modparam("htable", "htable",
'cid2hi=>size=8;autoexpire=600;dmqreplicate=1')
modparam("htable", "htable",
'xcid2count=>size=8;autoexpire=600;dmqreplicate=1')
On Wed, 30 Jun 2021 at 17:43, Daniel-Constantin Mierla <miconda(a)gmail.com>
wrote:
Hello,
do you replicate items in the htable via dmq? Does the htable have
autoexpire value set?
Cheers,
Daniel
On 30.06.21 13:54, George Diamantopoulos wrote:
Forwarding my reply to the list, using gmail's reply button set Henning
as the sole recipient :-\
---------- Forwarded message ---------
From: George Diamantopoulos <georgediam(a)gmail.com>
Date: Sat, 26 Jun 2021 at 02:25
Subject: Re: [SR-Users] Possible memory leak on 5.5.x (new)?
To: Henning Westerholt <hw(a)skalatan.de>
Hello Henning,
Thanks for your reply. Here's what has come up after a few hours:
shm55: https://pastebin.com/h9JCePmc
shm54: https://pastebin.com/Nx5xEEnA
It seems to me htable is the culprit? Are you seeing anything different?
54 has been running for 77020 seconds, 55 for 28521 (significantly less).
I'm going to turn it off until we figure something out...
BR,
George
On Fri, 25 Jun 2021 at 18:17, Henning Westerholt <hw(a)skalatan.de> wrote:
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda --
www.linkedin.com/in/miconda
Hello,
Good observation. Please run the memory statistics CLI commands to get
more hints about the module that might cause it (as per below link). Then
please report more details. If you can point to a particular module, you
can also open an issue on our tracker.
https://www.kamailio.org/wiki/tutorials/troubleshooting/memory
Cheers,
Henning
*From:* sr-users <sr-users-bounces(a)lists.kamailio.org> *On Behalf Of *George
Diamantopoulos
*Sent:* Friday, June 25, 2021 4:53 PM
*To:* Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org>
*Subject:* [SR-Users] Possible memory leak on 5.5.x (new)?
Hello all,
I'm still investigating the (most likely non-kamailio-related) memory
leak of my previous message to the list, there have been no developments so
far. I'll update if anything changes.
This concerns a new finding which seems to affect kamailio 5.5.x. I have
two kamailio instances receiving the same traffic via round-robin. I
upgraded only one of them to 5.5.1 and left the other to 5.4.6 as I feared
of any issues arising. I was lucky to do so, because with identical
configuration, 5.5.x seems to run out of SHM very quickly. Here are links
to graphs produced by our monitoring system:
Old kamailio (no memory leak): https://pasteboard.co/K8fVBiD.png
New kamailio (possible leak): https://pasteboard.co/K8fVS9N.png
The configuration uses mtree, htable, vars and vns extensively. Has
anyone come across anything similar? Let me know if I can provide any
further information to help disect this. Thanks!
BR,
George
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda --
www.linkedin.com/in/miconda
-- 
